Scope: this covers how the OpenStack lab and its Ceph cluster are actually monitored — the exporter stack, what Ceph exposes natively, the Prometheus recording rules that make the dashboards fast, a tour of the real dashboard panels (with screenshots), day-2 Ceph status commands, and how to load-test the Ceph cluster itself with dd/fio. For how the cloud and its network/BGP are built, see OpenStack Lab Deployment; for what runs on top of it, see Kubernetes on OpenStack. This article doesn’t repeat either — it’s monitoring-only.
1. The exporter stack
Every metric on these dashboards comes from one of these, laid down by a single Ansible playbook (monitoring.yml) that targets the whole OpenStack host group plus one dedicated monitoring VM:
| Exporter | What it covers | Where it runs |
|---|---|---|
node_exporter | Standard host metrics (CPU, memory, disk, network) for each physical OpenStack host | Every OpenStack host, as a pinned-tag Docker container under a long-lived systemd unit |
| cadvisor | Container-level RAM/CPU for every LXC container OSA creates (one per infra service) and any Ceph containers | Every OpenStack host |
| process-exporter | Per-process RAM/CPU — what’s actually holding memory: ceph-osd, mysqld, rabbitmq/beam.smp, nova-*, ovs, qemu — beyond what Nova’s own VM-allocation view shows | Every OpenStack host |
lvm-exporter (hansmi/prometheus-lvm-exporter) | LVM volume-group/logical-volume usage — node_exporter has no native LVM collector, needed for both OpenStack block-storage VGs and Ceph OSD VGs | Every OpenStack host |
| openstack-exporter | Centralized Nova/Neutron/Cinder/Glance/Keystone API and resource metrics — queries Keystone once for the whole cloud, not per-node | The monitoring VM itself, not the OpenStack cluster (a separate inventory owns it) |
| ceph-mgr Prometheus module | Ceph cluster health, capacity, IOPS/throughput, per-pool and (optionally) per-daemon perf counters | Every Ceph mon/mgr node, port 9283 |
Two operational details worth knowing before you go looking for “missing” metrics:
- node_exporter’s image tag is pinned inside an existing systemd unit, never recreated. The unit was laid down years ago; bumping the version is a regex replace of the image tag in the unit file followed by a restart — nothing else about the host is touched. If you’re chasing a stale node_exporter build info, check the systemd unit file directly rather than assuming a fresh container spec.
- ceph-mgr’s Prometheus endpoint on standby mons returns HTTP 200 with an empty body — only the currently-active mgr returns a real metrics payload. That’s expected Ceph behavior, not a broken exporter; don’t page on it.
2. Ceph-side Prometheus configuration
Beyond just enabling the module, a few explicit ceph config set calls shape what the mgr actually exposes:
# Per-pool RBD stats only for the pools that matter (avoids scraping every pool in the cluster)
ceph config set mgr mgr/prometheus/rbd_stats_pools glance,cinder,nova,kuberbd
# Per-daemon perf counters (OSD op latency, mon session counts, etc.) — off by default,
# turned on deliberately since this cluster is small enough that the extra cardinality is fine
ceph config set mgr mgr/prometheus/exclude_perf_counters false
ceph mgr module enable dashboard
ceph mgr module enable devicehealth
ceph mgr module disable nfs # not used, one less module to reason about
# SMART-based device health, scraped periodically
ceph device scrape-health-metrics
ceph device check-healthThe dashboard module also gets a dedicated read-only user (separate from the admin account) so the Ceph web dashboard can be shared without handing out cluster-admin.
3. Prometheus recording rules
Recording rules exist for one reason here: pre-computing an expensive, high-cardinality join once so every dashboard panel and alert downstream can query a cheap precomputed series instead of repeating the same join boilerplate. The real one worth understanding end-to-end is the per-VM CPU/memory attribution chain:
# process-exporter groups every qemu guest process by groupname="qemu-vm-<instance-uuid>".
# This rule attaches the human project NAME to each OpenStack server by joining on tenant_id.
- record: qemu_vm:server_info:by_uuid
expr: |
openstack_nova_server_status * on(tenant_id) group_left(project_name) (
label_replace(
label_replace(openstack_identity_project_info, "tenant_id", "$1", "id", "(.*)"),
"project_name", "$1", "name", "(.*)"
) * 0 + 1
)
# Real per-VM CPU, pulled out of the qemu-vm-<uuid> process group and joined to
# name/hypervisor/project via the rule above.
- record: qemu_vm:cpu_cores:by_uuid
expr: |
label_replace(
sum by (node, groupname) (rate(namedprocess_namegroup_cpu_seconds_total{groupname=~"qemu-vm-.*"}[5m])),
"uuid", "$1", "groupname", "qemu-vm-(.*)"
) * on(uuid) group_left(name, hypervisor_hostname, project_name) (qemu_vm:server_info:by_uuid * 0 + 1)
- record: qemu_vm:memory_resident_bytes:by_uuid
expr: |
label_replace(
sum by (node, groupname) (namedprocess_namegroup_memory_bytes{groupname=~"qemu-vm-.*", memtype="resident"}),
"uuid", "$1", "groupname", "qemu-vm-(.*)"
) * on(uuid) group_left(name, hypervisor_hostname, project_name) (qemu_vm:server_info:by_uuid * 0 + 1)
# Same CPU figure, normalized to % of that VM's own hypervisor's real physical core count —
# lets you compare load across hosts with different core counts on one 0-100 scale.
- record: qemu_vm:cpu_percent_of_host:by_uuid
expr: |
qemu_vm:cpu_cores:by_uuid * 100
/ on(node) group_left() (count by (node) (count by (node, cpu) (node_cpu_seconds_total{app="openstack",tier="host"})))The mechanism this depends on: every qemu guest process carries -uuid <instance-uuid> in its own command line, and that UUID matches Nova’s own openstack_nova_server_status{uuid=...} series exactly — so a process-exporter cmdline-regex capture is directly joinable to the real instance name/project/hypervisor with no separate lookup service needed.
A second, simpler recording rule (now, expr: time()) exists purely so alert-rule expressions can reference a stable “current time” series without repeating time() inline in every rule.
4. Dashboard tour
Two real, recent changes to the Hypervisor Overview dashboard show the actual iteration this monitoring stack goes through:
- Merged the qemu and ceph-osd CPU/Memory panels, normalized to %. These hosts are hyperconverged (Ceph OSD + Nova compute on the same physical machines), so “VM CPU” and “Ceph OSD CPU” used to be two separate panels per host. They’re now one panel each, with the legend distinguishing
qemuvsceph-osdso the source is unambiguous — and every CPU-breakdown panel on that row now shows % of that host’s real physical core count instead of raw core counts, so hosts with different core counts read on the same 0–100 scale. - Added “Total Other CPU/Memory per host” panels, and bumped the top-N process breakdown from 12 to 15. “Total Other” is the sum of everything except qemu and ceph-osd — i.e. real system/host overhead — shown as its own panel next to the VM/OSD ones, separate from the top-N individual-process breakdown table below it.
The cluster-level “traffic light” panels — the ones worth checking first during an incident — are aggregate stat/gauge panels with Grafana thresholds (green/yellow/red), not per-host or per-VM breakdowns:
Ceph Cluster — Advanced, “Cluster State” row: health status, available capacity, OSD/MGR/monitor counts, and firing-alert counts by severity, all at a glance.
OpenStack — Cluster Status: per-service availability (green/gray blocks per API) alongside live resource counts (VMs, networks, volumes, images, floating IPs, security groups).
The aggregate blocks above are actually a rollup — each OpenStack API service on this cloud runs as its own LXC container on all 3 control-plane hosts (no dedicated controller tier, see the OpenStack Lab Deployment article), so the dashboard also breaks every service down into 3 individual per-host traffic lights:
OpenStack — Cluster Status, “OpenStack LXC Containers — Service Availability” row: same principle repeats for every service on the dashboard (Keystone, Nova API/Metadata, noVNC, Neutron, Cinder, Glance, Placement, Horizon, Aodh, Gnocchi, Masakari, Memcached, Zookeeper, Galera, RabbitMQ) — 3 lights each, so a single host losing one service is visible immediately instead of being hidden behind an aggregate “mostly up” status.
5. Ceph day-2: checking cluster status by hand
Dashboards are for trends; these are the commands actually run against a mon node when something looks off or right after a change:
ceph -s # or: ceph status — health, mon/mgr/osd counts, pg states, IO rate, at a glance
ceph health detail # when health isn't HEALTH_OK, this says exactly which check is failing and why
ceph osd tree # OSD topology + up/down/in/out state, grouped by host — first stop for "which OSD"
ceph osd df # per-OSD usage % and PG count — spot a lopsided/nearly-full OSD before it pages you
ceph df # cluster-wide and per-pool capacity/usage
ceph crash ls -f json | jq -r '.[].crash_id' | xargs --no-run-if-empty -I{} ceph crash archive {}
# archive known crashes so `ceph health` stops reporting them as newThat crash-archive one-liner runs both right after any cluster-config change and again at the end, bracketing the change so any crash that appears during it is caught rather than silently archived away with the pre-existing ones.
6. Ceph performance testing with dd and fio
Run these from a Ceph client, never from an OSD node. A test on an OSD host measures local disk I/O, not the actual replicated write path a real workload takes (network + replication + the OSD’s own journal/WAL). Use a VM with an RBD-backed volume, or a scratch pod with an RBD or CephFS PVC mounted (kubectl run --rm -it cephperf --image=... with the PVC attached, or kubectl exec into an existing pod that already has one — several of this repo’s k8s addons already mount CephFS/RBD PVCs, e.g. WordPress’s wp-content volume, so any pod using csi-cephfs-sc/csi-rbd-sc works).
6.1 Quick sanity check: dd
dd if=/dev/zero of=testfile bs=1M count=1024 oflag=direct
# 1024 x 1M = 1GiB, oflag=direct bypasses the page cache so you're actually hitting Ceph,
# not just writing into RAM and reporting a fake numberdd is single-threaded, sequential-only, and gives you exactly one number (MB/s). It’s a fine 10-second gut check that the path isn’t obviously broken, but it says nothing about IOPS, latency, or how the cluster behaves under concurrent/random access — which is what almost every real workload actually looks like. Reach for fio for anything you’d actually make a capacity decision from.
6.2 Realistic testing: fio
Database-like workload — small-block random read/write, the pattern that matters for anything backed by InnoDB or similar (4K matches InnoDB’s page size, and DB access patterns are overwhelmingly random rather than sequential):
fio --name=db-randrw --rw=randrw --bs=4k --iodepth=32 --numjobs=4 \
--size=1G --runtime=60 --time_based --group_reportingWhat to look at in the output: IOPS and latency percentiles (p99/p99.9 matter more than the average for anything DB-shaped — a long tail is what actually causes query timeouts), not the throughput number.
Big-file / large-sequential workload — the pattern for backup dumps and media files (relevant here since this is the same Ceph cluster backing CephFS/RBD volumes like WordPress’s media/backup storage elsewhere in this repo’s k8s addons):
fio --name=bigfile-seq --rw=write --bs=1M --iodepth=16 --numjobs=1 \
--size=10G --runtime=60 --time_based --group_reportingWhat to look at here: sustained MB/s — this is the number that tells you how long a multi-gigabyte backup dump or media restore will actually take against this cluster.
These are lab-scale numbers, useful for capacity-planning intuition and catching regressions — not a substitute for Ceph’s own built-in per-OSD benchmark if you want to isolate a single slow disk: ceph tell osd.<id> bench runs a sequential write test directly against that one OSD, bypassing the client-side network/replication path entirely.
Summary
- Six exporters cover host, container, per-process, LVM, OpenStack-API, and Ceph metrics (§1–2)
- Recording rules pre-join qemu process data to real instance/project/hypervisor identity once, so every downstream panel/alert stays cheap (§3)
- Cluster-level stat panels with Grafana thresholds are the first place to look during an incident (§4)
ceph -s/health detail/osd tree/dffor day-2 status checks (§5)ddfor a 10-second sanity check,fio(randrw 4K for DB-like, sequential 1M for bulk) for real capacity-planning numbers,ceph tell osd.<id> benchto isolate a single OSD (§6)
Comments are closed.