Ceph Deep Dive: RADOS, BlueStore, CRUSH, and Running a Sovereign Storage Cluster

CephFS: From Zero to Hero — The Distributed Filesystem, End to End

CephFS in production: MDS architecture, pool design, quotas, snapshots, subvolume groups, multi-MDS scaling, Kubernetes CSI, and operations.

Teaser: Ceph is the storage backbone behind OpenStack, the Sovereign Cloud Stack, and thousands of self-hosted clusters — yet most engineers treat it as a black box. This deep dive unpacks the RADOS object store, BlueStore's on-disk design, and CRUSH placement so you can design, deploy, and debug a cluster with confidence — including what Squid 19.2 and the upcoming Tentacle 20.2 change for production.

Ceph turns a pile of commodity disks into one elastic storage pool with three protocols: RBD (block), RGW (S3-compatible object), and CephFS (POSIX file). The trick is that all three are just thin frontends over the same distributed object store, RADOS. If you understand RADOS, you understand Ceph — and 90% of production failures become predictable.

The RADOS Core: Everything Is an Object

RADOS (Reliable Autonomic Distributed Object Store) is a self-healing, self-managing key-value object store. Each object is identified by a 128-bit name, and objects live in pools that define replication, erasure coding, and placement rules.

The critical design decision: clients compute data placement themselves. There is no central metadata server on the I/O path. A client takes an object name, hashes it into a Placement Group (PG), and asks the CRUSH algorithm which OSDs should hold that PG. This is why Ceph scales linearly — the OSDs are the only thing that moves data, and the cluster map is the only shared state.

The Cluster Map

Every client and daemon holds a copy of the cluster map, versioned like a distributed database. It contains:

• The monitor map (who are the MONs, what's the quorum)
• The OSD map (every OSD, its state, its weight, its device class)
• The PG map (every PG, its state, its acting set)
• The CRUSH map (the physical topology — hosts, racks, failure domains)

When any of these change, the map epoch bumps and the change is gossiped out. Stale maps are the root cause of most "misdirected I/O" and "slow requests" incidents — which is why monitor tuning and ceph osd pool application hygiene matter more than raw throughput.

Pools, Replication, and Erasure Coding

A pool is where the policies live:

• Replicated pools — each object is stored N times on N OSDs in different failure domains. Default size=3, min_size=2. Simple, low CPU cost, higher space cost.
• Erasure-coded pools — objects are split into K data chunks + M parity chunks. EC 2+1 stores 1.5× the data (vs 3× for replicas) but costs CPU on every write and read-modify-write. EC is ideal for cold data, RGW buckets, and large objects.

CRUSH: Data Placement Without a Coordinator

CRUSH (Controlled Replication Under Scalable Hashing) is what makes Ceph different from every system with a metadata broker. It's a deterministic pseudo-random function over:

CRUSH(object, PG, cluster_map, rule) → ordered list of OSDs

The rule encodes your failure-domain policy. A common rule: "pick 3 different hosts, then 1 OSD per host" — guaranteeing that a host failure never takes out all replicas of any object.

The bucketing hierarchy is the physical reality you encode: root → datacenter → room → row → rack → host → OSD. Each bucket has a weight; OSD weights are proportional to disk capacity (a 4 TB OSD weighs 4× a 1 TB OSD).

CRUSH Weight vs OSD Usage — the Classic Trap

Here's where production clusters burn people: CRUSH weights describe the target distribution, not current usage. When you add a new OSD, CRUSH rebalances a portion of PGs onto it, but the amount depends on weight and PG count — and it's bounded per PG movement. The balancer module (enabled by default since Pacific) periodically re-weights and optimizes, but on big clusters you still want to:

1. Keep PG counts roughly equal per OSD (ceph pg dump → check pg_num per pool is a power of two)
2. Use osd_crush_update_on_start: true and tune osd_recovery_max_active for controlled backfill
3. Monitor recovery_rate and throttle recovery during business hours

BlueStore: The Modern OSD Backend

Since Luminous, BlueStore owns the data path. It replaces the old FileStore (a POSIX filesystem on top of a filesystem) with a direct block-device layout:

• Data is written as raw extents on the block device — no double filesystem overhead
• Metadata lives in RocksDB, stored on a small fast device (NVMe/SSD) if available
• BlueFS is the minimal filesystem that gives RocksDB its files
• WAL (write-ahead log) absorbs small writes before they hit the data region

This "fast device for metadata, big HDDs for data" split is why Ceph HDD clusters perform far better than anyone expects: the metadata hot path lives on SSD/NVMe, while bulk data streams to the disks.

Squid 19.2 Changes That Matter

Released 26 September 2024, Squid (v19.2.x) is the current long-term stable series. The release notes highlight:

• BlueStore optimized for snapshot-heavy workloads — RocksDB LZ4 compression enabled by default, improving both average performance and fast-device space usage
• More flexible erasure-code configurations and better scrub scheduling
• Crimson tech preview — the next-generation OSD in Seastar (RBD on replicated pools only)
• CephFS subvolume quiesce — pause write I/O on a subtree for crash-consistent snapshots of distributed apps
• RGW user accounts + IAM APIs — self-service users, keys, groups, roles, policies for AWS compatibility
• rbd-wnbd multiplexing — one daemon per Windows host instead of one per image
• ceph tell --daemon-output-file — stream large command output to a daemon-local file, avoiding monitor memory spikes

cephadm: The Only Deployment Method That Matters

Since Octopus, cephadm (containerized, systemd-managed, SSH-orchestrated) is the supported deployment path. Manual ceph-deploy clusters are legacy. Key operations:

# Bootstrap a single-node cluster (detects the host, deploys MON+MGR+OSD)
cephadm bootstrap --mon-ip 192.168.42.10

# Add an OSD on a specific device
ceph orch device zap <host> /dev/sdb --force
ceph orch daemon add osd <host>:/dev/sdb

# Deploy RGW with a realm+zone (needed for multisite later)
ceph orch apply rgw rgw.site1 --placement "3 host1 host2 host3" --port 7480

# Deploy MDS for CephFS
ceph fs volume create cephfs --placement="2 host1 host2"

# NVMe/TCP gateway for high-performance RBD
ceph orch apply nvmeof nvmeof.gw1 --placement "2 host1 host2"

The ceph orch layer manages daemon placement as a desired state, which is the single biggest quality-of-life improvement in Ceph history: no more hand-edited systemd units or manual config distribution.

Upgrade Discipline

Ceph requires upgrading one daemon type at a time, in order: MONs first, then MGR, then OSDs (by default), then MDS/RGW/clients. ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.5 handles it, but verify:

• ceph versions — all daemons on the same version before proceeding
• ceph health detail — no PG_AVAILABILITY or OSD_* warnings
• Back up the ceph.conf, client keys, and the MON store before starting

Day-2 Operations: The Metrics That Predict Outages

The MGR hosts the Prometheus endpoint (<mgr-ip>:9283/metrics). The alerts that actually catch real incidents:

The "full" cliff is non-linear. Ceph blocks writes at mon_osd_full_ratio (default 0.95) but degrades much earlier: recovery and rebalancing slow down as free space shrinks, and nearfull warnings (0.85) should trigger a capacity plan, not a "wait and see".

RBD, RGW, and CephFS in Production

RBD (Block)

The bread and butter for OpenStack/K8s. Key concepts: image (thin-provisioned block device), snapshot (instant, copy-on-write), clone (snapshot-based COW image), and rbd-mirror (async replication to a second cluster — the standard DR pattern, with a journal or snapshot-based sync).

Squid's diff-iterate improvement makes QEMU live disk sync dramatically faster — this is what powers consistent VM backups without agent-in-VM.

RGW (Object / S3)

The S3 gateway runs as a stateless frontend over RADOS. For production:

• Multisite requires realm → zonegroup → zone structure; one zone per site, sync agents replicate buckets asynchronously
• The Squid fix for the RGWBucketFullSyncCR infinite loop (source bucket deleted mid-sync) matters if you do active bucket deletions across sites
• IAM user accounts (Squid) replace the legacy S3-style key management for self-service tenants
• TLS 1.3 ciphersuites config (ssl_ciphersuites) for the Beast frontend

CephFS (File)

MDS is the metadata service; one active MDS per filesystem is the normal mode, with standbys for failover. Squid's quiesce API is the notable new tool: pause I/O on a subtree so distributed apps get crash-consistent snapshots — think databases or job queues on CephFS.

What Tentacle (v20.2) Brings — Released 2026

Tentacle is the 20th stable release (v20.2.2, June 16, 2026). The headline features for operators:

• Native SMB support — a new smb MGR module deploys Samba-backed SMB shares on CephFS, with CTDB clustering and a cephfs-proxy daemon for scale. This finally makes Ceph a credible replacement for Windows file servers and NAS appliances.
• CephFS case-insensitive / normalized directory entries — configurable per-directory
• NVMe/TCP gateway groups and multiple namespaces — dashboard-managed, with DHCHAP controller keys, namespace encryption, and secure listeners
• Dashboard overhaul — multi-cluster management, OAuth 2.0, RGW multisite automation, tiering, notifications, granular replication policy
• Fast EC restrictions — non-4K-aligned EC chunk sizes are rejected, closing a class of silent performance bugs
• NVMeoF fast failover — beacon grace reduced from 10s to 7s
• Rocky 10 package-based installs supported from v20.2.2

For the digital-sovereignty crowd this matters: Ceph + OSISM remains the reference backend for the Sovereign Cloud Stack, and the SMB/NVMeoF work widens the "leave the hyperscaler" envelope considerably.

Design Checklist for a New Cluster

1. Failure domains: encode racks (crush rule with rack as failure domain) or accept host-level only. Never put 3 replicas on one host.
2. Network separation: a dedicated cluster network for OSD↔OSD replication/backfill; public network for clients. One 10GbE NIC for both is a latency accident waiting to happen.
3. PG sizing: total PGs ≈ (OSDs × 100) / replication_factor, spread across pools as powers of two. pg_autoscaler (default on) helps, but set target_size_ratio per pool.
4. BlueStore fast devices: give every OSD an NVMe/SSD for block.db/block.wal if you can't afford all-flash — it's the highest-ROI upgrade for HDD clusters.
5. Monitors: odd count, min 3, on separate hosts, never alongside heavy OSDs or in the same failure domain. mon_clock_drift warnings mean NTP is broken.
6. Capacity discipline: alert at 70% used, plan at 80%, panic at 85%. Autoscaler plus ceph osd reweight-by-utilization are your friends.
7. Auth: CephX everywhere; rotate client keys; scope keys per tenant/application (mon caps, osd caps, mgr caps).
8. Backup: RBD images → rbd export or mirroring; RGW → bucket replication or s3 sync to another cluster; CephFS → snapshots + cephfs-mirror (since Pacific).

Debugging in the Trenches

# The three commands that answer 90% of questions
ceph health detail
ceph pg dump_stuck inactive,unclean,undersized
ceph osd tree   # + ceph osd perf for per-OSD latency

# Slow requests — where is the time going?
ceph daemon osd.<id> dump_historic_ops
ceph daemon osd.<id> dump_ops_in_flight

# Is the balancer doing its job?
ceph balancer status
ceph osd df tree

# CRUSH map inspection
ceph osd crush rule dump
ceph osd crush tree

When an OSD is slow, check in order: disk health (smartctl), BlueStore compaction (ceph daemon osd.N compact), RocksDB WAL device saturation, and finally the cluster network — ceph osd perf plus dump_ops_in_flight will separate a dying disk from a saturated NIC.

Ceph rewards the engineer who understands its one big idea: placement without coordination. Master the cluster map, the CRUSH rule, and BlueStore's device layout, and the rest is operational hygiene — monitor the right metrics, keep recovery throttled, respect the full-ratio cliff. With Squid stable and Tentacle adding SMB and NVMeoF polish, there has never been a better time to run your storage in your own data centre.