CephFS: From Zero to Hero — The Distributed Filesystem, End to End
~10 min read
DevOpsStoragecephcephfsdistributed-storageposixmdskubernetescsisnapshotsquotasreadwritemany
Save products you love by clicking the heart icon.
Evidenzbasierte Testing-Praxis aus Produktions-Codebasen — property-basierte Invarianten, gezielte Fehlerinjektion, Contract-Tests, benchmark-verifizierte Performance und Qualitäts-Gates, die durchgesetzt werden, nicht nur versprochen.
A deep dive into OSISM — the platform that orchestrates OpenStack, Ceph, and Kubernetes into a unified, sovereign private cloud. Covers architecture, components, and the full installation routine.
CephFS is the POSIX filesystem that runs on top of RADOS — the same object store that powers Ceph's block (RBD) and object (RGW/S3) interfaces. Where a filesystem like ext4 or XFS lives on one disk, CephFS lives on a cluster: metadata is served by dedicated Metadata Servers (MDS), data is striped across OSDs, and clients talk to both directly. There is no single node that holds your files.
The architecture explains the operational model in one glance:
CephFS is the right tool when you need a shared POSIX namespace with locks: multiple writers on the same files, directory quotas, snapshots, and subdirectories with independent lifecycles. If you only need single-writer volumes, RBD is simpler; if you only need buckets, RGW/S3 is simpler. CephFS is what you want when many machines need the same files, concurrently.
CephFS has no standalone mode — you start with a RADOS cluster. The fastest production-grade path in 2026 is cephadm (containerized daemons managed by systemd) or Rook (the same daemons as Kubernetes pods). On bare metal, cephadm is the standard:
# Bootstrap a single-node cluster, then add hosts
cephadm bootstrap --mon-ip 10.0.0.10
ceph orch host add node-2
ceph orch host add node-3
# Deploy the monitor and manager quorum
ceph orch apply mon node-1,node-2,node-3
ceph orch apply mgr node-1,node-2,node-3
# Add OSDs — one per disk, never partition a disk for Ceph
ceph orch device zap node-1 /dev/sdb --force
ceph orch apply osd --all-available-devices
Two configuration decisions matter before you create any filesystem:
auth_cluster_required, auth_service_required, auth_client_required). There is no reason to turn it off.A CephFS needs two pools: one for metadata (small objects, many operations, needs low latency) and one for data. The data pool is where your files actually live; the metadata pool holds directory entries, inode attributes, and the MDS journal.
# Metadata pool — replicated, small
ceph osd pool create cephfs_metadata 32 replicated
ceph osd pool application enable cephfs_metadata cephfs
# Data pool — replicated (EC data pools work too, see Part 4)
ceph osd pool create cephfs_data 128 replicated
ceph osd pool application enable cephfs_data cephfs
# Create the filesystem (one MDS to start)
ceph fs new cephfs cephfs_metadata cephfs_data
ceph fs status
ceph fs status should show one MDS in active, the data pool, and the metadata pool. That is the entire bootstrap — the filesystem now exists and is mountable.
The kernel client is the production choice; ceph-fuse is the fallback when you cannot load kernel modules (containers, macOS, older kernels):
# Kernel client — needs the cephx key
mount -t ceph node-1:6789,node-2:6789,node-3:6789:/ /mnt/cephfs \
-o name=admin,secretfile=/etc/ceph/admin.keyring
# FUSE client
ceph-fuse -m node-1:6789,node-2:6789,node-3:6789 /mnt/cephfs
Use all MON addresses in the mount string. The client needs to reach a monitor on startup; a single hard-coded MON address is a single point of failure for every mount in your fleet.
CephFS quotas are directory quotas: set on a directory, enforced on everything below it. Two dimensions: byte limit and file count.
# 100 GiB and 100k files on /mnt/cephfs/projects/alpha
ceph fs set-quota /mnt/cephfs/projects/alpha 100G 100000
ceph fs get-quota /mnt/cephfs/projects/alpha
Quotas are the mechanism that makes multi-tenant filesystems safe: each team gets a directory with a quota, and no team can exhaust the pool for everyone else. The enforcement is best-effort with a grace period — a client that ignores the quota signal can write slightly over before being blocked — so leave headroom on the pool, not on the quotas.
CephFS snapshots are asynchronous, copy-on-write, and free to create until data actually changes. No pool provisioning, no agent, no quiescing:
# Snapshots live in the hidden .snap directory
mkdir /mnt/cephfs/projects/alpha/.snap/pre-migration
ls /mnt/cephfs/projects/alpha/.snap/
rmdir /mnt/cephfs/projects/alpha/.snap/pre-migration # delete the snapshot
Snapshots are per-directory and recursive. They are the native primitive behind backup tools (K8up and Velero both snapshot CephFS volumes through the CSI) and behind "oops, that migration deleted a directory" recovery.
Subvolumes are the modern building block: a directory with its own inode, quota, and snapshot layout, addressable by name. Subvolume groups add a second level of organization, and the CSI driver creates one subvolume per PVC inside a group.
# Group for Kubernetes volumes, one subvolume per PVC
ceph fs subvolumegroup create cephfs k8s
ceph fs subvolume create cephfs pvc-workspace --group k8s --size 20G
ceph fs subvolume ls cephfs --group k8s
ceph fs subvolume snapshot create cephfs pvc-workspace --group k8s snap-1
Subvolumes matter operationally because they give you per-volume quotas, per-volume snapshots, and clean deletion — a plain directory tree with a thousand PVCs would be an unmanageable mess.
One active MDS can serve a very large filesystem, but metadata throughput is finite. The fix is more MDS daemons — CephFS shards the namespace (by directory tree) across them:
# Allow up to 4 active MDS daemons
ceph fs set cephfs max_mds 4
ceph fs status # watch ranks come up: 0, 1, 2, 3
# Let the balancer distribute the tree
ceph mds balancer mode distributed
The MDS is memory- and CPU-bound, not disk-bound (metadata lives on OSDs). Sizing rule of thumb: start with one active MDS per ~10–20k files or per ~100k metadata operations per second, and let ceph fs status + the balancer tell you when to add more. Standby MDS daemons (one per active rank) fail over in seconds when an active MDS dies.
The reason most teams meet CephFS in 2026 is Kubernetes. The ceph-csi-cephfs driver turns the filesystem into a dynamic provisioner with ReadWriteMany volumes — the only way to get true multi-pod, multi-node file storage without NFS.
# StorageClass — the fsName/pool pair selects the filesystem
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cephfs-workspaces
provisioner: cephfs.csi.ceph.com
parameters:
clusterID: <ceph-cluster-id>
fsName: cephfs
pool: cephfs_data
csi.storage.k8s.io/snapshotter-secret-name: csi-cephfs-secret
csi.storage.k8s.io/snapshotter-secret-namespace: default
reclaimPolicy: Retain
allowVolumeExpansion: true
# A RWX workspace volume — exactly the pattern used by a
# shared Nix builder in the reference cluster
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nix-workspace
namespace: nix-builder
spec:
accessModes: [ReadWriteMany]
storageClassName: cephfs-workspaces
resources:
requests:
storage: 20Gi
Two details in that StorageClass are load-bearing:
fsName + pool pin the volume to a specific filesystem and data pool. The CSI driver creates one subvolume per PVC (with a size quota from the PVC's storage request) — the "subvolume group" from Part 2 is what keeps a hundred PVCs tidy.allowVolumeExpansion: true lets you grow PVCs without recreating them; CephFS subvolume quotas are dynamic, so expansion is instant.VolumeSnapshot API with the snapshotter secret referenced above — K8up, Velero, and plain kubectl snapshots all land as CephFS subvolume snapshots.The real-world reference for this section: a three-node bare-metal cluster (Ceph Reef 18.2.8, cephadm under systemd) running K3s, where a Nix builder consumes a 50 GiB RBD volume for /nix and a 20 GiB CephFS RWX volume for /workspace — the filesystem being built is shared read-write between builder pods on different nodes. Storage classes for both (rbd and cephfs) are provisioned by ceph-csi with Retain reclaim policy and snapshotting wired up. That one workspace volume is the entire reason CephFS is on the cluster: RWX, no NFS, no single point of failure.
The killer feature pairing: CephFS RWX + subvolume snapshots gives you multi-writer storage and point-in-time recovery through the standard Kubernetes APIs — no storage-adjacent sidecar daemons to operate.
ceph fs status # MDS ranks, active/standby, laggy clients
ceph fs dump | head # filesystem config, max_mds, session timeout
ceph mds stat # per-rank state, up since, caps
ceph osd perf # commit/apply latency per OSD — your data path
ceph pg stat # placement group health — the cluster's pulse
Watch specifically: MDS memory (metadata cache; a small default cache on a busy filesystem causes thrashing), client eviction (ceph tell mds.* client evict when a hung client blocks a directory), and full pools — a CephFS whose data pool hits full_ratio stops accepting writes cluster-wide, not just in one directory.
ceph health goes WARN (degraded) and returns to HEALTH_OK when the backfill finishes. A filesystem on a healthy pool is unaware any of this happened.The reference cluster's layout follows exactly this: three nodes, MON/MGR/MDS co-located on all three, eight OSDs spread 4/2/2. Losing any one node keeps quorum, keeps the MDS active (standby on another node), and keeps the data readable.
min_size. size 3 tolerates two concurrent OSD failures; min_size 2 keeps serving writes during a single failure. Never set min_size 1 on a filesystem pool — you will serve un-replicated writes and regret it.k=4,m=2 = 1.5× raw capacity for 2-failure tolerance) at the cost of CPU and small-write performance. Enable it only when capacity, not latency, is the constraint.mount -o rsize/wsize matter for large sequential workloads; the MDS cache size (mds_cache_memory_limit) matters for metadata-heavy workloads like build trees — the Nix builder case.commit/apply latency stays in the low single-digit milliseconds because of it.ceph balancer. Let the balancer rebalance PGs across OSDs periodically (ceph balancer mode crush-compat; ceph balancer on) to keep utilization even — an unbalanced cluster makes one hot OSD the bottleneck for everyone.| Symptom | First thing to check | Likely fix |
|---|---|---|
| Mount hangs | ceph -s quorum, MON reachability | MON address list in mount string; firewall on public network |
| Writes stall cluster-wide | ceph df pool FULL | Add OSDs or raise mon_osd_full_ratio (carefully); check quotas aren't misconfigured |
| One directory is slow | ceph tell mds.<rank> client ls | A client with an unresponsive mount holds caps; evict it |
ls is slow, reads are fast | MDS memory / mds_cache_memory_limit | Raise the metadata cache; add an MDS rank |
| Replication eats all bandwidth | ceph osd tree + node NIC stats | Cluster network missing or sharing the public VLAN |
| Snapshots missing after backup | CSI snapshotter secrets | csi.storage.k8s.io/snapshotter-secret-* in the StorageClass |
Before touching any "tuning" knob, check ceph health detail. Most production CephFS problems are not tuning problems — they are full pools, failed OSDs waiting for backfill, or clock skew breaking MON quorum. Fix those first; tuning is the last 10%.
A production-grade CephFS deployment, condensed:
min_size.ceph balancer on, ceph pg autoscaler on — let the cluster do the boring work.CephFS is the least glamorous of Ceph's three interfaces and the one that delivers the most value per operator-hour once it is running: a self-healing, snapshotting, quota-enforcing POSIX filesystem that spans your whole cluster. Start with a three-node cluster and one workspace volume; the rest scales from there.