SeaweedFS: Deep Dive!

Ceph Deep Dive: RADOS, BlueStore, CRUSH, and Running a Sovereign Storage Cluster

Everything a senior engineer needs to know about Ceph in 2026: the RADOS architecture, BlueStore internals, CRUSH data placement, cephadm operations, and what Squid 19.2 and Tentacle 20.2 change for production clusters.

The One-Binary Object Store

SeaweedFS is a distributed object store written in Go that fits in a single static binary called weed. It is the anti-Ceph in the best possible way: where Ceph is a platform with monitors, managers, and a dashboard, SeaweedFS is a set of small daemons you can run on a Raspberry Pi — and scale to tens of petabytes.

Its superpower is the data path. A client asks the master where to write, gets a volume server address, and then talks to that volume server directly — the master is not in the data path, so throughput scales with the number of volume servers, not with a central coordinator.

The four moving parts:

• Master — knows every volume server (heartbeats every few seconds), assigns volume IDs, and hands clients the address of the volume server that holds their data. Tiny memory footprint; a master's metadata is the volume map, not your files.
• Volume servers — store logical volumes: fixed-size (default 30 GB), append-only files with 4 KB-aligned entries. Deleting a file leaves a hole; a background compaction reclaims the space. One volume server can hold many volumes (-max=32 in the reference deployment).
• Filer — the filesystem layer: directories, names, and file metadata (stored in an embedded DB, or MySQL/Postgres for scale), chunking your files into volume-server entries. Without a filer there is no namespace — only raw volume IDs.
• S3 gateway — weed s3 implements the S3 API on top of the filer, which is what lets K8up, rclone, and every S3 tool in existence use the cluster.

Part 1 — Zero: Running It Before You Understand It

Download the binary, run four commands, and you have a cluster:

# Master: manages volume servers (port 9333)
weed master -mdir=/data/master &

# Volume server: stores data (port 8080)
weed volume -dir=/data/volumes -max=32 -mserver=localhost:9333 &

# Filer: the namespace (port 8888)
weed filer -master=localhost:9333 &

# S3 gateway on the filer (port 8333)
weed s3 -filer=localhost:8888

Upload a file with plain HTTP — no SDK required:

# Assign a file ID from the master, then PUT to the volume server
weed upload -server=localhost:9333 ./photo.jpg
# → photo.jpg saved to http://localhost:8080/3,016bc0e4ac

# Or go through the filer for a real path
curl -F file=@photo.jpg http://localhost:8888/photos/

Two minutes in, you have already used the core design: the master handed you a volume server, and the data went there directly. That direct path is the whole architecture.

Part 2 — Architecture: Logical Volumes, Replication, Erasure Coding

2.1 Append-only logical volumes

A volume server does not manage files; it manages logical volumes — fixed-size blobs (default 30 GB, set with -volumeSizeLimitMB=30000 in the reference cluster) written strictly append-only. Each file entry is 4 KB-aligned and addressed by a compact file ID: volume id, file key, and a cookie (the cookie is the write token that makes URLs hard to guess). Deletion marks space as free; a periodic compaction (weed shell → volume.gc / volume.compact) rewrites the volume, reclaiming the holes.

Why append-only? Because it turns random small writes into sequential big writes — the cheapest possible disk pattern — and makes replication trivially consistent: you replicate an append, not a random update.

2.2 Replication codes

Replication is per-volume, set at creation with a three-digit code:

The reference cluster runs -defaultReplication=000 deliberately: the data it stores (Kubernetes backups) is already restic-encrypted, and the second copy lives on the off-cluster mirror (Part 5). Replication in SeaweedFS is a tool for availability, not the only copy you will ever have — design the chain, don't let the tool decide it for you.

2.3 Erasure coding for cold data

For capacity efficiency, volumes can be erasure-coded (weed volume -dataCenter ... -ec): configurable data + parity shards (e.g. 4+2), stored across volume servers. EC trades CPU for raw capacity — roughly 1.5× efficiency at 2-failure tolerance, the same arithmetic Ceph users know from replicated vs. EC pools. The difference: with SeaweedFS you can EC specific old volumes while keeping hot volumes replicated.

Part 3 — The S3 Gateway and Tiering

weed s3 turns the filer into an S3-compatible endpoint: buckets, multipart uploads, lifecycle rules, and the full S3 client ecosystem (aws cli, s3cmd, rclone, restic) work unchanged.

aws --endpoint-url http://s3.example.internal:8333 s3 mb s3://backups
aws --endpoint-url http://s3.example.internal:8333 s3 cp k8up-backups/ s3://backups/ --recursive

Beyond plain S3, three features matter in production:

• Tiered storage — volume servers can be tagged by tier (hot NVMe, warm SATA, cold archive); the filer moves data between tiers by policy. Cheap for backups, fast for active data.
• The filer is the single metadata source — all S3 operations funnel through the filer's namespace, so "where is this bucket?" has one answer, not a partition map to consult.
• weed mount — a FUSE mount gives you POSIX access to the same namespace, which makes the store usable as a shared filesystem for tools that do not speak S3 (or as a crash-consistent backup target).

Part 4 — In Kubernetes: The Reference Deployment

The reference deployment runs SeaweedFS 3.78 inside a K3s cluster as four small workloads in a dedicated backup namespace:

The master is configured with -defaultReplication=000 and -volumeSizeLimitMB=30000; the volume server with -max=32 volumes. Every workload gets startup, liveness, and readiness probes against its HTTP endpoints (/cluster/status for the master, /status for volumes) — a volume server that cannot answer /status is restarted before it can serve bad data.

A NetworkPolicy restricts the namespace: only the backup operator, the mirror job, and the ingress can reach the S3 port; the master and volume ports are cluster-internal only.

4.1 K8up: backups of backups

The store exists to hold K8up backups — the Kubernetes backup operator that wraps restic. One Schedule CRD drives the whole lifecycle:

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: backup-schedule
spec:
  backend:
    repoPasswordSecretRef: { name: backup-repo, key: password }
    s3:
      endpoint: http://seaweedfs-s3.backup.svc.cluster.local:8333
      bucket: k8up-backups
      accessKeyIDSecretRef: { name: seaweedfs-s3-creds, key: accessKey }
      secretAccessKeySecretRef: { name: seaweedfs-s3-creds, key: secretKey }
  backup:
    schedule: "42 0 * * *" # nightly, after quiet hours
  check:
    schedule: "0 4 * * 1" # weekly restic check
  prune:
    schedule: "30 4 * * 1" # weekly retention prune

PVCs opt in with the annotation k8up.io/backup: "true". Restic talks S3 to the filer gateway; the encrypted snapshots land as filer data striped across the volume server. The layout is deliberately layered: application data → PVC (Ceph RBD) → K8up/restic → SeaweedFS → off-cluster mirror — four independent copies in three places, none of them on the same failure domain as the app.

4.2 The mirror: rclone to the off-cluster host

A CronJob runs nightly at 01:00 — after the 00:42 backup — and syncs the bucket to a dedicated off-cluster mirror host:

rclone sync source:k8up-backups dest:k8up-backups \
  --stats=1m --transfers=4 --checkers=8 \
  --contimeout=60s --timeout=300s --retries=3

rclone sync makes the destination match the source: deletions propagate, so the mirror never accumulates stale snapshots beyond the retention the backup schedule already enforces. Four parallel transfers and eight checkers keep a multi-terabyte store moving without saturating the link.

Part 5 — Hero: The TLS Saga (What Can Actually Go Wrong)

The mirror endpoint is exposed over HTTPS with a self-signed certificate — and the first integration attempt failed with TLS errors. The investigation found four stacked problems, all of them instructive:

1. The certificate had CN=localhost and no IP SAN for the mirror host. The mirror client (rclone → AWS SDK) rejects certificates whose hostname or IP is not in the SAN list. Classic.
2. The mirror host's app template refused to use the correct certificate — an API limitation of the NAS-style platform it runs on, not something you can click around.
3. SeaweedFS with TLS enabled bound only to IPv6, breaking Docker's IPv4 port publishing — the container was listening, but on a socket nothing could reach.
4. The platform's entrypoint script injected an unsupported -master flag into the weed s3 command, which is why the S3 gateway crashed on startup in the first place.

The fix was a hand-rolled Docker container that sidesteps the platform entirely:

docker run -d --name seaweedfs-s3 \
  --network host \
  --entrypoint /usr/bin/weed \        # skip the broken wrapper script
  -v /etc/seaweedfs/ssl:/etc/seaweedfs/ssl:ro \
  seaweedfs/seaweedfs:3.78 \
  s3 -filer=<filer-address> -cert=/etc/seaweedfs/ssl/tls.crt -key=/etc/seaweedfs/ssl/tls.key

With the certificate regenerated to include the host's IP SAN, the container bound via host networking (no Docker port publishing involved), and the entrypoint bypassed:

• ✅ https://127.0.0.1:30304/ — TLS works locally
• ✅ https://<mirror-host>:30304/ — TLS works remotely
• ✅ Certificate carries the IP SAN
• ✅ The endpoint answers AccessDenied — proving the TLS handshake completed and only the (expected) S3 auth rejected the request

The lessons generalize far beyond SeaweedFS:

• SANs are the contract. Any TLS client that validates hostnames — rclone, restic, the AWS SDK, curl — rejects a certificate whose SAN list does not contain the exact name or IP it connected to. CN=localhost is not a fix; it is a lie.
• "It works in Docker" and "it works on this platform" are different statements. Host networking, entrypoint wrappers, and template limitations each add a layer where configuration silently goes wrong. Probe the actual endpoint from the actual client.
• Verify with the failing tool. The final check that mattered was not openssl s_client but the mirror job's own client completing a handshake — AccessDenied beats TLS error any day.

Operations: What to Watch

The whole system stays tiny: one master, one volume server, one filer — all of them smaller than a single Ceph MON. That is the point. SeaweedFS is the storage you run when the workload speaks S3 and you want the operational surface of one binary, not a platform.

SeaweedFS vs. the Alternatives

If you already run Ceph, RGW is free — see the CephFS article for the filesystem side. If you do not already run Ceph and your workload speaks S3, SeaweedFS delivers the same class of capability with a fraction of the machinery.

The Hero Checklist

1. Master + volume servers + filer as separate, probe-checked workloads.
2. Deliberate replication decision per dataset — 000 only with an external copy.
3. S3 gateway exposed through the filer; bucket lifecycle set before day one.
4. K8up (or Velero) writing restic snapshots into it; opt-in via PVC annotations.
5. Nightly rclone mirror to an off-cluster host with TLS and verified SANs.
6. Scheduled compaction; monitored volume fill; a weekly restic check.
7. The mirror is tested by restore, at least quarterly — a backup you never restore is a rumor.

SeaweedFS rewards the engineer who respects its two ideas: keep the master out of the data path, and keep the copies deliberate. Do that, and a store that fits in one binary quietly becomes the most reliable component in your stack.

Related Articles

• CephFS: From Zero to Hero — the POSIX-filesystem sibling for workloads that need shared file semantics
• K3s Production Upgrades, Backup & Disaster Recovery — the cluster lifecycle the K8up schedules protect
• Disaster Recovery for Self-Hosted Services — where the encrypted mirror fits in the recovery hierarchy
• The Open-Source Infrastructure Stack for Production, 2026 — how the storage layers compose into a full stack