MinIO in Production: S3-Compatible Object Storage with Automated Backup

Ceph Deep Dive: RADOS, BlueStore, CRUSH, and Running a Sovereign Storage Cluster

Everything a senior engineer needs to know about Ceph in 2026: the RADOS architecture, BlueStore internals, CRUSH data placement, cephadm operations, and what Squid 19.2 and Tentacle 20.2 change for production clusters.

Why MinIO

MinIO has become the de-facto standard for self-hosted S3-compatible storage. It provides a drop-in replacement for AWS S3 that runs anywhere — laptop, bare metal, Kubernetes — with the same API semantics.

The key architectural decision: MinIO replaces local filesystem storage with an S3 API. This decouples your application data from the server it runs on. Backups, replication, and disaster recovery become S3-native operations rather than filesystem scripts.

Architecture

S3 API Compatibility

MinIO implements the same S3 API that AWS S3 uses. Any tool that speaks S3 works with MinIO:

import boto3

client = boto3.client(
    "s3",
    endpoint_url="http://minio:9000",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="<password>",
)
client.create_bucket(Bucket="data")
client.upload_file("backup.sql", "data", "backups/daily.sql")

This compatibility extends to all S3 features MinIO supports:

• Presigned URLs (temporary access without credentials)
• Multipart uploads (large file support)
• Bucket policies and IAM-style access control
• Object locking and retention
• Event notifications (via bucket webhooks or AMQP)

What MinIO does NOT support (compared to AWS S3):

• S3 Select (server-side filtering)
• S3 Object Lambda
• Glacier/Deep Archive storage classes
• Cross-region replication (single-region only)

For most self-hosted workloads, the supported feature set is complete.

MinIO Configuration Decisions

Root Credentials

MinIO requires a root user (access key + secret key). Unlike AWS IAM, there's no separate root account — the credentials you set in .env are the superadmin:

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=a-strong-32-character-password

Important limitations:

• Root user cannot be deleted or disabled
• Root credentials cannot have their permissions reduced
• Create separate access keys for applications via the web console
• Rotate root password periodically using mc admin user info

Storage Layout

MinIO maps a local filesystem path (/data) to the S3 API. Each bucket becomes a subdirectory:

/data/
├── .minio.sys/          # MinIO internal metadata
├── data/                # S3 bucket "data"
│   ├── uploads/
│   └── archives/
├── backups/             # S3 bucket "backups"
└── logs/                # S3 bucket "logs"

This flat file structure means you can browse MinIO data directly on disk (read-only — never write directly to /data outside MinIO).

Erasure Coding (Multi-Disk)

For production deployments with multiple disks, MinIO's erasure coding provides RAID-like resilience without RAID overhead:

# With 4 data + 2 parity drives, tolerate 2 drive failures
docker compose run --rm minio server /disk{1..6}

Erasure coding uses Reed-Solomon encoding — data is split into N data shards and M parity shards. Storage overhead is M/N. For 4+2, overhead is 50% but tolerates 2 simultaneous drive failures.

Nginx Proxy Configuration

The Nginx proxy in this stack serves three purposes:

1. Large Upload Support

MinIO's default configuration has conservative limits. The Nginx proxy overrides these for applications that need large uploads:

client_max_body_size 5G;
client_body_buffer_size 10M;
proxy_request_buffering off;

Setting proxy_request_buffering off is critical for uploads over 1GB — without it, Nginx buffers the entire request body to disk before sending to MinIO, causing writes for large uploads to stall.

2. Rate Limiting

Object storage is bandwidth-intensive. Rate limiting prevents any single client from saturating the network connection:

limit_req zone=minio_api burst=200 nodelay;

The burst size of 200 allows short spikes while enforcing a sustained rate limit of 100 requests/second.

3. Connection Tuning

S3 API connections can be long-lived (especially for multipart uploads):

proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;

These prevent Nginx from killing idle S3 connections prematurely.

Backup Strategy

The backup-agent container implements a layered backup strategy:

Layer 1: Local Mirroring (mc mirror)

mc mirror --overwrite --watch local/data remote/data

The --watch flag enables continuous sync — changed files are mirrored immediately. For scheduled backups (e.g., nightly), omit --watch and run on a cron schedule.

Layer 2: Offsite Backup (S3-to-S3)

Configure a remote endpoint (Backblaze B2, AWS S3, Wasabi) and mc mirror all buckets to the remote. The advantage of object-to-object backup vs. file-level backup:

• No intermediate storage: Data streams directly from MinIO to the backup target
• Incremental by default: mc mirror transfers only changed objects
• Versioning preserved: Object metadata and version IDs are retained
• No local disk impact: Backup agent runs in a separate container with no local storage requirements

Layer 3: Lifecycle Policies

Automatically expire old data using bucket lifecycle rules:

{
  "Rules": [
    {
      "ID": "expire-old-backups",
      "Status": "Enabled",
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

This runs at the MinIO level — no external cron job required.

Production Checklist

• Set strong MINIO_ROOT_PASSWORD (24+ characters, alphanumeric + symbols)
• Configure HTTPS via the SSL Reverse Proxy stack
• Enable erasure coding for multi-disk resilience (use minio server /disk{1..N})
• Set up monitoring — MinIO exposes Prometheus metrics at /minio/v2/metrics/cluster
• Configure bucket versioning for accidental-delete protection
• Pin MINIO_VERSION to a specific release, not :latest
• Test backup restore — mc mirror remote/data local/data in reverse
• Set mc admin heal on a weekly cron for degraded drive detection
• Limit bucket access — create per-application access keys, don't share root

Key Takeaways

• MinIO is a drop-in S3 replacement — any S3 SDK or CLI tool works without modification
• The Nginx proxy is essential — without it, large uploads time out and single clients can saturate bandwidth
• Backup should be S3-to-S3 — mc mirror is more reliable than filesystem-level backup for object storage
• Erasure coding replaces RAID — use 4+2 (4 data + 2 parity) for production resilience
• Never write to MinIO's data directory directly — always use the S3 API