VM/Container/Pod Resource Tuning: A Comprehensive Production Guide

Container Building and Hardening: A Production-Ready Guide

Comprehensive guide to modern container image building tools, patterns, and security hardening techniques for production deployments.

Introduction

You've seen the symptoms:

• Databases pause under load because vm.swappiness is too aggressive
• Nginx hits "too many open files" despite seemingly reasonable traffic
• OOM killer terminates the wrong pod because requests weren't set
• CPUs throttle at 60% due to noisy neighbors
• PostgreSQL sheds connections when kernel.shm* isn't tuned for shared memory

Modern infrastructure layers introduce countless tunable knobs: kernel parameters, Docker daemon settings, container limits, Kubernetes resource policies, and workload-specific adjustments. Getting these right is the difference between a system that holds 10× the load and one that collapses at scale.

This guide codifies the tuning patterns from production experience, organized by isolation layer with workload-specific cheat sheets.

Part 1: The Isolation Hierarchy

Resource controls operate at distinct layers, each with its own knobs and trade-offs:

Each layer can override the layer above it, but defaults flow down incorrectly if not configured intentionally.

Part 2: Host/VM Kernel Tuning

Critical Network Parameters

# /etc/sysctl.d/99-production.conf
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
# Security
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1

Apply immediately:

sudo sysctl -p /etc/sysctl.d/99-production.conf

Memory Management

# /etc/sysctl.d/99-memory.conf
vm.swappiness = 10               # Don't swap unless critical
vm.overcommit_memory = 1         # K8s nodes expect this
vm.overcommit_ratio = 100        # Full overcommit
vm.dirty_ratio = 20              # Start blocking writes at 20%
vm.dirty_background_ratio = 5    # Background writeback at 5%
vm.min_free_kbytes = 1048576     # 1GB reserve on 64GB node

Filesystem

# /etc/sysctl.d/99-fs.conf
fs.file-max = 2097152            # Global open file limit
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 819200  # For etcd, collectd

Per-Process Limits (/etc/security/limits.conf)

# /etc/security/limits.conf
*       soft    nofile  1048576
*       hard    nofile  1048576
*       soft    nproc   65535
*       hard    nproc   65535
root    soft    nofile  1048576
root    hard    nofile  1048576

Part 3: Docker Daemon Tuning

daemon.json Production Defaults

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  },
  "default-ulimits": {
    "nofile": { "Name": "nofile", "Hard": 1048576, "Soft": 65535 },
    "nproc": { "Name": "nproc", "Hard": 65535, "Soft": 4096 }
  },
  "live-restore": true,
  "userns-remap": "default",
  "no-new-privileges": false,
  "max-concurrent-downloads": 10,
  "max-concurrent-uploads": 10,
  "data-root": "/var/lib/docker"
}

Container Runtime Defaults

Docker's default resource roots are generous but can lead to noisy neighbors:

docker run -d \
  --name myapp \
  --memory=512m \
  --memory-reservation=400m \
  --memory-swap=-1 \
  --pids-limit=200 \
  --ulimit nofile=65535:65535 \
  --ulimit nproc=4096:8192 \
  --security-opt no-new-privileges \
  --security-opt seccomp=default.json \
  myapp:latest

Part 4: Kubernetes Resource Controls

QoS Classes Explained

Kubernetes assigns QoS based on requests and limits:

# Guaranteed (best for DBs, latency-critical services)
resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    cpu: "2"
    memory: "8Gi"

# Burstable (typical for stateless services)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "4Gi"

# ❌ BestEffort (avoid in production)
resources: {} # or omit entirely

Production rule: Always set requests. At least assign requests.memory = limits.memory / 2 for non-latency-critical workloads.

Memory Swapping (Kubelet)

On cgroup v2, memory.swap accounting is enabled. Ensure kubelet config:

# /var/lib/kubelet/config.yaml
memorySwap:
  swapBehavior: NoSwap # NoSwap, LimitedSwap, UnlimitedSwap

• NoSwap (recommended): Memory limits act as hard limits; OOM kills container
• UnlimitedSwap: Container can swap to disk; limits.memory is ignored for swap
• LimitedSwap: Swap limited proportionally to memory reservation

Huge Pages

apiVersion: v1
kind: Pod
metadata:
  name: hugepages-pod
spec:
  containers:
    - name: app
      image: myapp:latest
      resources:
        limits:
          Hugepages-2Mi: 512Mi # 256 × 2Mi pages
          memory: "1Gi"
      volumeMounts:
        - mountPath: /hugepages
          name: hugepage
  volumes:
    - name: hugepage
      emptyDir:
        medium: HugePages

Use for: databases (PostgreSQL, MySQL), DPDK applications, in-memory caches.

CPU Manager Policy

# /var/lib/kubelet/config.yaml
cpuManagerPolicy: static # none (default), static
cpuManagerReconcilePeriod: 5s

• static: Pods with requests.cpu = integer cores get exclusive cores. Ideal for low-latency workloads.
• none: Default; cores are shared (CFS quotas).

# Pod requesting exclusive CPU 0-1
resources:
  requests:
    cpu: "2" # Must be integer
  limits:
    cpu: "2"

Topology Manager Policy

# /var/lib/kubelet/config.yaml
topologyManagerPolicy: best-effort # none, best-effort, restricted, single-numa-node

• none: No NUMA awareness
• best-effort: Prefer resources on same NUMA node; fit silently if not
• restricted: Reject pod if NUMA placement fails
• single-numa-node: Reject if resources span multiple NUMA nodes (most aggressive)

# Pods requesting aligned resources
resources:
  requests:
    cpu: "4"
    memory: "8Gi"
    cpu: "4" # Topology manager aligns with same NUMA node
    devices.kubernetes.com/mock: "1" # Device affinity

LimitRange (Namespace-wide Defaults)

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
    - type: Container
      default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      max:
        cpu: "4"
        memory: "8Gi"
      min:
        cpu: "50m"
        memory: "64Mi"
      maxLimitRequestRatio:
        cpu: "10"
        memory: "4"

• maxLimitRequestRatio: Prevents a pod from reserving 10× memory but using 0.1× (resource hoarding).

Part 5: Workload-Specific Cheat Sheets

PostgreSQL

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-tuning
data:
  # Postgres expects shared memory segments aligned with kernel
  # Adjust kernel.shm* if deploying to bare metal VM
  # /etc/sysctl.d/99-postgres.conf:
  #   kernel.shmmax = 4294967296         # 4GB
  #   kernel.shmall = 1048576           # 4GB pages
  #   kernel.sem = 250 32000 100 128
  postgresql.conf: |
    shared_buffers = 4GB               # 25% of RAM
    effective_cache_size = 12GB        # 75% of RAM
    work_mem = 64MB                    # per-operation sort memory
    maintenance_work_mem = 1GB
    max_connections = 200
    wal_buffers = 128MB
    checkpoint_completion_target = 0.9
---
apiVersion: v1
kind: Pod
metadata:
  name: postgres
spec:
  containers:
    - name: postgres
      image: postgres:16-alpine
      resources:
        requests:
          cpu: "2"
          memory: "8Gi"
        limits:
          cpu: "4"
          memory: "16Gi" # OOM target is 2× requests
      securityContext:
        fsGroup: 999 # Fix permission issues on /var/run/postgresql
      volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: data
        - mountPath: /var/run/postgresql
          name: run
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: postgres-pvc
    - name: run
      emptyDir: {}

Redis

resources:
  requests:
    cpu: "500m"
    memory: "4Gi"
  limits:
    cpu: "2"
    memory: "8Gi"
# In redis.conf:
# maxmemory 6gb
# maxmemory-policy allkeys-lru

Nginx

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "1"
    memory: "512Mi"
# In nginx.conf:
# worker_processes auto;
# worker_connections 4096;  # Watch open file limits
# worker_rlimit_nofile 65535;
# multi_accept on;
# access_log off;  # For high-throughput, pipe to syslog

Kafka

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    cpu: "8"
    memory: "32Gi"
# In server.properties:
# log.dirs = /kafka/data
# num.network.threads = 8
# num.io.threads = 16
# socket.send.buffer.bytes = 102400
# socket.receive.buffer.bytes = 102400
# log.flush.interval.messages = 10000
# log.flush.interval.ms = 1000
# num Partitions: 2× broker count

Elasticsearch

resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    cpu: "4"
    memory: "16Gi"
# In jvm.options:
# -Xms8g
# -Xmx8g  # Don't exceed 50% of pod memory for heap
# In elasticsearch.yml:
# bootstrap.memory_lock: true
# indices.queries.cache.size: 10%
# indices.fielddata.cache.size: 20%

JVM Applications

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "2"
    memory: "4Gi"
# JVM args:
# -Xms512m              # Start with 512MB heap
# -Xmx1536m             # Limit heap to 50% of pod memory (non-heap overhead)
# -XX:+UseG1GC
# -XX:MaxGCPauseMillis=200
# -XX:+PrintGCDetails
# -XX:+PrintGCDateStamps
# -Xloggc:/logs/gc.log

ML / GPU Workloads

apiVersion: v1
kind: Pod
metadata:
  name: ml-worker
spec:
  runtimeClassName: nvidia # For NVIDIA GPU support
  containers:
    - name: ml-app
      image: pytorch:24.01-cuda12.1
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
          nvidia.com/gpu: 1
        limits:
          cpu: "8"
          memory: "32Gi"
          nvidia.com/gpu: 1
      env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
        - name: NVIDIA_VISIBLE_DEVICES
          value: "all"

Part 6: Security-Tuned Production

Pod Security Standards Enforced

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Hardened Pod Example

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
          add:
            - NET_BIND_SERVICE # If needed
        seccompProfile:
          type: RuntimeDefault
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "512Mi"
          ephemeral-storage: "1Gi"
      volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: run
          mountPath: /var/run
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: "100Mi"
    - name: run
      emptyDir:
        sizeLimit: "50Mi"

seccomp RuntimeDefault

The RuntimeDefault profile blocks dangerous syscalls:

• kexec_load, module_load, delete_module (kernel persistence)
• ptrace, process_vm_readv/writev (process manipulation)
• mount, umount, pivot_root (filesystem remapping)
• clock_settime, settimeofday (system time manipulation)
• bpf (unprivileged BPF is blocked by kernel.unprivileged_bpf_disabled=1)

Use custom profiles only when required (e.g., some FUSE workloads).

Part 7: Observability & Debugging

Check Current Kernel Settings

# Network
sysctl net.core.somaxconn net.ipv4.tcp_tw_reuse net.ipv4.ip_local_port_range

# Memory
sysctl vm.swappiness vm.dirty_ratio vm.min_free_kbytes

# Filesystem
sysctl fs.file-max fs.inotify.max_user_watches

# Per-node
cat /proc/sys/vm/swappiness
cat /proc/sys/net/core/somaxconn

Inspect cgroup v2 Limits

# For a container, find its cgroup path
docker inspect --format '{{.State.Pid}}' myapp | xargs cat /proc/$$/cgroup

# Check limits
cat /sys/fs/cgroup/memory.max           # Memory limit
cat /sys/fs/cgroup/cpu.max              # CPU quota (e.g., "200000 100000")
cat /sys/fs/cgroup/pids.max             # PID limit
cat /sys/fs/cgroup/memory.swap.max      # Swap limit

Pod QoS Assignment

kubectl get pod -o jsonpath='{.metadata.name}{"\t"}{.status.qosClass}{"\n"}'

# Guaranteed? Look at .spec.containers[].resources
kubectl get pod -o json | jq '.items[] | select(.metadata.name=="myapp") |
  {
    name: .metadata.name,
    qos: .status.qosClass,
    resources: .spec.containers[].resources
  }'

Detect High CPU Throttling

# From Prometheus
rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])

# Alert if > 0.5 (50% throttled)

Detect High FD Usage

# Check per-container fd count
find /proc/*/fd 2>/dev/null | xargs -I{} sh -c 'echo $(dirname $(dirname {})): $(ls {}/fd | wc -l)' | sort -rn | head

# Or use cgroup
cat /proc/{PID}/limits  | grep "Max open files"

Check OOM Kills

# On host
dmesg | grep -i "killed process"

# On K8s
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilled

Part 8: Common Pitfalls

❌ Missing requests on All Pods

# Bad
resources:
  limits:
    memory: "512Mi"
    cpu: "1"

# becomes BestEffort if requests not set

Result: First to evict; unpredictable scheduling; node resource exhaustion.

kernel.shmmax Too Low

# PostgreSQL fails to start:
# FATAL: could not create shared memory segment: Invalid argument

Fix: Adjust kernel parameters for database pods:

# /etc/sysctl.d/99-database.conf
kernel.shmmax = 68719476736     # 64GB
kernel.shmall = 16777216        # 64GB / 4096 (page size)

--pids-limit Not Set

# Fork bomb in one container can crash the entire node

Fix: Set limits on daemon or container:

{
  "default-pids-limit": 200
}

Or per-container:

docker run --pids-limit 200 myapp

Memory Swap Equals Unlimited

# Default: limits.memory=512Mi, memory-swap=unlimited
# → Container can swap to disk (slow) and exhaust host swap

Fix: Set --memory-swap=-1 to disable swapping, or kubelet memorySwap.swapBehavior: NoSwap.

runAsNonRoot + fsGroup Mismatch

# PostgreSQL sleeps at startup, can't read/write /var/run/postgresql

Fix: Set fsGroup to match DB user:

securityContext:
  runAsUser: 999
  runAsGroup: 999
  fsGroup: 999

Time-Back-Forward Time Sync

# Kubernetes operations break when clocks are set back in time

Fix: Use NTP with maxslewrate or makestep to allow large forward jumps but reject backward jumps:

# /etc/chrony.conf
maxslewrate 1000         # Allow 1000x faster clock catch-up
makestep 1.0 -1          # Step clock if offset > 1 second (forward only)

seccomp Profile Missing

# Security review fails: "RuntimeDefault not set"

Fix: Explicitly set profile:

podSecurityContext:
  seccompProfile:
    type: RuntimeDefault

Part 9: Prometheus Alerts RCA

Memory Exhaustion

# Alert: Pod Memory Usage Nearing Limit
- alert: PodMemoryHigh
  expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Pod {{ $labels.pod }} memory > 90% of limit"

# Alert: OOM Kill
- alert: PodOOMKilling
  expr: rate(kube_pod_container_status_restarts_total{reason="OOMKilled"}[5m]) > 0
  for: 0m
  annotations:
    summary: "Container {{ $labels.container }} in pod {{ $labels.pod }} OOM killed"

RCA:

1. Check limits.memory is appropriate (is app memory-intensive?)
2. Look for memory leaks: kubectl logs ... + heap dump (JVM) or ps + pmap (native)
3. If intentional load, increase requests.memory and limits.memory

CPU Throttling

# Alert: High CPU Throttling
- alert: HighCPUThrottling
  expr: |
    rate(container_cpu_cfs_throttled_periods_total[5m]) /
    rate(container_cpu_cfs_periods_total[5m]) > 0.5
  for: 10m
  annotations:
    summary: "Container {{ $labels.container }} throttled > 50%"

RCA:

1. Add more cores (increase limits.cpu)
2. Move latency-critical pods to dedicated nodes with nodeSelector
3. Consider CPU Manager static policy for guaranteed exclusive cores

Connection Backlog

# Alert: TCP Listen Drops / SYN Flood
- alert: TCPListenDrops
  expr: rate(netstat_Tcp_ListenDrops[5m]) > 10
  for: 5m
  annotations:
    summary: "High TCP listen drops (increase net.core.somaxconn?)"

RCA:

1. Increase net.core.somaxconn on node
2. Tune net.ipv4.tcp_max_syn_backlog
3. Consider connection pooling / load balancer scaling

Summary: Tuning Checklist

Host/VM Layer

• Network: somaxconn=4096, tcp_max_syn_backlog=8192, tw_reuse=1
• Memory: swappiness=10 (DB), dirty_ratio=20, min_free_kbytes=1-2%
• Filesystem: file-max=2097152, inotify.max_user_watches=819200
• Security: kexec_load_disabled=1, unprivileged_bpf_disabled=1, yama.ptrace_scope=2

Docker Daemon

• default-ulimits set (nofile, nproc)
• userns-remap=default (UID remap)
• live-restore=true (daemon restart resilience)
• log-opts.max-size=50m (log rotation)

Kubernetes Namespace

• LimitRange defined (default requests + limits)
• ResourceQuota defined (prevent namespace exhaustion)
• pod-security.kubernetes.io/enforce=restricted
• NetworkPolicy default-deny + explicit allow

Pod/Container

• requests.cpu and requests.memory set always
• limits.cpu and limits.memory set (or reason documented)
• securityContext.runAsNonRoot
• allowPrivilegeEscalation=false
• readOnlyRootFilesystem=true with tmpfs mounts
• seccompProfile.type=RuntimeDefault
• pids-limit set (on daemon or container)
• volumeMounts for /tmp, /var/run, /var/cache

Workload-Specific

• PostgreSQL: kernel.shm* tuned via host sysctls
• Redis: maxmemory + maxmemory-policy set
• Nginx: worker_connections, worker_rlimit_nofile aligned
• Kafka: Multi-broker, partition planning
• JVM: -Xms/-Xmx = 25-50% of pod memory
• ML/GPU: runtimeClassName: nvidia, nvidia.com/gpu requests

Key Takeaways

1. Tune at every layer: sysctl → kubelet → pod → container → application
2. Always set requests: BestEffort is first to evict and unpredictable
3. QoS matters: Guaranteed for latency-critical; Burstable for general workloads
4. Security goes hand-in-hand: cap-drop ALL, no-new-privileges, seccomp=RuntimeDefault reduce attack surface without performance cost
5. Observe before you tune: Use Prometheus, dmesg, sysctl, and container logs to identify bottlenecks
6. Workload-specific defaults matter: PostgreSQL needs shared memory, Redis needs maxmemory, JVM needs headroom beyond heap
7. Document drift: Reapply sysctls via systemd units (--file=/etc/sysctl.d/*.conf) and kubelet configs

A well-tuned environment holds 10× the traffic, recovers gracefully from failures, and remains secure against common container escape vectors.

Further Reading

• Linux Kernel Documentation: man sysctl, man cgroups
• Kubernetes Documentation: Resource Management, Pod Security Standards
• Docker Docs: daemon.json reference, runtime options
• CIS Benchmarks: Docker, Kubernetes
• eBPF for Observability: bcc-tools, bpftrace for advanced containers debug