Guide: Self-Hosted AI — Ollama, Open WebUI and LiteLLM

Guide: MinIO as an S3-Compatible Backup Target

Self-hosted S3-compatible object storage for backups: MinIO behind an Nginx proxy, with an rclone-based backup agent and versioned buckets. Your offsite backups without a cloud bill.

Guide: Production k3s Cluster with Rancher, Longhorn and MetalLB

A complete Kubernetes lab-to-production pattern: k3s in Docker with the embedded etcd datastore, Rancher for management, MetalLB for Layer-2 LoadBalancer IPs, Longhorn replicated storage, and Traefik

Run LLMs on your own GPU hardware: Ollama as the serving engine, Open WebUI as the ChatGPT-style frontend, and LiteLLM as an OpenAI-compatible router in front of everything — so local models and external APIs stay interchangeable.

What this is

A deployment guide with working examples — not a downloadable product. Everything below is the actual configuration I use, abridged to the parts that matter.

Deployment example

services:
  ollama:
    image: ollama/ollama:${OLLAMA_VERSION:-0.5.9}
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama_models:/root/.ollama
      - ./config/ollama:/etc/ollama:ro
    ports:
      - ${OLLAMA_PORT:-11434}:11434
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE:-5m}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-2}
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-3}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    healthcheck:
      test:
        - CMD
        - curl
        - -f
        - http://localhost:11434/api/tags
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - ai_network
  litellm:
    image: ghcr.io/berriai/litellm:${LITELLM_VERSION:-main-v1.16.18}
    container_name: litellm
    restart: unless-stopped
    depends_on:
      ollama:
        condition: service_healthy
    volumes:
      - ./config/litellm/proxy_config.yaml:/app/proxy_config.yaml:ro
    ports:
      - ${LITELLM_PORT:-4000}:4000
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:?LITELLM_MASTER_KEY is required}
      - DATABASE_URL=${LITELLM_DB_URL:-}
      - LITELLM_MODE=${LITELLM_MODE:-proxy}
      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
    command:
      - --config
      - /app/proxy_config.yaml
      - --port
      - "4000"
    healthcheck:
      test:
        - CMD
        - curl
        - -f
        - http://localhost:4000/health/liveliness
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - ai_network

Excerpt — ollama, litellm from the compose file. The remaining services (proxies, init jobs, exporters) follow the same pattern and mount their configuration from a config/ directory.

Essential environment variables

Copy .env.example to .env and at minimum set:

OLLAMA_VERSION=latest
OLLAMA_PORT=11434
OLLAMA_HOST=0.0.0.0
OLLAMA_KEEP_ALIVE=10m
OLLAMA_NUM_PARALLEL=2
OLLAMA_MAX_LOADED_MODELS=4

Production checklist

• Set strong, unique secrets in .env before the first start — never ship the example values
• Pin image versions and test upgrades on a staging copy first
• Configure and restore-test backups before you rely on them
• Put the stack behind TLS and restrict management UIs to a VPN or allowlist
• Wire the stack into monitoring and alerting from day one

Sizing the hardware, honestly

The VRAM math is simple enough to do on a napkin. Model weights at Q4 quantization take roughly half a byte per parameter: a 7B model needs about 4 GB, a 14B model about 8, a 32B model about 18. On top sits the KV cache, which grows with context length and concurrency - budget another 1-4 GB for realistic chat workloads. That is the whole rule: an 8 GB card runs 7-8B models comfortably, 16 GB opens 14B or long-context 8B, and 24 GB reaches the 32B class. Throughput expectations scale accordingly: single-digit tokens per second on CPU, 30-80 tokens per second on a modern consumer GPU for these sizes - interactive, not instant.

Choosing models day to day

Run one workhorse and one fallback, not a zoo. The workhorse is the biggest model that fits your VRAM with headroom for context; the fallback is a smaller quantized model that always fits, so the stack stays usable while you experiment. Q4_K_M-style quantization is the practical sweet spot - the perplexity cost over full precision is negligible for most tasks, the memory saving is not. Context length is the hidden tax: doubling it roughly doubles KV cache, which is why the 32B model at 4k context often outperforms a 14B model squeezed into 32k. Prompt quality moves results more than any of these knobs.

FAQ

What GPU do I need for local LLMs? VRAM is the deciding factor, not model marketing: an 8 GB card runs 7-8B models comfortably at quantized precision, 16-24 GB opens 13-32B territory. Ollama handles the quantization, and OLLAMA_NUM_PARALLEL controls how many requests share the card. CPU-only works for patient single users; interactive use wants a GPU.

Why put LiteLLM in front of Ollama? LiteLLM speaks the OpenAI API dialect to everything behind it, so applications get one endpoint and one credential format whether the model runs on your GPU or comes from an external API. Switching a feature from local to hosted becomes a model-name change, not a code change.

Can I mix local and cloud models in one setup? Yes - that is the router's reason to exist. Sensitive or high-volume workloads stay on local models, occasional heavy reasoning goes to an external API, and LiteLLM applies routing, fallbacks, and spend tracking uniformly across both.

Do models reload from disk on every request? No. OLLAMA_KEEP_ALIVE keeps loaded models resident after a request (5 minutes by default, configurable), and OLLAMA_MAX_LOADED_MODELS caps how many stay in VRAM simultaneously. Tuning these two removes the cold-start pause for interactive use.

A guide, not a product. This page is deployment documentation with working examples — there is no zip, no download, no support contract. You adapt the patterns to your own environment, and you own the result.

Want this running production-grade in your infrastructure instead? This is the core of the two-day Self-Hosted AI in Production workshop — details on the consulting page.