Self-Hosted AI Stack

Produktionsreife, selbstgehostete KI-Inferenz-Stack mit Ollama, Open WebUI und LiteLLM — einheitlicher API-Gateway, GPU-beschleunigte Inferenz und ein ChatGPT-kompatibles Chat-Interface.

Production-ready self-hosted AI inference stack with Ollama, Open WebUI, and LiteLLM — a unified API gateway, GPU-accelerated LLM inference, and a ChatGPT-compatible chat interface. Deploy in minutes on any Docker host with an NVIDIA GPU.

What's Included

• docker-compose.yml — 3 services: Ollama (LLM inference), Open WebUI (chat interface), LiteLLM (API gateway)
• Ollama config — GPU passthrough, model persistence, parallel request handling
• LiteLLM proxy config — OpenAI-compatible API, model routing, rate limiting, retry logic
• Open WebUI — ChatGPT-like interface with file uploads, RAG, and user management
• .env.example — All environment variables documented
• README.md — Architecture diagram, quick start, GPU setup, production checklist

Architecture

┌──────────────────────────────────────────────────────┐
│              Open WebUI (Port 3000)                   │
│         ChatGPT-like chat interface                   │
└──────────────────┬───────────────────────────────────┘
                   │  OpenAI-compatible API
                   ▼
┌──────────────────────────────────────────────────────┐
│              LiteLLM Proxy (Port 4000)                │
│         Model routing, rate limiting, retries         │
└──────────────────┬───────────────────────────────────┘
                   │  Native API
                   ▼
┌──────────────────────────────────────────────────────┐
│              Ollama (Port 11434)                      │
│         GPU-accelerated LLM inference engine          │
└──────────────────────────────────────────────────────┘

Why Route Through LiteLLM?

Instead of connecting Open WebUI directly to Ollama, LiteLLM acts as an API gateway:

• Unified API: Any tool that speaks OpenAI API works with your local models
• Model routing: Route requests to different models based on cost, speed, or capability
• Rate limiting: Protect your GPU from overload with per-model rate limits
• Retry logic: Automatic retries with cooldown on model failures
• Usage tracking: Optional PostgreSQL backend for request logging and analytics

Supported Models

| Model | Parameters | VRAM | Quality | Speed (RTX 3090) | | ----------------- | ---------- | ------- | -------------- | ------------------- | | Qwen 3.6 | 7.6B | 6 GB | Good | ~80 tok/s | | Mistral Small 3.1 | 24B | 16 GB | Very Good | ~40 tok/s | | DeepSeek V4 | 67B | 42 GB | Excellent | ~15 tok/s | | Llama 4 | 8B/70B | 6/42 GB | Good/Excellent | ~75/12 tok/s |

Quick Start

# 1. Configure
cp .env.example .env
# Edit .env — fill in WEBUI_SECRET_KEY, LITELLM_MASTER_KEY

# 2. Pull a model (requires 15-20GB for llama3.2)
docker compose run --rm ollama ollama pull llama3.2:latest

# 3. Start the stack
docker compose up -d

# 4. Open the dashboard
open http://localhost:3000    # Open WebUI
open http://localhost:4000    # LiteLLM proxy

Requirements

• Docker Engine 24+ with Docker Compose v2
• NVIDIA GPU with CUDA support (for GPU inference)
• NVIDIA Container Toolkit (nvidia-container-toolkit)
• 8GB+ VRAM (depends on model — see table above)
• 20GB+ disk space for model storage

Environment Variables

| Variable | Required | Default | Description | |---|---|---|---| | WEBUI_SECRET_KEY | Required | — | Open WebUI encryption key | | LITELLM_MASTER_KEY | Required | — | LiteLLM admin API key | | OLLAMA_KEEP_ALIVE | No | 5m | Time to keep models loaded in memory | | OLLAMA_NUM_PARALLEL | No | 2 | Parallel request handling per model | | OLLAMA_MAX_LOADED_MODELS | No | 3 | Max models kept in GPU memory | | ENABLE_SIGNUP | No | false | Allow new user registration | | DEFAULT_MODELS | No | llama3.2:latest | Default model in Open WebUI | | LITELLM_DB_URL | No | — | Optional PostgreSQL URL for usage tracking |

Production Checklist

• [ ] Set strong WEBUI_SECRET_KEY and LITELLM_MASTER_KEY
• [ ] Disable signup (ENABLE_SIGNUP=false)
• [ ] Configure HTTPS (reverse proxy with Caddy or Nginx)
• [ ] Set up persistent Open WebUI auth (WEBUI_AUTH=true)
• [ ] Monitor GPU memory: nvidia-smi -l 1
• [ ] Set up backup for Open WebUI data volume
• [ ] Pin container versions to specific tags (not latest or main)

Download

Download the zip, extract it, configure .env, run docker compose up -d.