Skip to content

Latest commit

 

History

History
458 lines (341 loc) · 9.85 KB

DOCKER.md

File metadata and controls

458 lines (341 loc) · 9.85 KB

Docker Support for llms-py

This document provides detailed information about running llms-py in Docker.

Quick Start

Using Pre-built Images

# Pull and run the latest image
docker pull ghcr.io/servicestack/llms:latest
docker run -p 8000:8000 -e OPENROUTER_API_KEY="your-key" ghcr.io/servicestack/llms:latest

Using docker-compose (Recommended)

  1. Create a .env file with your API keys:
OPENROUTER_API_KEY=sk-or-...
GROQ_API_KEY=gsk_...
GOOGLE_FREE_API_KEY=AIza...
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GROK_API_KEY=xai-...
DASHSCOPE_API_KEY=sk-...
ZAI_API_KEY=sk-...
MISTRAL_API_KEY=...
  1. Start the service:
docker-compose up -d
  1. Access the UI at http://localhost:8000

Files Created

Dockerfile

Multi-stage Docker build that:

  • Uses Python 3.11 slim base image
  • Builds the package from source
  • Installs bun runtime
  • Installs dotnet-sdk 10.0
  • Runs as non-root user for security
  • Includes health checks
  • Exposes port 8000
  • Default command: llms --serve 8000

.dockerignore

Excludes unnecessary files from the Docker build context to reduce image size and build time.

docker-compose.yml

Provides easy orchestration with:

  • Port mapping (8000:8000)
  • Environment variable support via .env file
  • Named volume for data persistence
  • Automatic restart policy
  • Health checks

docker-build.sh

Convenience script for building the Docker image locally:

./docker-build.sh [tag]

.github/workflows/docker-publish.yml

GitHub Actions workflow that:

  • Builds Docker images on push to main and tags
  • Publishes to GitHub Container Registry (ghcr.io)
  • Supports multi-architecture builds (amd64, arm64)
  • Creates image tags for branches, PRs, and semantic versions
  • Uses Docker layer caching for faster builds

Usage Examples

Basic Server

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

With Multiple API Keys

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="sk-or-..." \
  -e GROQ_API_KEY="gsk_..." \
  -e GOOGLE_FREE_API_KEY="AIza..." \
  -e ANTHROPIC_API_KEY="sk-ant-..." \
  ghcr.io/servicestack/llms:latest

With Persistent Storage

docker run -p 8000:8000 \
  -v llms-data:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

CLI Usage

# Single query
docker run --rm \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms "What is the capital of France?"

# List models
docker run --rm \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms --list

# Check provider
docker run --rm \
  -e GROQ_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms --check groq

Custom Port

# Run on port 3000
docker run -p 3000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

With Verbose Logging

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms --serve 8000 --verbose

Building Locally

Using the Build Script

./docker-build.sh

This builds the image as llms-py:latest.

Manual Build

docker build -t llms-py:latest .

Build with Custom Tag

./docker-build.sh v2.0.24

Docker Compose

Using Pre-built Image (Recommended for Users)

The default docker-compose.yml uses the pre-built image from GitHub Container Registry:

# Start services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

Building from Source (For Developers)

If you've cloned the repository and want to build from source, use docker-compose.local.yml:

# Build and start services
docker-compose -f docker-compose.local.yml up -d --build

# View logs
docker-compose -f docker-compose.local.yml logs -f

# Stop services
docker-compose -f docker-compose.local.yml down

# Rebuild and restart
docker-compose -f docker-compose.local.yml up -d --build

Data Persistence

The container stores configuration and analytics data in /home/llms/.llms.

On first run, the container will automatically create default llms.json and providers.json files in this directory.

Named Volume (Recommended)

docker run -p 8000:8000 \
  -v llms-data:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Local Directory

docker run -p 8000:8000 \
  -v $(pwd)/llms-config:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Custom Configuration Files

You can customize the behavior of llms-py by providing your own llms.json and providers.json configuration files.

Method 1: Mount a Local Directory (Recommended)

  1. Create a local directory with your custom config files:
# Option A: Use the provided extraction script (easiest)  
./docker-extract-configs.sh config

# Option B: Manual extraction
mkdir -p config
docker run --rm -v $(pwd)/config:/home/llms/.llms \
  ghcr.io/servicestack/llms:latest \
  llms --init

  1. Edit config/llms.json and config/providers.json to your preferences

  2. Mount the directory when running the container:

docker run -p 8000:8000 \
  -v $(pwd)/config:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Or with docker-compose, update the volumes section:

volumes:
  - ./config:/home/llms/.llms

Method 2: Mount Individual Config Files

Mount specific config files (read-only recommended to prevent accidental changes):

docker run -p 8000:8000 \
  -v $(pwd)/my-llms.json:/home/llms/.llms/llms.json:ro \
  -v $(pwd)/my-providers.json:/home/llms/.llms/providers.json:ro \
  -v $(pwd)/my-providers-extra.json:/home/llms/.llms/providers-extra.json:ro \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Or with docker-compose:

volumes:
  - ./my-llms.json:/home/llms/.llms/llms.json:ro
  - ./my-providers.json:/home/llms/.llms/providers.json:ro
  - ./my-providers-extra.json:/home/llms/.llms/providers-extra.json:ro

Method 3: Initialize and Extract Configs

  1. Run the container with a named volume to initialize default configs:
docker run --rm \
  -v llms-data:/home/llms/.llms \
  ghcr.io/servicestack/llms:latest \
  llms --init
  1. Extract the configs to customize them:
# Create a temporary container to copy files
docker run -d --name llms-temp -v llms-data:/home/llms/.llms ghcr.io/servicestack/llms:latest sleep 60
docker cp llms-temp:/home/llms/.llms/llms.json ./llms.json
docker cp llms-temp:/home/llms/.llms/providers.json ./providers.json
docker cp llms-temp:/home/llms/.llms/providers-extra.json ./providers-extra.json
docker rm -f llms-temp
  1. Edit the files and copy them back:
# After editing, copy back
docker run -d --name llms-temp -v llms-data:/home/llms/.llms ghcr.io/servicestack/llms:latest sleep 60
docker cp ./llms.json llms-temp:/home/llms/.llms/llms.json
docker cp ./providers.json llms-temp:/home/llms/.llms/providers.json
docker cp ./providers-extra.json llms-temp:/home/llms/.llms/providers-extra.json
docker rm -f llms-temp

What Can You Customize?

In llms.json:

  • Enable/disable providers
  • Add or remove models
  • Configure API endpoints
  • Set pricing information
  • Customize default chat templates
  • Configure provider-specific settings

In providers-extra.json:

  • additional list of providers and models

Example: Custom Provider Configuration

Create a custom llms.json with only the providers you want:

{
  "defaults": {
    "headers": {
      "Content-Type": "application/json"
    },
    "text": {
      "model": "llama3.3:70b",
      "messages": [
        {
          "role": "user",
          "content": ""
        }
      ]
    }
  },
  "providers": {
    "groq": {
      "enabled": true,
    }
  }
}

Then mount it:

docker run -p 8000:8000 \
  -v $(pwd)/custom-llms.json:/home/llms/.llms/llms.json:ro \
  -e GROQ_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Health Checks

The Docker image includes a health check that verifies the server is responding.

Check Container Health

docker ps

Look for the health status in the STATUS column.

View Health Check Details

docker inspect --format='{{json .State.Health}}' llms-server | jq

Multi-Architecture Support

The published Docker images support:

  • linux/amd64 (Intel/AMD x86_64)
  • linux/arm64 (ARM64/Apple Silicon)

Docker automatically pulls the correct image for your platform.

GitHub Container Registry

Images are automatically published to GitHub Container Registry on:

  • Push to main branch → ghcr.io/servicestack/llms:main
  • Tagged releases → ghcr.io/servicestack/llms:v2.0.24
  • Latest tag → ghcr.io/servicestack/llms:latest

Pull Specific Version

docker pull ghcr.io/servicestack/llms:v2.0.24

Pull Latest

docker pull ghcr.io/servicestack/llms:latest

Troubleshooting

Container Won't Start

Check logs:

docker logs llms-server

Permission Issues

The container runs as user llms (UID 1000). If mounting local directories, ensure they're writable:

mkdir -p llms-config
chmod 777 llms-config

Port Already in Use

Change the host port:

docker run -p 3000:8000 ...

API Keys Not Working

Verify environment variables are set:

docker exec llms-server env | grep API_KEY

Security Considerations

  • Container runs as non-root user (UID 1000)
  • Only port 8000 is exposed
  • No unnecessary packages installed
  • Multi-stage build reduces attack surface
  • Health checks ensure service availability

Performance

  • Multi-stage build keeps final image small
  • Layer caching speeds up rebuilds
  • aiohttp provides async performance
  • Health checks prevent routing to unhealthy containers