Deploy AI Proxy

Deploy AI Proxy as a Docker Compose stack on a Linux host inside your network. After the stack is running, you connect it to OpenLM Platform and set up providers in Configure AI Proxy.

Before you begin

Confirm these prerequisites:

• A Linux host that meets the system requirements.
• Docker Engine and the Docker Compose v2 plugin are installed. Confirm with docker compose version.
• Access to the AI Proxy container images, provided by OpenLM. If the images are in a private registry, sign in first with docker login.
• Network access as described in the system requirements.

Deploy the stack

Create a file named docker-compose.yml in a working directory on the host, for example ~/aiproxy/, with the following content:

services:
  mongodb:
    image: mongo:7
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    command: redis-server --save 60 1 --loglevel warning --requirepass openlm
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "openlm", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  aiproxyapi:
    image: public.ecr.aws/r3q3q2f4/olm-aiproxyapi:1.260217.1514
    ports:
      - "8088:8088"
    depends_on:
      mongodb:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION=mongodb://admin:admin@mongodb:27017
      - REDIS__CONNECTIONSTRING=redis,password=openlm
      - CLOUD_MODE=false
      - ENCRYPTION_KEY=default-dev-key
    restart: unless-stopped

  aiproxyui:
    image: public.ecr.aws/r3q3q2f4/olm-aiproxyui:1.260630.1443
    ports:
      - "8092:8092"
    depends_on:
      - aiproxyapi
    environment:
      - LB_FQDN=localhost
    restart: unless-stopped

volumes:
  mongodb_data:
  redis_data:

Review the values before you start the stack — in particular the database credentials, the encryption key, and the host ports for the API and UI. Change any default values to suit your environment. You configure the OpenLM Platform connection and providers later, in the UI.

Start the stack in the background. Compose pulls the images on the first run:

docker compose up -d

Confirm all containers report a running state:

docker compose ps

Verify the deployment

• Containers are running

  docker compose ps lists the API, UI, MongoDB, and Redis containers in a running state.

• The UI loads

  Open http://<aiproxy-host>:<ui-port>/aiproxy/ in a browser. The AI Proxy interface loads. Replace <aiproxy-host> and <ui-port> with your host and the UI port you mapped in docker-compose.yml.

• A test request is recorded

  Route a request through AI Proxy, then confirm a matching record appears in the UI under usage records. See Configure AI Proxy for how to route a client.

Update a deployment

To move to a newer AI Proxy version:

1. Update the image tags in docker-compose.yml to the new version.

2. Pull the updated images and recreate the containers. Compose recreates only what changed:

   docker compose pull
   docker compose up -d

Your usage data and configuration persist in the MongoDB volume across updates.

Troubleshooting

Containers do not start

Check the status and logs, then restart the stack:

docker compose ps
docker compose logs <service>
docker compose down
docker compose up -d

Replace <service> with the container name, for example the API or UI service.

MongoDB does not start

If MongoDB exits on startup, its stored data is usually incompatible with the running MongoDB version. Recreate the data volume.

This deletes stored usage data

The following command removes the MongoDB volume and erases all usage records. Back up the volume first if you need the data.

docker compose down -v
docker compose up -d

The UI returns a 500 error

The UI returns a 500 error when the API container is not ready or is misconfigured. Wait a few seconds for the API to become healthy, then reload. If the error persists, check the UI container logs:

docker compose logs <ui-service>

You cannot reach the UI or API

Confirm the host firewall allows the ports you mapped in docker-compose.yml, and confirm you are using those ports in the URL. Verify the containers are running with docker compose ps.