> ## Documentation Index
> Fetch the complete documentation index at: https://motiadev-docs-deployment-guide.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment

> Running iii in development, staging, and production.

This page covers running the iii engine and its workers in a production environment. For local
development, see the [Quickstart](../quickstart) and the [Engine](./engine) page.

## iii Cloud deployments

The `iii cloud` subcommand group will manage hosted iii deployments.

<Note>iii's cloud will be available soon.</Note>

## Platform guides

The below guide covers general deployment using docker. We also have step-by-step deploy guides for
common hosts. These guides build on the Docker patterns below.

* [Deploy to Railway](./deploy-railway)
* [Deploy to Fly.io](./deploy-fly)

## Pre-built images on Docker Hub

iii publishes the engine image to [`iiidev/iii`](https://hub.docker.com/r/iiidev/iii). You do not
need to compile anything to run the engine.

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
docker pull iiidev/iii:latest
```

The image is multi-arch (`linux/amd64` and `linux/arm64`), built on
`gcr.io/distroless/cc-debian12:nonroot`, and runs as UID `65532`. Its entrypoint is `/app/iii` and
its default command is `--config /app/config.yaml`, so you only supply a `config.yaml`. There is no
shell, package manager, or other userland in the image.

<Note>
  Pin an immutable tag or digest for production (for example `iiidev/iii@sha256:...`) so a deploy is
  reproducible. `latest` changes when a new version ships.
</Note>

## Deploy with Docker

### What gets deployed

The generated image packages the **engine and your `config.yaml`**, not your application code. Your
application is made of workers, and for deployment what matters is where each worker runs:

* Workers configured in `config.yaml` run in-process in the engine (for example `iii-http`,
  `iii-state`, `iii-queue`, `iii-cron`, `iii-stream`, `iii-observability`). Enable them by listing
  them in `config.yaml`.
* Workers you run yourself (Node, Python, Rust) connect to the engine over WebSocket as separate
  processes. The engine image is distroless and carries no language runtime, so it does not host
  them. Deploy each as its own container and point `III_URL` at the engine (`ws://iii:49134` on the
  compose network). See
  [Creating Workers / Connecting to the engine](../creating-workers/workers#connecting-to-the-engine).

`iii project generate-docker` does not scaffold a sample app. A bare `iii project init` produces a
minimal `config.yaml` with no functions or endpoints. To start from working code, scaffold a
template first (`iii project init my-app --template quickstart`), then generate the Docker assets.
See the [Quickstart](../quickstart) for the sample workers.

### Generate the assets

Generate the Docker assets with the CLI and bring them up with Compose. Use
`iii project init --docker` for a fresh project, or `iii project generate-docker` to add Docker
assets to an existing one:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
iii project generate-docker
```

Both forms emit three files at the project root: `Dockerfile`, `docker-compose.yml`, and `.env`.
Re-running the generator does not overwrite existing files, so edits you make to the templates
stick.

The generated `Dockerfile` builds against `iiidev/iii:latest` and copies your `config.yaml` into the
image:

```dockerfile theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
FROM iiidev/iii:latest

ENV III_EXECUTION_CONTEXT=docker

WORKDIR /app
COPY config.yaml /app/config.yaml

EXPOSE 49134 3111 3112 9464
ENTRYPOINT ["/app/iii"]
CMD ["--config", "/app/config.yaml"]
```

Start the stack:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
docker compose up -d
```

The transport ports are:

| Port  | Surface                                              |
| ----- | ---------------------------------------------------- |
| 49134 | SDK WebSocket (worker connections)                   |
| 3111  | REST API                                             |
| 3112  | Stream API                                           |
| 49135 | Browser SDK / RBAC listener (when enabled in config) |

The compose file ships commented-out Redis and RabbitMQ services that you uncomment when your
workers need external adapters. See
[Scale out with Redis and RabbitMQ](#scale-out-with-redis-and-rabbitmq).

<Note>
  Port `9464` was the Prometheus metrics endpoint in older releases and still appears in the image's
  `EXPOSE` list and the generated assets. The current engine has no Prometheus exporter, so nothing
  listens on it. Leave it unmapped and export metrics over OTLP. See
  [Observability](#observability).
</Note>

### Mount the config and data

Mount `config.yaml` read-only, and put every file-based `file_path` on a named volume so data
survives container replacement:

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
services:
  iii:
    build:
      context: .
    ports:
      - "49134:49134"
      - "3111:3111"
      - "3112:3112"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - iii_data:/data
    environment:
      - RUST_LOG=info
      - III_EXECUTION_CONTEXT=docker
    restart: unless-stopped

volumes:
  iii_data:
```

Point adapters at the mounted path, for example `file_path: /data/state_store.db`. Point the
`configuration` worker at the volume too. It defaults to `./data/configuration`, which resolves to
`/app/data/configuration` (relative to the engine's `WORKDIR /app`), **not** the `/data` volume, so
on the non-root image it is the first thing to crash-loop with
`failed to create configuration directory './data/configuration': Permission denied`:

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: configuration
    config:
      adapter:
        name: fs
        config:
          directory: /data/configuration
```

The image runs as UID `65532`, so `/data` must be writable by that user. The stock `iiidev/iii`
image does **not** pre-create `/data`, so Docker mounts a fresh **named volume as `root:root`** and
the engine cannot write to it. Use one of:

* a **bind mount** to a host directory you `chown 65532:65532`, or
* an engine image that creates `/data` (and `/app/data`) owned by `65532`, after which a named
  volume inherits that ownership.

## Build from source

Use this only when you are building a custom engine or running on a platform without the pre-built
image. Build the binary from the repository root:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
cargo build --release
```

This produces a single binary at `target/release/iii`. Run it against a config:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
./target/release/iii --config ./config.yaml
```

To ship your own binary on the official base image, copy it over the entrypoint at `/app/iii`:

```dockerfile theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
FROM iiidev/iii:latest
COPY target/release/iii /app/iii
COPY config.yaml /app/config.yaml
ENTRYPOINT ["/app/iii"]
CMD ["--config", "/app/config.yaml"]
```

<Note>
  Build for the target platform. The base image is Debian-based (glibc), so a musl binary will not
  run on it, and an `arm64` binary will not run on an `amd64` host. Cross-compile with the matching
  target (for example `--target x86_64-unknown-linux-gnu`) or build on the target architecture.
</Note>

## Configure workers and adapters

The engine is stateless. State is held in workers, and each stateful worker selects a storage
**adapter** in `config.yaml`. The defaults are in-memory, tuned for local development, so choosing
durable adapters is the main production step.

| Worker              | Stores                     | Adapters                                                                   | Default             |
| ------------------- | -------------------------- | -------------------------------------------------------------------------- | ------------------- |
| `iii-state`         | Key/value state            | `kv` (`in_memory` / `file_based`), `redis`, `bridge`                       | `kv` in-memory      |
| `iii-stream`        | Realtime streams           | `kv` (`in_memory` / `file_based`), `redis`, `bridge`                       | `kv` in-memory      |
| `iii-queue`         | Background jobs / topics   | `builtin` (`in_memory` / `file_based`), `rabbitmq`, `redis` (publish-only) | `builtin` in-memory |
| `iii-cron`          | Scheduled-task bookkeeping | `kv` (`in_memory` / `file_based`)                                          | `kv` in-memory      |
| `configuration`     | Runtime config entries     | `fs` (file-based, defaults to `./data/configuration`)                      | file-based          |
| `iii-pubsub`        | Pub/sub fan-out            | `local`, `redis`                                                           | `local`             |
| `iii-http`          | Nothing (request routing)  | n/a                                                                        | n/a                 |
| `iii-observability` | Traces / metrics / logs    | `memory` or `otlp` exporters                                               | `memory`            |

When a `kv`-backed adapter runs in-memory, the engine logs
`DO NOT USE IN_MEMORY STORE_METHOD IN PRODUCTION - DATA WILL BE LOST ON SHUTDOWN` on boot. Treat
that as a release blocker. The `iii-queue` `builtin` adapter is in-memory by default, so jobs are
lost on restart until you make it file-based or move it to RabbitMQ.

For each worker's full config fields, see the
[Worker manifest reference](../creating-workers/worker-manifest) and the
[Engine configuration](./engine#configuration-file-structure) page.

### Make storage durable with file\_based

File-based storage is the simplest durable option: no extra services, only a writable path on a
mounted volume. Set `store_method: file_based` and a `file_path` for each `kv`-backed worker.

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-state
    config:
      adapter:
        name: kv
        config:
          store_method: file_based
          file_path: /data/state_store.db

  - name: iii-stream
    config:
      adapter:
        name: kv
        config:
          store_method: file_based
          file_path: /data/stream_store

  - name: iii-queue
    config:
      adapter:
        name: builtin
        config:
          store_method: file_based
          file_path: /data/queue_store

  - name: iii-cron
    config:
      adapter:
        name: kv
        config:
          store_method: file_based
          file_path: /data/cron_store
```

A file-based `kv` adapter flushes on a `save_interval_ms` cadence (default 5000 ms). A hard kill
between flushes can drop the most recent writes; use Redis when you need stronger durability.

<Note>
  The storage adapter is restart-tier and persisted. After first boot, `iii-state` writes its live
  settings to `./data/configuration/iii-state.yaml` (the default `fs` configuration adapter), and
  the `config.yaml` adapter block becomes seed-only. Editing the adapter in `config.yaml` after that
  has no effect. The persisted entry wins on the next boot. To change an adapter later, edit the
  persisted file or call `configuration::set`. On a fresh deploy with an empty `./data`, the
  `config.yaml` block is the seed.
</Note>

### Scale out with Redis and RabbitMQ

File-based storage is single-instance. To run more than one engine replica, or for real-time fan-out
across processes, move stateful workers to an external backend. This step is optional. A single
replica on file-based storage is a valid production setup.

Match the backend to the workload:

* `iii-state`, `iii-stream`, `iii-pubsub` → Redis, for shared key/value, stream fan-out, and
  pub/sub.
* `iii-queue` → RabbitMQ, for durable cross-instance delivery with retries and DLQs. The queue
  `redis` adapter is publish-only for named queues (no named-queue consumption, retries, or DLQs),
  so it does not replace `rabbitmq` or the single-instance `builtin` adapter.

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-state
    config:
      adapter:
        name: redis
        config:
          redis_url: redis://redis:6379

  - name: iii-stream
    config:
      adapter:
        name: redis
        config:
          redis_url: redis://redis:6379

  - name: iii-pubsub
    config:
      adapter:
        name: redis
        config:
          redis_url: redis://redis:6379

  - name: iii-queue
    config:
      adapter:
        name: rabbitmq
        config:
          amqp_url: amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@rabbitmq:5672
```

Uncomment the `redis` and `rabbitmq` services in the generated `docker-compose.yml` and add a
`depends_on` so the engine waits for the backends to be healthy:

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
services:
  iii:
    # ...
    env_file: .env # forwards RABBITMQ_USER / RABBITMQ_PASS into the engine
    depends_on:
      redis:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_PASS}
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "check_running"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  redis_data:
  rabbitmq_data:
```

<Note>
  Reference each backend by its compose service name (`redis://redis:6379`,
  `amqp://...@rabbitmq:5672`), not `localhost`. Inside the engine container, `localhost` is the
  engine. Keep credentials in environment variables; `${VAR:default}` in `config.yaml` is expanded by
  the **engine process**, so the variables must be in the engine container's environment. The
  generated `.env` carries `RABBITMQ_USER` / `RABBITMQ_PASS`, but Compose only uses `.env` to
  substitute the compose file itself; it does **not** forward those into the container. Add
  `env_file: .env` (or an explicit `environment:` list) to the `iii` service, or the engine boots
  with `Environment variable 'RABBITMQ_USER' not set`. Switching a stateful worker to a remote
  adapter after first boot also requires updating the persisted configuration entry, not only the
  `config.yaml` block (see the note above).
</Note>

## Adjust the worker manifest for production

A worker's `iii.worker.yaml` describes how the engine provisions and starts it. The development
defaults favor fast iteration; production wants the opposite. Adjust the manifest before you ship:

* `scripts.start`: drop watch mode so a code change does not restart the process and a crash is a
  real signal: `node dist/index.js`, not `npx tsx watch src/index.ts`.
* `scripts.install`: install from a lockfile without dev dependencies: `npm ci --omit=dev`.
* `env`: inject production configuration. Keep secrets out of the file and supply them from the
  environment. For **engine-managed** workers, the manifest keys `III_URL` and `III_ENGINE_URL` are
  ignored; the engine sets the connection URL. For workers you run yourself (Compose services, Fly
  apps, Railway services), set `III_URL` in the container environment and pass it to
  `registerWorker(process.env.III_URL)`.
* `runtime.base_image`: pin a specific base image so builds are reproducible.
* `resources`: set explicit `cpus` and `memory` for the worker's sandbox (defaults are 2 vCPU / 2048
  MiB, capped at 4 / 4096).

See the [Worker manifest reference](../creating-workers/worker-manifest) for every field.

## Deploy a multi-worker product with Compose

A real product composes several workers: the engine, its adapters (Redis, RabbitMQ), your own
workers, and registry workers. The iii registry at [workers.iii.dev](https://workers.iii.dev) hosts
the installable workers: `database` (PostgreSQL/MySQL/SQLite), `storage` (S3/GCS/R2/local),
`harness`, `iii-sandbox`, and the rest, each added with `iii worker add <name>`.

Workers reach the engine in one of two ways, and which one you use decides how it deploys:

* **Connect over WebSocket**: your own workers (and any worker you choose to run yourself) run as
  their own container and point `III_URL` at the engine service. This is the portable,
  Compose-native pattern.
* **Engine-managed** (added with `iii worker add`): the engine starts them from `config.yaml`.
  Workers with `deploy: binary` run as plain child processes; workers with `deploy: bundle` or
  `deploy: image` boot libkrun guests and need `/dev/kvm`.

### Engine, adapters, and your own workers

The engine runs the in-process workers from its `config.yaml` (`iii-http`, `iii-state`, `iii-queue`,
...); each of your own workers runs as its own service and connects with `III_URL=ws://iii:49134`
(the engine's service name on the Compose network).

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
services:
  iii:
    image: iiidev/iii:latest
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - iii_data:/data
    ports:
      - "3111:3111" # REST (front this with a reverse proxy in production)
    env_file: .env # forwards RABBITMQ_USER / RABBITMQ_PASS into the engine
    environment:
      - III_EXECUTION_CONTEXT=docker
    restart: unless-stopped

  # Your worker (your image, your code) connects to the engine over the network.
  app-worker:
    build: ./worker # a Node/Python/Rust image that runs your worker
    environment:
      - III_URL=ws://iii:49134
    depends_on:
      - iii
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes: [redis_data:/data]
    restart: unless-stopped

  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_PASS}
    volumes: [rabbitmq_data:/var/lib/rabbitmq]
    restart: unless-stopped

volumes:
  iii_data:
  redis_data:
  rabbitmq_data:
```

The worker's image is an ordinary language image that installs the SDK and runs your code. A minimal
Node example:

```dockerfile theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
FROM node:22-alpine
WORKDIR /app
COPY package.json ./
RUN npm install          # depends on iii-sdk
COPY . ./
CMD ["node", "index.mjs"]
```

```javascript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
import { registerWorker } from "iii-sdk";
const worker = registerWorker(process.env.III_URL); // ws://iii:49134
// worker.registerFunction(...) / worker.registerTrigger(...)
```

Reference adapters by service name in `config.yaml` (`redis://redis:6379`,
`amqp://...@rabbitmq:5672`); see
[Scale out with Redis and RabbitMQ](#scale-out-with-redis-and-rabbitmq). Add as many of your own
workers as the product needs. Each is another service that connects over `III_URL`.

### Registry workers (engine-managed)

Workers added with `iii worker add` are started by the engine from `config.yaml`. How they run
depends on the worker's top-level `deploy:` field in its `iii.worker.yaml` on
[workers.iii.dev](https://workers.iii.dev):

| `deploy:`          | Runtime                              | Needs `/dev/kvm`? | Typical workers                                   |
| ------------------ | ------------------------------------ | ----------------- | ------------------------------------------------- |
| `binary`           | plain child process via `iii-worker` | no                | `database`, `shell`, `storage` (binary build)     |
| `bundle` / `image` | libkrun micro-VM guest               | yes               | `iii-sandbox`, OCI workers, older `bundle` builds |

Declare them in `config.yaml` alongside the in-process workers:

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: database
    config:
      databases:
        primary:
          url: ${DATABASE_URL}
  - name: storage
    config:
      buckets:
        uploads:
          provider: s3
          bucket: my-bucket
          region: us-east-1
```

The `iii-worker` daemon (`iii worker ...`) ships with the install and is what the engine calls to
start registry workers. Running **binary** registry workers in a container takes more than the stock
distroless image:

* **The stock `iiidev/iii` image cannot run them.** It is distroless: engine-only, no shell, and no
  `libcap-ng.so.0` for `iii-worker`. Build on a glibc base (for example `debian-slim`) with both
  `iii` and `iii-worker`.
* **Bake workers at build time** with `RUN iii worker add <name>` in a shell-capable build stage.
  Binary workers land in `~/.iii/workers/`; bundle workers also need `~/.iii/rootfs/<lang>` (large).
* **`deploy: bundle` / `deploy: image` workers need `/dev/kvm`.** On hosts without it (Fly Machines,
  Railway containers, Docker Desktop on macOS), use `deploy: binary` workers or run libkrun
  workloads on a KVM-capable Linux host. On those hosts, `iii-sandbox` may register but guest
  execution (`sandbox::run`, OCI invocations) does not work.

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
services:
  iii:
    image: your-engine-image # glibc base with iii + iii-worker + baked binary workers
    devices:
      - /dev/kvm # required only for deploy: bundle / deploy: image workers
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - iii_data:/data
    restart: unless-stopped
```

<Note>
  Docker Desktop on macOS does not expose `/dev/kvm` to its Linux VM, so managed-worker micro-VMs do
  not boot inside Compose on a Mac. Run them on a Linux host with KVM, or run the engine directly on
  the macOS host (Apple Silicon), where the virtualization is available.
</Note>

<Note>
  Each registry worker has its own config schema (database connection, storage backend,
  credentials). Look it up on the worker's page at [workers.iii.dev](https://workers.iii.dev), and
  supply secrets from the environment.
</Note>

## Hardening

### Bind interfaces deliberately

Inside a container, bind `0.0.0.0` so published ports are reachable, and control exposure at the
Docker, orchestrator, and reverse-proxy layers. On a bare host, bind `127.0.0.1` and let a local
reverse proxy reach the engine. Do not expose `49134`, `3111`, or `3112` directly to the internet.

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-http
    config:
      host: 0.0.0.0 # container; use 127.0.0.1 on a bare host
      port: 3111
      default_timeout: 30000
      concurrency_request_limit: 1024
```

### Restrict CORS

The development default allows all origins. Restrict it to the origins that call your API:

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-http
    config:
      cors:
        allowed_origins:
          - https://app.example.com
        allowed_methods:
          - GET
          - POST
          - PUT
          - DELETE
          - OPTIONS
```

### Keep secrets in the environment

Use `${VAR}` / `${VAR:default}` placeholders in `config.yaml` and supply the values through the
container environment or your orchestrator's secret store. The generated `.env` (which holds the
RabbitMQ password) is git-ignored; do not bake secrets into the image.

## RBAC

Workers connect over WebSocket. The main engine port (`49134`) is trusted, internal traffic with no
access control. To accept worker connections from outside your trust boundary, run a second
`iii-worker-manager` listener with role-based access control and expose only that port.

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-worker-manager
    config:
      host: 0.0.0.0
      port: 49134 # main, trusted listener

  - name: iii-worker-manager
    config:
      host: 0.0.0.0
      port: 49135 # public RBAC listener
      rbac:
        auth_function_id: my-project::auth-function
        expose_functions:
          - match("api::*")
          - metadata:
              public: true
```

How it gates access:

* `expose_functions` is the allowlist of function IDs reachable through this listener, by wildcard
  `match("...")` pattern or by `metadata` filter. A function that matches no filter is denied; a
  caller invoking it gets `function '<id>' not allowed (add to rbac.expose_functions)`.
* `auth_function_id` runs once per WebSocket upgrade. It receives the request headers, query params,
  and client IP, and returns per-session allow/forbid lists plus a context object. Throwing rejects
  the connection.
* `middleware_function_id` (a sibling of `rbac`, at the `config` level) runs before every invocation
  for audit logging, rate limiting, or payload enrichment.

<Info>
  Expose only the RBAC listener externally. Keep the main engine port (`49134`) and stream port
  (`3112`) on a private network, enforced with firewall rules or network policies. Channels work
  through the RBAC listener at `/ws/channels/{channel_id}` on the same port.
</Info>

## Reverse proxies

The engine does not terminate TLS. Put a reverse proxy in front of it to handle TLS and route the
transport surfaces. Each surface is a separate port, and each WebSocket surface is served at the
root path of its port:

| Surface                 | Internal port | Path on that port                     |
| ----------------------- | ------------- | ------------------------------------- |
| REST API                | 3111          | your registered routes, at root       |
| Stream API (WS)         | 3112          | `/`                                   |
| Worker connections (WS) | 49134         | `/` (channels at `/ws/channels/{id}`) |
| RBAC listener (WS)      | 49135         | `/` (channels at `/ws/channels/{id}`) |

### Subdomain per surface

Giving each surface its own hostname needs no path rewriting, and the WebSocket roots line up.

```nginx theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# REST API
server {
    listen 443 ssl;
    server_name api.example.com;
    location / {
        proxy_pass http://127.0.0.1:3111;
    }
}

# Worker / RBAC WebSocket
server {
    listen 443 ssl;
    server_name ws.example.com;
    location / {
        proxy_pass http://127.0.0.1:49135;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;
    }
}
```

### Single host with path prefixes

To serve everything from one hostname, route by path and strip the prefix so each WebSocket surface
is reached at its root. With nginx, a trailing slash on `proxy_pass` strips the matched prefix:

```nginx theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
server {
    listen 443 ssl;
    server_name example.com;

    location /ws/ {
        proxy_pass http://127.0.0.1:49134/;   # trailing slash strips "/ws/"
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;
    }

    location /stream/ {
        proxy_pass http://127.0.0.1:3112/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;
    }

    location / {
        proxy_pass http://127.0.0.1:3111;
    }
}
```

The equivalent in Caddy, which handles WebSocket upgrades automatically and strips the matched
prefix with `handle_path`:

```
example.com {
    handle_path /ws/* {
        reverse_proxy 127.0.0.1:49134
    }
    handle_path /stream/* {
        reverse_proxy 127.0.0.1:3112
    }
    handle {
        reverse_proxy 127.0.0.1:3111
    }
}
```

<Note>
  These are example configurations. See the [Caddy](https://caddyserver.com/docs/) and
  [Nginx](https://nginx.org/en/docs/) documentation for full TLS, header, and timeout options.
</Note>

<Note>
  Long-lived WebSocket connections need a generous proxy read timeout. The nginx default (60 s)
  drops idle worker and stream sockets; set a high `proxy_read_timeout`. Caddy keeps streaming
  connections open by default.
</Note>

## Health checks

The engine has no HTTP health endpoint, and the image has no shell or networking utility, so
`HEALTHCHECK` directives that shell out (for example `CMD-SHELL` with `nc` or `curl`) do not work.
Probe from outside the container:

* A TCP check against the REST port (`3111`) is a reliable liveness signal. In Kubernetes, use a
  `tcpSocket` probe.
* A reverse proxy can run upstream health checks against the REST port.
* For a readiness signal that reflects worker availability, register an HTTP-triggered function that
  returns `200` and probe that route.

## Observability

The `iii-observability` worker collects traces, metrics, and logs through one of two exporters:

* `memory` (default): keeps recent observability data in the engine, queryable through its
  observability functions. Bounded by `memory_max_spans` and retention settings.
* `otlp`: forwards traces, metrics, and logs to an OTLP collector. Use this in production.

Pick the port to match the protocol. This is the common mistake. Traces and metrics export over
OTLP/**gRPC** by default (collector port `4317`); logs always export over OTLP/**HTTP** to
`<endpoint>/v1/logs` (collector port `4318`). So a single default-protocol `endpoint` can't satisfy
all three: pointed at `4317` it delivers traces and metrics but drops logs.

To send all three signals to one endpoint, switch the engine to HTTP and use the collector's HTTP
port (`4318`):

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
  - name: iii-observability
    config:
      enabled: true
      service_name: iii
      exporter: otlp
      endpoint: http://otel-collector:4318 # collector HTTP port
      metrics_enabled: true
      metrics_exporter: otlp
      logs_enabled: true
      logs_exporter: otlp
```

```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# in the engine service's environment (Compose): forces all signals onto HTTP/protobuf
environment:
  - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Without that env var, use `endpoint: http://otel-collector:4317` (gRPC) and accept that logs need
the HTTP path. Metrics export on a 60-second interval. TLS is enabled automatically for `https://`
endpoints.

<Note>
  `metrics_exporter` accepts only `memory` or `otlp`. Setting `prometheus` (removed in a prior
  release) stops the engine from starting: `unknown variant 'prometheus', expected 'memory' or
      'otlp'`. There is no Prometheus scrape endpoint on port `9464`.
</Note>

## Deployment checklist

**Image and build**

* [ ] Pinned an immutable image tag or digest, not `latest`.
* [ ] If building from source, compiled for the target arch and libc and placed the binary at
  `/app/iii`.

**Configuration and persistence**

* [ ] No `DO NOT USE IN_MEMORY` warnings in the boot logs.
* [ ] `iii-state`, `iii-stream`, and `iii-cron` use `file_based` or `redis`; `iii-queue` uses
  `file_based` `builtin` or `rabbitmq`.
* [ ] Every `file_path` resolves to a mounted volume writable by UID `65532`.
* [ ] Adapter changes account for the persisted configuration entry (a fresh deploy seeds from
  `config.yaml`; an existing one reads `./data/configuration`).
* [ ] For multiple replicas, all stateful workers use an external backend referenced by service
  name, with credentials from the environment.

**Workers**

* [ ] Worker manifests use production `start` / `install` scripts (no watch mode, lockfile
  installs).
* [ ] Worker `env` and `resources` set; secrets injected from the environment.
* [ ] Your own workers run as their own services pointing `III_URL` at the engine.
* [ ] Engine-managed registry workers use a glibc-based engine image with `iii-worker` and
  `libcap-ng0` (not the distroless stock image). `/dev/kvm` only when using `deploy: bundle` or
  `deploy: image` workers.

**Networking and security**

* [ ] Engine binds `0.0.0.0` in containers, `127.0.0.1` on bare hosts.
* [ ] Only required ports are published; `49134` and `3112` stay private.
* [ ] CORS restricted to real origins.
* [ ] External worker connections go through an RBAC listener with an auth function.
* [ ] TLS terminated at a reverse proxy; WebSocket upgrade headers and long read timeouts set.

**Operations**

* [ ] External health checks configured (TCP `3111` or `tcpSocket`), not in-container shell probes.
* [ ] Observability exporter set to `otlp` with a collector endpoint, or `memory` accepted for
  scope.
* [ ] Restart policy set (`restart: unless-stopped` or the orchestrator equivalent).