Administration

For the team that runs an Oqim deployment: the platform admin console, the oqimctl command, configuration, backups, health checks, metrics and logs.

Admin console#

Platform administrators manage the whole deployment at /admin, or on an admin. host routed to it. Opening it takes a platform-admin user, a console session (API keys can't use it) and two-factor authentication: an administrator without it is asked to enroll on the first visit, and every new session asks for a code.

Section	What you can do
Overview	Platform numbers, 14 days of delivered and failed messages, worker health and open abuse reports.
Organizations	Search tenants, create one for a customer with its owner, change plans, suspend with a reason, reactivate.
Users	Find users across organizations; disable and re-enable them.
Telegram accounts	Every connected account with its organization, status and pace; disable one, and restore it later.
Proxies	Every organization's proxies with their health checks and the account each one serves; check one now, or disable one that's being misused.
Campaigns	Every campaign with its progress; pause or cancel one.
Workers	Each worker and scheduler process: heartbeat, version, concurrency, active jobs, throughput, uptime.
Queues	Depth, retries, latency and seven days of history per queue; pause and resume a queue, list the tasks that are stuck, and retry or drop them (dropping needs a written reason).
Support access	Open a read-only session on one organization to debug its problem. You give a reason and a length; the organization is notified, sees who looked and why, and can read every request the session made. It cannot change anything and cannot reach this console.
Abuse reports	Review automatic and manual reports; resolve, dismiss or suspend the organization.
Audit log	Every organization's audit trail, plus platform-admin actions, with their details.
Billing	Plans side by side and each organization's 30-day usage.
Limits, AI budgets and AI cost	Per organization: override the plan's limits for one customer without moving them to another plan, set their AI spending caps, and see what they spent by business, agent, provider and model. Every change needs a written reason and appears in the organization's own audit log.
Platform settings	Sign-ups, policy version, consent rules, pacing bounds, abuse thresholds and maintenance mode, which makes the platform read-only for customers while you work.

Every admin action is written to the audit log with the actor type PLATFORM_ADMIN, including in the affected organization's trail.
Suspending an organization pauses its running and scheduled campaigns, stops its messaging and makes it read-only; its members see your reason in their console. Reactivating doesn't resume those campaigns; the organization does.
A campaign you pause, or an account you disable, can't be resumed by its organization. The organization is notified and told to contact support. Restore the account from the console; resume the campaign with POST /api/v1/admin/campaigns/{id}/resume.
Disabling a proxy needs a reason, which the organization sees. Oqim stops connecting through the proxy at once: the account using it stops sending and its campaigns pause, and it never falls back to a direct connection. The organization can give the account another proxy or a direct connection; only you can enable the proxy again.
New organizations need an owner who already has an Oqim sign-in.
Settings changes reach every service within about five seconds.

Automatic brakes#

Every 30 seconds the scheduler looks for signs that messages are reaching people who didn't ask for them. The campaign rules pause the running campaign, and every rule opens an abuse report for you to review. The campaign thresholds are under Platform settings → Abuse protection.

Rule	Trips when (defaults)	What happens
High failure rate	Failed and unreachable sends are more than 35% of a campaign's attempts, once it has made 40.	The campaign pauses and a `HIGH_FAILURE_RATE` report opens.
Opt-out spike	At least 5 recipients opted out after the campaign's message reached them, and they make up 5% or more of its deliveries.	The campaign pauses and an `OPT_OUT_SPIKE` report opens with the counts, split by where the opt-outs were recorded (opt-out link, the organization, an import).
Restricted accounts	An organization has three or more accounts Telegram restricted.	An `ACCOUNT_RESTRICTIONS` report opens, at most one a day per organization.

The organization is notified of a pause and can resume the campaign. After an automatic pause, only what happens since then counts toward the same rule, so a resumed campaign is judged on its new sends: opt-outs from messages sent before the pause don't pause it again.

oqimctl#

oqimctl performs operator tasks and reads the same environment variables as the services. With Docker Compose, run it in the worker container, which has SESSION_KEY: docker compose exec worker oqimctl ….

Command	What it does
`oqimctl gen-keys`	Prints fresh `APP_KEY`, `SESSION_KEY` and `INTERNAL_TOKEN` values.
`oqimctl migrate`	Applies database migrations. The API also migrates when it starts; concurrent runs wait for each other.
`oqimctl create-admin -email E -password P -name N`	Creates a platform administrator, or promotes an existing user and resets their password. The password needs at least 10 characters.
`oqimctl seed-demo`	Creates the demo organization Silk Road Events with simulated accounts, recipients and campaigns (owner `demo@oqim.dev`). It needs `SESSION_KEY`. For local development only.

After create-admin, sign in to the console, open /admin and set up two-factor authentication.

Configuration#

Services read their configuration from environment variables and report every missing required one together at startup.

Variable	Used by	Default	Description
`APP_ENV`	all	development	`development` or `production`. Production turns on secure cookies and HSTS by default.
`HTTP_ADDR`	api	:8080	Listen address of the REST API.
`INTERNAL_ADDR`	worker	:8090	Listen address of the worker's internal Telegram gateway.
`GATEWAY_URL`	api	http://localhost:8090	Where the API reaches the worker gateway.
`INTERNAL_TOKEN`	api, worker	required	Shared secret that authenticates the API to the worker gateway.
`DATABASE_URL`	all	required	Postgres connection string.
`REDIS_URL`	all	redis://localhost:6379/0	Redis for queues, rate limits and live events.
`APP_KEY, APP_KEY_ID`	all	required, k1	32-byte key (base64) that seals app secrets and signs tokens and opt-out links. The ID labels it for rotation.
`SESSION_KEY, SESSION_KEY_ID`	worker only	required, s1	32-byte key (base64) that seals Telegram sessions. Never give it to the API.
`TELEGRAM_DRIVER`	worker	simulator	`simulator` or `mtproto` for real Telegram.
`TELEGRAM_APP_ID, TELEGRAM_APP_HASH`	worker	—	The operator's api_id and api_hash from my.telegram.org. Required with `mtproto`.
`PUBLIC_APP_URL`	api	http://localhost:3000	The console's public URL, used in opt-out and invitation links.
`ALLOWED_ORIGINS`	api	PUBLIC_APP_URL	Comma-separated origins allowed to make cookie-authenticated changes. Include the admin host.
`COOKIE_SECURE`	api	true in production	Mark the session cookie Secure. Leave it on behind HTTPS.
`COOKIE_DOMAIN`	api	host only	Set to `.example.com` to share the session between the console and admin hosts.
`STORAGE_DIR`	api, worker	./data/attachments	Where attachments are stored. The API and workers must share it.
`WORKER_CONCURRENCY`	worker	20	Tasks one worker process runs at once.
`METRICS_ADDR`	api, worker, scheduler	:9101, :9102, :9103	Listen address for Prometheus metrics; the defaults differ per service. Keep it private.

Also available: SESSION_TTL_HOURS (console session lifetime, default 336), LOG_LEVEL=debug, and WORKER_ID to name a worker process. The web app takes API_URL at build time: where it forwards /api/v1.

Hosts#

One web deployment serves three surfaces:

Host	Serves
`app.example.com`	The console, and the API at /api/v1
`admin.example.com`	The admin console (/admin)
`docs.example.com`	These docs (/docs)

Sign-in, invitation and opt-out pages work on every host. Locally, /admin and /docs work directly. To share one sign-in between the console and admin hosts, set COOKIE_DOMAIN to the parent domain and list both hosts in ALLOWED_ORIGINS.

infrastructure/nginx/oqim.conf is an example edge configuration: it terminates TLS for all three hosts and forwards everything to the web app, which proxies /api/v1 to the API. Whatever you put in front, allow request bodies of at least 25 MB for imports and attachments, and turn response buffering off for /api/v1/events/stream.

Backups and recovery#

On a server deployed with deploy.sh (see deploy/README.md), the database is dumped with pg_dump in its custom format into .deploy/backups/ next to the checkout:

./deploy.sh backup makes a dump now, named oqim-YYYYMMDD-HHMMSS-manual.dump.
Every deploy makes one before it applies migrations, named oqim-YYYYMMDD-HHMMSS-before-BUILD.dump. The last 10 dumps are kept (KEEP_BACKUPS changes that), and ./deploy.sh --no-backup skips the dump.
The dumps stay on the server, so copy them somewhere else on a schedule. They cover Postgres only, not attachment files: on the local volume, back up oqim_attachments yourself; on S3, turn on the bucket's versioning or your provider's backups.

Restoring a dump

Stop the services that write to the database, restore with pg_restore, then start them again:

Shell

./deploy.sh dc stop api worker scheduler
./deploy.sh dc exec -T postgres pg_restore -U oqim -d oqim --clean --if-exists < .deploy/backups/oqim-YYYYMMDD-HHMMSS-….dump
./deploy.sh dc start api worker scheduler

Rolling back

./deploy.sh rollback runs the previous build again (or ./deploy.sh rollback TAG a given kept one) without rebuilding. It doesn't undo migrations: the older build runs against the newer schema, which is why migrations only ever add. To get the data back as it was before a deploy, restore the dump that deploy made.

Health checks#

Endpoint	Returns
`GET /healthz`	Liveness. Always `200` with `{"status":"ok"}` while the process serves requests.
`GET /readyz`	Readiness. `200` when Postgres and Redis answer, otherwise `503` with `postgres_unavailable` or `redis_unavailable`.

Both are served by the API at its root, outside /api/v1. Workers report health through their heartbeat, visible under Workers in the admin console: online within 20 seconds of the last heartbeat, degraded within 90 seconds or while erroring, offline after that.

Metrics#

The API, workers and scheduler expose Prometheus metrics on METRICS_ADDR: :9101, :9102 and :9103 by default. docker compose --profile monitoring up starts a Prometheus and a Grafana that scrape them.

Metric	Type	Labels	Meaning
`jobs_created_total`	counter	–	Send jobs created by the scheduler.
`jobs_completed_total`	counter	result	Send jobs finished, by result.
`jobs_failed_total`	counter	result	Send jobs that ended without delivery, by result.
`campaigns_running`	gauge	–	Campaigns in RUNNING.
`queue_depth`	gauge	queue	Tasks waiting per queue (pending, scheduled and retry).
`worker_active`	gauge	–	Tasks running in this worker.
`worker_errors_total`	counter	–	Unexpected worker errors.
`telegram_errors_total`	counter	code	Telegram errors by type, e.g. FLOOD_WAIT or PEER_FLOOD.
`account_restrictions_total`	counter	–	Accounts moved to RESTRICTED.
`proxy_failures_total`	counter	–	Failed proxy health checks.
`api_requests_total`	counter	method, route, status	HTTP requests.
`api_latency_seconds`	histogram	method, route	HTTP request latency.

Alerts worth having

PromQL

# Any account restricted in the last hour
increase(account_restrictions_total[1h]) > 0

# The messages queue keeps growing
deriv(queue_depth{queue="messages"}[15m]) > 0

# More than 2% of API requests failing
sum(rate(api_requests_total{status=~"5.."}[5m])) / sum(rate(api_requests_total[5m])) > 0.02

# Workers hitting unexpected errors
rate(worker_errors_total[10m]) > 0

Logs#

Every service writes structured JSON to standard output, one event per line, tagged with service and env. The API logs each request with its method, route pattern, status, duration, request ID and client IP, plus the actor and organization when authenticated. Set LOG_LEVEL=debug for more detail.

API request log line

{"time":"2026-09-26T09:30:00.412Z","level":"INFO","msg":"http","service":"api","env":"production","method":"POST","route":"/api/v1/campaigns/{id}/pause","status":200,"ms":18,"request_id":"api-01/Kx9e2QmPzR-000042","ip":"203.0.113.24","actor":"usr_01j9r2n0d5c7b9a1e3f5g7h9jk","org":"org_01j9r2m6k4n8q3v5w7x9y1z2a3"}

Errors the API hides from clients (internal_error) are logged in full with the request path, so search the logs by path and time when a user reports one. See Security for what Oqim stores.