Skip to content
HeliosDB HeliosProxy

HeliosProxy Admin API Reference

HeliosProxy Admin API Reference

The Admin API provides REST endpoints for monitoring, management, HA/migration control, and SQL routing. It runs on a dedicated TCP listener, separate from the PostgreSQL client port.

Default address: 127.0.0.1:9090 (loopback). Configurable via admin_address in proxy.toml or --admin on the command line.

All responses use Content-Type: application/json unless otherwise noted. The two exceptions are the embedded web UI (GET / and /ui, served as text/html) and the edge SSE stream (GET /api/edge/subscribe, text/event-stream).

Source of truth for everything below is the dispatch match in src/admin.rs.

Security Model

Bind default

The admin API is privileged (it can execute SQL, disable nodes, force chaos faults, cut over migrations). It therefore defaults to loopback (127.0.0.1:9090). If you set a non-loopback admin_address (e.g. 0.0.0.0:9090), the proxy refuses to start unless one of the following is also true:

  • admin_token is set (bearer-token auth is enabled), or
  • admin_allow_insecure = true is set (explicit opt-in to an unauthenticated non-loopback bind).

This guard lives in ProxyConfig::validate (src/config.rs) and only fires for non-loopback binds; a loopback bind with no token is allowed.

Bearer-token authentication

When admin_token is set, every route requires Authorization: Bearer <token>, verified with a constant-time compare, except the token-exempt liveness paths (below) and the static web-UI shell (GET /, /ui) — the shell holds no privileged data and injects the token into its own API calls client-side (see Web UI). Requests without a valid token get 401 Unauthorized with body {"error":"missing or invalid admin bearer token"}.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/nodes

When admin_token is unset, no route requires a token (rely on the loopback bind for protection).

Token-exempt health paths

The auth gate exempts a fixed set of GET liveness paths so orchestrators can probe without the token:

/health   /healthz   /livez   /readyz

The static web-UI shell (GET /, /ui) is also served without a token (it holds no privileged data); see Web UI.

All four are routed and token-exempt. The z-suffixed paths are Kubernetes-style aliases of the slash-form health routes and return byte-for-byte the same responses:

PathAlias ofResponse
/health—200 {"status":"ok"}
/healthz/health200 {"status":"ok"}
/livez/health/live200 {"alive":true}
/readyz/health/ready200 if ≥1 healthy backend, else 503

The slash-form /health/live and /health/ready routes are not token-exempt, so when admin_token is set they require the bearer token like any other route — use the z-suffixed aliases (/livez, /readyz) for unauthenticated probes.

Recommendation for Kubernetes probes: point the liveness probe at /healthz (or /livez) and the readiness probe at /readyz — all three are open (no token) and always routed.

Connection & request caps

The admin listener is hardened against slow-loris and oversized-request abuse with hard caps (constants in src/admin.rs):

CapValueMeaning
MAX_ADMIN_CONNS256Max concurrent admin connections (semaphore-bounded).
ADMIN_READ_TIMEOUT15 sBounds the request-line/header/body read phase; a stalled reader is dropped.
MAX_ADMIN_HEADERS100Max header lines per request.
MAX_ADMIN_HEADER_BYTES64 KiBMax total header bytes.
MAX_ADMIN_BODY_BYTES8 MiBMax request body; larger Content-Length is rejected.
ADMIN_SSE_WRITE_TIMEOUT30 sPer-write timeout on the long-lived edge SSE stream (paired with a 15 s heartbeat) so a wedged subscriber can’t hold a connection permit forever.

Feature gating

The default build (default = ["pool-modes"]) compiles in only a subset of routes. Feature-gated routes are always present in the dispatch table, but return 503 Service Unavailable with an explanatory error message when their cargo feature is not compiled in (a full build uses --features all-features). A handful of routes are always routed but return 503 when the corresponding subsystem is not configured at runtime (mirror/migration, branch databases, plugin manager).

Endpoint Summary

Auth column: token = requires bearer token when admin_token is set; open = token-exempt.

MethodPathPurposeFeature gateAuth
GET/healthLiveness ({"status":"ok"})—open
GET/healthzLiveness — alias of /health—open
GET/health/liveLiveness ({"alive":true})—token
GET/livezLiveness — alias of /health/live—open
GET/health/readyReadiness — 200 if ≥1 healthy backend, else 503—token
GET/readyzReadiness — alias of /health/ready—open
GET/metricsServer metrics (JSON)—token
GET/metrics/prometheusServer metrics (Prometheus text, wrapped in JSON text)—token
GET/versionProxy version—token
GET/capabilitiesPer-subsystem compiled/enabled/wired manifest + strict_config flag (since 1.9.0); also at /api/capabilities—token
GET/configCurrent configuration snapshot—token
GET/topologyPrimary + healthy/unhealthy node sets in one call—token
GET/nodesAll backend nodes with health—token
GET/nodes/{addr}Single node health (404 if unknown)—token
POST/nodes/{addr}/enableRe-enable a node into routing—token
POST/nodes/{addr}/disableRemove a node from routing—token
GET/sessionsActive client session count—token
GET/poolsPer-node connection pool stats—token
POST/api/sqlExecute SQL with transparent write routing—token
GET/pluginsLoaded WASM plugins (503 if manager not attached)wasm-pluginstoken
GET/admin/kv/{plugin}/{key}Read a plugin KV value ({"plugin","key","value"}; 404 if absent)wasm-pluginstoken
GET/admin/kv/{plugin}/List a plugin’s KV keys (trailing slash)wasm-pluginstoken
PUT/admin/kv/{plugin}/{key}Set a plugin KV value (UTF-8 body; 413 on cap breach)wasm-pluginstoken
DELETE/admin/kv/{plugin}/{key}Delete a plugin KV value (idempotent 200)wasm-pluginstoken
GET/anomaliesAnomaly-detector recent events (?limit=N)anomaly-detectiontoken
GET/analytics, /api/analyticsTop queries + slow-query log (?limit=N)query-analyticstoken
GET/api/chaosRead current chaos overrides—token
POST/api/chaosInject/clear a fault (force_unhealthy/restore/reset)—token
POST/api/replayReplay a journal window against a target backend (503 when tr_enabled = false)(default build)token
POST/api/shadowDual-execute a query and diff the results(default build)token
GET/api/circuitPer-node circuit-breaker statecircuit-breakertoken
GET/api/edgeEdge/geo cache + registered-edge statsedge-proxytoken
POST/api/edge/registerRegister an edge with the home proxyedge-proxytoken
POST/api/edge/invalidateBroadcast a table-level invalidationedge-proxytoken
GET/api/edge/subscribeLong-lived SSE invalidation stream (?edge_id=…)edge-proxytoken
GET/api/migration/statusTraffic-mirror / migration status (503 if mirror off)—token
POST/api/migration/snapshotSnapshot-bootstrap named tables into the mirror—token
POST/api/migration/cutoverPromote the mirror target to primary—token
POST/api/migration/cutover/rollbackRevert a cutover to the original primary—token
GET/api/branch, /branchList branch databases (503 if branching off)—token
POST/api/branch, /branchCreate a branch database—token
DELETE/api/branch, /branchDrop a branch database (?name=…)—token
GET/, /uiEmbedded admin web UI (HTML) — static shell only; its API calls are gated—open

The /api/migration/* and /api/branch routes also accept the same paths without the /api prefix (e.g. /migration/cutover, /branch) — both spellings are wired.

Any path/method not in this table returns 404 {"error":"Not found"}.

Health Endpoints

GET /health, GET /healthz

Basic liveness. Always 200 while the process is running. Token-exempt — both spellings are routed and open, so either is a correct target for an unauthenticated liveness probe (/healthz is the Kubernetes-conventional alias).

Terminal window
curl http://localhost:9090/health
curl http://localhost:9090/healthz
{ "status": "ok" }

GET /health/live, GET /livez

Simple alive indicator. Always 200. /health/live requires the bearer token when admin_token is set (it is not in the token-exempt list); its /livez alias is token-exempt — prefer /livez for unauthenticated probes.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/health/live
curl http://localhost:9090/livez
{ "alive": true }

GET /health/ready, GET /readyz

Readiness. Returns 200 if at least one backend node is healthy, 503 otherwise. /health/ready requires the bearer token when admin_token is set; its /readyz alias is token-exempt — prefer /readyz for unauthenticated probes.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/health/ready
curl http://localhost:9090/readyz


```json
{ "ready": true, "message": "Proxy is ready" }
{ "ready": false, "message": "Proxy is not ready" }

Topology & Node Management

GET /topology

Joins the config (node roles) with live health so a controller can read the current primary and the healthy/unhealthy node sets in a single, non-blocking round-trip. Designed for polling.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/topology

GET /nodes

List all configured backend nodes with their current health status.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/nodes
[
  {
    "address": "db-primary.internal:5432",
    "healthy": true,
    "last_check": "2026-07-08T10:15:30.123Z",
    "failure_count": 0,
    "last_error": null,
    "latency_ms": 0.5,
    "replication_lag_bytes": null
  }
]
FieldTypeDescription
addressstringNode address in host:port format.
healthyboolWhether the node is currently passing health checks.
last_checkstringISO 8601 timestamp of the most recent health check.
failure_countu32Consecutive health-check failures; resets to 0 on success.
last_errorstring | nullError from the most recent failed health check.
latency_msf64Round-trip latency of the most recent successful check.
replication_lag_bytesu64 | nullReplication lag in bytes (standby/replica only).

GET /nodes/{address}

Single node health. {address} is the node’s host:port string. 404 {"error":"Node not found"} if unknown.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:9090/nodes/db-primary.internal:5432

POST /nodes/{address}/enable — POST /nodes/{address}/disable

Enable re-admits a node into routing; disable removes it (new queries stop routing to it; in-flight work is allowed to finish). Useful for draining a single backend for maintenance.

Terminal window
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:9090/nodes/db-replica-1.internal:5432/disable
# → {"status":"disabled"}


curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:9090/nodes/db-replica-1.internal:5432/enable
# → {"status":"enabled"}

Draining the whole proxy is a different operation: send SIGUSR2 for a graceful drain (bounded by shutdown_drain_timeout_secs). There is no /drain route.

Failover & Chaos

Failover between backends is automatic (health-driven; in-session Transaction Replay is governed by tr_mode and tr_enabled, with the write journal behind /api/replay shipping in the default build). There is no /failover endpoint. To force a failover for testing, mark a node unhealthy via the chaos API.

GET /api/chaos

Read the current chaos overrides — “what is broken on purpose right now”.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/api/chaos

POST /api/chaos

Inject or clear a controlled fault. Supported actions: force_unhealthy, restore, reset.

Terminal window
# Force the primary unhealthy → triggers automatic failover to a standby
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"action":"force_unhealthy","target_node":"db-primary.internal:5432"}' \
  http://localhost:9090/api/chaos


# Restore that node
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"action":"restore","target_node":"db-primary.internal:5432"}' \
  http://localhost:9090/api/chaos


# Clear all chaos overrides at once
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"action":"reset"}' \
  http://localhost:9090/api/chaos

GET /api/circuit

Per-node circuit-breaker state (closed / open / half-open), so an operator can see which backends the breaker has tripped. 503 unless built with --features circuit-breaker.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/api/circuit

Metrics, Sessions, Pools, Version, Config

GET /metrics

Server metrics in JSON.

{
  "connections_accepted": 15234,
  "connections_rejected": 0,
  "connections_closed": 15100,
  "connections_active": 134,
  "queries_processed": 892451,
  "bytes_received": 45623891,
  "bytes_sent": 189234567,
  "failovers": 1
}

connections_active is computed as accepted − closed. connections_rejected counts connections refused because the [limits] max_client_connections cap was saturated (they are also counted in accepted/closed); cancel requests are never refused.

Transaction Replay counters (1.6.0–1.7.0)

Both /metrics and /metrics/prometheus additionally expose the in-session failover counters. The counter set arrived with the tr_mode data-path work in 1.6.0; 1.6.1 and 1.7.0 changed what they report, not their names — commit-outcome and delivery safety in 1.6.1, response-digest verification, the isolation and cap refusals, and one shared recovery deadline in 1.7.0:

CounterMeaning
tr_failovers_totalIn-session failovers handled (a backend fault re-homed onto a healthy primary).
tr_statements_reexecuted_totalStatements transparently re-executed after a fault.
tr_transactions_replayed_totalExplicit transactions replayed from their BEGIN (select / transaction modes).
tr_replay_failures_totalReplays abandoned because a replayed statement failed (client receives 40001).
tr_unknown_outcome_errors_totalFaults surfaced as 08007 transaction_resolution_unknown rather than retried.
tr_replay_cap_exceeded_totalTransactions marked non-replayable by [limits] tr_max_replay_statements / tr_max_replay_bytes.
tr_session_set_cap_exceeded_totalSessions whose SET/RESET tracking stopped at [limits] tr_max_session_set_statements (distinct variables). A subsequent failover on such a session is refused with 08006 rather than re-homed with incomplete state.

See Transaction Replay for what each mode guarantees and Configuration for the [limits] caps these counters report against.

Journal, topology and admission counters (1.9.0–2.0.0)

/metrics/prometheus also exports:

CounterSinceMeaning
heliosdb_proxy_journal_committed_total1.9.0Transactions the recovery journal recorded as committed.
heliosdb_proxy_journal_rolled_back_total1.9.0Captured transactions the backend rolled back (never enter committed history).
heliosdb_proxy_journal_statements_total1.9.0Statements appended to the recovery journal.
heliosdb_proxy_admission_waited_total1.9.0Client admissions that queued on the max_client_connections cap (client_admission_wait_secs).
heliosdb_proxy_admission_timeout_total1.9.0Admissions whose bounded wait expired (a subset of rejections).
heliosdb_proxy_reconnect_attempts_total1.9.0Jittered waits in the primary-select recovery loops — the size of a failover’s reconnect wave.
heliosdb_proxy_topology_conflicting_primaries_total2.0.0Provider polls that found more than one node claiming write authority (resolved by timeline or failed closed). Also conflictingPrimariesTotal on GET /topology.
heliosdb_proxy_backend_capacity_waits_total2.0.0Backend connections refused at max_connections (53300) and retried after closing an idle pooled connection.
heliosdb_proxy_backend_capacity_refusals_total2.0.0Connections refused with 53300 after the backend stayed full for the whole [pool] acquire_timeout_secs.

GET /metrics/prometheus

Prometheus text exposition format, wrapped in a JSON text field:

{ "text": "# HELP heliosdb_proxy_connections_total ...\n..." }

This is not directly scrapable as raw Prometheus text — the payload is JSON with the exposition string in text, so a scraper must unwrap the text field. This endpoint is served unconditionally; there is no separate build flag that changes its format (the observability feature only pulls in the prometheus/opentelemetry crates and wires no exporter — see the feature-flags reference).

GET /sessions

{ "active_sessions": 42 }

GET /pools

Per-node connection pool statistics (active/idle/pending, lifetime created/closed). Useful for watching a single-node drain complete.

GET /version

{ "version": "2.0.0", "build_time": "2.0.0" }

Both fields derive from the crate version (CARGO_PKG_VERSION); build_time is the version string, not a wall-clock timestamp.

GET /config

Returns a snapshot of the running configuration (secrets such as admin_token and TLS material that serialize as None are omitted).

SQL Execution API

POST /api/sql

Execute a SQL query through the proxy with transparent write routing (TWR): writes go to the primary, reads are load-balanced across healthy standby/replica nodes. The proxy forwards to the backend’s HTTP SQL API and returns the result plus routing metadata.

Terminal window
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT * FROM users LIMIT 10"}' \
  http://localhost:9090/api/sql
{
  "query_type": "read",
  "routed_to": "db-standby-1.internal:5432",
  "node_role": "standby",
  "result": { "columns": ["id","name"], "rows": [[1,"Alice"]], "row_count": 1 }
}

Request body

FieldTypeRequiredDescription
querystringyesSQL to execute.
paramsarraynoParameters (reserved for prepared-statement support).

Write classification — routed to the primary when the statement starts with: INSERT, UPDATE, DELETE, MERGE, UPSERT, CREATE, ALTER, DROP, TRUNCATE, GRANT, REVOKE, VACUUM, REINDEX, BEGIN, COMMIT, ROLLBACK, SAVEPOINT. Everything else (chiefly SELECT) is a read. node_role is reported as primary / standby / replica.

Error responses include {"error":"No healthy primary node available"} and {"error":"Empty query"}.

Observability & Diagnostics (feature-gated)

GET /plugins

Loaded WASM plugins — name, version, description, hooks, state, invocation count. Returns 503 {"error":"plugin manager not attached"} when the proxy runs without a plugin manager, and 503 {"error":"wasm-plugins feature not compiled in"} in a build without --features wasm-plugins.

/admin/kv/{plugin}/{key} — plugin runtime KV

Read, write, delete, and list a loaded plugin’s key-value state — the same per-plugin namespace the plugin sees through its kv_get / kv_set host imports. Operators use it to push runtime config (budgets, region maps, mask rules, allowlists) without restarting the proxy. All four verbs sit behind the normal admin bearer gate.

MethodPathBehavior
GET/admin/kv/{plugin}/{key}200 {"plugin","key","value"}, or 404 {"error":"key not found"}
GET/admin/kv/{plugin}/200 {"plugin","keys":[...]} — the trailing slash lists the namespace; an optional ?prefix= filters the listing
PUT/admin/kv/{plugin}/{key}200 {"ok":true}, or 413 when a cap is exceeded
DELETE/admin/kv/{plugin}/{key}200 {"ok":true} — idempotent (200 even when the key is absent); deleting a namespace’s last key frees its slot
  • {key} may contain /. The first path segment after /admin/kv/ is the plugin name; everything after it is the key (e.g. budget/tenant-a). Any query string is stripped before the split, so ?… never leaks into a key or plugin name; on a list request ?prefix=<p> (percent-decoded) filters the returned keys.
  • Values are UTF-8 text. PUT bodies are decoded with String::from_utf8_lossy (the admin body limit still applies), and GET returns the value as a JSON string. Store binary blobs base64-encoded.
  • Caps guard against runaway writes; all four are tunable in [plugins] and 0 means unlimited. kv_max_value_bytes (default 65536) bounds a single key’s OR value’s length; kv_max_keys_per_plugin (default 1024) bounds the distinct keys per namespace; kv_max_plugins (default 256) bounds how many <plugin> namespaces can exist at once (so a token-holder cannot exhaust memory by writing to unboundedly-many namespace names); and kv_max_total_bytes (default 67108864 / 64 MiB) bounds the TOTAL retained size across all namespaces (each entry’s key + value bytes plus each live namespace’s name bytes) — the single backstop that keeps the whole store within a survivable ceiling regardless of the per-axis product kv_max_plugins × kv_max_keys_per_plugin × kv_max_value_bytes (which can otherwise reach tens of GiB). Overwriting an existing key never trips the key-count cap, and writing to an already-present namespace never trips the namespace cap. A PUT that would exceed a cap returns 413 ({"error":"kv_max_value_bytes exceeded"} for an oversized body, or {"error":"kv_max_value_bytes, kv_max_keys_per_plugin, kv_max_plugins, or kv_max_total_bytes exceeded"}).
  • Keys containing ? are not addressable here. The query string is stripped before the plugin/key split (that is what lets ?prefix= filter a listing), so a key that itself contains ? — a plugin can create one through kv_set — cannot be read or deleted through this endpoint: the strip eats everything from the first ?, and path segments are not percent-decoded, so %3F does not reach it either. Such a key still appears in the trailing-slash listing. Avoid ? in KV keys.
  • 400 on a malformed path (/admin/kv/{plugin} with no key segment) or an empty {plugin} segment (/admin/kv//{key}).
  • 405 on an unsupported method.
  • 503 {"error":"plugin runtime not enabled"} when no plugin manager is attached (plugins disabled in config).
  • 501 {"error":"proxy built without the wasm-plugins feature"} in a build without --features wasm-plugins — note this is 501, not the 503 other feature-gated routes use, because the entire KV subsystem is absent from the binary.
Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" -X PUT \
  http://localhost:9090/admin/kv/helios-plugin-cost-governor/budget/tenant-a \
  --data-raw '{"queries_per_minute":1000}'
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:9090/admin/kv/helios-plugin-cost-governor/budget/tenant-a
# {"plugin":"helios-plugin-cost-governor","key":"budget/tenant-a","value":"{\"queries_per_minute\":1000}"}

GET /anomalies

Recent events from the in-process anomaly detector (SQL-injection heuristics, auth bursts, rate spikes, novel query shapes), newest-first. Optional ?limit=N clamps the response (default 100). 503 unless built with --features anomaly-detection.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" "http://localhost:9090/anomalies?limit=20"

GET /analytics (alias /api/analytics)

Top queries by call count plus the slow-query log. Optional ?limit=N (default 50). 503 unless built with --features query-analytics.

Terminal window
curl -H "Authorization: Bearer $ADMIN_TOKEN" "http://localhost:9090/api/analytics?limit=50"

HA / Time-Travel

POST /api/replay

Replay journaled history against a target backend (typically a staging DB) — for failover validation, hydrating staging from prod, or forensics. Body is a ReplayRequestBody. 503 {"error":"transaction replay disabled (tr_enabled = false)"} when TR is disabled at runtime; Transaction Replay ships in the default build.

Body fields: from / to (RFC 3339), target_host, target_port, optional target_user / target_password / target_database, optional mode, and (for committed_history) optional after_commit_seq.

mode selects one of two replays (since 1.9.0); any other value is 400:

modeWhat is replayedOrderOn failure
omitted / "time_window"The statement text of every committed transaction whose statements fall in [from, to]timestamp order, flattened across transactions, each statement independentcontinues; partial: true, first error reported
"committed_history"Every transaction whose commit was observed in [from, to] (and commit_seq > after_commit_seq)commit order, one transaction at a time, each in its own BEGIN … COMMIT on one connection, parameters re-sent in their captured formatrolls that transaction back and stops; partial: true, stopped_at says where

Committed-history replay never leaves a partial transaction on the target: a transaction the journal could not capture completely (a COPY … FROM STDIN, an EXECUTE of a session-scoped prepared statement, a statement over [journal] max_statement_bytes, a per-transaction cap) is refused before it starts. last_commit_seq in the response is the resume point for the next call.

{
  "mode": "committed_history",
  "transactions_replayed": 12,
  "statements_replayed": 41,
  "transactions_selected": 15,
  "last_commit_seq": 1042,
  "partial": true,
  "deadline_exceeded": false,
  "stopped_at": { "tx_id": "…", "commit_seq": 1043, "sequence": 2,
                  "error": "duplicate key value violates unique constraint" },
  "elapsed_ms": 813,
  "from": 1756684800,
  "to": 1756771200,
  "coverage": {
    "retained_transactions": 3,
    "retained_entries": 7,
    "retained_bytes": 512,
    "max_journals": 50000,
    "committed_transactions": 1042,
    "committed_entries": 3901,
    "committed_bytes": 1048576,
    "max_committed_transactions": 50000,
    "max_committed_bytes": 268435456,
    "commit_seq_high": 1042,
    "dropped_transactions": 0,
    "survives_restart": true,
    "transaction_boundaries": true,
    "parameter_values": true,
    "outcomes": true
  }
}

A time_window response keeps its previous shape (statements_replayed, failures, partial, deadline_exceeded, first_error) plus the same coverage block.

What the journal contains (since 1.9.0): real transactions as the backend reported them. On both the simple and the extended protocol, every data-changing statement the backend completed is journaled with its CommandComplete tag, the extended-protocol Bind parameter values byte for byte with their text/binary format and the declared type OIDs, the source identity (client address, user, database, backend, tenant) and — once the backend answered COMMIT — a global commit sequence. Rejected statements, rolled-back transactions, work undone by ROLLBACK TO SAVEPOINT and two-phase transactions never enter committed history. Reads are not journaled, so a read with side effects (nextval, set_config, a volatile function) is not reproduced. survives_restart is true when [journal] dir is configured (see configuration.md). The commit order is the order this proxy observed the commit responses, which is a total order over everything routed through it but not necessarily the backend’s WAL order for commits that raced on different sessions.

POST /api/shadow

Run a query against a source and a shadow backend in parallel and diff the results — used for major-version-upgrade validation, schema-migration canaries, and replica-drift detection. Body is a ShadowRequestBody. Available in the default build (not gated by tr_enabled).

The two sides now execute concurrently: the shadow runs in its own task and connection while the primary is awaited, so its latency overlaps instead of adding to the request (since 1.9.0). Two optional body fields bound the comparison: max_rows (default 10000) and max_bytes (default 16 MiB). A result over either ceiling is returned with "budget_exceeded": true and is deliberately not certified clean ("is_clean": false) even when counts and digests match. The comparison is an order-independent digest and retains no copy of either result set beyond what the backend client materialises.

Edge / Geo Mode (feature edge-proxy)

All edge routes return 503 {"error":"edge-proxy feature not compiled in"} in a build without --features edge-proxy.

GET /api/edge

Home-side stats: registered edges and cross-region cache stats.

POST /api/edge/register

Register an edge with the home proxy (ack-only compatibility path; the long-lived stream is /subscribe).

POST /api/edge/invalidate

Broadcast a table-level invalidation to subscribed edges (last-write-wins TTL coherence). Handy for ops drills.

GET /api/edge/subscribe

Long-lived Server-Sent Events stream of invalidations. Requires an ?edge_id=<id> query parameter (optional region, base_url); a missing edge_id gets 400. This route is intercepted before the normal one-shot dispatch (a Content-Length-framed JSON response cannot hold a stream open), but the bearer-token gate still applies — an unauthenticated subscribe gets the same 401. Each write is bounded by ADMIN_SSE_WRITE_TIMEOUT (30 s) and a 15 s heartbeat keeps the connection permit from leaking.

Terminal window
curl -N -H "Authorization: Bearer $ADMIN_TOKEN" \
  "http://localhost:9090/api/edge/subscribe?edge_id=eu-west-1"

Migration / Traffic Mirror

These routes are always compiled in but return 503 {"error":"traffic mirroring not enabled"} unless a [mirror] migration is configured. Each path also accepts the non-/api spelling.

GET /api/migration/status

Mirror lag/backlog/drop counters plus a cutover_active flag. migration_ready is true only when lag == 0, dropped == 0 and errors == 0: a mirrored write whose apply failed stays visible in both errors and lag, and blocks an unforced cutover.

POST /api/migration/snapshot

Snapshot-bootstrap named tables from the source into the mirror. Body: {"tables":["orders","users"]} — an empty/missing tables array returns 400. Response reports per-table rows copied and a rows_copied total.

POST /api/migration/cutover

Promote the mirror target to primary — new connections route there. If the mirror is not migration_ready (backlog/drops present) the call returns 409 with the current status; pass force=true (query string or {"force":true} body) to override.

Terminal window
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  "http://localhost:9090/api/migration/cutover?force=true"

POST /api/migration/cutover/rollback

Revert a cutover so new connections route to the original primary again.

Branch Databases

Always compiled in; each returns 503 {"error":"branch databases not enabled"} unless [branch] is configured.

Terminal window
# List
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:9090/api/branch


# Create (base optional; defaults to the configured base database)
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"feature-x","base":"main"}' \
  http://localhost:9090/api/branch


# Drop
curl -X DELETE -H "Authorization: Bearer $ADMIN_TOKEN" \
  "http://localhost:9090/api/branch?name=feature-x"

Creating without a name returns 400 {"error":"provide 'name'"}; dropping without ?name= returns 400 {"error":"provide ?name=<branch>"}.

Web UI

GET / and GET /ui serve an embedded admin dashboard (single HTML file compiled into the binary). The static shell is token-exempt — it is served even when admin_token is set, because the HTML itself carries no privileged data; every API call the dashboard makes (/nodes, /metrics, /api/sql, …) still goes through the bearer gate.

When admin_token is set, the dashboard handles auth entirely client-side:

  • On first load, the first API call returns 401 and the page prompts once for the admin token (the same value as admin_token in proxy.toml).
  • The token is kept in the tab’s sessionStorage under the key helios_admin_token — it dies with the tab, is never written to disk, and never appears in the URL. A wrapped window.fetch injects Authorization: Bearer <token> into every request (a caller-supplied Authorization header still wins).
  • A token button in the header bar clears the saved token and reloads, so a wrong token can be re-entered without closing the tab.

When admin_token is unset, the dashboard works with no prompt (rely on the loopback bind for protection).

Error Handling

All JSON errors share the shape {"error":"<description>"}.

StatusMeaning
200Success.
400Bad request (malformed input, missing required field/param).
401Missing/invalid admin bearer token (only when admin_token is set).
404Unknown node address, or an unrouted path.
409Migration cutover blocked (mirror not migration_ready); retry with force=true.
500Internal server error.
503Not ready / no healthy backends, or a feature/subsystem is not compiled in / not enabled.

Usage Examples

Liveness / readiness in a script

Terminal window
# Liveness — open path, no token needed
curl -sf http://localhost:9090/health >/dev/null && echo "alive"


# Readiness — token required when admin_token is set
if curl -sf -H "Authorization: Bearer $ADMIN_TOKEN" \
     http://localhost:9090/health/ready >/dev/null; then
  echo "ready"
else
  echo "NOT ready" >&2; exit 1
fi

Drain a single backend for maintenance

Terminal window
NODE=db-standby-1.internal:5432
AUTH="Authorization: Bearer $ADMIN_TOKEN"


curl -X POST -H "$AUTH" "http://localhost:9090/nodes/$NODE/disable"


while true; do
  active=$(curl -s -H "$AUTH" http://localhost:9090/pools | \
    jq --arg n "$NODE" '.[] | select(.node == $n) | .active_connections')
  [ "$active" = "0" ] && break
  echo "waiting for $active connections to drain..."; sleep 5
done


# ...maintenance...
curl -X POST -H "$AUTH" "http://localhost:9090/nodes/$NODE/enable"

Force a failover drill

Terminal window
AUTH="Authorization: Bearer $ADMIN_TOKEN"
# Mark the primary unhealthy → automatic failover kicks in
curl -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"action":"force_unhealthy","target_node":"db-primary.internal:5432"}' \
  http://localhost:9090/api/chaos


curl -s -H "$AUTH" http://localhost:9090/topology | jq '.currentPrimary'


# Restore when done
curl -X POST -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"action":"reset"}' http://localhost:9090/api/chaos

Prometheus scrape

scrape_configs:
  - job_name: heliosproxy
    metrics_path: /metrics/prometheus
    static_configs:
      - targets: ["heliosproxy:9090"]

Note: /metrics/prometheus wraps the exposition text in a JSON text field, so the scrape job must unwrap text before parsing. There is no build flag that emits raw Prometheus text (the observability feature adds the prometheus/opentelemetry crates but wires no exporter). If admin_token is set, add the bearer token to the scrape job’s authorization config.

See Also