Operations

How to configure, run and observe Querdian. This documents the binary as it exists in this repository. There is no published release, container image or Helm chart yet (they belong to the packaging tasks of the plan); the sections on deployment therefore describe what the binary needs rather than a packaged install. See policy-guide.md for the policy resources and troubleshooting.md for diagnosing denials.

Process model#

One binary, querdian, runs three things:

• the proxy listener (--listen, default :8080): plain HTTP and WebSocket. It does not terminate TLS; put an ingress, gateway or service mesh in front of it and make it the only way clients reach the backends.
• the management listener (--management-listen, default :9090): GET /metrics, GET /healthz, GET /readyz. It is unauthenticated; do not expose it outside the cluster.
• the controller: informers on ClusterAccessPolicy, AccessPolicy and Namespace, a 250 ms debounce, snapshot compilation and, on the leader only, status writing.

Every replica builds its own snapshot and serves traffic independently. Replicas share no state.

Static configuration#

Process settings are flags; structured settings come from one YAML file, read once at startup (--config, default /etc/querdian/config.yaml; mount it from a ConfigMap and restart the pods to change it). Unknown fields are an error, and any validation failure stops the process before it listens. Policies reference backends by name, so changing backend names invalidates policies (UnknownBackend).

Sample configuration#

auth:
  issuer: https://keycloak.example.com/realms/observability
  audiences: [querdian]
  claims:
    username: preferred_username
    groups: groups
backends:
  - name: metrics
    type: prometheus
    url: http://thanos-query.monitoring:10902
    pathPrefix: /prometheus
    namespaceLabel: namespace
  - name: logs
    type: loki
    url: http://loki-gateway.monitoring
    pathPrefix: /loki
    namespaceLabel: k8s_namespace_name
    tenantHeader: X-Scope-OrgID
    tenantFederation: tenantIdLabel
  - name: traces
    type: tempo
    url: http://tempo-query-frontend.monitoring:3200
    pathPrefix: /tempo
    namespaceLabel: resource.k8s.namespace.name
    tenantHeader: X-Scope-OrgID
    tenantFederation: none
    traceByID: filter
limits:
  maxPolicyTerms: 16
  maxFanout: 16
  maxMergeBytes: 67108864
  upstreamTimeout: 5m

Grafana data sources use the proxy URL plus the pathPrefix (for example https://querdian.example.com/loki) with the user’s OAuth token forwarded.

auth#

Tokens are verified for signature, expiry, issuer and audience. There is no token cache, no introspection and no revocation check beyond token expiry.

backends[]#

limits#

All must be positive. What happens when one trips is listed under Limits.

Flags#

Kubernetes access uses the usual controller-runtime discovery (in-cluster service account, or KUBECONFIG / ~/.kube/config for local runs). Shutdown on SIGINT/SIGTERM drains in-flight requests for up to 30 s.

Deployment notes#

Until a chart exists, a deployment needs:

• CRDs: apply config/crd/policy.querdian.dev_clusteraccesspolicies.yaml and config/crd/policy.querdian.dev_accesspolicies.yaml (generated by make manifests). The CRDs carry the CEL validations (at least one subject; exactly one of unrestricted or matchers).
• RBAC for the service account:
  • get, list, watch on clusteraccesspolicies, accesspolicies (group policy.querdian.dev) and on namespaces;
  • update, patch on clusteraccesspolicies/status and accesspolicies/status;
  • get, list, watch, create, update, patch on coordination.k8s.io leases in the namespace of the pod (the lease is named querdian-leader-election). The service account needs no access to workloads or secrets.
• Network isolation: backends must not be reachable by clients except through Querdian (NetworkPolicy or equivalent). The security model depends on it.
• Several replicas behind one Service. The pods are stateless.
• Backend settings for cross-tenant requests (multi-tenant query federation) if you use several tenants per user, see policy-guide.md. Tempo streaming over gRPC is out of scope and must be disabled in the Grafana Tempo data source.
• A traceparent header from clients is honoured as the parent of the request span; towards upstreams only W3C trace context is sent.

Readiness and health#

Until then the proxy answers 503 unavailable (snapshot_not_ready or auth_unavailable). After readiness, readiness stays true while the Kubernetes API is unreachable: the last good snapshot keeps being used (see Upgrade and failure behaviour).

Scaling#

• Each request costs one parse and rewrite (see querdian_rewrite_duration_seconds; typically microseconds to milliseconds) plus streaming the response. Scale horizontally on CPU.
• Fan-out multiplies upstream load: a log query for a user with N terms is N upstream requests, up to maxFanout. Watch querdian_policy_terms and querdian_upstream_requests_total. Prefer policy shapes that simplify to one term (the single-label merge helps).
• Memory: streaming requests use constant memory. Merged requests buffer up to maxMergeBytes each, so plan for peak concurrent merged requests x maxMergeBytes. Reduce the value if you expect many concurrent fan-outs. There is no built-in rate limit or concurrency cap; apply one at the ingress.
• The snapshot is held per replica in memory and rebuilt completely on every policy or namespace label change (about 250 ms after the last change). Snapshot size grows with the number of rules.
• WebSocket tails keep one connection per client plus one per term open.

Leader election#

--leader-elect=true (default) uses a Kubernetes Lease (querdian-leader-election, released on shutdown, controller-runtime default timings). Only the status writer is leader-gated: exactly one replica patches status.conditions. All replicas build snapshots and serve requests, so leader loss never interrupts enforcement; it only pauses status updates until a new leader acquires the lease. With a single replica or for local runs you may use --leader-elect=false. querdian_policies and querdian_snapshot_* are exported by every replica.

Limits and what happens when they trip#

An upstream failure of any branch of a fan-out fails the whole request with 502 (the upstream status and the first 256 bytes of its error body are included in the message).

Observability#

Metrics#

Served on the management port. Labels: endpoint is the route template (for example /api/v1/label/:name/values), never the raw path; unknown paths use unknown.

Go and process collectors are also exported.

Tracing#

Configured exclusively through the standard OTEL_* environment variables (OTEL_SERVICE_NAME default querdian, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_EXPORTER, OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, OTEL_RESOURCE_ATTRIBUTES).

• If neither an OTLP endpoint nor OTEL_TRACES_EXPORTER is set, spans are discarded.
• If OTEL_TRACES_SAMPLER is unset the sampler is parentbased_traceidratio with ratio 1.0.
• Propagation is fixed in code: inbound W3C tracecontext and baggage, outbound tracecontext only. OTEL_PROPAGATORS is ignored on purpose, so client baggage never reaches a backend.
• Every response carries X-Trace-Id and every error body a traceId; use it to find the trace.

Span tree of one proxied request:

Privacy flags, both off by default:

• --tracing.record-user: enduser.id on the server span and user in the request log line.
• --tracing.record-queries: querdian.query.original and querdian.query.rewritten on the server span. Queries can contain sensitive values; enable only where the trace backend has equivalent access controls. Queries are never written to the log.

Without these flags no username and no query text is recorded; querdian.policies records the names of the matched policies.

Logging#

log/slog to stderr, JSON by default (--log-format=text for humans). Every proxied request produces one request line with backend, route, decision, reason, status, duration and, when the request is traced, trace_id and span_id; plus user if --tracing.record-user is set. A querdian started line is written at startup.

Upgrade and failure behaviour#

• Pre-release. The API group is policy.querdian.dev/v1alpha1 and may change incompatibly before a stable version; read the changelog of each release once releases exist.
• Order: apply updated CRDs first (kubectl apply --server-side -f config/crd), then roll the deployment. The static configuration is read at startup, so a configuration change needs a rollout.
• Rolling updates are safe for availability: a new pod is ready only after its first snapshot and issuer discovery. During a rollout old and new replicas may enforce slightly different behaviour if the new version changed rewriting rules; there is no cross-replica coordination.
• Policy changes reach each replica within the watch latency plus ~250 ms. In-flight requests and open WebSocket tails continue under the policy they started with, and an open tail is not closed when the token expires.
• Kubernetes API outage: replicas keep serving with the last snapshot (they do not fail open or closed). A failed rebuild keeps the previous snapshot and increments querdian_snapshot_rebuilds_total{result="error"}; a new pod cannot become ready.
• Issuer outage: tokens can only be verified with keys the verifier has cached; unreachable issuer discovery at startup keeps the pod unready (auth_unavailable).
• Removed backend names or labels: policies referencing an unknown backend become UnknownBackend and grant nothing.