Skip to main content
Access required. Airhouse is proprietary and not open source. The container image and Helm chart are available only to customers with an agreement with Oxygen. Contact Oxygen to get access, or see Getting access.
This page walks through a reference production deployment on AWS, similar to the setup Oxygen runs. It’s an example to adapt, not a turnkey module. Every name, size, and region below is a placeholder. Replace <region>, <account-id>, and names like airhouse-data-<env> with your own. For the general installation steps and all deployment options, see Installation.

Architecture

Terraform: AWS resources

The snippets assume you already have a VPC and an EKS cluster, for example from the terraform-aws-modules/vpc and terraform-aws-modules/eks modules, referenced here as module.vpc and module.eks.

KMS key and S3 bucket

s3.tf
Airhouse’s maintenance jobs delete Parquet files that are no longer referenced. If you enable S3 versioning for extra protection, add a lifecycle rule that expires noncurrent versions. Otherwise deleted files keep accruing storage cost.

RDS for PostgreSQL

rds.tf
After the instance is up, create the two Postgres roles Airhouse uses. Run this once as the master user, for example from a one-off Kubernetes Job or a bastion host. RDS IAM authentication applies to roles that are granted rds_iam.
bootstrap.sql

IAM role for EKS Pod Identity

One role serves both planes. It grants read and write on the bucket, use of the KMS key, and IAM database login for the two control-plane users.
iam.tf
The cluster needs the EKS Pod Identity Agent add-on. If you use IRSA instead, annotate the service accounts with the role ARN through serviceAccount.controlPlane.annotations and serviceAccount.dataPlane.annotations, and use an IRSA trust policy.

Admin API token in Secrets Manager

secrets.tf
Your backend reads the same secret when it calls the Admin API to create databases, users, or service accounts.

Kubernetes: secrets and Helm values

Sync the admin token with External Secrets

admin-token-external-secret.yaml
If you use password authentication for Postgres instead of IAM, sync the airhouse_cp and airhouse_catalog_admin passwords the same way, and reference them with controlPlaneDatabase.passwordSecretRef and catalogDatabase.adminPasswordSecretRef.

Helm values

values-aws.yaml

Install with Argo CD

airhouse-application.yaml
Without Argo CD, run helm install airhouse oci://ghcr.io/oxy-hq/helm-charts/airhouse --version <chart-version> -n airhouse -f values-aws.yaml.

Client access and TLS

  • Clients inside the cluster connect to the HAProxy Service on port 5445, for example host=airhouse-haproxy.airhouse.svc port=5445 for a release named airhouse in the airhouse namespace.
  • Restricting port 5445: the chart ships NetworkPolicies for the control plane and data plane only. HAProxy has none, so any pod in the cluster can reach port 5445 (clients still need valid credentials). networkPolicy.allowedNamespaces doesn’t change this: it only opens port 8080 on the two planes. To limit which namespaces can connect, add a policy like this one. It’s not part of the chart. Its pod selector matches the labels the chart puts on HAProxy pods:
    haproxy-networkpolicy.yaml
    Add a rule for port 5446 if you enable haproxy.analyticsBackend. If you expose HAProxy through a load balancer, allow the load balancer’s source ranges too.
  • Clients outside the cluster: set haproxy.service.type: LoadBalancer and add AWS Load Balancer Controller annotations for a Network Load Balancer. Prefer an internal scheme, reachable over VPN or peering, to an internet-facing one:
  • TLS for clients: HAProxy passes TCP through, so TLS terminates on the data plane. Enable it with a kubernetes.io/tls Secret and point the server at the mounted files:
    Enable TLS for any client traffic that leaves the cluster. Connections from Airhouse to RDS use TLS (sslMode: require), and connections to S3 use HTTPS (useSsl: true).
  • Admin API: keep it private. Reach it from your backend inside the cluster (control-plane Service, port 8080), or with kubectl port-forward for manual administration. Don’t expose it through an ingress.

Operations

Metrics and alerts

With metrics.serviceMonitor.enabled, Prometheus or a compatible scraper (for example VictoriaMetrics’ agent) collects /metrics from both planes. The scraper’s namespace must be in networkPolicy.allowedNamespaces, or the chart’s NetworkPolicies block the scrape and none of the alerts below can fire. metrics.prometheusRule.enabled installs the chart’s alert rules, including:
  • AirhouseQueryAvailabilityBurning and AirhouseQueryAvailabilitySlowBurn: wire-protocol query error budget.
  • AirhouseQueryLatencyP99High
  • AirhouseAdminAvailabilityBurning and AirhouseMintErrorRateHigh: Admin API and credential minting.
  • AirhouseCircuitBreakerOpen: a pod is shedding all queries after repeated backend failures.
  • AirhouseDataPlaneRestart and AirhouseControlPlaneLivenessRestart
  • AirhouseQueryQueueSaturated, AirhouseWorkerPoolExhausted, and AirhouseQueriesSpillingToDisk: capacity pressure.
  • AirhouseCPDBPoolSaturated and AirhouseSweeperLag
In addition, the reference setup pages on these availability conditions, using kube-state-metrics. The Deployment names below assume a Helm release named airhouse:
airhouse-availability-rules.yaml
A data-plane pod that fails readiness but stays alive doesn’t restart on its own. kubectl rollout restart of the data-plane Deployment clears it. For RDS, CloudWatch alarms on DBLoad (above 50% and 100% of vCPUs), CPUUtilization (above 80%), DatabaseConnections, ReadLatency and WriteLatency (above 20 ms), FreeableMemory, and FreeStorageSpace are a good baseline.

Logs and traces

  • Logs are JSON (logging.format: json). Collect them with your cluster’s log agent, such as an OpenTelemetry Collector or Fluent Bit. Keep logging.logStatementText: false if SQL text may contain sensitive values.
  • Traces: set tracing.enabled: true and tracing.endpoint to an OTLP/HTTP collector, for example http://<otel-collector>.<namespace>.svc:4318.

Backups

  • Catalogs and registry: RDS automated backups (14 days here) and snapshots cover every Airhouse database’s catalog and the registry. Keep deletion_protection on, and test restores periodically.
  • Data: Parquet files in S3 are immutable once written. S3 versioning, or replication to a second bucket, protects against accidental deletes. Restore the catalog and data to the same point in time, because a catalog references specific data files.
  • Time travel window: snapshot expiry (below) sets how far back time travel can reach.

Maintenance jobs

DuckLake needs periodic maintenance: compacting small files, expiring old snapshots, and deleting files that are no longer referenced. By default the control plane runs this in process every 6 hours and keeps 30 days of snapshots (controlPlane.vacuum). The reference setup moves it into the chart’s vacuum CronJob, so a large merge can’t compete with the control plane for memory. The CronJob is the one job that expires snapshots and deletes files:
Choose retentionDays to match how far back you need time travel and point-in-time restores. The chart also ships a native compaction drain (compactionDrain) for working through a large backlog of small files. It’s a manual Job: keep it suspended, and trigger it on demand instead of scheduling it. Leave its expiry, cleanup, and reap switches off so it doesn’t duplicate the vacuum CronJob’s work:
A dry run measures and reports the backlog without changing data. Setting dryRun: false makes the drain rewrite Parquet files, so make that a separate, deliberate change: read the dry-run report, pause the vacuum CronJob (vacuum.cronjob.suspend: true) so the two don’t work on the same catalogs at once, run one drain, then resume the CronJob.

Sizing starting point

To resize, keep maxConcurrentQueries × memoryPerSessionMB + ~1 GB and maxTenantInstances × memoryPerSessionMB (which must stay under podMemoryBudgetMB) below resources.limits.memory, and keep maxConcurrentQueries × threadsPerSession near the CPU limit. For example, a 10 GiB pod fits 8 queries at 1 GiB. Watch AirhouseQueriesSpillingToDisk and pod memory. Add replicas for more concurrency, and raise pod memory together with memoryPerSessionMB for larger queries.

Next steps