Deploy with OpenTofu

The aperturedata/aperturedb-terraform package is an OCI module for OpenTofu. It creates native Kubernetes resources for distributed ApertureDB in an existing cluster. The default topology has three database pods, three ZooKeeper pods, two database gateways, two HTTP proxies, one checkpointer, a Web UI, and telemetry.

Prerequisites

  • OpenTofu 1.10 or later for OCI module sources. Use tofu for the OCI source shown below, even though the package name contains terraform.

  • Kubernetes 1.23 or later and a kubeconfig for the intended cluster.

  • An object storage bucket and credentials; see Deploying ApertureDB.

  • Persistent volume provisioning for ZooKeeper and a 100Gi monitoring volume when telemetry is enabled.

  • Sufficient cluster capacity: each database pod and the checkpointer requests 8Gi of memory by default, and each Lenz pod requests 1Gi, in addition to the other components.

  • LoadBalancer support for the example below.

Configure the Kubernetes provider. The HTTP service is included; no ingress controller or Helm provider is needed. Use a separate namespace per deployment because application resource names are fixed within a namespace.

Configure the module

In a new directory, create main.tf. Replace <package-version> with the published package tag you want, and replace the bucket name and hostname. This example references secrets you create separately in the namespace:

terraform {
  required_version = ">= 1.10.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.33"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

module "aperturedb" {
  source = "oci://registry-1.docker.io/aperturedata/aperturedb-terraform?tag=<package-version>"

  namespace        = "aperturedb"
  create_namespace = false
  master_key       = { existing_secret = "adb-master-key" }

  storage = {
    type            = "s3"
    bucket          = "<your-bucket-name>"
    existing_secret = "database-storage-credentials"
  }

  service = { type = "LoadBalancer" }
  http = {
    host       = "aperturedb.example.com"
    tls        = true
  }

  telemetry = {
    enabled               = true
    auth_password_secret  = "adb-monitoring"
    admin_password_secret = "adb-grafana"
  }
}

output "database_load_balancer" {
  value = module.aperturedb.lenz_external_load_balancer
}

output "http_load_balancer" {
  value = module.aperturedb.http_load_balancer
}

Create the namespace and secrets

Create the namespace if it does not already exist:

kubectl create namespace aperturedb

Create four local files: master-password.txt, monitoring-password.txt, grafana-password.txt, and storage-credentials.json. Each password file must contain a different strong password without a trailing newline. The JSON file contains the S3 credentials described in Deploying ApertureDB. Then run:

chmod 600 master-password.txt monitoring-password.txt grafana-password.txt storage-credentials.json
kubectl -n aperturedb create secret generic adb-master-key \
  --from-file=password=master-password.txt
kubectl -n aperturedb create secret generic adb-monitoring \
  --from-file=password=monitoring-password.txt
kubectl -n aperturedb create secret generic adb-grafana \
  --from-file=password=grafana-password.txt
kubectl -n aperturedb create secret generic database-storage-credentials \
  --from-file=ADB_STORAGE_CREDENTIALS=storage-credentials.json

For Google Cloud Storage, set storage.type = "gs" and put the service account JSON in storage-credentials.json. The OpenTofu storage secret key is ADB_STORAGE_CREDENTIALS; the Helm chart uses a different key.

Alternatively, the module can create its namespace with create_namespace = true and create secrets from inline values: master_key.password, storage.credentials (a map containing ADB_STORAGE_CREDENTIALS), telemetry.auth_password, and telemetry.admin_password. Inline credentials are stored in OpenTofu state. Protect state, plan files, and local credential files, and keep them out of source control.

For private workload images, create a registry secret in the namespace and set image_pull_secrets = ["my-registry-secret"] in the module.

Configure TLS

The module uses an existing TLS secret, inline certificate and key, cert-manager, or a generated self-signed certificate, in that order. To use a certificate covering your web and TCP hostnames:

kubectl -n aperturedb create secret tls adb-tls \
  --cert=cert.pem --key=key.pem

Add tls = { existing_secret = "adb-tls" } to the module. Other options are tls.cert and tls.key, or tls.cert_manager with a configured issuer. Without these settings, the self-signed fallback is for evaluation; clients must explicitly trust it or use an evaluation-only certificate verification override. The generated private key is stored in OpenTofu state.

Plan, apply, and connect

Initialize the module and providers, review the plan, then apply it:

tofu init
tofu validate
tofu plan -out=aperturedb.tfplan
tofu apply aperturedb.tfplan
kubectl -n aperturedb get pods,svc,pvc
tofu output database_load_balancer
tofu output http_load_balancer

Point the DNS record for http.host at the address from http_load_balancer. Native clients use lenz_external_load_balancer. The TLS certificate must cover all hostnames used by web and native clients.

  • Web UI: https://aperturedb.example.com/.

  • REST API: https://aperturedb.example.com/api/.

  • Native clients: the lenz-external service’s external address on port 55555, with TLS enabled. In-cluster clients can use lenz on that port.

  • Grafana: https://aperturedb.example.com/grafana/ with telemetry enabled. Log in as admin using the Grafana secret’s password.

Use admin with the master key in the runnable client check at Connect and verify the deployment. See Deployment settings and process boundaries for server and gateway settings. Check pod events, PVC status, and container logs if startup does not complete. A pending external address requires checking LoadBalancer support; HTTP routing also depends on DNS and http.host.

Outputs by service type

Module outputs become visible to tofu output only after you expose them in your root configuration. Use the output section that matches each Service; native service.type and http.service.type can differ.

LoadBalancer outputs

Use the existing database_load_balancer and http_load_balancer outputs in the example above. Their values are ingress lists containing an ip or hostname. They can be empty while a load balancer is provisioning, and stay empty for NodePort or ClusterIP Services.

NodePort outputs

Add these root outputs to read the allocated ports, including ports assigned automatically by Kubernetes:

output "database_node_port" {
  value = module.aperturedb.lenz_ports.tcp.node_port
}

output "http_node_ports" {
  value = {
    http  = module.aperturedb.http_ports.http.node_port
    https = try(module.aperturedb.http_ports.https.node_port, null)
  }
}

Connect to a reachable cluster node address using these ports. The module does not select or output a node address. http_node_ports.https is null when http.tls is disabled.

ClusterIP outputs

Use Service DNS and the Service port, rather than node_port:

output "database_cluster_endpoint" {
  value = {
    host = "${module.aperturedb.lenz_service}.${module.aperturedb.namespace}.svc.cluster.local"
    port = module.aperturedb.service_port
  }
}

output "http_cluster_endpoint" {
  value = {
    host  = "http-proxy.${module.aperturedb.namespace}.svc.cluster.local"
    ports = {
      http  = module.aperturedb.http_ports.http.port
      https = try(module.aperturedb.http_ports.https.port, null)
    }
  }
}

These addresses are reachable within the cluster. Substitute your cluster domain if it differs from cluster.local. Configure http.host and TLS for the hostname used by HTTP clients. The native certificate must also cover the Service hostname when clients connect through that name. lenz_external_service provides the other native Service name for every Service type; lenz_service always names the internal ClusterIP Service.

NodePort access

For direct access through cluster nodes, configure distinct, available HTTP and HTTPS NodePorts and allow them through your firewall:

http = {
  host = "aperturedb.example.com"
  service = {
    type            = "NodePort"
    http_node_port  = 30080
    https_node_port = 30443
  }
}

Point http.host at a reachable node. With these example ports, the Web UI is at https://aperturedb.example.com:30443/; append api/ or grafana/ for the API or dashboards. HTTP on port 30080 redirects to that HTTPS address. Choose ports from your cluster’s configured NodePort range, normally 30000–32767. To expose native clients through NodePort as well, set service.type to NodePort and inspect lenz-external for its allocated port.

Behind an external load balancer serving ports 80/443, use NodePort and set http.public_port to 443. Leave the HTTP and HTTPS NodePorts at zero for automatic allocation. Redirects and Grafana URLs use public port 443. Both database and HTTP Services are always created; use ClusterIP for private access.

Workflow routes and request limits

Use http.routes to expose workflow HTTP services under the deployment’s web hostname. Each route specifies a path prefix, an in-cluster service name, and its HTTP port. For example:

http = {
  host = "aperturedb.example.com"
  routes = [{
    path = "/notebooks"
    host = "workflow-notebooks-ingresses"
    port = 8888
  }]
}

Paths must be unique and omit the trailing slash; /api and /grafana are reserved. Requests retain their path prefix and query string. WebSockets and streamed responses are supported, and route updates take effect automatically.

http.body_limit limits each request body to 256 MiB by default. Oversized requests receive 413. http.timeout_seconds controls upstream inactivity and defaults to 300 seconds; workflow routes and Grafana allow one hour between response reads. Active requests have up to five minutes to finish during shutdown. Long-lived sessions should reconnect after an update.

Valid replacements in the configured TLS Secret are loaded automatically. Existing connections can finish with the previous certificate; invalid replacements leave the last valid certificate active.

Settings reference

Set only the attributes you want to change in the module block. Omitted attributes of an object keep their defaults, so zookeeper = { min_session_timeout = 30000 } keeps the other ZooKeeper settings.

Network

Network settings

Attribute

Default

Purpose

service.type

LoadBalancer

Type of the lenz-external native database Service: LoadBalancer, NodePort, or ClusterIP.

service.port

55555

Published native database port.

service.node_port

0

Fixed node port when service.type is NodePort; 0 assigns one automatically.

service.annotations, service.load_balancer_class, service.load_balancer_source_ranges

{}, "", []

Load balancer annotations, class, and allowed client CIDRs for lenz-external.

http.host

None; required

Web hostname used for redirects, Grafana URLs, and generated certificates.

http.extra_hosts

[]

Additional web hostnames, also included in generated and cert-manager certificates.

http.tls

true

Serve HTTPS and redirect HTTP to it; false serves HTTP only.

http.public_port

0

Public web port behind an external load balancer; 0 derives it from the Service.

http.body_limit

268435456

Maximum HTTP request body size in bytes.

http.timeout_seconds

300

Upstream inactivity timeout for API and Web UI requests.

http.routes

[]

Workflow routes; see Workflow routes and request limits.

http.service.type

LoadBalancer

Type of the http-proxy Service.

http.service.http_node_port, http.service.https_node_port

0, 0

Fixed HTTP and HTTPS node ports; see NodePort access.

http.service.annotations, http.service.load_balancer_class, http.service.load_balancer_source_ranges

{}, "", []

Load balancer annotations, class, and allowed client CIDRs for http-proxy.

Native client connections always require TLS; the module has no equivalent of the Compose LNZ_FORCE_SSL setting.

Credentials, TLS, and images

Credential, TLS, and image settings

Attribute

Default

Purpose

namespace

None; required

Namespace for all resources.

create_namespace

false

Create the namespace and delete it on tofu destroy.

master_key.password, master_key.existing_secret

None; one required

Database admin password, inline or from an existing Secret.

storage.type

s3

Object storage provider: s3 or gs.

storage.bucket

""

Object storage bucket for database content.

storage.credentials, storage.existing_secret

{}, ""

Storage credentials, as a map containing ADB_STORAGE_CREDENTIALS or from an existing Secret.

storage.extra_env

{}

Additional non-secret storage variables for database and checkpointer pods.

tls.existing_secret

""

Existing kubernetes.io/tls Secret for web and native TLS.

tls.cert, tls.key

"", ""

Inline PEM certificate chain and private key.

tls.cert_manager.enabled, tls.cert_manager.issuer_ref.name, tls.cert_manager.issuer_ref.kind

false, "", ClusterIssuer

Issue the certificate with cert-manager.

image_pull_secrets

[]

Names of existing registry Secrets for private images.

registry_credentials.enabled, registry_credentials.registry, registry_credentials.username, registry_credentials.password

false, Docker Hub, "", ""

Create the aperturedb-registry pull Secret from inline credentials.

image.pull_policy

IfNotPresent

Pull policy for ApertureData images.

See Create the namespace and secrets and Configure TLS for Secret keys and certificate precedence. The ?tag= value in source selects the module package and matching versions of all ApertureData service images.

Capacity and logging

The package sets a 4096 MiB native message limit and an 8192 MiB shared retained-byte budget (ADB_QUERY_QUEUE_BYTES_MB). Concurrent requests and responses share that budget. Use the database extra-environment settings to adjust both limits together; see Configuration File for required headroom.

Capacity and logging settings

Attribute

Default

Purpose

aperturedb.max_concurrency

512

Query workers per database process (ADB_QUERY_THREADS) and the fallback gateway concurrency; independent of accepted connections.

lenz.adb_max_concurrency

null

Independent gateway concurrency override. Omitted or null follows aperturedb.max_concurrency. An override must be a positive integer.

aperturedb.task_executor_threads

100

Database executor threads.

aperturedb.storage_max_gi

4096

Graph-store storage limit in GiB; 0 means unlimited.

aperturedb.storage_operations_limit

100

Concurrent object storage operations per database process.

aperturedb.descriptorset_flusher_interval

1800

Interval in seconds for the background descriptor index refresh.

aperturedb.session_token_max_age_sec

36000

Session token lifetime in seconds.

aperturedb.refresh_token_max_age_sec

360000

Refresh token lifetime in seconds.

aperturedb.health_check

true

Enable the database health port and use it for readiness checks.

lenz.timeout_factor

4

Scales the gateway’s network timeouts; raise it for slow clients or very large requests.

lenz.workflows_secret

""

Existing Secret mounted into each gateway at /etc/lenz/workflows.

zookeeper.tick_time

8000

ZooKeeper tick, in milliseconds, from 1 to 10000.

zookeeper.min_session_timeout

20000

Lower bound in milliseconds on the session timeout ZooKeeper negotiates with a client.

zookeeper.publish_not_ready_addresses

true

Publish pod DNS records on the ZooKeeper headless Service before pods pass their readiness checks so peers can find each other during startup. Set to false to publish only ready pods.

zookeeper.max_time_to_wait_for_epoch

30000

Milliseconds a newly elected leader waits for a voting majority to agree on an epoch before a voter returning to election can trigger another election. Set to -1 to disable this early retry or 0 to allow it immediately.

For example, set these attributes in the module block to configure the gateway independently of the database query threads:

aperturedb = { max_concurrency = 256 }
lenz       = { adb_max_concurrency = 128 }

This sets LNZ_ADB_MAX_CONCURRENCY to 128 while the database’s ADB_QUERY_THREADS remains 256. Leaving lenz.adb_max_concurrency unset or null preserves the existing fallback; both values retain their defaults when neither setting is changed.

The module sets the other database settings listed for Compose, including ADB_LOG_LEVEL=0 and ADB_KVGD_DB_SYNC=false. See Configuration File for database configuration details.

Workloads and storage

This module argument sets CPU and memory for each ZooKeeper server and allows its pods to run on nodes tainted dedicated=isvi-cpu:NoSchedule. You can set resources and tolerations independently.

zookeeper = {
  resources = {
    requests = {
      cpu    = "250m"
      memory = "512Mi"
    }
    limits = {
      cpu    = "1"
      memory = "2Gi"
    }
  }
  tolerations = [
    { key = "dedicated", operator = "Equal", value = "isvi-cpu", effect = "NoSchedule" }
  ]
}

Set db_ephemeral_volume to give each pod its own generic ephemeral volume. Database pods mount it at /aperturedb/db and checkpointer pods at /checkpointer/db.

db_ephemeral_volume = {
  size          = "100Gi"
  storage_class = "premium-rwo"
}

The default null uses the container filesystem. size is a positive Kubernetes storage quantity for each pod’s separate ReadWriteOnce claim. An omitted or empty storage_class uses global.storage_class, then the cluster default. The class must support dynamic provisioning. Deleting a pod, including scaling it to zero with inactive, deletes its claim; the underlying volume follows the class’s reclaim policy. Distributed database content remains in the object storage bucket. See generic ephemeral volumes.

Replace <workload> with a name from the table. The replica count applies to the workload, and resource requests and limits apply to its main container.

Default replicas and container resources

Workload

replicas

resources.requests

resources.limits

aperturedb

3

{ cpu = "10m", memory = "8Gi" }

{ cpu = "64", memory = "128Gi" }

checkpointer

1

{ cpu = "10m", memory = "8Gi" }

{ cpu = "64", memory = "128Gi" }

http

2

{ cpu = "10m", memory = "64Mi" }

{ cpu = "1", memory = "512Mi" }

lenz

2

{ cpu = "10m", memory = "1Gi" }

{ cpu = "64", memory = "32Gi" }

tools

0

{ cpu = "25m", memory = "64Mi" }

{ cpu = "1", memory = "256Mi" }

webui

1

{ cpu = "25m", memory = "32Mi" }

{ cpu = "250m", memory = "128Mi" }

zookeeper

3

{ cpu = "250m", memory = "1Gi" }

{ cpu = "1", memory = "2Gi" }

Use an odd replica count for ZooKeeper and at least one replica for the HTTP proxy. Web UI and Tools each accept 0 or 1. Setting Web UI to 0 removes its container and HTTP route. Telemetry can still run in the shared web pod, which uses the webui scheduling settings. Setting Tools to 0 keeps its Deployment scaled down. At 1, it runs an administration pod with the deployment’s ZooKeeper server list and prefix.

When telemetry is enabled, Grafana, Loki and Prometheus each run in the shared web pod. Promtail runs in every database pod. Set their resources independently.

Default telemetry resources per container

Module input

Requests (CPU, memory)

Limits (CPU, memory)

telemetry.grafana.resources

100m, 256Mi

1, 1Gi

telemetry.loki.resources

100m, 256Mi

2, 2Gi

telemetry.prometheus.resources

100m, 256Mi

2, 2Gi

telemetry.promtail.resources

25m, 64Mi

500m, 256Mi

For aperturedb, checkpointer, and lenz, each CPU and memory quantity has its own default. For telemetry containers, http, tools, and webui, supplying a requests or limits map replaces that map’s defaults. Web UI resource settings apply only to the web UI container in the shared web pod. Omitting zookeeper.resources uses the defaults above. Supplying it replaces the whole object, so a requests-only override leaves limits unset and an empty object clears both maps.

Workload and storage settings

Attribute

Default

Purpose

inactive

false

Scale all workloads to zero. Managed persistent claims remain; pod-owned generic ephemeral claims are deleted with their pods.

global.storage_class

""

Storage class for the monitoring claim and fallback for ZooKeeper and database scratch claims. An empty value uses the cluster default.

db_ephemeral_volume

null

One ReadWriteOnce claim per database or checkpointer pod. Set size to a positive quantity string and optionally set storage_class.

zookeeper.persistence.size, zookeeper.persistence.storage_class

1Gi, ""

Capacity and storage class of each ZooKeeper pod’s persistent ReadWriteOnce claim. An empty class falls back to global.storage_class.

zookeeper.prefix

db_root

ZooKeeper path for this deployment’s coordination data.

<workload>.pod_annotations

{}

Map of annotation names to strings on the workload’s pod template.

<workload>.affinity

null

Kubernetes provider affinity settings. Use node_affinity, pod_affinity, and pod_anti_affinity. Each nested block is a list of objects.

<workload>.tolerations

[]

List of objects with optional string fields key, operator, value, effect, and toleration_seconds. The operator defaults to Equal. For NoExecute, omit the duration to tolerate indefinitely or set it to "0" for immediate eviction.

<workload>.node_selector

{}

Map of node label names to string values. A pod can run on nodes that match every listed label.

<workload>.extra_env

{}

Additional environment variables for aperturedb, checkpointer, or lenz, such as ADB_HTTPS_* settings.

Monitoring

Monitoring settings

Attribute

Default

Purpose

telemetry.enabled

true

Deploy Prometheus, Grafana, Loki, and Promtail log collection.

telemetry.retention_days

28

Metrics and log retention in days.

telemetry.auth_password, telemetry.auth_password_secret

None; one required with telemetry

Monitoring bearer password used to collect metrics. An existing Secret takes precedence and must contain the key password. When upgrading, move telemetry.auth.password or telemetry.auth.existing_secret to the corresponding input.

telemetry.admin_password, telemetry.admin_password_secret

None; one required with telemetry

Grafana admin password. An existing Secret takes precedence and must contain the key password. When upgrading, move telemetry.grafana.admin_password to telemetry.admin_password or telemetry.grafana.existing_secret to telemetry.admin_password_secret.

Grafana, Loki, and Prometheus share the web Pod, alongside the Web UI when it is enabled. Their shared ReadWriteOnce monitoring claim requests 100Gi and uses global.storage_class or the cluster default. Promtail runs in each database Pod and forwards its logs to Loki.

Upgrade and remove

When upgrading older module inputs, replace webui.enabled with webui.replicas (0 for false, 1 for true). Remove tools.enabled and set tools.replicas to the desired number of running pods, either 0 or 1.

In 0.20.0, move any configured telemetry.persistence.storage_class to global.storage_class. Keep the storage class of the existing monitoring claim when upgrading. ZooKeeper and database scratch claims also use the global class unless you configure their storage classes separately.

Review the compatibility notes in Release Notes for your target version, then complete Changing a deployment. The following commands refresh dependencies and apply the configuration:

tofu init -upgrade
tofu plan -out=aperturedb.tfplan
tofu apply aperturedb.tfplan

init -upgrade refreshes provider selections within the configured constraints. Review changes to the provider lock file before applying them.

To remove the resources managed by this configuration, run tofu destroy and review its proposed deletions. It removes the active monitoring PVC, whose underlying data follows the storage reclaim policy. ZooKeeper PVCs created by StatefulSet claim templates remain when their namespace remains. The module does not delete the object storage bucket. In this example the namespace and separately created secrets also remain; if you instead let the module manage the namespace, destroying it deletes the namespace and its remaining resources, including retained claims.