Deploy with OpenTofu
The aperturedata/aperturedb-terraform package is an OCI module for OpenTofu.
It creates native Kubernetes resources for distributed ApertureDB in an existing
cluster. The default topology has three database pods, three ZooKeeper pods,
two database gateways, two HTTP proxies, one checkpointer, a Web UI, and telemetry.
Prerequisites
OpenTofu 1.10 or later for OCI module sources. Use
tofufor the OCI source shown below, even though the package name containsterraform.Kubernetes 1.23 or later and a kubeconfig for the intended cluster.
An object storage bucket and credentials; see Deploying ApertureDB.
Persistent volume provisioning for ZooKeeper and a 100Gi monitoring volume when telemetry is enabled.
Sufficient cluster capacity: each database pod and the checkpointer requests 8Gi of memory by default, and each Lenz pod requests 1Gi, in addition to the other components.
LoadBalancer support for the example below.
Configure the Kubernetes provider. The HTTP service is included; no ingress controller or Helm provider is needed. Use a separate namespace per deployment because application resource names are fixed within a namespace.
Configure the module
In a new directory, create main.tf. Replace <package-version> with the
published package tag you want, and replace the bucket name and hostname.
This example references secrets you create separately in the namespace:
terraform {
required_version = ">= 1.10.0"
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.33"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
module "aperturedb" {
source = "oci://registry-1.docker.io/aperturedata/aperturedb-terraform?tag=<package-version>"
namespace = "aperturedb"
create_namespace = false
master_key = { existing_secret = "adb-master-key" }
storage = {
type = "s3"
bucket = "<your-bucket-name>"
existing_secret = "database-storage-credentials"
}
service = { type = "LoadBalancer" }
http = {
host = "aperturedb.example.com"
tls = true
}
telemetry = {
enabled = true
auth_password_secret = "adb-monitoring"
admin_password_secret = "adb-grafana"
}
}
output "database_load_balancer" {
value = module.aperturedb.lenz_external_load_balancer
}
output "http_load_balancer" {
value = module.aperturedb.http_load_balancer
}
Create the namespace and secrets
Create the namespace if it does not already exist:
kubectl create namespace aperturedb
Create four local files: master-password.txt, monitoring-password.txt,
grafana-password.txt, and storage-credentials.json. Each password file
must contain a different strong password without a trailing newline. The JSON
file contains the S3 credentials described in Deploying ApertureDB. Then run:
chmod 600 master-password.txt monitoring-password.txt grafana-password.txt storage-credentials.json
kubectl -n aperturedb create secret generic adb-master-key \
--from-file=password=master-password.txt
kubectl -n aperturedb create secret generic adb-monitoring \
--from-file=password=monitoring-password.txt
kubectl -n aperturedb create secret generic adb-grafana \
--from-file=password=grafana-password.txt
kubectl -n aperturedb create secret generic database-storage-credentials \
--from-file=ADB_STORAGE_CREDENTIALS=storage-credentials.json
For Google Cloud Storage, set storage.type = "gs" and put the service
account JSON in storage-credentials.json. The OpenTofu storage secret key
is ADB_STORAGE_CREDENTIALS; the Helm chart uses a different key.
Alternatively, the module can create its namespace with
create_namespace = true and create secrets from inline values:
master_key.password, storage.credentials (a map containing
ADB_STORAGE_CREDENTIALS), telemetry.auth_password, and
telemetry.admin_password. Inline credentials are stored in OpenTofu
state. Protect state, plan files, and local credential files, and keep them
out of source control.
For private workload images, create a registry secret in the namespace and
set image_pull_secrets = ["my-registry-secret"] in the module.
Configure TLS
The module uses an existing TLS secret, inline certificate and key, cert-manager, or a generated self-signed certificate, in that order. To use a certificate covering your web and TCP hostnames:
kubectl -n aperturedb create secret tls adb-tls \
--cert=cert.pem --key=key.pem
Add tls = { existing_secret = "adb-tls" } to the module. Other options
are tls.cert and tls.key, or tls.cert_manager with a configured
issuer. Without these settings, the self-signed fallback is for evaluation;
clients must explicitly trust it or use an evaluation-only certificate
verification override. The generated private key is stored in OpenTofu state.
Plan, apply, and connect
Initialize the module and providers, review the plan, then apply it:
tofu init
tofu validate
tofu plan -out=aperturedb.tfplan
tofu apply aperturedb.tfplan
kubectl -n aperturedb get pods,svc,pvc
tofu output database_load_balancer
tofu output http_load_balancer
Point the DNS record for http.host at the address from
http_load_balancer. Native clients use lenz_external_load_balancer.
The TLS certificate must cover all hostnames used by web and native clients.
Web UI:
https://aperturedb.example.com/.REST API:
https://aperturedb.example.com/api/.Native clients: the
lenz-externalservice’s external address on port55555, with TLS enabled. In-cluster clients can uselenzon that port.Grafana:
https://aperturedb.example.com/grafana/with telemetry enabled. Log in asadminusing the Grafana secret’s password.
Use admin with the master key in the runnable client check at
Connect and verify the deployment. See Deployment settings and process boundaries for
server and gateway settings. Check pod events, PVC status, and container logs
if startup does not complete. A pending external address requires checking
LoadBalancer support; HTTP routing also depends on DNS and http.host.
Outputs by service type
Module outputs become visible to tofu output only after you expose them
in your root configuration. Use the output section that matches each Service;
native service.type and http.service.type can differ.
LoadBalancer outputs
Use the existing database_load_balancer and http_load_balancer outputs
in the example above. Their values are ingress lists containing an ip or
hostname. They can be empty while a load balancer is provisioning, and
stay empty for NodePort or ClusterIP Services.
NodePort outputs
Add these root outputs to read the allocated ports, including ports assigned automatically by Kubernetes:
output "database_node_port" {
value = module.aperturedb.lenz_ports.tcp.node_port
}
output "http_node_ports" {
value = {
http = module.aperturedb.http_ports.http.node_port
https = try(module.aperturedb.http_ports.https.node_port, null)
}
}
Connect to a reachable cluster node address using these ports. The module
does not select or output a node address. http_node_ports.https is
null when http.tls is disabled.
ClusterIP outputs
Use Service DNS and the Service port, rather than node_port:
output "database_cluster_endpoint" {
value = {
host = "${module.aperturedb.lenz_service}.${module.aperturedb.namespace}.svc.cluster.local"
port = module.aperturedb.service_port
}
}
output "http_cluster_endpoint" {
value = {
host = "http-proxy.${module.aperturedb.namespace}.svc.cluster.local"
ports = {
http = module.aperturedb.http_ports.http.port
https = try(module.aperturedb.http_ports.https.port, null)
}
}
}
These addresses are reachable within the cluster. Substitute your cluster
domain if it differs from cluster.local. Configure http.host and TLS
for the hostname used by HTTP clients. The native certificate must also cover
the Service hostname when clients connect through that name.
lenz_external_service provides the other native Service name for every
Service type; lenz_service always names the internal ClusterIP Service.
NodePort access
For direct access through cluster nodes, configure distinct, available HTTP and HTTPS NodePorts and allow them through your firewall:
http = {
host = "aperturedb.example.com"
service = {
type = "NodePort"
http_node_port = 30080
https_node_port = 30443
}
}
Point http.host at a reachable node. With these example ports, the Web UI is
at https://aperturedb.example.com:30443/; append api/ or grafana/ for
the API or dashboards. HTTP on port 30080 redirects to that HTTPS address.
Choose ports from your cluster’s configured NodePort range, normally 30000–32767.
To expose native clients through NodePort as well, set service.type to
NodePort and inspect lenz-external for its allocated port.
Behind an external load balancer serving ports 80/443, use NodePort and set
http.public_port to 443. Leave the HTTP and HTTPS NodePorts at zero for automatic
allocation. Redirects and Grafana URLs use public port 443. Both database and
HTTP Services are always created; use ClusterIP for private access.
Workflow routes and request limits
Use http.routes to expose workflow HTTP services under the deployment’s web
hostname. Each route specifies a path prefix, an in-cluster service name, and its
HTTP port. For example:
http = {
host = "aperturedb.example.com"
routes = [{
path = "/notebooks"
host = "workflow-notebooks-ingresses"
port = 8888
}]
}
Paths must be unique and omit the trailing slash; /api and /grafana are
reserved. Requests retain their path prefix and query string. WebSockets and
streamed responses are supported, and route updates take effect automatically.
http.body_limit limits each request body to 256 MiB by default. Oversized
requests receive 413. http.timeout_seconds controls upstream inactivity and
defaults to 300 seconds; workflow routes and Grafana allow one hour between
response reads. Active requests have up to five minutes to finish during
shutdown. Long-lived sessions should reconnect after an update.
Valid replacements in the configured TLS Secret are loaded automatically. Existing connections can finish with the previous certificate; invalid replacements leave the last valid certificate active.
Settings reference
Set only the attributes you want to change in the module block. Omitted
attributes of an object keep their defaults, so
zookeeper = { min_session_timeout = 30000 } keeps the other ZooKeeper
settings.
Network
Attribute |
Default |
Purpose |
|---|---|---|
|
|
Type of the |
|
|
Published native database port. |
|
|
Fixed node port when |
|
|
Load balancer annotations, class, and allowed client CIDRs for
|
|
None; required |
Web hostname used for redirects, Grafana URLs, and generated certificates. |
|
|
Additional web hostnames, also included in generated and cert-manager certificates. |
|
|
Serve HTTPS and redirect HTTP to it; |
|
|
Public web port behind an external load balancer; |
|
|
Maximum HTTP request body size in bytes. |
|
|
Upstream inactivity timeout for API and Web UI requests. |
|
|
Workflow routes; see Workflow routes and request limits. |
|
|
Type of the |
|
|
Fixed HTTP and HTTPS node ports; see NodePort access. |
|
|
Load balancer annotations, class, and allowed client CIDRs for
|
Native client connections always require TLS; the module has no equivalent of
the Compose LNZ_FORCE_SSL setting.
Credentials, TLS, and images
Attribute |
Default |
Purpose |
|---|---|---|
|
None; required |
Namespace for all resources. |
|
|
Create the namespace and delete it on |
|
None; one required |
Database |
|
|
Object storage provider: |
|
|
Object storage bucket for database content. |
|
|
Storage credentials, as a map containing |
|
|
Additional non-secret storage variables for database and checkpointer pods. |
|
|
Existing |
|
|
Inline PEM certificate chain and private key. |
|
|
Issue the certificate with cert-manager. |
|
|
Names of existing registry Secrets for private images. |
|
|
Create the |
|
|
Pull policy for ApertureData images. |
See Create the namespace and secrets and Configure TLS for Secret keys
and certificate precedence. The ?tag= value in source selects the
module package and matching versions of all ApertureData service images.
Capacity and logging
The package sets a 4096 MiB native message limit and an 8192 MiB shared retained-byte
budget (ADB_QUERY_QUEUE_BYTES_MB). Concurrent requests and responses share that
budget. Use the database extra-environment settings to adjust both limits together;
see Configuration File for required headroom.
Attribute |
Default |
Purpose |
|---|---|---|
|
|
Query workers per database process ( |
|
|
Independent gateway concurrency override. Omitted or |
|
|
Database executor threads. |
|
|
Graph-store storage limit in GiB; |
|
|
Concurrent object storage operations per database process. |
|
|
Interval in seconds for the background descriptor index refresh. |
|
|
Session token lifetime in seconds. |
|
|
Refresh token lifetime in seconds. |
|
|
Enable the database health port and use it for readiness checks. |
|
|
Scales the gateway’s network timeouts; raise it for slow clients or very large requests. |
|
|
Existing Secret mounted into each gateway at |
|
|
ZooKeeper tick, in milliseconds, from 1 to 10000. |
|
|
Lower bound in milliseconds on the session timeout ZooKeeper negotiates with a client. |
|
|
Publish pod DNS records on the ZooKeeper headless Service before pods
pass their readiness checks so peers can find each other during startup.
Set to |
|
|
Milliseconds a newly elected leader waits for a voting majority to agree
on an epoch before a voter returning to election can trigger another
election. Set to |
For example, set these attributes in the module block to configure the gateway independently of the database query threads:
aperturedb = { max_concurrency = 256 }
lenz = { adb_max_concurrency = 128 }
This sets LNZ_ADB_MAX_CONCURRENCY to 128 while the database’s
ADB_QUERY_THREADS remains 256. Leaving
lenz.adb_max_concurrency unset or null preserves the existing fallback;
both values retain their defaults when neither setting is changed.
The module sets the other database settings listed for Compose, including
ADB_LOG_LEVEL=0 and ADB_KVGD_DB_SYNC=false. See Configuration File
for database configuration details.
Workloads and storage
This module argument sets CPU and memory for each ZooKeeper server and allows
its pods to run on nodes tainted dedicated=isvi-cpu:NoSchedule. You can set
resources and tolerations independently.
zookeeper = {
resources = {
requests = {
cpu = "250m"
memory = "512Mi"
}
limits = {
cpu = "1"
memory = "2Gi"
}
}
tolerations = [
{ key = "dedicated", operator = "Equal", value = "isvi-cpu", effect = "NoSchedule" }
]
}
Set db_ephemeral_volume to give each pod its own generic ephemeral volume.
Database pods mount it at /aperturedb/db and checkpointer pods at /checkpointer/db.
db_ephemeral_volume = {
size = "100Gi"
storage_class = "premium-rwo"
}
The default null uses the container filesystem. size is a positive
Kubernetes storage quantity for each pod’s separate ReadWriteOnce claim.
An omitted or empty storage_class uses global.storage_class, then the
cluster default. The class must support dynamic provisioning.
Deleting a pod, including scaling it to zero with inactive, deletes its
claim; the underlying volume follows the class’s reclaim policy. Distributed
database content remains in the object storage bucket. See
generic ephemeral volumes.
Replace <workload> with a name from the table. The replica count applies to
the workload, and resource requests and limits apply to its main container.
Workload |
|
|
|
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Use an odd replica count for ZooKeeper and at least one replica for the HTTP
proxy. Web UI and Tools each accept 0 or 1. Setting Web UI to 0
removes its container and HTTP route. Telemetry can still run in the shared
web pod, which uses the webui scheduling settings. Setting Tools to 0
keeps its Deployment scaled down. At 1, it runs an administration pod with
the deployment’s ZooKeeper server list and prefix.
When telemetry is enabled, Grafana, Loki and Prometheus each run in the shared web pod. Promtail runs in every database pod. Set their resources independently.
Module input |
Requests (CPU, memory) |
Limits (CPU, memory) |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
For aperturedb, checkpointer, and lenz, each CPU and memory quantity
has its own default. For telemetry containers, http, tools, and webui, supplying a
requests or limits map replaces that map’s defaults. Web UI resource
settings apply only to the web UI container in the shared web pod.
Omitting zookeeper.resources uses the defaults above. Supplying it replaces
the whole object, so a requests-only override leaves limits unset and an
empty object clears both maps.
Attribute |
Default |
Purpose |
|---|---|---|
|
|
Scale all workloads to zero. Managed persistent claims remain; pod-owned generic ephemeral claims are deleted with their pods. |
|
|
Storage class for the monitoring claim and fallback for ZooKeeper and database scratch claims. An empty value uses the cluster default. |
|
|
One ReadWriteOnce claim per database or checkpointer pod. Set |
|
|
Capacity and storage class of each ZooKeeper pod’s persistent
ReadWriteOnce claim. An empty class falls back to |
|
|
ZooKeeper path for this deployment’s coordination data. |
|
|
Map of annotation names to strings on the workload’s pod template. |
|
|
Kubernetes provider affinity settings. Use
|
|
|
List of objects with optional string fields |
|
|
Map of node label names to string values. A pod can run on nodes that match every listed label. |
|
|
Additional environment variables for |
Monitoring
Attribute |
Default |
Purpose |
|---|---|---|
|
|
Deploy Prometheus, Grafana, Loki, and Promtail log collection. |
|
|
Metrics and log retention in days. |
|
None; one required with telemetry |
Monitoring bearer password used to collect metrics. An existing Secret
takes precedence and must contain the key |
|
None; one required with telemetry |
Grafana |
Grafana, Loki, and Prometheus share the web Pod, alongside the Web UI when it
is enabled. Their shared ReadWriteOnce monitoring claim requests 100Gi and
uses global.storage_class or the cluster default. Promtail runs in each
database Pod and forwards its logs to Loki.
Upgrade and remove
When upgrading older module inputs, replace webui.enabled with
webui.replicas (0 for false, 1 for true). Remove tools.enabled
and set tools.replicas to the desired number of running pods, either 0 or 1.
In 0.20.0, move any configured telemetry.persistence.storage_class to
global.storage_class. Keep the storage class of the existing monitoring
claim when upgrading. ZooKeeper and database scratch claims also use the
global class unless you configure their storage classes separately.
Review the compatibility notes in Release Notes for your target version, then complete Changing a deployment. The following commands refresh dependencies and apply the configuration:
tofu init -upgrade
tofu plan -out=aperturedb.tfplan
tofu apply aperturedb.tfplan
init -upgrade refreshes provider selections within the configured
constraints. Review changes to the provider lock file before applying them.
To remove the resources managed by this configuration, run tofu destroy
and review its proposed deletions. It removes the active monitoring PVC,
whose underlying data follows the storage reclaim policy. ZooKeeper PVCs
created by StatefulSet claim templates remain when their namespace remains.
The module does not delete the object storage bucket. In this example the
namespace and separately created secrets also remain; if you instead let the
module manage the namespace, destroying it deletes the namespace and its
remaining resources, including retained claims.