GetMetrics

Global allocation hooks report process totals under kUnknownC and kUnknownCpp. The server enforces explicit request and transfer budgets.

Retrieve the current metric values.

Any authenticated user can call this command; no additional CRUD permission or input/output blobs are required.

Parameters

Parameter

Type / required

Default and behavior

format

string / optional

Defaults to json. Allowed: json or prometheus.

metrics

array of objects / optional

Omitted or empty: all metric families. Otherwise select the named families and optional complete label sets.

Each metrics element is an object with these fields:

Parameter

Type / required

Default and behavior

family

string / required

Exact metric family name; no default. Unknown names select no family.

labels

array of string maps / optional

Omitted or empty: all series in the family. Each object must exactly match the complete label set, including values, of a series. Multiple objects select alternative complete sets.

A label filter is exact, not a subset match. For a series with both method and status labels, filtering only method does not select that series. An empty object in labels selects a series without labels.

Response and recovery

Success returns status: 0, version (the server version), and values. For json, values contains the metric families and series. Floating-point values are strings and may be "inf", "-inf", or "nan"; integer counts remain JSON integers. For prometheus, values is a string in Prometheus text exposition format. Unknown family names or unmatched label sets give a successful result without those metrics.

Invalid formats, missing family names, or non-string label values fail validation; correct the selector. See Responses for authentication and validation error envelopes.

Details

A metric represents a single specific measurable phenomenon such as the size of a database or the number of queries that have run. Metrics come in a variety of types:

  • A counter is a single cumulative value that can only increase over time. As their name implies, counters can be used to ‘count’ discrete events as they occur, for example the number of queries received. The value of a counter reflects the amount accumulated since the ApertureDB process launched.

  • A gauge is a numerical value that can go up or down. The value of a gauge generally reflects some current state, for example the length of a queue.

  • A histogram provides a breakdown of discrete samples into buckets. It also includes a ‘count’ (counter) and ‘sum’ (gauge) representing the total number of samples taken and sum of all sample values respectively. Histograms are used to describe the distribution of events according to their size, such as query processing times or response sizes.

  • A summary is similar to a histogram but instead of fixed buckets it computes running quantiles over a sliding time window. Summaries are more expensive to compute than other metric types and are used less frequently.

ApertureDB metrics are organized into families. A family describes a class of related metrics that can be compared directly with each other. Metrics within a family are differentiated using labels and must share the same type.

Query timing

graph_read_query_seconds and graph_write_query_seconds break graph work into phases identified by the operation label. Their exec phase includes authorization. Sum the phases within these families when calculating graph work.

graph_authorization_seconds measures authorization separately, with operation="read" for read queries, operation="write" for write queries, and operation="snapshot" for identity lookups used by error reporting. The read and write measurements are already included in graph execution time; do not add them to the phase totals. This family replaces the previous authorization series in the graph phase families and the earlier query_seconds{operation="process_query",suboperation="auth"} series.

Client sessions

client_session_total counts successful committed authentication and token refresh requests, with status="started" and status="refreshed". Both series exist at zero before the first request. Failed or cancelled transactions do not increment them. These are cumulative events, not active-session counts; use rate to graph sessions per second.

Descriptor index cache

The descriptor_index_cache_ families report the current graph’s shared in-memory descriptor index cache. They are sampled when metrics are requested, including changes caused by searches and background builds without graph writes. Values cover all descriptor indexes and engines on this server. Aggregate families have no labels. The two reason families below have one fixed reason label; no descriptor class names, index identities or revisions become cache metric labels.

Family suffix

Type

Meaning

budget_bytes

Gauge

Configured cache budget, in bytes.

charged_bytes

Gauge

Memory reserved for active builds, resident snapshots, and evicted snapshots still held by readers.

entries

Gauge

Resident snapshots available for lookup.

queued

Gauge

Builds waiting for the background worker.

pending

Gauge

Queued builds plus the active build.

hits_total

Counter

Lookups that acquired a cached snapshot.

misses_total

Counter

Lookups without a snapshot, including cache lock contention.

builds_total

Counter

Completed snapshots published to the cache.

failures_total

Counter

Build or admission exceptions, including queue/observation allocation failures.

rejected_total

Counter

Requests rejected or deferred by admission checks, lock contention, queue limits, or insufficient available cache budget.

misses_by_reason_total

Counter

Misses partitioned by reason: disabled, contention, stopping, pending, absent.

rejected_by_reason_total

Counter

Rejections partitioned by reason: invalid, disabled, oversize, contention, stopping, queue_full, stale_revision, pinned_budget, deferred.

automatic_bypasses_total

Counter

Automatic jobs successfully queued after replacing an entry in the full admission observation table; excludes resident/pending coalescing.

For example, descriptor_index_cache_charged_bytes is the full family name for charged memory. Its value includes a build’s reservation before the snapshot is published, and can remain nonzero after eviction while searches hold the snapshot. It excludes LMDB’s mapped pages, search scratch memory, allocator overhead and fixed control memory, including the lazy admission observation table (at most 32 namespaces, about 18 KiB on supported 64-bit platforms). It is therefore not a measure of total descriptor-related process RAM.

Miss reasons are selected in priority order: disabled caching, lock contention, shutdown, then an exact revision pending or absent. A contention result does not inspect the entry or pending maps. Admission first checks invalid arguments, disabled caching, oversized reservation, contention and shutdown; an exact resident/pending request then coalesces before queue capacity is checked. Automatic admission additionally checks stale revisions and reuse evidence. The worker can reject a stale revision or fail to reserve its budget after LRU eviction because existing leases still hold memory (pinned_budget).

deferred means automatic admission needs more reuse evidence for that revision. It is expected policy behavior, not a build failure. Admission normally requires 32 eligible absent observations or a later observation at least one second into the window. Idle time does not queue work. When the bounded table is full, a new namespace replaces the least recently observed entry and qualifies immediately; automatic_bypasses_total counts successful queueing through that path, not lookups, published builds or a cache being disabled.

Each miss or rejection increments one reason and its aggregate counter. Their sums agree after queries and the worker quiesce; live metric requests can observe updates between separately sampled atomic counters. An unavailable revision returned by a loader, a build discarded during shutdown and coalesced requests do not increment rejection counters. Exceptions are counted as failures.

Counters describe the current cache instance and reset on cache reconfiguration or server restart. Gauges and counters report zero when no cache is attached. A miss immediately uses the transactional index; it does not wait for a build.

Examples

The following query returns the connection_queue_seconds histogram in JSON. The example responses use zero sample counts and sums. A running server reports its current measurements; family metadata and bucket boundaries have the structure shown here:

[{
    "GetMetrics": {
        "format": "json",
        "metrics": [{
            "family": "connection_queue_seconds"
        }]
    }
}]

Successful response:

[{
    "GetMetrics": {
        "status": 0,
        "values": {
            "families": [{
                "help": "Time spent in the connection queue, in seconds",
                "metrics": [{
                    "buckets": [{
                        "count": 0,
                        "max": "0.0001"
                    }, {
                        "count": 0,
                        "max": "0.001"
                    }, {
                        "count": 0,
                        "max": "0.01"
                    }, {
                        "count": 0,
                        "max": "0.1"
                    }, {
                        "count": 0,
                        "max": "1"
                    }, {
                        "count": 0,
                        "max": "10"
                    }, {
                        "count": 0,
                        "max": "inf"
                    }],
                    "count": 0,
                    "sum": "0"
                }],
                "name": "connection_queue_seconds",
                "type": "Histogram"
            }]
        },
        "version": "0.19.15"
    }
}]

The same metrics are available in Prometheus text format:

[{
    "GetMetrics": {
        "format": "prometheus",
        "metrics": [{
            "family": "connection_queue_seconds"
        }]
    }
}]
[{
    "GetMetrics": {
        "status": 0,
        "values": "# HELP connection_queue_seconds Time spent in the connection queue, in seconds\n# TYPE connection_queue_seconds histogram\nconnection_queue_seconds_count 0\nconnection_queue_seconds_sum 0\nconnection_queue_seconds_bucket{le=\"0.0001\"} 0\nconnection_queue_seconds_bucket{le=\"0.001\"} 0\nconnection_queue_seconds_bucket{le=\"0.01\"} 0\nconnection_queue_seconds_bucket{le=\"0.1\"} 0\nconnection_queue_seconds_bucket{le=\"1\"} 0\nconnection_queue_seconds_bucket{le=\"10\"} 0\nconnection_queue_seconds_bucket{le=\"+Inf\"} 0\n",
        "version": "0.19.15"
    }
}]

The following are the metric families exposed by ApertureDB.

  • commands_per_query : Number of commands per ApertureDB query

  • user_command_total : Number of commands executed by a user

  • client_connection_seconds : The time taken to establish and/or destroy a client connection, in seconds

  • connection_failed_attempts : The number of connection attempts that failed to negotiate protocol

  • connection_queue_seconds : Time spent in the connection queue, in seconds

  • cpu_cycles_total : Elapsed CPU cycles

  • page_faults_total : The number of page faults encountered by ApertureDB

  • memory_bytes : Bytes of memory in use

  • storage_graph_bytes : Bytes of storage used for graph

  • storage_images_bytes : Bytes of storage used for images

  • storage_videos_bytes : Bytes of storage used for videos

  • storage_blobs_bytes : Bytes of storage used for blobs

  • storage_total_bytes : Bytes of storage used

  • num_cores : The number of concurrent threads supported on this machine

  • query_seconds : Time taken per query, in seconds

  • api_bytes_transferred : The amount of data sent and received by the ApertureDB API, in bytes

  • log_messages_total : The number of log messages generated

  • audit_events_total : The number of audit events generated

  • region_intersections_by_resulting_pieces : The number of region intersection operations performed broken down by number of convex pieces in the intersection

  • region_intersection_seconds_by_resulting_pieces : The time spent computing region intersections broken down by number of convex pieces in the intersection

  • region_intersection_pieces_total : The total number of convex pieces generated by region intersection operations that fell within the specified range

  • iou_values : The distribution of intersection over union (IoU) values computed between pairs of regions

  • region_intersection_seconds : A histogram of the time spent per region intersection operation

  • client_session_total : The number of successful client session creations and refreshes

  • graph_process_command_seconds : Time to process graph commands in seconds

  • graph_query_commands_total : Count of graph commands processed within a query

  • lmdb_page_size_bytes : The size in bytes of a single LMDB page

  • lmdb_depth : The current maximum depth (height) of the LMDB b-tree

  • lmdb_pages : The current number of pages in the LMDB B-tree

  • lmdb_entries : The current number of data items in the LMDB B-tree

  • memory_heap_allocated_bytes_total : The total number of bytes of heap memory allocated

  • memory_heap_freed_bytes_total : The total number of bytes of heap memory deallocated

  • query_handlers : The number of individual query handlers.

  • page_size : The size of a memory page, in bytes.

  • query_executor_workers : Shared request/query CPU workers.

  • query_executor_task_capacity : Bound on admitted queued/running/retained task handles.

  • query_executor_budget_bytes : Shared retained task/input/scratch/result byte allowance.

  • query_executor_pending : Admitted task reservations, including retained completed results.

  • query_executor_retained_bytes : Currently charged task/input/scratch/result bytes.

  • query_executor_submitted_total : Accepted callbacks, including inline fallbacks.

  • query_executor_completed_total : Finished callbacks, including failures and inline work.

  • query_executor_failed_total : Callbacks that threw, including cancellation exceptions.

  • query_executor_rejected_total : Task reservation attempts rejected before materialization.

  • query_executor_inline_total : Callbacks run inline after admission fallback.

  • query_executor_cancel_requested_total : Accepted unconsumed tasks receiving a group stop request.

  • query_executor_abandoned_total : Queued callbacks discarded during pool shutdown.

  • graph_owner_links : Committed ownership links by fixed ownership kind.

  • graph_owner_traversals_total : Ownership traversal attempts by kind and direction, including empty and interrupted traversals.

  • graph_owner_links_visited_total : Ownership links delivered to traversal callbacks by kind and direction.

  • graph_filter_work_total : Executed filter work by category and phase: constant plans, selected index plans, residual row evaluations and returned proven rows. Index estimates and metadata reads are not row evaluations.

  • graph_filter_seconds : The time taken processing graph filters, in seconds.

  • graph_filter_entities_total : The number of entities processed by graph filters.

  • graph_read_query_seconds : Read graph operations in seconds

  • graph_write_query_seconds : Write graph operations in seconds

  • graph_authorization_seconds : Authorization time within graph execution or snapshot identity lookup

  • graph_txns_behind : Transactions this node is behind the cluster, as of its last sync

Typed filtering, ownership and query tasks

graph_filter_seconds and graph_filter_entities_total retain their entity_type, filter_type and histogram stage labels. All node and customer connection series also carry class, node_type, src_type, src_class, dst_type and dst_class. Nonapplicable values are empty. Names come from the executing snapshot. A rename creates new display-label series; old reader snapshots continue to use the original labels.

graph_filter_work_total uses the same category labels with entity_type and one fixed phase label, without filter_type or stage:

  • constant_plans counts completed class-count/constant decisions, including eligible count-only commands.

  • index_plans counts executions of selected index accesses. Estimating a discarded access does not increment this phase.

  • residual_rows counts actual predicate evaluations on rows, including a reached evaluation that throws.

  • proven_rows counts materialized matches accepted without row evaluation.

Zero residual work establishes that no row predicate ran. Metadata, identity, index-estimate and projection reads can still occur. These counters aggregate work within a category and do not label individual expressions or requests.

Ownership has separate graph_owner_links gauges and graph_owner_traversals_total / graph_owner_links_visited_total counters. Their fixed kind values are bounding_box_image, polygon_image, frame_video and clip_video. Traversals also carry direction=owners or direction=children. Link gauges publish after successful commit, including replay; aborted writers and older publications cannot replace committed counts. Traversal counters include empty calls and links delivered before early stop or an exception. Physical ownership storage uses the existing LMDB metrics for owners-by-child and children-by-owner. Owners remain absent from GetSchema.

query_executor_ families expose the shared CPU pool without request labels. workers, task_capacity and budget_bytes are limits; pending and retained_bytes include admitted work and completed results retained until owner consumption. Counters report submitted_total, completed_total, failed_total, rejected_total, inline_total, cancel_requested_total and abandoned_total. A completion includes a failure. Stop requests count unconsumed accepted handles, rather than asserting that a callback had already stopped. Rejected reservations can keep serial execution; inline callbacks count as submitted and completed work. Queued callbacks discarded by shutdown count as abandoned.

The Descriptor Performance dashboard shows these native families and snapshot cache admission.