GetMetrics
Global allocation hooks report process totals under kUnknownC and
kUnknownCpp. The server enforces explicit request and transfer budgets.
Retrieve the current metric values.
Any authenticated user can call this command; no additional CRUD permission or input/output blobs are required.
Parameters
Parameter |
Type / required |
Default and behavior |
|---|---|---|
|
string / optional |
Defaults to json. Allowed: json or prometheus. |
|
array of objects / optional |
Omitted or empty: all metric families. Otherwise select the named families and optional complete label sets. |
Each metrics element is an object with these fields:
Parameter |
Type / required |
Default and behavior |
|---|---|---|
|
string / required |
Exact metric family name; no default. Unknown names select no family. |
|
array of string maps / optional |
Omitted or empty: all series in the family. Each object must exactly match the complete label set, including values, of a series. Multiple objects select alternative complete sets. |
A label filter is exact, not a subset match. For a series with both method
and status labels, filtering only method does not select that series.
An empty object in labels selects a series without labels.
Response and recovery
Success returns status: 0, version (the server version), and values.
For json, values contains the metric families and series. Floating-point
values are strings and may be "inf", "-inf", or "nan"; integer
counts remain JSON integers. For prometheus, values is a string in
Prometheus text exposition format. Unknown family names or unmatched label
sets give a successful result without those metrics.
Invalid formats, missing family names, or non-string label values fail validation; correct the selector. See Responses for authentication and validation error envelopes.
Details
A metric represents a single specific measurable phenomenon such as the size of a database or the number of queries that have run. Metrics come in a variety of types:
A counter is a single cumulative value that can only increase over time. As their name implies, counters can be used to ‘count’ discrete events as they occur, for example the number of queries received. The value of a counter reflects the amount accumulated since the ApertureDB process launched.
A gauge is a numerical value that can go up or down. The value of a gauge generally reflects some current state, for example the length of a queue.
A histogram provides a breakdown of discrete samples into buckets. It also includes a ‘count’ (counter) and ‘sum’ (gauge) representing the total number of samples taken and sum of all sample values respectively. Histograms are used to describe the distribution of events according to their size, such as query processing times or response sizes.
A summary is similar to a histogram but instead of fixed buckets it computes running quantiles over a sliding time window. Summaries are more expensive to compute than other metric types and are used less frequently.
ApertureDB metrics are organized into families. A family describes a class of related metrics that can be compared directly with each other. Metrics within a family are differentiated using labels and must share the same type.
Query timing
graph_read_query_seconds and graph_write_query_seconds break graph work
into phases identified by the operation label. Their exec phase includes
authorization. Sum the phases within these families when calculating graph work.
graph_authorization_seconds measures authorization separately, with
operation="read" for read queries, operation="write" for write queries,
and operation="snapshot" for identity lookups used by error
reporting. The read and write measurements are already included in graph
execution time; do not add them to the phase totals. This family replaces the
previous authorization series in the graph phase families and the earlier
query_seconds{operation="process_query",suboperation="auth"} series.
Client sessions
client_session_total counts successful committed authentication and token
refresh requests, with status="started" and status="refreshed".
Both series exist at zero before the first request. Failed or cancelled transactions
do not increment them. These are cumulative events, not active-session counts;
use rate to graph sessions per second.
Descriptor index cache
The descriptor_index_cache_ families report the current graph’s shared
in-memory descriptor index cache. They are sampled when metrics are requested,
including changes caused by searches and background builds without graph writes.
Values cover all descriptor indexes and engines on this server. Aggregate families
have no labels. The two reason families below have one fixed reason label;
no descriptor class names, index identities or revisions become cache metric labels.
Family suffix |
Type |
Meaning |
|---|---|---|
|
Gauge |
Configured cache budget, in bytes. |
|
Gauge |
Memory reserved for active builds, resident snapshots, and evicted snapshots still held by readers. |
|
Gauge |
Resident snapshots available for lookup. |
|
Gauge |
Builds waiting for the background worker. |
|
Gauge |
Queued builds plus the active build. |
|
Counter |
Lookups that acquired a cached snapshot. |
|
Counter |
Lookups without a snapshot, including cache lock contention. |
|
Counter |
Completed snapshots published to the cache. |
|
Counter |
Build or admission exceptions, including queue/observation allocation failures. |
|
Counter |
Requests rejected or deferred by admission checks, lock contention, queue limits, or insufficient available cache budget. |
|
Counter |
Misses partitioned by |
|
Counter |
Rejections partitioned by |
|
Counter |
Automatic jobs successfully queued after replacing an entry in the full admission observation table; excludes resident/pending coalescing. |
For example, descriptor_index_cache_charged_bytes is the full family name
for charged memory. Its value includes a build’s reservation before the snapshot
is published, and can remain nonzero after eviction while searches hold the
snapshot. It excludes LMDB’s mapped pages, search scratch memory, allocator
overhead and fixed control memory, including the lazy admission observation
table (at most 32 namespaces, about 18 KiB on supported 64-bit platforms).
It is therefore not a measure of total descriptor-related process RAM.
Miss reasons are selected in priority order: disabled caching, lock contention,
shutdown, then an exact revision pending or absent. A contention result does not
inspect the entry or pending maps. Admission first checks invalid arguments,
disabled caching, oversized reservation, contention and shutdown; an exact
resident/pending request then coalesces before queue capacity is checked.
Automatic admission additionally checks stale revisions and reuse evidence.
The worker can reject a stale revision or fail to reserve its budget after LRU
eviction because existing leases still hold memory (pinned_budget).
deferred means automatic admission needs more reuse evidence for that
revision. It is expected policy behavior, not a build failure. Admission normally
requires 32 eligible absent observations or a later observation at least one
second into the window. Idle time does not queue work. When the bounded table is
full, a new namespace replaces the least recently observed entry and qualifies
immediately; automatic_bypasses_total counts successful queueing through that
path, not lookups, published builds or a cache being disabled.
Each miss or rejection increments one reason and its aggregate counter. Their sums agree after queries and the worker quiesce; live metric requests can observe updates between separately sampled atomic counters. An unavailable revision returned by a loader, a build discarded during shutdown and coalesced requests do not increment rejection counters. Exceptions are counted as failures.
Counters describe the current cache instance and reset on cache reconfiguration or server restart. Gauges and counters report zero when no cache is attached. A miss immediately uses the transactional index; it does not wait for a build.
Examples
The following query returns the connection_queue_seconds histogram in
JSON. The example responses use zero sample counts and sums. A running server
reports its current measurements; family metadata and bucket boundaries have
the structure shown here:
[{
"GetMetrics": {
"format": "json",
"metrics": [{
"family": "connection_queue_seconds"
}]
}
}]
Successful response:
[{
"GetMetrics": {
"status": 0,
"values": {
"families": [{
"help": "Time spent in the connection queue, in seconds",
"metrics": [{
"buckets": [{
"count": 0,
"max": "0.0001"
}, {
"count": 0,
"max": "0.001"
}, {
"count": 0,
"max": "0.01"
}, {
"count": 0,
"max": "0.1"
}, {
"count": 0,
"max": "1"
}, {
"count": 0,
"max": "10"
}, {
"count": 0,
"max": "inf"
}],
"count": 0,
"sum": "0"
}],
"name": "connection_queue_seconds",
"type": "Histogram"
}]
},
"version": "0.19.15"
}
}]
The same metrics are available in Prometheus text format:
[{
"GetMetrics": {
"format": "prometheus",
"metrics": [{
"family": "connection_queue_seconds"
}]
}
}]
[{
"GetMetrics": {
"status": 0,
"values": "# HELP connection_queue_seconds Time spent in the connection queue, in seconds\n# TYPE connection_queue_seconds histogram\nconnection_queue_seconds_count 0\nconnection_queue_seconds_sum 0\nconnection_queue_seconds_bucket{le=\"0.0001\"} 0\nconnection_queue_seconds_bucket{le=\"0.001\"} 0\nconnection_queue_seconds_bucket{le=\"0.01\"} 0\nconnection_queue_seconds_bucket{le=\"0.1\"} 0\nconnection_queue_seconds_bucket{le=\"1\"} 0\nconnection_queue_seconds_bucket{le=\"10\"} 0\nconnection_queue_seconds_bucket{le=\"+Inf\"} 0\n",
"version": "0.19.15"
}
}]
The following are the metric families exposed by ApertureDB.
commands_per_query : Number of commands per ApertureDB query
user_command_total : Number of commands executed by a user
client_connection_seconds : The time taken to establish and/or destroy a client connection, in seconds
connection_failed_attempts : The number of connection attempts that failed to negotiate protocol
connection_queue_seconds : Time spent in the connection queue, in seconds
cpu_cycles_total : Elapsed CPU cycles
page_faults_total : The number of page faults encountered by ApertureDB
memory_bytes : Bytes of memory in use
storage_graph_bytes : Bytes of storage used for graph
storage_images_bytes : Bytes of storage used for images
storage_videos_bytes : Bytes of storage used for videos
storage_blobs_bytes : Bytes of storage used for blobs
storage_total_bytes : Bytes of storage used
num_cores : The number of concurrent threads supported on this machine
query_seconds : Time taken per query, in seconds
api_bytes_transferred : The amount of data sent and received by the ApertureDB API, in bytes
log_messages_total : The number of log messages generated
audit_events_total : The number of audit events generated
region_intersections_by_resulting_pieces : The number of region intersection operations performed broken down by number of convex pieces in the intersection
region_intersection_seconds_by_resulting_pieces : The time spent computing region intersections broken down by number of convex pieces in the intersection
region_intersection_pieces_total : The total number of convex pieces generated by region intersection operations that fell within the specified range
iou_values : The distribution of intersection over union (IoU) values computed between pairs of regions
region_intersection_seconds : A histogram of the time spent per region intersection operation
client_session_total : The number of successful client session creations and refreshes
graph_process_command_seconds : Time to process graph commands in seconds
graph_query_commands_total : Count of graph commands processed within a query
lmdb_page_size_bytes : The size in bytes of a single LMDB page
lmdb_depth : The current maximum depth (height) of the LMDB b-tree
lmdb_pages : The current number of pages in the LMDB B-tree
lmdb_entries : The current number of data items in the LMDB B-tree
memory_heap_allocated_bytes_total : The total number of bytes of heap memory allocated
memory_heap_freed_bytes_total : The total number of bytes of heap memory deallocated
query_handlers : The number of individual query handlers.
page_size : The size of a memory page, in bytes.
query_executor_workers : Shared request/query CPU workers.
query_executor_task_capacity : Bound on admitted queued/running/retained task handles.
query_executor_budget_bytes : Shared retained task/input/scratch/result byte allowance.
query_executor_pending : Admitted task reservations, including retained completed results.
query_executor_retained_bytes : Currently charged task/input/scratch/result bytes.
query_executor_submitted_total : Accepted callbacks, including inline fallbacks.
query_executor_completed_total : Finished callbacks, including failures and inline work.
query_executor_failed_total : Callbacks that threw, including cancellation exceptions.
query_executor_rejected_total : Task reservation attempts rejected before materialization.
query_executor_inline_total : Callbacks run inline after admission fallback.
query_executor_cancel_requested_total : Accepted unconsumed tasks receiving a group stop request.
query_executor_abandoned_total : Queued callbacks discarded during pool shutdown.
graph_owner_links : Committed ownership links by fixed ownership kind.
graph_owner_traversals_total : Ownership traversal attempts by kind and direction, including empty and interrupted traversals.
graph_owner_links_visited_total : Ownership links delivered to traversal callbacks by kind and direction.
graph_filter_work_total : Executed filter work by category and phase: constant plans, selected index plans, residual row evaluations and returned proven rows. Index estimates and metadata reads are not row evaluations.
graph_filter_seconds : The time taken processing graph filters, in seconds.
graph_filter_entities_total : The number of entities processed by graph filters.
graph_read_query_seconds : Read graph operations in seconds
graph_write_query_seconds : Write graph operations in seconds
graph_authorization_seconds : Authorization time within graph execution or snapshot identity lookup
graph_txns_behind : Transactions this node is behind the cluster, as of its last sync
Typed filtering, ownership and query tasks
graph_filter_seconds and graph_filter_entities_total retain their
entity_type, filter_type and histogram stage labels. All node and
customer connection series also carry class, node_type, src_type,
src_class, dst_type and dst_class. Nonapplicable values are empty.
Names come from the executing snapshot. A rename creates new display-label
series; old reader snapshots continue to use the original labels.
graph_filter_work_total uses the same category labels with entity_type
and one fixed phase label, without filter_type or stage:
constant_planscounts completed class-count/constant decisions, including eligible count-only commands.index_planscounts executions of selected index accesses. Estimating a discarded access does not increment this phase.residual_rowscounts actual predicate evaluations on rows, including a reached evaluation that throws.proven_rowscounts materialized matches accepted without row evaluation.
Zero residual work establishes that no row predicate ran. Metadata, identity, index-estimate and projection reads can still occur. These counters aggregate work within a category and do not label individual expressions or requests.
Ownership has separate graph_owner_links gauges and
graph_owner_traversals_total / graph_owner_links_visited_total counters.
Their fixed kind values are bounding_box_image, polygon_image,
frame_video and clip_video. Traversals also carry direction=owners
or direction=children. Link gauges publish after successful commit, including
replay; aborted writers and older publications cannot replace committed counts.
Traversal counters include empty calls and links delivered before early stop or
an exception. Physical ownership storage uses the existing LMDB metrics for
owners-by-child and children-by-owner. Owners remain absent from
GetSchema.
query_executor_ families expose the shared CPU pool without request labels.
workers, task_capacity and budget_bytes are limits; pending and
retained_bytes include admitted work and completed results retained until
owner consumption. Counters report submitted_total, completed_total,
failed_total, rejected_total, inline_total,
cancel_requested_total and abandoned_total. A completion includes a
failure. Stop requests count unconsumed accepted handles, rather than asserting
that a callback had already stopped. Rejected reservations can keep serial
execution; inline callbacks count as submitted and completed work. Queued
callbacks discarded by shutdown count as abandoned.
The Descriptor Performance dashboard shows these native families and snapshot cache admission.