GetMetrics ========== Global allocation hooks report process totals under ``kUnknownC`` and ``kUnknownCpp``. The server enforces explicit request and transfer budgets. Retrieve the current metric values. Any authenticated user can call this command; no additional CRUD permission or input/output blobs are required. Parameters ---------- .. list-table:: :header-rows: 1 :widths: 22 22 56 * - Parameter - Type / required - Default and behavior * - ``format`` - string / optional - Defaults to json. Allowed: json or prometheus. * - ``metrics`` - array of objects / optional - Omitted or empty: all metric families. Otherwise select the named families and optional complete label sets. Each metrics element is an object with these fields: .. list-table:: :header-rows: 1 :widths: 22 22 56 * - Parameter - Type / required - Default and behavior * - ``family`` - string / required - Exact metric family name; no default. Unknown names select no family. * - ``labels`` - array of string maps / optional - Omitted or empty: all series in the family. Each object must exactly match the complete label set, including values, of a series. Multiple objects select alternative complete sets. A label filter is exact, not a subset match. For a series with both ``method`` and ``status`` labels, filtering only ``method`` does not select that series. An empty object in ``labels`` selects a series without labels. Response and recovery --------------------- Success returns ``status: 0``, ``version`` (the server version), and ``values``. For ``json``, ``values`` contains the metric families and series. Floating-point values are strings and may be ``"inf"``, ``"-inf"``, or ``"nan"``; integer counts remain JSON integers. For ``prometheus``, ``values`` is a string in Prometheus text exposition format. Unknown family names or unmatched label sets give a successful result without those metrics. Invalid formats, missing family names, or non-string label values fail validation; correct the selector. See :doc:`../Responses` for authentication and validation error envelopes. Details ------- A **metric** represents a single specific measurable phenomenon such as the size of a database or the number of queries that have run. Metrics come in a variety of types: * A **counter** is a single cumulative value that can only increase over time. As their name implies, counters can be used to 'count' discrete events as they occur, for example the number of queries received. The value of a counter reflects the amount accumulated since the ApertureDB process launched. * A **gauge** is a numerical value that can go up or down. The value of a gauge generally reflects some current state, for example the length of a queue. * A **histogram** provides a breakdown of discrete samples into buckets. It also includes a 'count' (counter) and 'sum' (gauge) representing the total number of samples taken and sum of all sample values respectively. Histograms are used to describe the distribution of events according to their size, such as query processing times or response sizes. * A **summary** is similar to a histogram but instead of fixed buckets it computes running quantiles over a sliding time window. Summaries are more expensive to compute than other metric types and are used less frequently. ApertureDB metrics are organized into **families**. A **family** describes a class of related metrics that can be compared directly with each other. Metrics within a family are differentiated using **labels** and must share the same type. Query timing ------------ ``graph_read_query_seconds`` and ``graph_write_query_seconds`` break graph work into phases identified by the ``operation`` label. Their ``exec`` phase includes authorization. Sum the phases within these families when calculating graph work. ``graph_authorization_seconds`` measures authorization separately, with ``operation="read"`` for read queries, ``operation="write"`` for write queries, and ``operation="snapshot"`` for identity lookups used by error reporting. The read and write measurements are already included in graph execution time; do not add them to the phase totals. This family replaces the previous authorization series in the graph phase families and the earlier ``query_seconds{operation="process_query",suboperation="auth"}`` series. Client sessions --------------- ``client_session_total`` counts successful committed authentication and token refresh requests, with ``status="started"`` and ``status="refreshed"``. Both series exist at zero before the first request. Failed or cancelled transactions do not increment them. These are cumulative events, not active-session counts; use ``rate`` to graph sessions per second. Descriptor index cache ---------------------- The ``descriptor_index_cache_`` families report the current graph's shared in-memory descriptor index cache. They are sampled when metrics are requested, including changes caused by searches and background builds without graph writes. Values cover all descriptor indexes and engines on this server. Aggregate families have no labels. The two reason families below have one fixed ``reason`` label; no descriptor class names, index identities or revisions become cache metric labels. .. list-table:: :header-rows: 1 * - Family suffix - Type - Meaning * - ``budget_bytes`` - Gauge - Configured cache budget, in bytes. * - ``charged_bytes`` - Gauge - Memory reserved for active builds, resident snapshots, and evicted snapshots still held by readers. * - ``entries`` - Gauge - Resident snapshots available for lookup. * - ``queued`` - Gauge - Builds waiting for the background worker. * - ``pending`` - Gauge - Queued builds plus the active build. * - ``hits_total`` - Counter - Lookups that acquired a cached snapshot. * - ``misses_total`` - Counter - Lookups without a snapshot, including cache lock contention. * - ``builds_total`` - Counter - Completed snapshots published to the cache. * - ``failures_total`` - Counter - Build or admission exceptions, including queue/observation allocation failures. * - ``rejected_total`` - Counter - Requests rejected or deferred by admission checks, lock contention, queue limits, or insufficient available cache budget. * - ``misses_by_reason_total`` - Counter - Misses partitioned by ``reason``: ``disabled``, ``contention``, ``stopping``, ``pending``, ``absent``. * - ``rejected_by_reason_total`` - Counter - Rejections partitioned by ``reason``: ``invalid``, ``disabled``, ``oversize``, ``contention``, ``stopping``, ``queue_full``, ``stale_revision``, ``pinned_budget``, ``deferred``. * - ``automatic_bypasses_total`` - Counter - Automatic jobs successfully queued after replacing an entry in the full admission observation table; excludes resident/pending coalescing. For example, ``descriptor_index_cache_charged_bytes`` is the full family name for charged memory. Its value includes a build's reservation before the snapshot is published, and can remain nonzero after eviction while searches hold the snapshot. It excludes LMDB's mapped pages, search scratch memory, allocator overhead and fixed control memory, including the lazy admission observation table (at most 32 namespaces, about 18 KiB on supported 64-bit platforms). It is therefore not a measure of total descriptor-related process RAM. Miss reasons are selected in priority order: disabled caching, lock contention, shutdown, then an exact revision pending or absent. A contention result does not inspect the entry or pending maps. Admission first checks invalid arguments, disabled caching, oversized reservation, contention and shutdown; an exact resident/pending request then coalesces before queue capacity is checked. Automatic admission additionally checks stale revisions and reuse evidence. The worker can reject a stale revision or fail to reserve its budget after LRU eviction because existing leases still hold memory (``pinned_budget``). ``deferred`` means automatic admission needs more reuse evidence for that revision. It is expected policy behavior, not a build failure. Admission normally requires 32 eligible absent observations or a later observation at least one second into the window. Idle time does not queue work. When the bounded table is full, a new namespace replaces the least recently observed entry and qualifies immediately; ``automatic_bypasses_total`` counts successful queueing through that path, not lookups, published builds or a cache being disabled. Each miss or rejection increments one reason and its aggregate counter. Their sums agree after queries and the worker quiesce; live metric requests can observe updates between separately sampled atomic counters. An unavailable revision returned by a loader, a build discarded during shutdown and coalesced requests do not increment rejection counters. Exceptions are counted as failures. Counters describe the current cache instance and reset on cache reconfiguration or server restart. Gauges and counters report zero when no cache is attached. A miss immediately uses the transactional index; it does not wait for a build. Examples -------- The following query returns the ``connection_queue_seconds`` histogram in JSON. The example responses use zero sample counts and sums. A running server reports its current measurements; family metadata and bucket boundaries have the structure shown here: .. TESTED_EXAMPLE: GetMetrics_1 .. code-block:: json [{ "GetMetrics": { "format": "json", "metrics": [{ "family": "connection_queue_seconds" }] } }] Successful response: .. TESTED_EXAMPLE: GetMetrics_1_response .. code-block:: json [{ "GetMetrics": { "status": 0, "values": { "families": [{ "help": "Time spent in the connection queue, in seconds", "metrics": [{ "buckets": [{ "count": 0, "max": "0.0001" }, { "count": 0, "max": "0.001" }, { "count": 0, "max": "0.01" }, { "count": 0, "max": "0.1" }, { "count": 0, "max": "1" }, { "count": 0, "max": "10" }, { "count": 0, "max": "inf" }], "count": 0, "sum": "0" }], "name": "connection_queue_seconds", "type": "Histogram" }] }, "version": "0.19.15" } }] The same metrics are available in Prometheus text format: .. TESTED_EXAMPLE: GetMetrics_2 .. code-block:: json [{ "GetMetrics": { "format": "prometheus", "metrics": [{ "family": "connection_queue_seconds" }] } }] .. TESTED_EXAMPLE: GetMetrics_2_response .. code-block:: json [{ "GetMetrics": { "status": 0, "values": "# HELP connection_queue_seconds Time spent in the connection queue, in seconds\n# TYPE connection_queue_seconds histogram\nconnection_queue_seconds_count 0\nconnection_queue_seconds_sum 0\nconnection_queue_seconds_bucket{le=\"0.0001\"} 0\nconnection_queue_seconds_bucket{le=\"0.001\"} 0\nconnection_queue_seconds_bucket{le=\"0.01\"} 0\nconnection_queue_seconds_bucket{le=\"0.1\"} 0\nconnection_queue_seconds_bucket{le=\"1\"} 0\nconnection_queue_seconds_bucket{le=\"10\"} 0\nconnection_queue_seconds_bucket{le=\"+Inf\"} 0\n", "version": "0.19.15" } }] The following are the metric families exposed by ApertureDB. .. include:: ../autogen/metric_families.rst Typed filtering, ownership and query tasks ------------------------------------------ ``graph_filter_seconds`` and ``graph_filter_entities_total`` retain their ``entity_type``, ``filter_type`` and histogram ``stage`` labels. All node and customer connection series also carry ``class``, ``node_type``, ``src_type``, ``src_class``, ``dst_type`` and ``dst_class``. Nonapplicable values are empty. Names come from the executing snapshot. A rename creates new display-label series; old reader snapshots continue to use the original labels. ``graph_filter_work_total`` uses the same category labels with ``entity_type`` and one fixed ``phase`` label, without ``filter_type`` or ``stage``: * ``constant_plans`` counts completed class-count/constant decisions, including eligible count-only commands. * ``index_plans`` counts executions of selected index accesses. Estimating a discarded access does not increment this phase. * ``residual_rows`` counts actual predicate evaluations on rows, including a reached evaluation that throws. * ``proven_rows`` counts materialized matches accepted without row evaluation. Zero residual work establishes that no row predicate ran. Metadata, identity, index-estimate and projection reads can still occur. These counters aggregate work within a category and do not label individual expressions or requests. Ownership has separate ``graph_owner_links`` gauges and ``graph_owner_traversals_total`` / ``graph_owner_links_visited_total`` counters. Their fixed ``kind`` values are ``bounding_box_image``, ``polygon_image``, ``frame_video`` and ``clip_video``. Traversals also carry ``direction=owners`` or ``direction=children``. Link gauges publish after successful commit, including replay; aborted writers and older publications cannot replace committed counts. Traversal counters include empty calls and links delivered before early stop or an exception. Physical ownership storage uses the existing LMDB metrics for ``owners-by-child`` and ``children-by-owner``. Owners remain absent from :doc:`GetSchema`. ``query_executor_`` families expose the shared CPU pool without request labels. ``workers``, ``task_capacity`` and ``budget_bytes`` are limits; ``pending`` and ``retained_bytes`` include admitted work and completed results retained until owner consumption. Counters report ``submitted_total``, ``completed_total``, ``failed_total``, ``rejected_total``, ``inline_total``, ``cancel_requested_total`` and ``abandoned_total``. A completion includes a failure. Stop requests count unconsumed accepted handles, rather than asserting that a callback had already stopped. Rejected reservations can keep serial execution; inline callbacks count as submitted and completed work. Queued callbacks discarded by shutdown count as abandoned. The Descriptor Performance dashboard shows these native families and snapshot cache admission.