Skip to main content

Query protocol

The native TCP/TLS and HTTP transports carry the JSON command array and ordered binary blobs described in API Description. The SDK negotiates the connection and sends blobs as raw bytes separately from the JSON.

Native query framing​

After connection setup, the client and server exchange these fields in order:

request  = WS token WS command-array WS BE32(blob-count) blob...
response = WS result-json WS BE32(blob-count) blob...
blob = BE64(byte-length) raw-bytes

BE32 and BE64 denote unsigned integers in network byte order (big-endian), with widths of four and eight bytes respectively.

WS means zero or more of the four ASCII JSON whitespace bytes: space (20 hex), tab (09), carriage return (0D), or line feed (0A). Whitespace is accepted before the token, between the token and command array, and after the JSON before the blob count. A response has no token.

The request token is a JSON string, including its double quotes and any required JSON escapes. Use "" when no session token is available, such as for an initial authentication request. The following command JSON must be an array. Response JSON is normally an array of command results; an error may instead be a JSON object. See Responses for result fields and error envelopes. JSON text uses UTF-8 and must be valid JSON without comments.

The blob count must be less than 224 (at most 16,777,215). Its first byte is therefore always zero, which distinguishes it from preceding whitespace. A four-byte zero count is required even when no blobs are sent. Every declared blob then has its own eight-byte size and exactly that many payload bytes. A zero-length blob still has an eight-byte zero size and occupies one position in the blob list.

Once the first count byte has been read, all following count bytes, size bytes, and payload bytes are binary. Do not skip whitespace there: any of those bytes can equal an ASCII whitespace byte. No separators or alignment padding appear between blobs. Any whitespace after a complete message is the leading whitespace of the next message.

For example, this Python expression constructs a complete request with no session token, one command, and no input blobs:

import struct

request = b'"" [{"GetStatus": {}}]' + struct.pack("!I", 0)

An input blob is added after the count with its byte length:

command_json = b'[{"AddBlob":{"class":"Document"}}]'
blob = b"\x00\n\xff"
request = (b'"" ' + command_json + struct.pack("!I", 1)
+ struct.pack("!Q", len(blob)) + blob)

The empty token in these examples only shows framing; commands that require authentication need a valid session token.

Read JSON structurally, respecting nested arrays, objects, quoted strings, and escaped characters. A closing bracket inside a JSON string does not end the JSON. Read the count and each size with an exact-width operation, and read each payload to its declared length. TCP reads and TLS records can split or combine any of these fields. A single read is not a message boundary.

The message ends after the last declared blob, or immediately after a zero count. A parser must preserve any bytes from the next message when a read contains both. A connection executes requests and returns responses in order; there are no request IDs or multiplexed responses. Wait for a complete response before starting another request on the same connection.

Lengths and counts are subject to configured resource limits in addition to their wire widths. The server checks message size, JSON size and nesting, token size, blob sizes, and retained metadata before allocating payloads. In particular, declaring many empty blobs still consumes the memory budget. max_connection_message_size_mb and query_queue_bytes_mb are described in Configuration File; a 64-bit blob length does not imply that an arbitrarily large blob will be accepted. Malformed, excessive, or truncated framing closes the connection. There is no in-stream resynchronization marker.

HTTP representation​

The HTTP connector posts queries to /api/ on Lenz and sets the required protocol headers. Use HTTPS when the connection requires encryption.

The request body uses multipart/form-data. It contains one query part holding the JSON command array, followed by any input files in blobs parts in input-blob order. Query parts may use Content-Type: application/json; blob files use Content-Type: application/octet-stream. Send the session token as Authorization: Bearer <token> rather than as a JSON body field. Omit authorization for an initial authentication request.

Query responses have content type Content-Type: multipart/mixed; boundary=<boundary>. The multipart body has:

  1. Exactly one first part with Content-Type: application/json, containing the entire response JSON array or error object.

  2. One application/octet-stream part per output blob, in blob-index order. Each binary part declares its byte length in Content-Length and carries raw bytes, including for an empty blob.

  3. The closing multipart boundary.

No output blobs means the JSON part is followed directly by the closing boundary. Blob indexes in JSON refer to these binary parts, starting at zero. The HTTP body does not contain native BE32 counts or BE64 sizes.

For example, a response with one three-byte blob has this body structure. Line endings shown here are CRLF, and <three raw bytes> stands for the actual binary payload:

--example-boundary
Content-Type: application/json

[{"FindBlob":{"status":0,"returned":1,"blobs_start":0}}]
--example-boundary
Content-Type: application/octet-stream
Content-Length: 3

<three raw bytes>
--example-boundary--

Use the boundary from the response header rather than assuming a fixed value. HTTP/1.1 responses use chunked transfer encoding; transfer chunks can split multipart headers and binary parts at any point. Decode HTTP transfer framing before parsing multipart framing. HTTP/1.0 responses use connection-close delimiting. Check JSON statuses even when HTTP status is 200; application errors can be valid multipart query responses.

Streaming, completion, and retries​

The server completes result selection and JSON metadata before sending output blobs. Eligible media retrieval and conversion run when the next blob is needed, and blob bytes are sent in bounded chunks. A slow receiver delays subsequent blob work. The 600-second network send timeout bounds socket writes. Waiting for server workers to prepare media or read the next chunk does not consume that write deadline. Sorting, ranking, top-k selection, and filters still finish before the response starts; receiving JSON does not mean all media has already been fetched or encoded.

Exact blob lengths are required before their payloads. Encoded outputs can therefore materialize one complete encoded object before sending that object. This retained output is subject to memory and response-size budgets. Lazy sending does not promise constant memory for an arbitrarily large encoded image or video. Frame selections based on time or fractional position can also require eager decoding to determine the final frame count and indexes; frame extraction can perform preflight work before JSON is ready.

In-flight responses protect media from reclamation on the serving database node. Cleanup on another cluster node or changes at an external media origin can still interrupt a media read and close the response stream.

After JSON has been sent, a later storage, conversion, resource, or connection failure cannot replace it with an error envelope. The server closes the stream. Treat a native response as complete only after receiving its count, every size, and all declared payload bytes. For HTTP, require the closing multipart boundary and successful completion of HTTP transfer framing. Valid JSON alone is not a complete response.

Do not automatically replay a query after a write or response failure when the server may have received it. Writes may already have committed before blob generation or delivery fails. Reconnect and verify the operation’s effects before retrying; use stable application identifiers and conditional adds where appropriate. Version negotiation is safe to repeat before sending a query. See Failures and retrying writes for application error handling.