16. LLM Connections

1. What an LLM Connection is

An LLM Connection (shipped in CMSDK 2.2.0) is a named, node-owned connection object to an LLM endpoint. It is configured from the reserved <llm> named-connection XML element, which contains an <llmvendor> vendor-descriptor sub-block describing the HTTP shape of the vendor's API. Once configured and connected, the connection offers two interaction modes: a blocking request/reply exchange (interact()) and a streaming exchange (interactStream()) that delivers decoded tokens incrementally.

Recognised vendor families for <llmvendor type="…"> are openai, anthropic, bedrock, google and custom. The custom type is for config-only vendors that fully specify their own HTTP shape via templates and headers.

2. The <llm> and <llmvendor> elements

<llm name="main" apikeyref="OPENAI_API_KEY" global="true" schemaversion="1"
     maxtokensperrequest="4096" maxrequestspermin="60"
     maxconcurrent="2" costceiling="10.0">
  <llmvendor type="openai" endpoint="https://api.openai.com/v1/chat/completions">
    <header name="Authorization" value="Bearer %apikey%"/>
    <requesttemplate>{"model":"%model%","messages":[...]}</requesttemplate>
    <responsetemplate>choices[0].message.content</responsetemplate>
    <models default="gpt-4o">
      <model name="gpt-4o"/>
      <model name="gpt-4o-mini"/>
    </models>
  </llmvendor>
</llm>

<llm> attributes

<llmvendor> attributes and children

3. Credentials: apikeyref indirection

apikeyref is an indirect reference, resolved at configure() time. Three forms are accepted:

A missing environment variable or unreadable file fails typed at connect() with LLM_ERR_SECRET_UNRESOLVED. The secret value (and the failure detail) never appears in logs, errors, or getSystemStatus(); the public API only exposes hasResolvedSecret(), never the value.

Correct — indirection via the environment:

<llm name="main" apikeyref="env:MY_LLM_KEY">
  <llmvendor type="openai" endpoint="https://api.openai.com/v1/chat/completions">
    <header name="Authorization" value="Bearer %apikey%"/>
  </llmvendor>
</llm>

Incorrect — rejected at configure() with LLM_ERR_CONFIG_INLINE_SECRET:

<!-- WRONG: inline secret. This does not configure; it fails typed. -->
<llm name="main" apikey="PUT-KEY-HERE-THIS-IS-REJECTED">
  <llmvendor type="openai" endpoint="https://api.openai.com/v1/chat/completions"/>
</llm>

4. Interaction modes

Mode A — blocking request/reply

connect() validates the configuration and the resolved secret, then marks the connection live and starts a fresh stats record. Endpoint reachability is checked lazily, on the first interact() — the underlying HTTP client is connectionless and blocking, so there is nothing to dial eagerly. interact() shapes the request from the <requesttemplate> (%model%/%input%) and the configured headers (%apikey%/%model%), sends it, and extracts the reply text via the <responsetemplate> JSON path. reset() drops the live state but retains the configuration and the final stats record (ended=true).

Mode B — streaming

interactStream() sends one request and delivers incremental decoded tokens (not raw wire bytes) to a callback as they arrive; on success an optional fullReply holds the assembled full text. The call blocks until the stream completes, errors, or the callback cancels by returning false (the call then returns LLM_ERR_STREAM_CANCELLED).

The streaming wire format is resolved once at configure() time — callers never branch on vendor:

5. Worked example

PsySpec declaration:

<llm name="assistant" apikeyref="env:MY_LLM_KEY" maxtokensperrequest="4096">
  <llmvendor type="anthropic" endpoint="https://api.anthropic.com/v1/messages">
    <header name="x-api-key" value="%apikey%"/>
    <requesttemplate>{"model":"%model%","max_tokens":1024,"messages":[{"role":"user","content":"%input%"}]}</requesttemplate>
    <responsetemplate>content[0].text</responsetemplate>
    <models default="claude-sonnet-4-5">
      <model name="claude-sonnet-4-5"/>
    </models>
  </llmvendor>
</llm>

C++ usage (there is no Python API for LLM Connections):

#include "LLMConnection.h"
using namespace cmlabs;

LLMConnection llm;
if (llm.configureFromString(xml) != LLM_OK) { /* typed error via getLastError() */ }
llm.selectModel("claude-sonnet-4-5");
if (llm.connect() != LLM_OK) { /* e.g. LLM_ERR_SECRET_UNRESOLVED */ }

// Mode A: blocking
std::string reply;
LLMResult res = llm.interact("Summarise the bake status.", reply);

// Mode B: streaming
bool onToken(const char* token, uint32 size, void* userData); // return false to cancel
std::string full;
res = llm.interactStream("Summarise the bake status.", onToken, NULL, &full);

LLMStats s = llm.stats();   // uptimeMS, bytesSent/Received, requestCount,
                            // tokensIn/Out, replyChunks, cost, ended
llm.reset();                // drop live state; stats retained, ended=true

6. Error handling

Every operation returns a typed LLMResult; bad configuration must never crash. LLMResultText() gives the human-readable name for logs and typed-error reporting, and getLastError() retains the most recent failure.

7. Honest limits (2.2.0)

8. See also

• Builders & Supervisors — the bake pipeline an LLM-assisted workflow feeds into.
• System Guide: Security & SSL — transport security and operational secret handling.