8. Distributed Admin

The moving parts

Every Psyclone instance starts a local Node which manages communication and bookkeeping: subscriptions, services, interfaces, the local components, and the local process spaces. A distributed system is simply one master node (started with the PsySpec) plus any number of satellite nodes on other machines — possibly running different operating systems — that the master configures remotely. Conceptually there are three layers an administrator manages:

Satellite mode: starting the workers

To spread a system across machines, start idle nodes on each worker — Psyclone with no spec, optionally with an explicit port:

# on each worker machine
./bin/linux64/Psyclone port=11000

An idle node waits to be claimed. Then, in the master’s PsySpec, declare the nodes and place components on them by name:

<node name="Node1" address="localhost" port="11000" />
<node name="Node2" address="otherhost" port="10000" />

<module name="Ping" node="Main">  ...  </module>
<module name="Pong" node="Node1"> ...  </module>

The local (master) node is implicit and referred to as Main. Components with no node= attribute run on the startup node. Placement works identically for whiteboards and catalogs (including their key/keytype, FileCatalog root and DataCatalog interval attributes) — storage components can live wherever the data is.

Code libraries are distributed globally by default; a node can override where a library comes from:

<node name="Node3" address="192.168.20.20" port="10000">
  <library name="otherlib" library="../path/mylib.dll" />
</node>

Spaces: process isolation per node

Each node auto-creates the internal space Root (all components as threads in one OS process). Declaring more spaces creates separate OS processes:

<space name="YTTMSpace" />
<module name="YTTM" space="YTTMSpace"> ... </module>

If a space crashes, the node restarts the space and recreates its components; private (persistent) data survives. In a multi-node system a plainly-declared space is created on all nodes; restrict it with <space node="Node1" name="YTTMSpace" /> and place the component with both attributes. External processes can also host a space (<space name="GUISpace" type="external" /> plus a placeholder module) — the external program links CMSDK and connects with the PsySpace API; its crash and reconnection are tolerated like internal spaces. See the User Guide distributed chapter for the developer-side detail.

Topology: keep high-volume dataflow local

The single most important distributed design rule: bandwidth-heavy pipelines stay on one node; only distilled results travel. A camera producing raw frames should be processed (detection, feature extraction) on the machine it is attached to, with only compact events (Human.Self.Detected, positions, classifications) crossing the node link. CoCoMaps ran exactly this shape: each robot processed its own vision and audio onboard and shared only observations, roles and tasks with the server. The interactive diagram below shows a typical production topology — drag nodes to explore.

Blue = modules, orange = storage catalogs/whiteboards, charcoal = node engines. Thick edges = high-volume local dataflow (never leaves the machine); thin edges = distilled cross-node messages over the node links.

Verifying a multi-node install: test=psymulti

2.2.0 ships multi-node bring-up as a verified path, and ships the verification with it. test=psymulti is a self-checking run-through that starts three real Psyclone nodes as separate OS processes and brings up one distributed model across them under Startup Supervisor orchestration. Run it on a fresh install before you trust a deployment:

# three nodes, all on this machine (peers are auto-spawned on free ports)
./bin/linux64/Psyclone test=psymulti

# variants
./bin/linux64/Psyclone test=psymulti nodes=4      # scale the ring
./bin/linux64/Psyclone test=psymulti short        # quick smoke run
./bin/linux64/Psyclone test=psymulti v logall     # verbose, incl. peer processes

The process you launch becomes Main on its own port and holds the Startup Supervisor and the full <node> list. The peers are started bare — exactly like a production satellite, i.e. Psyclone with no spec and only a port — and wait to be claimed by Main, which assigns their node IDs and forwards their per-node bring-up recipes. At the end of the run Main posts Psyclone.Shutdown, the peer processes are reaped, and the harness prints its own PASS/FAIL verdict lines plus a final === test=psymulti complete === banner with the node exit status.

Two sibling tests share the same harness, which is useful when triaging: test=psytest is the single-node equivalent (use it to separate “multi-node is broken” from “this install is broken”), and test=psylong drives the same three-node topology as a sustained cross-node stress with per-message sequence, ordering and timing accounting.

What the 3-node topology actually exercises

When bring-up fails: what to check

• Ports. The harness auto-selects free local ports for peers; a production spec does not. Confirm every declared <node> address/port is reachable from the master, that nothing else holds the port, and that no firewall or NAT sits in between. A satellite that never gets claimed almost always means the master could not reach it.
• Are the peers alive? Check that a bare Psyclone process is actually running on each worker and still running after Main connects. If a peer exits immediately, read its own log rather than the master’s.
• Stuck before Psyclone.Ready. Deferred components are normal and transient; components still deferred at the end mean a node named in the spec never appeared, or a component names a node that is not declared at all. Check spelling of every node= value against the <node name=...> list, remembering the master’s own node is Main.
• Libraries. Each node must be able to load every library its placed components need. A per-node failure to load a crank library shows up as components missing on exactly one node — use a per-node <library> override where a machine needs its own build.
• SSL. If node links are secured, a verification failure looks like a node that connects and then drops. Test once with verification settings from chapter 5, and remember self-signed certificates need an explicit allowselfsigned="yes".
• Clean slate. A previous crashed run can leave stale POSIX shared segments and orphaned peer processes behind (cleanup was hardened in 2.2.0, but a SIGKILLed process cannot clean up after itself). Confirm no orphan Psyclone processes remain before re-running.
• Isolate. Run test=psytest first. If the single-node run also fails, the problem is the install or the platform build, not distribution. Then re-run test=psymulti with v logall and compare the master and peer logs around the point where a node stops progressing.
• Confirm shape, not just exit code. Use PsyProbe’s Nodes section against the running system to check that every expected node is present with its expected components, and the System Node Communication matrix to confirm traffic is crossing links only where you intended.

Load balancing & migration

Placement is the first-order load-balancing tool: watch PsyProbe’s Node Performance tab (input/output rates, queues, CPU, memory over the last 1/10/30 s) and move components between nodes in the spec. Nodes can be added and removed while the system runs. Runtime component migration exists in the engine but is not yet a hands-off balancing feature (see the Roadmap note below) — today, rebalancing means editing placement in the spec and restarting the affected spaces/nodes.

Runtime module migration is mostly built into the engine (external modules are marked migratable, and per-component private data is designed to survive a move), and the spec accepts a per-component opt-in:

<module name="Analyser" migration="yes"> ... </module>

Roadmap The migration attribute is parsed and sets the component’s migrate-allow flag, but automatic runtime migration as a hands-off admin feature is still being finalised — on the current roadmap it becomes a builder-owned action (SYS_BUILDER_MOVE, driving the existing migration machinery); see Roadmap Features. For today’s production systems, treat migration as spec-driven re-placement plus space restarts, and design modules to be movable: keep state in private data or catalogs, not in process-local globals.

Talking between separate systems

Multi-node is one system spread out. Sometimes you instead want several independent Psyclone systems (one per robot, per site…) that cooperate. The mechanism is the remote query: identical to a local <query> plus a host and port:

<!-- local -->
<query name="Master" source="CCMMaster" />
<!-- remote: another Psyclone system -->
<query name="Master" source="CCMMaster" host="other.hostname" port="10010" />

The remote component receives and answers as normal; the reply is routed back transparently. A component can detect that a query came from another system by the extra entries in the query message: INTERSYSTEM_IDENTIFICATION (string), INTERSYSTEM_ADDRESS (uint32 IP), INTERSYSTEM_PORT (uint16) and INTERSYSTEM_SOURCENAME. Chapter 9 shows this pattern federating a whole robot fleet through proxy catalogs.

Production checklist

• Ports & reachability: every declared <node> address/port must be reachable from the master at startup; pick fixed ports and open only those.
• Security: node-to-node and remote-query links carry your dataflow — apply the SSL configuration from chapter 5 (verification is on by default; self-signed requires an explicit allowselfsigned="yes").
• Libraries: ensure each node can load every library its components need (global by default, per-node override available); mind OS differences (.dll vs .so is handled automatically by name).
• Clocks: DataMessage timestamps are synchronised across computers to microsecond resolution by the platform, but sane NTP on all machines keeps logs and external correlation honest.
• Observability: use PsyProbe’s System Node Communication matrix to verify that high-volume types are not crossing node links, and Node Performance to spot overloaded machines before users do.
• Failure drills: kill a space and a satellite node in staging; confirm space restart and system behaviour match expectations before relying on them in production.