10. Distributed Systems

Nodes: one system, many computers

Every Psyclone instance starts a local Node that manages communication and bookkeeping: subscriptions, services, interfaces, local components and local process spaces. To distribute a system, start idle satellite nodes on the other computers — simply run Psyclone with no spec (optionally port=6789) — then declare them in the master PsySpec:

<node name="Node1" address="localhost" port="11000" />
<node name="Node2" address="otherhost" port="10000" />

The local node is implicit and is referred to as Main. Components are placed with the node= attribute; anything without one runs on the startup node:

<module name="Ping" node="Main"> ... </module>
<module name="Pong" node="Node1"> ... </module>
<whiteboard name="WB2" node="Node1" key="count" keytype="integer" />
<catalog name="MyFiles" node="Node2" type="FileCatalog" root="./" />

Nodes can mix operating systems — a single system can span Windows and Linux machines. Crank libraries are global by default; override per node when a machine needs its own build:

<node name="Node3" address="192.168.20.20" port="10000">
  <library name="otherlib" library="../path/mylib.dll" />
</node>

Multi-node bring-up

As of 2.2.0, bringing up a distributed model across several nodes — each a separate OS process, possibly on a separate machine — is a verified, shipped path rather than something you assemble by hand. You still write one spec on the master; the difference is what happens when it is parsed:

• Bring-up groups the declared components by their node= attribute and orders creation per node. Components naming a node that has not yet been claimed are deferred, not failed, so a satellite that is slow to appear does not abort the run.
• Each node’s share of the model is dispatched to that node and created there, under the Startup Supervisor on the master.
• Cross-node bookkeeping is handled for you: message- and component-type ID agreement between nodes, a deterministic tie-break when two nodes reserve the same context ID, and confirmation that each node has the component information it needs before the system is declared ready.
• Trigger groups whose members live on different nodes are joined up across the node links.

The practical consequence for you as a developer: Psyclone.Ready on a multi-node system means what it says — every declared component on every node actually exists. Until then, components on other nodes may not be reachable yet, so do not send work in a module constructor; wait for readiness or for the Psyclone.SystemStatus broadcast the master emits on a five-second cadence.

Spaces: process separation

Within a node, each Space is a separate OS process. Every node auto-creates the internal space Root (all its components run as threads in one process). Declaring more spaces buys crash isolation:

<space name="YTTMSpace" />
<module name="YTTM" space="YTTMSpace"> ... </module>

If a component in a space crashes, the node restarts that space and recreates its components — the rest of the system keeps running, and the components' private data survives. This is how commercial-grade and experimental code safely coexist. A plainly-declared space is created on all nodes; restrict it with <space node="Node1" name="YTTMSpace" />. Whiteboards and catalogs can be placed in spaces too.

External spaces

An external space lets a third-party program host Psyclone components inside its own process. Declare the space with type="external" and give the module a crank name only:

<space name="GUISpace" type="external" />
<module name="CoCoMapsGUI" space="GUISpace">
  <crank name="CoCoMapsGUI" />
</module>

The external program links CMSDK and connects in:

PsySpace* space = new PsySpace("GUISpace");
space->connect(sysid);
space->start();
PsyAPI* api = space->getCrankAPI("CoCoMapsGUI");  // retry in a loop on failure
// ...then use api exactly like an internal module crank...

space = cmsdk.PsySpace("GUISpace")
space.connect(sysid)
space.start()
api = space.getCrankAPI("CoCoMapsGUI")  # retry in a loop on failure
# ...then use api exactly like an internal module crank...

Monitor with space->isConnected() / space->hasShutdown(); on failure delete the space (but not the PsyAPI — it is owned by the PsySpace) and retry. External-space crashes are tolerated and reconnectable just like internal ones. A full walk-through is in Tutorials.

Topology at a glance

The interactive graph below shows a typical two-computer deployment: drag nodes around, zoom, and note the boundaries — hosts contain Psyclone nodes, nodes contain spaces, spaces contain components.

Blue = modules, orange = whiteboards/catalogs, charcoal = nodes; dashed edge = network link between nodes.

Communication between separate Psyclone systems

Nodes join computers into one system. Sometimes you instead want several independent Psyclone systems — for example one per robot — to cooperate. That is done with remote requests: a <query> pointed at a component in another system by host and port:

<!-- local query -->
<query name="Master" source="CCMMaster" />
<!-- remote query: same code, different spec -->
<query name="Master" source="CCMMaster" host="other.hostname" port="10010" />

The remote component receives and answers the query exactly as if it were local; the reply is routed back transparently. Because queries are referenced by name in code, switching a component from a local to a remote catalog is a pure spec change — the CoCoMaps slave robots ran a proxy catalog under the same name as the master's CCM so no module ever knew the world model lived on another machine.

Recognising a remote query

A catalog that wants to know where a query came from can check for the extra entries stamped onto remote queries:

Migration and load balancing

Components can be assigned to nodes manually as above, and the platform is designed for runtime migration of components between nodes for load balancing — marking a module migration="yes" flags it as movable. The operational side (when and how migration happens, monitoring node load, adding/removing nodes on a live system, and securing node links with SSL) is administration territory: see the System Guide chapter on distributed administration.