Distributed execution¶

Orchestra can drive a workflow that lives on another site. The trick is a single contract (OrchestraClientInterface) with two implementations behind it: one in-process, one over HTTP. A caller depends only on the interface, so the same code runs same-site and cross-site; the container binds whichever half the site provides.

flowchart LR
  caller["Caller<br/>(ECA action, inbox,<br/>task handler)"] --> contract{{OrchestraClientInterface}}
  contract -.->|same site| local[LocalOrchestraClient]
  contract -.->|remote site| remote[RemoteOrchestraClient]
  local --> engine[(Workflow engine)]
  remote -->|OAuth2 + HTTP| api[orchestra_server]
  api --> serverClient[LocalOrchestraClient] --> serverEngine[(Engine on the server)]

The contract¶

orchestra defines OrchestraClientInterface and what both ends of the wire share, so a site that only calls a remote Orchestra installs it without the engine. The engine, orchestra_engine, depends on it and ships LocalOrchestraClient, which wraps the engine and entity storage in process. A site that runs the engine and is called by nobody needs nothing more. The surface is small and transport-friendly: values cross as scalars and plain arrays, never entity objects, so a remote half can rebuild them from JSON:

Operation	Meaning
`startProcess(definition_id, variables)`	Start an instance; returns its ID.
`signalToken(token_id)`	Resume a parked token (the completion primitive).
`getStepView(resume_handle, actor_proof, context)`	The step a resume handle names as a person is shown it, if they may see it: its title, description, the messages shown with it, and the outcomes it is completed with.
`checkStepCompleteAccess(resume_handle, actor_proof)`	Whether a person may complete that step.
`resumeStep(resume_handle, payload, variables, scope, context, actor_proof)`	Complete that step; answers resumed, refused, moved on or expired.
`cancelInstance(instance_id)`	End a running instance, cascading as a local cancel does.
`getInstance(instance_id)`	An instance with its live tokens, or `NULL`.
`listInstances(filters)`	One bounded page of instance summaries, newest first.
`getVariables(instance_id)`	The instance's variables as a name→value map.
`setVariable(instance_id, name, value)`	Create or replace one variable.

Every operation is scoped to the current tenant: an instance, token or variable in another tenant is invisible and cannot be acted on.

The HTTP API¶

orchestra_server exposes the contract over HTTP under /orchestra-api, guarded by OAuth2 (the client-credentials grant). Its controllers are a thin transport: each delegates to a LocalOrchestraClient and serializes the result; no workflow logic lives there.

Method & path	Operation
`POST /orchestra-api/process/{definition_id}/start`	`startProcess`
`GET /orchestra-api/instances`	`listInstances`
`GET /orchestra-api/instance/{instance_id}`	`getInstance`
`POST /orchestra-api/token/{token_id}/signal`	`signalToken`
`POST /orchestra-api/step/view`	`getStepView`
`POST /orchestra-api/step/access`	`checkStepCompleteAccess`
`POST /orchestra-api/step/resume`	`resumeStep`
`POST /orchestra-api/instance/{instance_id}/cancel`	`cancelInstance`
`GET /orchestra-api/instance/{instance_id}/variables`	`getVariables`
`PUT /orchestra-api/instance/{instance_id}/variable/{name}`	`setVariable`
`GET /orchestra-api/definitions`	`listDefinitions`

Three more endpoints are not part of the contract. They are how two sites set up and keep working together:

Method & path	What it does
`POST /orchestra-api/handshake`	Tells a caller whether the two sites can work together.
`POST /orchestra-api/presence/redeem`	Redeems a presence handoff code for its presence key.
`GET /orchestra-api/presence/status`	Says how long a presence key has left, and extends it.

Two permissions gate them, so a consumer that only reads never gains the ability to drive a workflow:

Permission	Endpoints
`access orchestra api`	The four reads: `listInstances`, `getInstance`, `getVariables`, `listDefinitions`. And the three above.
`drive orchestra api`	The writes: `startProcess`, `signalToken`, `setVariable`, `cancelInstance`, and the two step endpoints.

A consumer's role needs both to use the whole surface. Granted only the read permission, it is answered 403 on the write endpoints, before the controller runs.

Resuming a step from another site¶

A step a person acts on hands out a resume handle: the link of an External interaction, or a task's reference. A calling site that shows the step's form itself, say a webform on the forms site, sends that handle back to complete the step:

Before showing the step, it asks getStepView(), with the context it is about to show, such as the submission it reopens. The answer says whether the person may see the step with that context, and gives what this site's own pages show of it: the title, the description and the messages shown with it, as HTML the calling site filters before showing it.
Before showing the form, it may ask checkStepCompleteAccess(), so a person who may not complete the step is told before they type anything. An expired link is told apart, so the person can be asked for a new one.
On submit, it calls resumeStep() with the handle, the payload, the answers as variables, and a context naming what was submitted, such as ['webform' => 'leave_request', 'submission' => '42']. The payload is what this site's own pages send: the outcome alone, or the outcome with what the step records beside it, such as a review's comment. A step without outcomes gets the comment alone.

orchestra_client_webform does both for a webform: see A webform on a site without the engine.

The engine site decides with the same code its own pages use. It checks the person, then asks the step's interaction whether the context is its own: a webform step accepts only its own webform, or the submission it shows. A step whose interaction cannot check a context refuses one, so a handle copied from another step cannot complete it. Nothing is written unless the step is resumed.

Who the person is comes from the identity proof sent beside the handle, as for a start. Without a proof, the step is answered for nobody, never for the calling site's own account: a link step is completed with its handle alone, as in a browser, and a task is refused. The handle always travels in the request body, never in the URL.

The full contract (request and response shapes, status codes, and the OAuth2 scheme) is described in OpenAPI 3.1 in openapi.yaml, with a rendered reference alongside it in api-reference.html (a ReDoc page that loads the spec).

Tenant binding¶

A consumer is bound to one tenant by an immutable orchestra_tenant field added to its OAuth consumer. A tagged tenant resolver (ConsumerTenantResolver) reads that binding on each request, so every API call a consumer makes is automatically scoped to its tenant: the controller never resolves a tenant itself, and a consumer cannot reach across tenants.

The remote client¶

orchestra_client ships RemoteOrchestraClient, which speaks a remote server's HTTP API. It obtains a bearer token via client-credentials (the secret held by the Key module, never in config) and calls /orchestra-api. The two halves of the contract are symmetric: neither the caller nor the server knows which side the other is on.

A remote per Orchestra¶

Which Orchestra a client speaks to is an orchestra_remote configuration entity, and a site holds one per engine it calls: a host, the consumer identity to authenticate as, the Key holding that consumer's secret, and how long a call to it is worth waiting for. They are managed at Configuration > Web services > Remote Orchestras by holders of administer orchestra remotes, and each carries its own access token in state, so a site calling several Orchestras never offers one host the token it agreed with another.

Code that knows which Orchestra it means asks RemoteClientFactory for that remote's client. Code that names none gets the site's default remote: the one marked as such, or the only one when a site holds one. A site holding several and marking none is refused rather than answered with whichever sorts first, because which host a call reaches is not a thing to decide by accident.

The client contract itself is still bound site-wide here, to that default remote, and resolved per call rather than when the service is built, so building a service that holds the contract never fails for want of an outgoing remote.

Which leaves three ways to name an Orchestra, and a site can use all three at once, because a site that answers other Orchestras and calls one of its own is exactly what a pair of peers is:

OrchestraClientInterface is the Orchestra this site works through: its own engine, or the default remote where orchestra_client is installed. It is what code that does not care is written against.
LocalOrchestraClient is this site's own engine, named outright. The API controller holds this one: what it serves is always started, read and resumed here, and a controller holding the contract instead would answer an inbound call by forwarding it to whichever site this one happens to call.
RemoteClientFactory is that Orchestra, by remote id. The remote child node holds this one.

That contract, and nothing else. ProcessControlInterface is never rebound: it is aliased to orchestra_engine.engine, whose only implementation is the local WorkflowEngine, so code depending on the engine rather than on the client keeps driving the local site whatever else is enabled.

orchestra_content is the one to know about. An attachment holds its instance as an entity reference to a local orchestra_instance, so a binding can only ever name a local run, and a start that binds one is therefore a start on the local engine. Enabling it on a consumer alongside orchestra_client gets a local start where a remote one was meant, which surfaces as a workflow that does not exist when the workflows live on the server. Submodules that drive the engine belong on the server; a consumer holds the client.

The site-wide binding is chosen by OrchestraClientResolver, in orchestra. Each client is tagged orchestra.client with a priority: the engine's own client sits below the default, and orchestra_client's remote one at it, so on a site that has both the remote wins. The contract is built from the resolver rather than aliased to a client, so the binding does not depend on the order module service files are read in, and a site that installed the contract alone still builds its container: the status report says it has no client, and anything asking for one is refused.

Encrypted transport¶

The client secret and the bearer token travel to the base URL, so it must use https. The connection form enforces this on save, and the client re-checks it at request time, so a base URL imported through configuration cannot slip past the form. Two exceptions are allowed: a loopback host (localhost, 127.0.0.1, ::1 or a *.localhost name), for a remote you run locally in development; and any host listed in a settings.php array, for a trusted internal peer reached by a name that is not resolvable as https (a Docker service name, a private-network address):

$settings['orchestra_client.allow_http_hosts'] = ['orchestra-internal', '10.0.0.5'];

The list lives in settings.php, not configuration, on purpose: allowing plain http for a host is an operator decision that a configuration import must not be able to introduce.

The same rule holds for the address a caller announces, and there it matters more. A caller shaking hands is telling this site where to post, unprompted, from cron — so an unencrypted address would put every notice and its signature on the wire in clear, and an internal one would turn an API consumer into a way to make this site reach into its own network. A handshake naming an address this site will not post to is refused there and then, rather than at the moment a notice fails to land, and delivery asks again before each post, because the handshake is not the only way an address can reach a consumer.

What that check does not do is tie the address to the caller: nothing here knows what hostname a consumer's site has, so a caller may name any address the rule allows, and this site will post small signed bodies there as its runs move. Granting a consumer the API is granting it that, so grant it to sites you would give an account to.

Agreeing terms¶

A caller knows a remote's address and its credentials, and nothing else: not whether the two versions understand each other, not which tenant those credentials act in over there, and not where that site should send word of what its runs are doing. Each is answerable only by the other end.

So the two ends shake hands when the remote is saved, and the form says how it went.

The calling site sends:

the wire version it speaks;
the address where it receives notices;
the address where it takes presence handoff codes back;
its own address, where a step showing one of its forms sends the person.

The receiving site answers with:

its own wire version;
the tenant those credentials act in;
a secret for signing notices;
the identity scheme the two sites use;
the identity proof schemes it grants the calling site.

The calling site refuses an answer that leaves out any of these. It records the answer on the remote, except the secret. The secret is kept in state, because configuration is exported, and a secret in an export ends up in a repository.

The version is checked in both directions. The remote refuses a caller it is too new to read; the caller refuses a remote too new for it, which that end has no way to know. WireVersion carries a minimum as well as a current, so the range between them is a statement that both ends can still hold a conversation rather than a version label.

A failed handshake does not fail the save. A remote is routinely configured before the site it names is reachable, and refusing to store the address until it answers would make the first of two sites impossible to set up. The form warns, and saving again is the retry. A refused handshake records nothing at all: half an agreement would read as an agreement.

The answer is recorded when the remote is saved and is not asked again on its own. So after either site is upgraded, the two are working from terms agreed before that, and saving the remote again is what re-asks.

Being told what a remote run is doing¶

Without this a caller finds out on its next sweep, so a child finishing just after one runs stays invisible until the next. A notice is the same news earlier — it is an optimization of polling, never a replacement, and the run is reconciled either way.

The child site queues word of a run on its lifecycle transitions and delivers it separately. Queued rather than sent, because those transitions are dispatched inside the advancing worker's transaction: an outbound call there would announce runs that then roll back, and a caller whose host hangs would hold the worker advancing everybody else's runs. Delivery has its own queue for the same reason — one unreachable caller must not hold up work here.

A caller that started an instance with notify_status is also sent an instance.status_changed notice each time the status that instance passes up changes: the status it shows, or one a child below it passes through it (see child_status in subprocesses). The remote subprocess node asks for it when its "Child's status" setting is Show or Pass through, and the caller's site applies the status at the parent waiting on that child, as its "Child's status" setting says, when a status with that machine name exists there for the parent's tenant; a name it does not have changes nothing. Statuses therefore have to share machine names across the two sites to travel. A child site older than this ignores the flag, and the parent keeps its own node's status. Like every notice this is news sooner rather than an answer: one that arrives before the parent has finished parking on the remote node finds nobody waiting, and the parent shows an older status until the next change.

notify_variables works the same way for variables: a list of the instance's variable names, and the caller is sent an instance.variables_changed notice, carrying variables (name and new value), each time a write changes any of them: one notice per write, carrying every one of them that changed. The remote subprocess node asks for the child variables its output map hands back when it is set to pass its child's variables up as they change, and the caller's site writes only those, under the parent's names, while the parent still waits on that child. A name the map does not hold changes nothing, whatever a notice carries. Like a status notice, one that arrives before the parent has finished parking on the remote node finds nobody waiting; the parent then holds the older value until the next change, or until the child completes and the output map runs.

Each notice is delivered, and retried after a failure, on its own, so a status or variables notice can land after a newer one. Each carries sequence, when the change happened on the child's site in microseconds, and the caller's site applies it only over something recorded earlier: every process variable records when it was written (changed) and every instance when its status was recorded (status_changed), whoever wrote them. A late notice therefore changes nothing once the parent holds a newer value, whether that came from the same child, from another child mapping onto the same variable, or from the parent itself. The status a node set to Show records while it waits counts as older than any status from its child. Ordering across two sites rests on their clocks agreeing, which time synchronization keeps to milliseconds. A variables notice is sent only when the value changed, not each time a step writes the value the variable already holds.

Delivery retries on the engine's own attempt budget with a wait that doubles. When the budget is spent the notice is given up on, and the caller, the event and the last error are recorded against the run as well as logged: a notice dropped in silence is indistinguishable from a run that never moved, and a log rotates while the question an operator arrives with is about the run. It is not an incident — an incident belongs to a dead-lettered token and blocks its instance from completing, and there is no stuck branch here.

Each notice is signed with the secret that caller was given, and the timestamp is signed with the body rather than sent beside it — a timestamp anybody can edit bounds nothing, and a captured notice could be replayed for as long as the secret lived. The tolerance is symmetric, because a sender whose clock runs fast cannot be told from somebody holding a notice to use later.

The caller listens on one address per remote, so an arriving notice names its sender by where it arrived rather than by anything in the body a sender chose. A secret one remote leaks cannot be used to forge a notice from another, and a secret outlives only the relationship it was agreed for: deleting a remote, pointing it at another host, or giving it other credentials drops it, so the host that was replaced cannot go on signing notices accepted as that remote's. Every way of failing to be that remote — a wrong signature, an expired one, a site that has never shaken hands — is refused alike, because telling them apart would say whether a handshake exists.

A verified notice is re-dispatched locally as RemoteNoticeEvent and answered with a bare 204. What a site does about a child finishing is its own business, and telling the far end more would describe this site's workflows to whoever posted. The remote subprocess node is one listener: a notice that a child ended has the parent waiting on it polled once the 204 has been sent, and a status notice is shown or passed up at that parent at once (see notify_status above).

Those addresses are open, so refusing an unsigned post still costs the work of checking. Configuration › Web services › Orchestra notices caps how often one remote is answered, and how often the addresses answer in total; the second catches somebody working through each address in turn, which the first never sees. Status notices count toward both like any other, one for each status a child passes up, so a remote whose children change status often needs room for them. Both ship at no limit and count nothing until request limits are turned on for the whole site, at Configuration › Web services › Orchestra request limits (administer orchestra request limits), and while they are off the status report says so.

Whose run it is, on both sites¶

An example to start with. Marie fills a request form on a forms site. The workflow that handles the request runs on another site, the workflow site. In this page, the forms site is the calling site, and the workflow site is the receiving site.

The run on the workflow site should belong to Marie. Then Marie can follow it, and the steps meant for Marie are theirs to complete.

Why a user id cannot cross¶

A run records its owner as a user id. User ids are numbered by each site on its own. User 42 on the forms site can be somebody else on the workflow site, or nobody. So the calling site never sends a user id. It sends a name that both sites read the same way, such as the account name. How the two sites agree on that name is explained in how the two sites identify the same person.

The remote_subprocess node sends a name by default. Whoever the parent run belongs to can then see the part of their work that runs on the other site. The node can also send no name. See who the child instance belongs to.

Names come back the same way. When the calling site reads its runs, each initiator is reported as a name, never as a user id. A run the receiving site started for itself reports nobody: that person is somebody on the receiving site, and the calling site has no business knowing who.

A start carries a proof, not a bare name¶

A start does not just say "this is Marie". It carries an identity proof: a scheme, and a credential for that scheme. The receiving site checks the credential with the verifier of that scheme. The person it proves becomes the run's initiator.

This matters because the initiator owns the run. Owning a run means being allowed to read it. So a calling site whose proofs are believed can give people access to the runs it starts. The receiving site therefore decides, for each consumer, which proofs it accepts. The setting is Identity proofs accepted, on the consumer. There are two schemes:

The calling site's word (asserted). The credential is Marie's name. The receiving site believes it because it trusts the calling site. If an attacker takes over the calling site, they can make anybody the initiator of anything. Grant it only to a site you trust as much as the receiving site.
The person, present (presence). The credential is a key Marie got by signing in on the receiving site. The calling site keeps it only in their browser. An attacker who takes over the calling site can act only for the people using it at that time, and only until their keys expire. The calling site's database holds nothing that can act for anybody. See the presence proof.

A consumer granted no scheme proves nobody, and every run it starts belongs to nobody. The handshake gives the calling site the list of granted schemes, so it never sends another one. Each start checks the list again, because the calling site's copy can be out of date.

A start sent without any proof belongs to nobody. It never belongs to the API account that sent it: that account is a machine, and it can act for many people.

A proof that fails stops the start¶

A proof that proves nobody refuses the start, and nothing is created. These are all refused:

a scheme the consumer was not granted;
a name no account on the receiving site has;
an account the receiving site has blocked.

Why refuse instead of starting a run that belongs to nobody? Because the calling site asked for somebody. A run that belongs to nobody would look started, but Marie could not see it, and nobody would be told.

The refusals say different amounts on purpose. A scheme that was not granted is refused before anything is looked up. The refusal reveals nothing, not even whether the account exists. A name that matches no account is refused with the name repeated, so the calling site can fix it. That tells the calling site nothing new: it is already trusted with that name.

How the two sites identify the same person¶

Which part of an account means "the same person" on both sites depends on how the sites are set up. Some examples:

Two sites signing people in through the same identity provider know each other's people by the subject that provider issues.
Two sites that copy their accounts to each other know them by uuid.
An organization running both sites may know them by an employee number.

So each way of identifying people is a plugin, called an identity scheme. Orchestra ships one: the account name.

A scheme matches exactly, or it matches nobody. The initiator owns the run, so a guess at a near match could give somebody a run that is not theirs. No scheme guesses.

A pair of sites uses one identity scheme, or none. The receiving site sets it, for each consumer, under Identity scheme. The receiving site decides because it is the one reading the names. It tells the calling site at the handshake. No call names a scheme. Every name the calling site sends is read with the pair's scheme, and every person reported back is identified with it. So the two sites never read the same name in two different ways.

Two sites that share no accounts choose none. A name from that calling site then proves nobody, and runs reported back to it have nobody as their initiator.

A pair can identify its people with a scheme one of the two sites does not have installed. The calling site keeps the scheme the receiving site chose, and its remote form warns about it. Until the plugin is installed on the calling site, the pair identifies nobody. It never falls back to another scheme. The same holds the other way round: if the receiving site loses the plugin a consumer uses, it identifies nobody for that consumer until the plugin is back.

The presence proof¶

Marie is filling the request form on the forms site. The forms site needs to prove to the workflow site that it is really Marie. Here is what happens:

The forms site sends Marie to the workflow site. The form page links to the forms site's presence start, with the remote and the page to come back to: /orchestra-client/presence?remote=workflows&return_to=/form/request. The forms site remembers that page under a random value, the state. It then sends Marie's browser to the workflow site's presence page, /orchestra/presence. The address carries:
the consumer, which names the forms site;
the return address, /orchestra-client/presence/return on the forms site. It must be the one the forms site announced at the handshake;
the state;
Marie's name on the forms site, when the two sites identify people.
Marie signs in on the workflow site. They sign in the usual way. On a site using OpenID Connect, that is the provider's login, which they do not even see while their provider session lasts. The first time, they approve the forms site acting for them while they use it. The consumer can be set to skip that question. If their account names differ on the two sites, they are always asked to confirm.
The workflow site sends Marie back with a one-time presence handoff code. It creates a presence key for Marie and the forms site, and keeps only a hash of it. It sends Marie's browser back with the presence handoff code, which hands the key over. The code works once, only for the forms site, and only for two minutes.
The forms site redeems the code for the key. It accepts the code only for a round trip Marie started. It redeems the code with the workflow site, server to server: POST /orchestra-api/presence/redeem. Redeeming counts as the key's first use. The answer holds the key, how long it lives unused, and how long it has left. The forms site keeps the key only in a cookie in Marie's browser, which no script can read. Its session also records that the key was fetched for Marie, and when it expires. So if somebody else signs in next on the same browser, the key is never sent for them.
Each call for Marie carries the key. The workflow site checks that the key is still valid and belongs to the forms site. Once half the key's lifetime is gone, it extends the key. Every answer to a call whose key it checked says how long the key has left, in the X-Orchestra-Presence-Expires-In header. The forms site records that in Marie's session, and never sends the key past it; the cookie holding the key lasts the browser session. Once Marie stops using the forms site, the key expires after the lifetime set on the consumer, on both sites at once.

The forms site takes the workflow site's word for the expiry rather than counting. A key used with 20 minutes left is not extended, so counting would get it wrong. Showing a form does not reach the workflow site, so it extends nothing either. Instead, when the form is shown and the key has used up half its lifetime, the forms site asks how long it has left: GET /orchestra-api/presence/status. Asking is a use of the key, so it extends it. For example, with keys living 30 minutes unused, a key with 10 minutes left is extended to 30 minutes again, so Marie has time to fill the form in. A key with 25 minutes left is not asked about. So the workflow site is asked at most once every 15 minutes per person.

If the workflow site refuses a key, for example because Marie withdrew the forms site, the refusal carries the code identity_proof_expired, and the header says zero. The forms site forgets the key, and sends Marie for a fresh one before they type anything.

None of this applies to the asserted scheme. There, the forms site's word proves the person, and there is no key to expire.

Each visit to the presence page can create a key. So the workflow site can limit how often one person, and everybody together, may visit it. The limits are on its Presence settings page, and they apply while request limits are turned on.

Marie sees the sites acting for them on their account, under Sites acting for you. Withdrawing one stops its keys at once.

Whose runs a caller may read¶

A caller acts in one tenant and cannot reach another. Two callers can share a tenant, though, and that is an ordinary deployment — a staging caller and a production caller against one tenant, or two peers — so the tenant alone let each of them read and drive the other's runs.

So a run records which caller asked for it, and every read and write is scoped to that as well as to the tenant. A run belonging to another caller reads as absent rather than as refused, because a refusal would confirm that a run the caller may not see exists. A run this site started for itself carries no caller and stays reachable: the tenant governs it, and hiding it would take the API away from the site that installed it.

The caller is not the initiator. The initiator is a person, the caller is a machine, and a run can have both, either or neither.

Choosing what to run over there¶

A step that runs its child on another Orchestra takes the child's machine name, and the two sites keep their own workflows, so that name used to be typed blind — with the mistake surfacing when the first token reached the step.

A caller can now ask a remote which workflows it will start, and what each one declares its runs begin with. Only the tenant those credentials act in, and only workflows a start would be accepted for right now: one that exists but could not be started as it stands is left out rather than listed and refused later, which on this boundary also keeps the listing from answering a question the start refusal is careful not to. What a workflow returns is declared nowhere, so only the input half of a variable map can be offered from this.

Asking twice¶

A start commits on the remote and then has to travel back. A caller whose connection drops in between cannot tell a request that never arrived from one that arrived and worked, and the safe reading — ask again — is the one that runs the work twice.

So a start may carry an idempotency key: the caller's own name for the attempt. The second ask is answered with the run the first one made. What identifies an attempt is the caller, the workflow and that key together — two callers naming an attempt the same thing have named two different attempts, and so has one caller reaching for the same order number when that order needs a second workflow. Concurrent asks can still race; what this closes is the sequential case it exists for.

A key longer than 128 characters is refused rather than cut to fit, for the reason a cut key is worse than no key at all: two attempts whose keys differ only past the cut would read as one attempt, and the second caller would be handed the first one's run.

The remote_subprocess node names its own launches, because it is the one starter in Orchestra that knows what identifies one. The name is derived, never stored: a retry happens precisely when the first attempt left no usable record, and the record of the child is exactly what rolls back with the failed advance, so a name written alongside it would not be there to read. A token's first child is named after the token; a child launched to replace one that failed is named after the child it replaces, whose id was committed when that child was booked. Not after how many replacements have happened, since an operator resolving an incident sets that count back to zero and the next replacement would then ask for the very child it was replacing.

Any other caller passes its own key, or none. Orchestra cannot invent one for you: only you know what "the same request" means, and a name derived from the workflow and the variables would collapse two deliberate starts of the same thing into one and tell the second caller it had succeeded, which is worse than a duplicate run because a duplicate is visible and a wrong answer is not.

This is the shape kessai uses for a payment provider (PaymentOperationKey), for the same reasons; what differs is only what identifies an operation.

Naming a run for the event that comes later¶

A run often ends up waiting for something that arrives on its own: a payment return, an inbound message, a reply from whatever the workflow is about. What arrives carries the reference the outside world knows the run by, an order number or a case reference, and not the id Orchestra minted for it. So a start may carry a correlation key, and the engine resolves an event against it, scoped to the tenant those credentials act in and to the runs still running. The key is not unique and is nobody's to reserve, so the lookup answers with every run still carrying it rather than with one.

Send it as correlation_key on the start request. It is a start-time value and there is no write for it afterwards, so a run started without one can never be found this way. A key longer than 255 characters is refused rather than cut to fit: the column holds 255, and a key cut short is one the event would never match, which fails silently and long after the request that caused it.

It is not the idempotency key above, whatever the two names suggest. That one answers whether this request has already been served: it is read once, before anything is created, and must resolve to at most one run. This one answers which live run an event is about: it is read for as long as the run is running, and several runs may legitimately share it. A start may carry both, and they are read at different moments for different questions.

It is for a run somebody else asked for. A caller starts a run because its own system has work to track, and that system knows the work by its own reference, never by the id Orchestra minted here. Anything Orchestra hands out itself needs no key: a parked token is reached through the doorway link the site issued for it, or by its token id over the API, and whoever has to act was given that address when the token parked.

A subprocess step names its child the same way, local or remote, when the author points its correlation_variable at a parent variable holding the reference (see subprocesses). The step reads it out of the variables it already loads for the input map, and a value too long for the column stops the launch rather than being cut to fit, whether it is the first or a re-launch: the sweep that re-launches a failed child raises the incident itself rather than retrying a refusal nothing would report.

Its child_label works the same way on both nodes too: the label the lists call the child by, with tokens read off the parent's branch, sent to the far site as the start request's label (see subprocesses). So does start_variables: the node's own variables, their tokens read off the parent's branch, travel with the input map's in the one start request.

A child process on another site¶

orchestra_remote_process adds a remote_subprocess node: the remote counterpart of the subprocess node, which runs its child on the site a remote names. It carries the same input and output maps, and the parent routes on what comes back exactly as it does for a local child.

Two things differ, and both come from the hop rather than from the node.

A synchronous child needs no wait. The start is one HTTP request, so a remote running synchronously has drained the child before it answers: the node reads the result and advances in that same request. A child still running parks the parent, and a cron sweep asks each remote about the run its parent recorded, resuming the parent with the mapped output when it ends. The sweep is bounded, fifty parents a run, because each of them is an HTTP call; a cursor moves it through them, so every parent comes round. The resumes it makes are left to the queue worker rather than run inside the cron request, as every other sweep in the engine does it. A remote that gives no answer at all, because it cannot be reached or does not reply before its timeout, is called once per sweep: the parents after it that wait on the same remote are not asked again that run, and each has the unanswered ask counted as though it had been made. Fifty calls to a host that is down would otherwise hold the cron run for fifty timeouts, and every hook and queue after this one with it.

When the remote sends notices, the parent does not wait for the sweep at all: a notice that its child ended has that one parent polled as soon as this site has answered the notice. The child is still asked about, as on any sweep, so a notice that is late or repeated changes nothing the sweep would not have.

The two sites commit separately. A parent that rolls back after the start returned leaves that child standing on the other site. The start is not part of this site's transaction and cannot be made to be, which is why the node records what it started before parking: a retried advance finds the record and does not pay for the child twice.

What an ending gives the parent is the node's rule, and it is the same rule the local subprocess node follows. A failed child is re-launched while the node's relaunch_attempts budget lasts, after the relaunch_backoff delay if one is set; the sweep is what launches it, because the sweep is where this site learns the child failed at all. A canceled child is not re-launched: somebody stopped that one on purpose. With nothing left to try, a child that did not complete routes through the node's state variable, and with no state variable to route on the branch stops with an incident. The sweep raises that one even when the far side drained the child inside the start call, so the answer does not depend on how fast the child was.

A parent that moves on without its child, because a timeout resumed it, an operator canceled the branch, or another branch superseded it, has that child canceled for it. The engine does this itself for a local child, as part of canceling the token that launched it; a child on another site is out of its reach, so the sweep ends it instead.

That is the one thing in the sweep worth trying again. A lost resume is caught by the next poll, but a lost cancellation leaves a run alive on the other site for good, holding its tasks open with nobody reading them. So an unreachable remote is retried on the site's own advance budget, the same number of goes the engine gives a step before it stops trying; once that is spent the parent stops being asked about and what was left behind is recorded in the audit log and the Orchestra channel, naming the remote and the run.

A child that had already finished over there is not a failure and is not retried: the parent moving on just after its child ended is an ordinary race, which is why the sweep reads the child's state before canceling anything rather than reading a refusal as an unreachable host.

An ask that never brings anything back is eventually an operator's decision. That covers a host that stops answering and a host that answers to say it no longer has the run, purged there or reached through a remote now pointing elsewhere; neither leaves the parent anything to resume with. One failed ask is a host down for a minute, and a parent should wait through it. A host that never answers is a step stuck for ever with nothing saying why, so the failures are counted and, on the same budget, the parent is dead-lettered with an incident naming the remote and the run it was waiting on. That is what puts the decisions on the table: retry once the host is back (the record means a retry waits on the same child rather than starting a second one), resume the step with variables of your own, which is deliberately going on without the child's result, skip the step, cancel the branch, or fail the run. A parked step offers none of that, which is why it is halted rather than left waiting. While the incident is open the sweep leaves that parent alone, including its child: ending it is one of the decisions, not something to do before the operator has made one. A remote that answers again clears the failures counted before it, and so does the retry itself, which gives the parent its budget back the way the engine gives a re-queued token its attempts back.

A remote is not deleted while a workflow still calls it: the delete page lists the callers and refuses, the way the kernel refuses a workflow named as a subprocess, because the reference lives in node configuration where the dependency graph cannot see it. Disabling is how a remote is taken out of service instead, one click from the list, and a step naming a disabled remote fails and says so rather than reaching a host that is no longer meant to answer. The notices it posts are refused too, so a retired host no longer writes into the parents waiting on its children. The workflows keep the name, so enabling it again is all it takes to put them back to work. Disabling the site's default stops more than the steps that name it, because code naming no remote reaches that one: nothing is promoted in its place, those calls fail until it is back or another is marked as the default, and the list says so as it disables it.

What it records is the remote and the child's id there, never the id alone. Instance ids are minted per site, so 42 from another Orchestra names one of this site's runs just as well as its own.

Routing on what the child returned works as it does for a local child, and the caution there applies with more force here: the two sites' workflows are unrelated, nobody can open the child from this side to see what it writes, and a name that drifts on the far side simply stops arriving. Give "the child returned nothing" a flow of its own, on not_set. See Subprocesses.

The child's tenant is the tenant that remote authenticates into, which is decided on the far side; and the two sites keep their own workflows, so the node names the child by machine name rather than choosing from a list.