Skip to content

Incidents

An incident is an operational failure that halts one branch of a running process and flags it for a human to resolve, rather than tearing the whole process down. It is Orchestra's answer to the "poison token" problem: a task that throws every time it runs.

How a task failure is handled

When a token's task throws, the engine classifies the failure and reacts to it, rather than blindly retrying. A task signals the kind of failure by the exception it throws (see Extending Orchestra):

  • Retryable (RetryableException): a transient failure a later attempt may clear (a provider not settled yet, a brief outage). The engine retries the advance per the node's retry policy, and only once the attempts are exhausted without success does it dead-letter the token to an incident.
  • Outcome (OutcomeException): not an error but a definite outcome the workflow should handle. The exception carries an outcome value (and an optional reason); the engine records the outcome on the node's payload variable (and the reason on its message variable, if it declares one) and advances the token, so its outgoing flows route the outcome. No retry, no incident.
  • Fatal, or any unmarked exception (FatalException, or any other throwable): a failure a retry cannot fix, or one the task did not classify. The engine dead-letters the token to an incident at once, without retrying. Surfacing an unclassified failure immediately is safer than guessing it is transient (and cheaper than retrying something that cannot recover).

Dead-lettering moves the token to the terminal error state so it is no longer retried.

What dead-lettering does

What happens next is set by When a step fails for good, under When a step cannot go on on the Orchestra settings page (the on_unrecoverable_failure setting):

  • incident (the default): an incident is raised for the failed branch and the instance keeps running. Its other branches advance normally; only the failed branch is parked, in error, waiting for an operator. The instance cannot complete while an incident is open.
  • fail: the whole instance is failed and its remaining live tokens are canceled, the older all-or-nothing behavior.

Failing the whole process for one recoverable branch is usually the wrong response, so incident is the default.

A branch that dead-ends on its conditions

A task failure is not the only way a branch stops. A token that reaches a node which has outgoing flows and takes none of them (every condition evaluated false) has no successor either, so the branch ends there, and once it was the last live branch the whole run completes. Nothing failed, so without help nothing would be recorded: no incident, no status, no log entry, and the run reads exactly like a successful one.

That shape is always a fault, in the model or in the data:

  • a typo in a condition, or a condition comparing a variable that no longer exists under that name;
  • a variable a task or provider failed to write, so the comparison reads NULL;
  • an outcome value no flow matches (a task returning escalated where the flows route approved and rejected only);
  • a gateway whose branches are all guarded;
  • a split creating one branch per item that listed no item, such as a per_member split whose audience has no member (see One branch per item). The fix is in the items: give the audience a member, then Resume.

So the engine treats it as a dead-lettered branch: the token moves to error and an incident is raised, naming the node, every condition it evaluated and the value each one saw. That is usually the whole diagnosis at a glance:

Dead end at node n_validate: every outgoing flow condition evaluated false, so
the branch ends without reaching an end node. Evaluated: flow f_approve:
comparison validation.outcome == "approved" (saw NULL); flow f_reject:
comparison validation.outcome == "rejected" (saw NULL).

A condition tree is rendered the same way, down to the leaf that did not hold:

flow f_yes: any of [all of [comparison approved == TRUE (saw TRUE) [true],
comparison amount > 1000 (saw 500) [false]] [false], comparison vip == TRUE
(saw FALSE) [false]]

A node with no outgoing flows at all is untouched: that is a legitimate end (an end node, or any branch exit), and it completes as always.

The incident is a normal incident, with the usual actions, but their meanings differ a little here since nothing threw:

  • Resume is the usual fix: correct the variable the condition reads, and the node's flows are evaluated again.
  • Retry re-runs the node, which is what you want when the cause was upstream (a task that wrote the wrong value).
  • Cancel branch abandons the branch deliberately, letting the rest of the instance finish.
  • Skip re-evaluates the same flows, so unless the variables changed the branch dead-ends and reports again.

A site that ends branches on guarded flows on purpose can opt out with When a branch takes none of its outgoing flows, in the same section of the settings page (the on_dead_end setting):

  • incident (the default): raise the incident described above.
  • complete: consume the token and let the run complete. The dead end is still logged as a warning, with the same diagnostic, so it is never fully silent.

Prefer modeling the deliberate case explicitly instead: give the node a catch-all outgoing flow (unconditional, or the negation of the others) to an end node. Then the branch ends because the model says so, and a real dead end is still reported.

A child that ended with nowhere to send the run

A subprocess step waits on its child, and when the child completes the parent resumes with the child's mapped output. A child that fails or is canceled is different: there is no output to carry back, and what the parent does about it is the node's state_variable. Named, the parent resumes with __subprocess_failed__ or __subprocess_canceled__ in it and an outgoing flow condition routes the run.

With no state variable there is nothing to route on, and advancing the parent anyway would carry the run past a step whose work never happened, down whatever arrow follows, as if the child had worked. So the branch stops. Stopping silently would leave a run that can never move again with nothing anywhere saying why, so it stops with an incident. The child is already over, so there is nothing to wait for and nothing gained by waiting.

The same holds whether the child ran here or on another Orchestra. For a remote child the sweep is what raises it, including when the far side drained the child inside the start call: answering it there would make that one case an advance failure, and so subject to the site's failure policy, where the same child failing a minute later would raise an incident regardless.

A second, rarer way into the same incident: the node stopped being a subprocess step while the parent was parked on it, which a site running without version pinning allows. Nothing can then say what the child's ending means, so the branch stops the same way. Pinning runs to a version is what prevents it.

Resolving it. Retry runs the step again, which means another child (a new run on the other site, for a remote child), with the node's re-launch budget handed back whole. Resume writes the variables you give it and then runs the step again too, so it also means another child: it corrects what the next attempt reads, rather than standing in for the child that failed. Skip is what carries the run past the step without running it again, and Cancel branch abandons the branch. The lasting fix is usually to give the node a state variable and an outgoing flow for the failed case, which turns this from an incident into a branch the model handles.

A step nobody can take

A human step has to have somebody to offer itself to. When the audiences a node names resolve to nobody, the step is held closed rather than opened to everyone, because a restriction that found no one is not an invitation to the whole site. Held closed and left there, though, the work simply stops being anybody's, so that is an incident too, and for the same reason: it is the object an operator can act on.

It is caught at both moments it can happen.

When the step is reached. The engine resolves the node's audience before the token parks. If it comes back with nobody, the token halts and an incident is raised there and then, so no task is created for anybody to stumble on:

Node n_review names an audience that matched nobody, so the step could not be
offered to anyone. Fix the audience and retry, or resume with the variable it
reads.

Afterwards, when the audience empties. A task's candidates are resolved once and then frozen on it, so a pool can empty later: the last member leaves the role a task is pooled to, or every account it names is blocked or deleted. The task is then offered to tokens nobody carries, which shows in no inbox and can be claimed by nobody, while everything about the run still looks healthy. The cron recovery sweep asks each audience whether its own tokens still name anybody, and halts a step whose every candidate came back with nobody. The halt takes the task nobody could see with it, and is raised as an incident:

Node n_review was offered to Editor, which no live account carries any more, so
nobody is left to take the step. Widen the audience and retry.

The audience is named the way every task list names it: each dimension reads its own tokens back into labels, so the role above reads as its own label. An account that has been deleted leaves nothing to read, so its user: token stands in for the name nothing can supply any more.

A step is only reported when every candidate it carries has emptied: an audience is a union, so one token that still names somebody keeps the step reachable. A token from a custom assignment plugin that does not answer the question (see Extending Orchestra) is never reported as empty, and one such token keeps its whole step out of the sweep, since nothing can say the pool behind it is gone.

The repair is Retry once the audience is fixed: it re-runs the node, which resolves the audience afresh against the accounts as they now are. For an audience read from a process variable, Resume writes the corrected variable and retries in one action. Skip and Cancel branch are there for a step that turns out not to be needed after all.

A join that can no longer fire

A join that waits for its branches can find that the ones it still needs will never come. Firing anyway would pass on fewer branches than the workflow asks for, and waiting would hang the run with nothing to show for it, so the branch waiting there halts on an incident instead. It happens two ways.

A threshold join short of its count. Every branch the join counts has arrived, and they are fewer than its count: an inclusive split ran two reviewers where the join requires three approvals, say. The branch whose arrival settles it halts:

Join n_decision can no longer fire: every branch it counts has arrived (2), and
that is fewer than it requires. Fire the join to go on with the branches that
arrived, or lower its count (by editing the workflow, or migrating the instance
to a version that does) and retry.

A wait_all join missing a path drawn into it. A path into the join will not deliver: a split did not take its flow towards the join, a token went where the join can no longer be reached, or a token was canceled on the way. As soon as that happens, the join halts, while the other branches may still be running. The message says what happened (a matching join goes on without such a path, so it does not halt this way):

Join n_join can no longer fire: a path drawn into it will not deliver (the flow
from n_c to n_x was not taken). Fire the join to go on with what arrived, or
cancel the instance.

If the path was not meant to be required, the join should be a matching join.

Firing the join from this incident deals with the branches still running as its When it fires setting says: canceled, or left to finish and ignored when they arrive.

Another join can end up waiting for a branch that will never come without noticing it at once, for example after a branch was canceled straight in the database. Once nothing else in the run can move, one of the branches waiting at the join halts, on behalf of them all:

Join n_join can no longer fire: it waits for a branch that will never arrive
(canceled or ended elsewhere). Fire the join to go on with the branches that
arrived, or give it a join timeout (by editing the workflow, or migrating the
instance to a version that does) and retry.

A join whose timeout gave no deadline (its duration is missing, or is not a duration) is named as such instead, since giving it a timeout is not the fix. So is a branch the join has already fired without: a second token of a branch the fire already counted, from a timer that sent a branch twice down the same flow, say, with nothing left to pair with. Cancel branch drops it; firing the join would run the join's step, and everything after it, a second time. While a branch of the run is halted on an incident of its own, such a join raises nothing: retried, that branch may still arrive. Canceling it is what strands the join.

The actions mean this here:

  • Fire the join takes the place of Skip: the join fires with the branches that arrived, as if it had decided to, and the step's own work then runs.
  • Retry asks the join again, which is the fix once the instance runs a corrected join, a lower count or a timeout: an edit to the workflow reaches it only when it pins no version, so an instance that does is migrated first (see Versioning). Asked again unchanged, it halts again.
  • Resume writes corrected variables, then asks again. A quorum join tests its condition again on the halted branch only: each branch already waiting keeps the result it arrived with, whatever the variables say now.
  • Cancel branch and Fail instance as for any incident.

A token that reaches a join outside its split

A join counts the branches of one split, its opening split. A token that reaches the join without being in a branch of that split cannot be counted. It halts there:

Token 42 reached join n_join without a branch of split n_s, whose branches the
join counts. The workflow changed under the instance, for example by a
migration to a version that draws the split elsewhere. Cancel the branch.

A workflow with a path to the join that avoids the split is refused when it is saved. So this happens only when the model changed under a running instance. The usual cause is a migration to a version that draws the split somewhere else. The dry run of the migration warns about it, naming the join (see Versioning).

  • Cancel branch is the way out: the token belongs to no pass of the join, so nothing waits for it.
  • Retry asks the join again. Unless the instance now runs a version where the token is in a branch of the split, it halts again.
  • Fail instance as for any incident.

A join that refuses values that are not numbers

A collect line that computes a number (sum, avg, min or max) takes numbers only. When the join would fire and such a line reads a value that is not a number, the join does not fire: the branch whose arrival would have fired it halts, naming each value, the line that read it and the flow it came down:

Join n_budget did not fire: a collect line that computes a number read values
that are not numbers: amount|total_amount|sum read "to be confirmed" (text)
from flow f_dept_b. Fire the join without the values that are not numbers to
compute the line from the numbers alone, or correct the values or the workflow
and retry.

The other branches stay waiting at the join. The actions mean this here:

  • Fire the join without the values that are not numbers takes the place of Skip: the join fires, its lines are computed from the numbers alone, each value left out is logged, and the step's own work then runs. Only this action leaves a value out: Fire the join on a join that can no longer fire meets the same refusal, on an incident of its own.
  • Retry asks the join again, which is the fix once the instance runs a corrected line (list keeps any value), or once the values are numbers. Asked again unchanged, it halts again.
  • Resume writes corrected variables instance-wide, then asks again. A value a branch set for itself (token-scoped) is not corrected this way: the branch still reads its own.
  • Cancel branch and Fail instance as for any incident.

Bringing a retry forward

While a retryable failure is between attempts (waiting out its backoff, or for the next cron run), the token carries a spent attempt but is not yet an incident. Its row on the process instance page offers a Retry now action (for a user with the Resolve Orchestra incidents permission): it runs the pending retry immediately, so once the cause is fixed an operator does not have to wait for the backoff or cron. It is safe against the pending retry racing it, the advance is claimed exactly once, and the attempt still counts toward the node's limit, so a persistent failure still dead-letters to an incident.

A notification that could not be sent

An incident is also raised when a notification is given up and its notification profile says to raise one, which the shipped Important profile, and so every interaction link by default, does. The process may be waiting on someone the notification never reached, and an open incident is where operators look and what keeps the instance from completing. Its token is not in error: it is parked, waiting on that person, or has moved on. So the incident offers its own actions:

  • Send again: sends the notification anew, to the recipients it did not reach, as its profile says. An interaction link whose step has moved on since is not sent: it would open onto nothing.
  • Leave unsent: resolves the incident without sending, for an operator who reached the recipient another way.
  • Fail instance: as for any incident.

Resolving it lets the instance complete if nothing else is outstanding. A notification given up once its process has ended raises no incident, since nothing waits on it; it is recorded in the process's history instead.

Resolving an incident

Structure » Orchestra » Incidents lists the open incidents of every running instance, newest first, for whoever may see the instances list; an operator scoped to a tenant sees that tenant's only. Each row names its instance, which also lists its own incidents on its page.

A user with the Resolve Orchestra incidents permission sees an incident's recovery actions on both pages. Each acts on just that branch:

  • Retry: clears the failure count and puts the token back on its node, so the node runs again. Use it once the cause has been fixed. A step that waits for its branches to meet does not wait for them again if they already met there: they are gone, so the retry goes straight to the step's own work. If the failure undid their meeting, the step meets them again.
  • Resume: corrects one or more process variables, then retries. Alone among the five it opens a form first, showing the instance's variables as editable fields; the corrections are written and the branch retried on submit.
  • Skip: advances past the failing node without running its task, for when the step is no longer needed. At a join that can no longer fire, it reads Fire the join, and fires it instead. At a join that refuses values that are not numbers, it reads Fire the join without the values that are not numbers.
  • Cancel branch: abandons just the failed branch, letting the rest of the instance finish.
  • Fail instance: the explicit "give up on this run", failing the whole instance.

Resolving the last open incident lets the instance complete if nothing else is outstanding.

An incident can also be resolved without an operator. When a threshold or quorum join, or a join timeout, fires without waiting for the halted branch, and its setting cancels that branch, the branch is canceled and its incident resolved as canceled, as canceling the instance does (see After a fire).

The three that put the branch back in play, retry, resume and skip, carry the run as far as it goes in the same request when the instance runs synchronously, which is the shipped default: the operator is shown where the run got to, not where it failed, and a run with nothing else outstanding is already complete by the time the page comes back. A queued instance leaves the work for cron, as every other advance does in that mode. Cancel branch and fail instance leave no live token, so there is nothing to carry.

From code

The recovery actions are methods on WorkflowEngineInterface, so anything (a controller, a Views bulk action, a remote integration) can drive them:

$engine->retryIncident($incident, $uid);
$engine->resumeIncident($incident, ['amount' => 42], $uid);
$engine->skipIncident($incident, $uid);
$engine->cancelIncident($incident, $uid);
$engine->failFromIncident($incident, $uid);

// Any operation the incident's kind offers, by its name.
$engine->resolveIncident($incident, 'retry', $uid);

Each incident is of a kind, an IncidentKind plugin that says which operations it offers and what each does: token for a branch that failed (the five above), notification for a notification that could not be sent (retry sends it again, skip leaves it unsent, fail). An operation a kind does not offer is refused. A module adds a kind of its own by declaring the plugin, #[IncidentKind(id: 'my_kind', label: ...)] on a class extending IncidentKindBase that implements getOperations(), resolve() and getSummary(), and raising its incidents with IncidentStore::raise($instance, 'my_kind', $message, ...), whose details hold what it needs to resolve them. The instance page then lists them with their operations, and an open one keeps the instance from completing like any other. An operation that takes values, as a resume takes the corrected variables, needs a form of its own; every other one is applied as it is clicked.

Bringing a retry forward (before it becomes an incident) is a token operation:

// Re-run a token that failed and is waiting on a retry, now.
$engine->retryToken($token);

Incidents are orchestra_incident content entities, tenant-scoped like the rest of the runtime and exposed to Views as a base table, so a multi-tenant dashboard of open incidents is just a view.

Retry policy

How many times a failing advance is retried before dead-lettering is configurable. The site default is Attempts before a step is given up on, in the same section of the settings page (the max_advance_attempts setting), which is 3. A node can override it, and add a backoff between retries, through its Retry policy in the modeler:

  • Max attempts: the cutoff for this node; empty uses the site default.
  • Backoff: a delay between retries, in seconds or as an ISO-8601 duration (e.g. PT5M); empty retries immediately. The delay is honored by cron's queue runner.

Configuration

All three are set on the Orchestra settings page, under When a step cannot go on, and stored as:

# orchestra_engine.settings
on_unrecoverable_failure: incident   # or: fail
on_dead_end: incident                # or: complete
max_advance_attempts: 3              # at least 1

A node overrides the cutoff, and adds a backoff, on itself:

# a workflow node, under features.retry
features:
  retry:
    max_attempts: 5
    backoff: PT2M

A cutoff below one would mean a step is given up on before it has been tried, which the engine will not do whatever is stored: a node's override below one is ignored and the site-wide number is floored at one. Rather than leave it at that, the site-wide setting is refused below one, by the settings page and by the configuration schema both, because the page is one door onto it and an import or a drush config:set is another.