Incidents

An incident is an operational failure that halts one branch of a running process and flags it for a human to resolve, rather than tearing the whole process down. It is Orchestra's answer to the "poison token" problem: a task that throws every time it runs.

How a task failure is handled

When a token's task throws, the engine classifies the failure and reacts to it, rather than blindly retrying. A task signals the kind of failure by the exception it throws (see Extending Orchestra):

• Retryable (RetryableException): a transient failure a later attempt may clear (a provider not settled yet, a brief outage). The engine retries the advance per the node's retry policy, and only once the attempts are exhausted without success does it dead-letter the token to an incident.
• Outcome (OutcomeException): not an error but a definite outcome the workflow should handle. The exception carries an outcome value (and an optional reason); the engine records the outcome on the node's payload variable (and the reason on its message variable, if it declares one) and advances the token, so its outgoing flows route the outcome. No retry, no incident.
• Fatal, or any unmarked exception (FatalException, or any other throwable): a failure a retry cannot fix, or one the task did not classify. The engine dead-letters the token to an incident at once, without retrying. Surfacing an unclassified failure immediately is safer than guessing it is transient (and cheaper than retrying something that cannot recover).

Dead-lettering moves the token to the terminal error state so it is no longer retried.

What dead-lettering does

What happens next is set by the on_unrecoverable_failure setting:

• incident (the default): an incident is raised for the failed branch and the instance keeps running. Its other branches advance normally; only the failed branch is parked, in error, waiting for an operator. The instance cannot complete while an incident is open.
• fail: the whole instance is failed and its remaining live tokens are canceled, the older all-or-nothing behavior.

Failing the whole process for one recoverable branch is usually the wrong response, so incident is the default.

A branch that dead-ends on its conditions

A task failure is not the only way a branch stops. A token that reaches a node which has outgoing flows and takes none of them (every condition evaluated false) has no successor either, so the branch ends there, and once it was the last live branch the whole run completes. Nothing failed, so without help nothing would be recorded: no incident, no status, no log entry, and the run reads exactly like a successful one.

That shape is always a fault, in the model or in the data:

• a typo in a condition, or a condition comparing a variable that no longer exists under that name;
• a variable a task or provider failed to write, so the comparison reads NULL;
• an outcome value no flow matches (a task returning escalated where the flows route approved and rejected only);
• a gateway whose branches are all guarded.

So the engine treats it as a dead-lettered branch: the token moves to error and an incident is raised, naming the node, every condition it evaluated and the value each one saw. That is usually the whole diagnosis at a glance:

Dead end at node n_validate: every outgoing flow condition evaluated false, so
the branch ends without reaching an end node. Evaluated: flow f_approve:
comparison validation.outcome == "approved" (saw NULL); flow f_reject:
comparison validation.outcome == "rejected" (saw NULL).

A condition tree is rendered the same way, down to the leaf that did not hold:

flow f_yes: any of [all of [comparison approved == TRUE (saw TRUE) [true],
comparison amount > 1000 (saw 500) [false]] [false], comparison vip == TRUE
(saw FALSE) [false]]

A node with no outgoing flows at all is untouched: that is a legitimate end (an end node, or any branch exit), and it completes as always.

The incident is a normal incident, with the usual actions, but their meanings differ a little here since nothing threw:

• Resume is the usual fix: correct the variable the condition reads, and the node's flows are evaluated again.
• Retry re-runs the node, which is what you want when the cause was upstream (a task that wrote the wrong value).
• Cancel branch abandons the branch deliberately, letting the rest of the instance finish.
• Skip re-evaluates the same flows, so unless the variables changed the branch dead-ends and reports again.

A site that ends branches on guarded flows on purpose can opt out with the on_dead_end setting:

• incident (the default): raise the incident described above.
• complete: consume the token and let the run complete, the behavior before this policy existed. The dead end is still logged as a warning, with the same diagnostic, so it is never fully silent.

Prefer modeling the deliberate case explicitly instead: give the node a catch-all outgoing flow (unconditional, or the negation of the others) to an end node. Then the branch ends because the model says so, and a real dead end is still reported.

Bringing a retry forward

While a retryable failure is between attempts (waiting out its backoff, or for the next cron run), the token carries a spent attempt but is not yet an incident. Its row on the process instance page offers a Retry now action (for a user with the Resolve Orchestra incidents permission): it runs the pending retry immediately, so once the cause is fixed an operator does not have to wait for the backoff or cron. It is safe against the pending retry racing it, the advance is claimed exactly once, and the attempt still counts toward the node's limit, so a persistent failure still dead-letters to an incident.

Resolving an incident

A user with the Resolve Orchestra incidents permission sees an incident's recovery actions on the process instance page. Each acts on just that branch:

• Retry: clears the failure count and re-queues the token, so its node runs again. Use it once the cause has been fixed.
• Resume: corrects one or more process variables, then retries. Alone among the five it opens a form first, showing the instance's variables as editable fields; the corrections are written and the branch retried on submit.
• Skip: advances past the failing node without running its task, for when the step is no longer needed.
• Cancel branch: abandons just the failed branch, letting the rest of the instance finish.
• Fail instance: the explicit "give up on this run", failing the whole instance.

Resolving the last open incident lets the instance complete if nothing else is outstanding.

From code

The recovery actions are methods on WorkflowEngineInterface, so anything (a controller, a Views bulk action, a remote integration) can drive them:

$engine->retryIncident($incident, $uid);
$engine->resumeIncident($incident, ['amount' => 42], $uid);
$engine->skipIncident($incident, $uid);
$engine->cancelIncident($incident, $uid);
$engine->failFromIncident($incident, $uid);

Bringing a retry forward (before it becomes an incident) is a token operation:

// Re-run a token that failed and is waiting on a retry, now.
$engine->retryToken($token);

Incidents are orchestra_incident content entities, tenant-scoped like the rest of the runtime and exposed to Views as a base table, so a multi-tenant dashboard of open incidents is just a view.

Retry policy

How many times a failing advance is retried before dead-lettering is configurable. The site default is the max_advance_attempts setting (3). A node can override it, and add a backoff between retries, through its Retry policy in the modeler:

• Max attempts: the cutoff for this node; empty uses the site default.
• Backoff: a delay between retries, in seconds or as an ISO-8601 duration (e.g. PT5M); empty retries immediately. The delay is honored by cron's queue runner.

Configuration

# orchestra.settings
on_unrecoverable_failure: incident   # or: fail
on_dead_end: incident                # or: complete
max_advance_attempts: 3

# a workflow node, under its retry key
retry:
  max_attempts: 5
  backoff: PT2M