Jobs for programs¶

The optional Orchestra Job submodule (orchestra_job) lets a program outside Drupal do the work of a step. The step offers a job; a program asks the HTTP API for the jobs of a topic, takes some, does the work, and completes or fails each one.

The program calls Orchestra, never the other way round. It can run anywhere it can reach the site: behind a firewall, on a GPU machine, on a server Orchestra cannot see. It takes only as many jobs as it can handle, and needs no web address of its own.

Compare the HTTP request step, where Orchestra calls the other system, which must then be a web server Orchestra can reach.

An example¶

An invoice arrives. A program reads the scan, and the workflow routes on the total it found:

flowchart LR
  s([Start]) --> o["Read the invoice<br/>type: job, topic: invoice_ocr"]
  o -->|ocr.total < 500| a([Approved directly])
  o -->|otherwise| m([Manager approval])

n_read_invoice:
  type: job
  config:
    topic: invoice_ocr
    data: "scan_url\nsupplier|from"
    payload_variable: ocr
  features:
    retry: { max_attempts: 3, backoff: PT10M }

The program, in Python, with a consumer allowed to drive the API:

import requests, time

API = "https://orchestra.example/orchestra-api"
headers = {"Authorization": f"Bearer {get_token()}"}

while True:
    answer = requests.post(f"{API}/jobs/take", headers=headers,
                           json={"topic": "invoice_ocr", "max": 5, "lease_seconds": 600})
    jobs = answer.json()["jobs"]
    if not jobs:
        time.sleep(10)
        continue
    for job in jobs:
        try:
            total = read_total(job["data"]["scan_url"])
            requests.post(f"{API}/job/complete", headers=headers,
                          json={"handle": job["handle"], "data": {"total": total}})
        except UnreadableScan as e:
            requests.post(f"{API}/job/fail", headers=headers,
                          json={"handle": job["handle"], "message": str(e)})

The instance resumes with ocr.total. An unreadable scan fails the job: the step runs again ten minutes later, up to three times, then halts on an incident.

The step¶

Setting	What it does
Topic (`topic`)	The kind of work, as a machine name such as `invoice_ocr`. The programs taking the jobs of this topic do it.
Job data (`data`)	The variables sent with the job, one per line: `name`, or `name\\|field` to send it as another field. A variable the instance does not carry is left out.
Payload variable (`payload_variable`)	Where what the program sends back is written, whole or one variable per field, as for an event's data.

The step parks like a person's step does, so its timeout and timers and the instance's handlers work on it. A job nobody takes waits until one of them moves the step on.

The API¶

Three doors, under the drive orchestra api permission, for a consumer bound to a tenant as for the rest of the HTTP API. A program sees only its tenant's jobs. The full contract is in openapi.yaml.

POST /orchestra-api/jobs/take with topic, max (10 by default, 100 at most) and lease_seconds (300 by default, a day at most). Answers the oldest jobs of the topic that nobody holds, each with its handle, its data and when its lease runs out.
POST /orchestra-api/job/complete with handle and data, at most 64 KB as JSON, as an event's data.
POST /orchestra-api/job/fail with handle, message, and retry (true by default).

The handle travels in the body, never in a URL: it is a secret, and URLs end up in logs.

Leases¶

A job taken is leased: no other take gets it until the lease runs out. A program that dies leaves its jobs to run out, and the next take hands them to another program. Ask for a lease longer than the work takes.

Two programs taking at once never get the same job. The one that loses gets fewer jobs, or none.

A lease that ran out does not stop the program answering: its handle still works until another program takes the job. Then the late one is answered 404, and only the new handle completes the job.

Failing a job¶

Failing a job fails the step, as a step whose task throws (If a step fails). With retry, the step runs again while its retry policy has attempts left, and offers its job again then. A step runs once by default (max_parked_step_attempts), so give it a retry policy to have it run again. Then, or at once with retry: false, its failure handling decides: an incident, a failed instance, or its failure outcome.

A job is never handed out while its step waits for a retry.

A step that no longer waits¶

A step canceled, timed out or moved on by a handler no longer waits for its job. No take hands the job out: the first take to reach it, among the oldest jobs nobody holds, deletes it instead. A program still holding it gets 409 when it answers.

A job is deleted with its instance, a retention purge included, so the data it copied from the instance's variables goes with them.

A step moved to a new version¶

Say version 3 of a workflow renames the topic of its step from invoice_ocr to invoice_scan, and stops sending the supplier. You migrate the instances parked on that step from version 2 to version 3. Their jobs move to invoice_scan, with only the invoice: programs taking invoice_ocr no longer see them.

After a migration, each job step still parked compares its job with the step it now stands on, the same node or the node you mapped it to:

Another topic, or other data: the job is offered again, under the new topic and with the data the new step sends, read from the variables as they are at the migration. It has no lease any more: a program that took it before gets 404 when it answers, and a program taking the new topic gets it.
The same topic and data: the job stays as it was, lease included. The program holding it answers it as before.
A job failed with a retry: nothing is offered while the step waits for its retry. When the step runs again, it offers its job from the step it now stands on.
No job yet: a step mapped onto a job step from a step of another type offers its job at the migration.

A step moved onto a step of another type has no job any more. Its old job stays under the old topic: a program that takes it gets 409 when it answers, as for a step that no longer waits.