Durable Workflows

A sequence of steps that survives a crash, a deploy, and a step that fails on its third attempt at four in the morning. You declare the steps as JSON; we persist the run and advance it one step at a time, so it picks up exactly where it left off.

What it is, and when to reach for it

A definition is a versioned, ordered list of steps. A run is one execution of a definition. Both a run and each of its steps are rows in our control plane, and the engine advances a run by exactly one bounded step per tick, writing the step's outcome and the run's new position in the same transaction. That single property is what the word durable buys you:

• Resumable across restarts. A run interrupted by a crash or a deploy is picked up again by whichever instance claims it next, at the step it had reached, not from the beginning.
• Retries are built in. Each step has its own attempt budget and exponential backoff. Between attempts the run costs nothing: it is a row with a wake-up time.
• Waiting is free. A step can sleep for a month or park until an external event arrives. Nothing is held open while it waits.
• At-least-once, not exactly-once. If a worker dies after a step's side effect landed but before its outcome was committed, that step runs again. Write every step so that running it twice is harmless. The engine guarantees a step's recorded outcome is written once, not that the world outside it is touched once.

Reach for a workflow when you have a sequence with real state between the parts: charge → provision → email, an onboarding drip with days of waiting in it, a nightly ETL whose third stage sometimes times out, a saga that must be unwound if a later step fails. For a single scheduled side effect, a cron job is simpler and cheaper.

Authentication

Everything here is the project management API, not a data-plane API: it answers to an owner or admin session token for the project's organization, the same credential the dashboard and the CLI use. Project API keys (kbp_…, kbs_…) are not accepted.

Authorization: Bearer <your-session-token>

Every query is scoped by project, so a run id from another project, or from another workflow of the same project, is a 404, never a cross-tenant read or write.

Defining a workflow

A definition is published with PUT. The workflow's name is in the path and must match ^[a-z0-9][a-z0-9_-]{0,63}$. The body is { "enabled": true, "steps": [ … ] }; enabled defaults to true when omitted.

curl -X PUT https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "steps": [
      {
        "key": "charge",
        "kind": "webhook",
        "target": "https://payments.example.com/charge",
        "payload": {"amount": 4900, "currency": "eur"},
        "secret": "a-signing-secret",
        "max_attempts": 5,
        "timeout_seconds": 10,
        "compensate": {
          "kind": "webhook",
          "target": "https://payments.example.com/refund",
          "payload": {"amount": 4900, "currency": "eur"}
        }
      },
      {
        "key": "provision",
        "kind": "sql",
        "target": "update accounts set active = true where id = 42"
      },
      { "key": "cool_off", "kind": "sleep", "sleep_for": "30m" },
      {
        "key": "notify",
        "kind": "enqueue",
        "target": "emails",
        "payload": {"template": "welcome"}
      }
    ]
  }'

{"name":"order-pipeline","version":1,"enabled":true}

Versions are immutable

There is no edit. A PUT always publishes version max + 1, so a run that is already in flight keeps executing the exact version it started on and cannot be changed underneath itself. A new run resolves the highest enabled version at the moment it starts.

GET  /v1/projects/{ref}/workflows            # every workflow, newest version, step count
GET  /v1/projects/{ref}/workflows/{name}     # the newest version's full step list
PUT  /v1/projects/{ref}/workflows/{name}     # publish the next version  -> 201
PATCH /v1/projects/{ref}/workflows/{name}    # { "enabled": false }      -> 204

Step kinds

There are nine. Four do something (and are the only kinds allowed as a fan-out child or as a compensating action); the rest are control flow or waiting.

A run's output is the output of the last step that produced one, so the final step's result is what a completed run reports.

Attempts and timeouts

What validation enforces

The whole definition is checked before it is stored; a violation is a 400 (invalid_definition) naming the offending step, so a workflow the engine could not execute is refused at authoring time rather than halfway through a run.

• Keys. Every step needs a key matching ^[a-z0-9][a-z0-9_-]{0,63}$ (lower-case letters, digits, _, -; max 64 characters), and every key must be unique across the whole definition, fan-out children included. A key may not contain :, which is how derived internal keys can never collide with yours.
• Size. 1 to 200 steps. A target is at most 8 KiB.
• A fan-out must be immediately followed by its join, and a join must be immediately preceded by a fan-out. They are a pair, in adjacent positions; a join may not declare children.
• Children nest exactly one level. A fan-out child must be enqueue, sql, webhook or function, so a fan-out cannot contain another fan-out, a sleep, or anything else that would have to park.
• A sub-workflow may not name its own definition, and only a fanout step may declare children.
• Per-kind targets. A queue name matches ^[a-z][a-z0-9_]{0,47}$; a function name and a workflow name match ^[a-z0-9][a-z0-9_-]{0,63}$; a webhook target must be an http(s) URL with a host; a sql target must be non-empty; a sleep needs a positive sleep_for of at most 30 days; a wait_for_signal needs a signal matching ^[a-z0-9][a-z0-9_.-]{0,63}$.
• Compensation is only allowed where there is something to undo; see Compensation.

Fan-out and join

A fanout spawns its children, and the paired join holds the run until every one of them is finished. The children are unordered and each carries its own attempt budget; the engine advances up to eight runnable children per tick, so a wide fan-out drains over several ticks rather than all at once.

{
  "steps": [
    {
      "key": "notify_all",
      "kind": "fanout",
      "children": [
        {"key": "email",  "kind": "enqueue",  "target": "emails", "payload": {"template": "receipt"}},
        {"key": "crm",    "kind": "webhook",  "target": "https://crm.example.com/hooks/order"},
        {"key": "search", "kind": "function", "target": "reindex-order", "max_attempts": 5}
      ]
    },
    {"key": "notified", "kind": "join"}
  ]
}

When the barrier clears, the join's output is an object keyed by child step key: {"email":{"msg_id":7},"crm":{"delivered":true},"search":…}. If any child exhausts its attempts, the join fails and the run fails; there is no partial-success policy.

Running a workflow

Four things can start a run. All four resolve the workflow's highest enabled version at start time, and all four write the same kind of run row.

1. The API

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline/runs \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{"input": {"order_id": 42}, "idempotency_key": "order-42"}'

{"id":"6f1c…-…","status":"pending"}

Both fields are optional: input defaults to {} and must be a JSON value; idempotency_key is at most 200 characters.

2. Cron

A cron job with target_kind: "workflow" starts a run of the named workflow on every fire, with the job's payload as the run input. Each scheduled fire is deliberately a distinct run (no idempotency key), so a nightly job produces one run per night.

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/cron-jobs \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-etl",
    "schedule": "17 2 * * *",
    "target_kind": "workflow",
    "target": "order-pipeline",
    "payload": {"mode": "full"}
  }'

3. A queue message

Bind one of the project's pgmq queues to a workflow and every message on that queue starts a run, with the message body as the run input.

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/workflow-queue-triggers \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{"queue": "orders", "workflow": "order-pipeline"}'

{"id":"…","queue":"orders","workflow":"order-pipeline","enabled":true,"created_at":"…"}

GET    /v1/projects/{ref}/workflow-queue-triggers
POST   /v1/projects/{ref}/workflow-queue-triggers
DELETE /v1/projects/{ref}/workflow-queue-triggers/{id}

A queue may be bound to only one workflow (a second binding is a 409), and names beginning kethosbase_ are reserved for the platform's own queues. Delivery is at-least-once with the queue's visibility timeout; the message id is used as the run's idempotency key, so a redelivered message resolves to the run that already exists. A message that cannot start a run, most often because the workflow has no enabled version yet, is retried and then archived to the queue's archive table. While Workflows is switched off for the project the bound queue is simply not drained: your messages stay where they are and start runs if the capability is switched back on.

4. A row change

Bind a table to a workflow and an insert, update or delete starts a run. Creating the binding installs a capture trigger on the table, so the table must already exist; deleting the binding removes it.

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/workflow-row-triggers \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "table": "orders",
    "events": ["insert", "update"],
    "filter": {"status": "paid"},
    "workflow": "order-pipeline"
  }'

GET    /v1/projects/{ref}/workflow-row-triggers
POST   /v1/projects/{ref}/workflow-row-triggers
DELETE /v1/projects/{ref}/workflow-row-triggers/{id}

events is a subset of insert, update, delete. filter is optional: a flat JSON object of column → scalar (string, number, boolean or null), at most 16 keys and 4 KiB, matched by containment against the row: the post-change row, except on a delete, where it is the row that was removed. A change that does not match never leaves your database. The run input is the captured change:

{
  "trigger_id": "…",
  "action": "insert",
  "table": "orders",
  "record": { … the new row … },
  "old": null,
  "ts": 1785000000.123
}

A single table can be bound to several workflows (one binding each); the same table bound twice to the same workflow is a 409. A change whose captured payload would exceed 1 MB is replaced by a marker carrying "oversized": true instead of the row, so a large write never fails and the run still starts.

Idempotency keys

An idempotency key makes “start this workflow once” safe under an at-least-once trigger. Starting the same (workflow, key) twice returns the id of the existing run instead of starting a second one. Without a key, every start is a new run. The queue and row-change triggers set one for you (derived from the message id, and from the binding plus the message id, respectively); the API takes yours; cron deliberately sets none.

Waiting

sleep

A sleep step parks the run until the delay elapses. It is a wake-up time on a row, so a long sleep costs nothing while it waits, up to a maximum of 30 days per step. Resolution is the engine tick, so treat a sleep as “at least this long”, not as an alarm clock.

{"key": "wait_a_day", "kind": "sleep", "sleep_for": "24h"}

wait_for_signal

A wait_for_signal step parks the run until something outside it delivers a signal of that name: an approval, a delivery confirmation, a human clicking a button. The signal's payload becomes the step's output, and the run's.

{"key": "await_approval", "kind": "wait_for_signal", "signal": "approved", "timeout_seconds": 86400}

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline/runs/<run-id>/signals \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{"signal": "approved", "payload": {"by": "u_123"}, "idempotency_key": "approval-42"}'

{"id":"…","run_id":"…","signal":"approved","created_at":"…"}

• An early signal is not lost. A signal is a row addressed to (run, name). One delivered before the run reaches its wait step sits there unconsumed and is taken the moment the run arrives. Nothing has to be listening at delivery time.
• Consumed once. Each signal is claimed by exactly one waiting step; a worker that claimed one and then crashed re-reads its own claim rather than eating a second. One definition can wait on several different signal names over its life.
• Resume is prompt but poll-backed. Delivering to a run that is already parked wakes it immediately; a parked run also re-checks roughly every 30 seconds, which is what closes the gap when a signal lands while the run is mid-advance.
• A wait is always bounded. timeout_seconds is the total wait measured from the first park; with none set the default is 7 days, and the ceiling is 30 days. On expiry the step dead-letters and the run fails, without consuming the retry budget, because re-waiting for a signal that never came is the same wait again, not a retry.
• signal must match ^[a-z0-9][a-z0-9_.-]{0,63}$, and the payload is at most 32 KiB of JSON. Delivering to a run that has already finished is a 409, not a silent success; nothing would ever consume it.

Sub-workflows

A subworkflow step starts another of the project's workflows and waits for it. The child is an ordinary run: it appears in that workflow's run list, is advanced and retried by the same engine, and can be inspected and cancelled on its own.

{
  "key": "fulfil",
  "kind": "subworkflow",
  "target": "fulfilment",
  "payload": {"warehouse": "eu-1"},
  "max_attempts": 2,
  "timeout_seconds": 3600
}

• The child runs the target's highest enabled version, resolved when it starts, with payload as its input.
• The parent parks while the child runs, and records the child's id as its step output. A successful child's output becomes the parent's run output.
• A failed child honours the parent step's attempt budget: with attempts left the parent backs off and starts a brand-new child run; with the budget spent the step dead-letters and the parent run fails. A child that a human cancelled is never retried; it dead-letters at once. A wait that times out cancels the abandoned child before retrying, so a retry never leaves two children running.
• Depth is capped at 5. A root run is depth 0 and each child is one deeper; a run at the ceiling is refused before it can start another. Direct self-invocation is refused at authoring time, and the depth cap is what terminates an indirect cycle (a → b → a) that no single definition can reveal.
• Cancelling a parent does not cascade to a running child. Cancel the child yourself if you want it stopped.

When a step fails

Retries and backoff

A failed attempt bumps the step's attempt counter and re-parks it with an exponential backoff (2s, 4s, 8s, 16s…, capped at one hour) and the run's wake-up time follows the step's. Nothing is held open in between. Each fan-out child retries on its own budget, independently of its siblings.

Dead-lettering

When a step's max_attempts are spent, the step is marked failed and the run fails, carrying that step's error as the run's last_error. A failed run is terminal: there is no resume and no manual retry, so start a new run once you have fixed the cause. A handful of errors skip the retries entirely because repeating them could not help: a wait that timed out, a sub-workflow child that was cancelled, a step whose static configuration cannot succeed.

Compensation (saga rollback)

A step may declare a compensate action: the side effect that undoes it. When a run fails after earlier steps have already succeeded, the engine does not leave the world half-changed; it walks backwards from the failure point and runs each completed step's compensating action.

{
  "key": "charge",
  "kind": "webhook",
  "target": "https://payments.example.com/charge",
  "payload": {"amount": 4900},
  "compensate": {
    "kind": "webhook",
    "target": "https://payments.example.com/refund",
    "payload": {"amount": 4900}
  }
}

What it does, precisely:

• Backwards, one step at a time, starting at the step before the one that failed. The failed step is not compensated; the engine never recorded it as done. Only steps recorded as succeeded are undone; a step with no compensate declared is skipped.
• A fan-out is undone through its children. The fan-out itself has no side effect and may not declare a compensation; each child may, and the succeeded children are unwound together (a bounded batch per tick), because that is how they ran. There was no sibling ordering going in and there is none coming out.
• Not compensatable: fanout, join, sleep and wait_for_signal; declaring one on any of them is a 400 at publish time. A subworkflow step may declare one, but it undoes the parent's step; the child has already succeeded and is a separate, finished run that is not driven backwards.
• A compensating action is an ordinary step of one of the four side-effecting kinds, with its own max_attempts, timeout_seconds, target, payload and secret. It may not set its own key (the row is recorded as compensate:<step key>), declare children, nest another compensation, or set sleep_for or signal.
• Make it idempotent. Because the engine is at-least-once, a compensating action can run more than once, and the step it is undoing may itself have half-landed.
• The rollback survives a restart. The cursor moves in the same transaction as each compensating step's outcome, so an interrupted unwind resumes where it stopped instead of restarting from the failure point.
• If a compensation itself keeps failing, it exhausts its own attempt budget and the walk stops there. The run ends failed with its rollback cursor parked on the step that could not be undone and last_error naming it (compensation of step "charge" failed, rollback stopped: …). It deliberately does not carry on to earlier steps, because that would claim an unwinding that never happened. This is the state that wants a human.
• A compensated run is still a failed run. There is no separate status; the rollback is reported alongside it (below).

A run that fails for a structural reason, a definition that will not load, a step ceiling breached, fails terminally without compensating: a definition that cannot be read forwards cannot be trusted to be read backwards.

Inspecting runs

curl https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline/runs \
  -H "Authorization: Bearer <your-session-token>"

curl https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline/runs/<run-id> \
  -H "Authorization: Bearer <your-session-token>"

{
  "id": "6f1c…",
  "workflow": "order-pipeline",
  "version": 1,
  "status": "failed",
  "cursor": 1,
  "output": {"delivered": true},
  "compensation": {"state": "compensated"},
  "last_error": "ERROR: relation \"accounts\" does not exist (SQLSTATE 42P01)",
  "steps": [
    {"key":"charge","kind":"webhook","status":"succeeded","attempt":1,"max_attempts":5,"output":{"delivered":true}},
    {"key":"compensate:charge","kind":"webhook","status":"succeeded","attempt":1,"max_attempts":3,"compensating":true},
    {"key":"provision","kind":"sql","status":"failed","attempt":3,"max_attempts":3,"error":"ERROR: relation \"accounts\" does not exist (SQLSTATE 42P01)"}
  ]
}

The run list returns the 100 most recent runs of that workflow, newest first. A single run returns every step row, including fan-out children (which carry a parent_step_id) and compensation rows (which carry "compensating": true, so you never have to parse a key to tell them apart).

If the project has a log drain configured, run and step transitions are shipped to it under the workflow source: run.started, run.succeeded, run.failed, run.cancelled, step.failed (an attempt that will be retried) and step.dead_lettered (the budget spent). Step starts and successes are deliberately not emitted, and no event carries a payload; the full history stays behind this API.

Stopping things

Cancel a run

curl -X POST https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline/runs/<run-id>/cancel \
  -H "Authorization: Bearer <your-session-token>"

{"id":"6f1c…","status":"cancelled"}

Only a live run (pending, running or waiting) can be cancelled; an already-finished run is a 409 (run_finished) rather than a silent success, so a caller retrying a cancel learns the run had already ended.

• A step that was already executing when the cancel landed finishes its own attempt and its outcome is recorded; the side effect really happened, and the history says so. The run never advances past it.
• Cancelling does not roll back. Compensation is triggered by failure only. Cancelling a run that is mid-rollback stops the unwind where it is, and the run reports compensation.state: "incomplete".
• Cancelling a parent does not cancel a running sub-workflow child; cancel it separately.

Disable a definition

curl -X PATCH https://api.kethosbase.com/v1/projects/<ref>/workflows/order-pipeline \
  -H "Authorization: Bearer <your-session-token>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Disabling stops new runs from every trigger: API, cron, a bound queue, a bound table, because all four resolve the highest enabled version and there no longer is one. Runs already in flight are untouched and finish on the version they pinned; to stop one of those, cancel it.

The flag is applied to every version of the name, on purpose: disabling only the newest would silently resurrect an older one. PATCH with {"enabled": true} reverses it. There is deliberately no DELETE: run history references a definition by name and version, and dropping one out from under an in-flight run would fail it on its next advance.

Limits and defaults

The per-project concurrency cap and the tick are shared-box fairness limits: a burst of runs is drained steadily rather than all at once, so treat the tick as the floor on how quickly a run moves between steps.

Errors

Limits of the model

• No data flow between steps. Payloads are static and a step's output is not piped into the next one; see the note under Attempts and timeouts. Carry state in your own tables.
• No resume or manual retry of a failed run, and no “run this step again”. Start a new run.
• No edit in place. Every change publishes a new version; old versions stay for the runs that pinned them.
• Runs are listed per workflow, 100 at a time, newest first; there is no project-wide run feed and no filtering or paging on this surface.
• No partial-success join. One failed fan-out child fails the join and the run.
• No conditionals or loops. The graph is an ordered list with one fan-out/join shape; branching belongs inside a function or sql step.
• No SDK or DSL. Definitions are JSON over this API (and the dashboard's Workflows section, which is the same API with a form in front of it).
• Compensation cannot itself be a sub-workflow: an explicit “undo workflow” is not built.