What is durable execution, and why is AI bringing it back?

In 1972, an IBM manual for mainframe programmers promised that with checkpoints, "the time-consuming alternative of rerunning an entire job is eliminated." The idea is now called durable execution. A program records each step as it finishes, so after a crash or a failed call it resumes from the last finished step.

This year it became one of the busiest corners of AI infrastructure. In September, Temporal raised $550 million at a $12.55 billion valuation and Restate raised $20 million, both pitching infrastructure for AI agents. Cloudflare, AWS and Vercel have each shipped a durable runtime in the last 18 months. Restate's CEO, Stephan Ewen, told TechCrunch his product "never was built for agents in the beginning, but it just happened to be a perfect match for all these problems that agents surface."

What is durable execution?

Take a support team that gets 500 tickets a day. For each one, an AI agent looks up the customer, checks the help docs and drafts a reply. The draft goes to the team's Slack channel for someone to review. The job that runs it is a loop:

for ticket in tickets:
    draft = triage_agent(ticket)
    post_to_slack("#support", draft)

About 380 tickets in, the model provider starts returning "overloaded" errors and the job dies. Anthropic's status page lists 13 incidents in September, six of them major, so a long job will meet one sooner or later. Run the job again and it starts at ticket one, pays for 380 agent runs a second time, and posts 380 duplicate drafts.

Durable execution wraps each step so its result is recorded the first time it succeeds:

for ticket in tickets:
    draft = step(
        f"triage:{ticket.id}",
        lambda: triage_agent(ticket),
    )
    step(
        f"post:{ticket.id}",
        lambda: post_to_slack("#support", draft),
    )

On the next run the code starts from the top, and every finished step hands back its recorded result without running. AWS describes its version as code that "runs from the beginning but skips over completed checkpoints." The code has to call the same steps in the same order every time, which engines call determinism.

Where did durable execution come from?

IBM documented checkpoint/restart for mainframe batch jobs in 1968, and a 1987 paper called Sagas worked out how to split a long job into steps that can each be undone. The cloud versions came out of Amazon, Uber and Microsoft in the 2010s, and in 2023 Uber reported more than 12 billion Cadence executions a month.

For most of that history it ran payments, rides and deploys. In November 2023, Chris Riccomini wrote in his Materialized View newsletter that "the current market can't sustain this many startups."

Why is AI bringing it back?

AI made each step expensive. In the systems Cadence was built for, a step is usually a quick call to another service. In an AI job it is a model call, or an agent run made of a dozen of them, and each one takes seconds to minutes and is billed by the token. The providers behind them have bad days too. OpenAI's status page lists 25 incidents between September 13 and October 1. Starting the ticket job over pays for every finished run again, at Anthropic and OpenAI list prices:

Durable execution in a qbash Task

In qbash the triage agent is an inline prompt called triage-ticket with Agent mode on, which gives it tools to call. The Python Task that feeds it tickets calls it once per ticket and passes the HubSpot account and memory folder it may use to look up the customer and search the help docs:

def triage_and_post(ticket):
    draft = qbash.ai.run_prompt(
        "triage-ticket",
        {"ticket": ticket},
        connectors=["hubspot"],
        memory=["Help docs"],
    )
    qbash.integrations.slack.create_message(
        channel="#support-drafts",
        text=f"*{ticket['subject']}*\n{draft}",
    )

tickets = qbash.inputs.get("tickets")

qbash.parallel(triage_and_post, tickets, concurrency=5, retry=3)

qbash.ai.run_prompt waits for the agent to finish and returns its draft. retry=3 covers a blip, but a provider that stays down for ten minutes still fails the run, and running it again starts at ticket one. The durable version adds two arguments:

qbash.parallel(
    triage_and_post,
    tickets,
    concurrency=5,
    retry=3,
    checkpoint="tickets",
    checkpoint_key="id",
)

checkpoint records each ticket as it finishes, keyed by its id. Set the Task's expected runtime to "Up to 12 hours" and turn on auto retry, and a failed run retries itself and skips every ticket it has already recorded.

Durable workflows: exactly-once execution with checkpointing covers the same idea with a checkout, where the step that must not repeat is charging a card.

What durable execution does not fix

A step that was halfway through when the process died runs again on the retry, and with five tickets in flight that can be five duplicate drafts. retry=3 repeats a ticket inside the run too, so a Slack call that times out after the message went through posts it again. The fix is the one payments teams have always used: make the step safe to repeat, for example by checking for an existing post first. Durability also does nothing for a wrong answer. If the agent misreads a ticket, the checkpoint keeps the wrong draft.

Plenty of jobs don't need it. A Hacker News commenter argued that "the only thing the durable execution engine is buying you is an optimization against re-running some slow tasks," and for a nightly job that finishes in two minutes that is fair. AI jobs make hundreds of slow, paid calls that can fail, and post their results where a repeat does damage, which is most of why the money showed up.

The concurrency and state docs cover qbash.checkpoint, qbash.parallel and auto retry, and the agent runtime docs cover agent prompts.