You charged the customer's card. Then the inventory API went down. The order is half done, and retrying from the top charges the card a second time. This video shows how checkpointing in the Tasks SDK gets every step of a workflow to run exactly once, no matter how many attempts the run takes.
What's covered
- Why a plain retry makes a partial failure worse in a multi-step workflow
- How
aisle.checkpointrecords finished steps so a retry skips them - A live demo with a planted failure in
reserve_inventorythat kills the first two runs - What auto retry in task settings does once the steps are checkpointed
The demo is a checkout with four steps that all have to happen: validate the order, charge the payment, reserve inventory, send the confirmation. Each step is wrapped in a checkpoint, so its result is stored the first time it succeeds:
charge = aisle.checkpoint.get_or_set(
"charge_payment",
lambda: charge_payment(order),
)
The checkpoint store is scoped to a run and its retries. A retry re-runs the whole script from the top, but a step that already finished returns its stored result instead of executing again.
In the video, reserve_inventory is rigged to fail on the first two attempts, which is what a flaky inventory API looks like. The first run validates the order, charges the card, and dies. With auto retry on, the second run skips both finished steps and fails at inventory again. The third run reserves inventory and sends the confirmation. The card was charged once across all three attempts.
The full checkpoint API, including aisle.parallel's built-in checkpointing for fan-out work, is in the concurrency and state docs.
Video transcript
Let's talk about durable workflows or durable execution. Example, let's say you have an e-commerce site. When a customer buys something, you need to do four things. You need to validate the order, you got to charge payment, you got to reserve inventory, and you need to send a confirmation. If any one of these steps get skipped, it's really bad.
You also don't want to double charge or double reserve inventory, that's also bad. You want each of these steps to happen exactly once. So, how do you do it? Well, there's a trick. What you do is you run the sequence over and over and over again, but each time you keep track of which ones you actually succeeded at and you don't do those again.
Eventually, they all get finished. So, in the Tasks SDK, we have something called checkpointing, and this is basically what it does. Each time you run this thing, you can say, "Hey, this step was finished and don't do it again next time." So, I am not connected to a real payment processor here. I've just created stubbed methods uh called validate order, charge payment, reserve inventory, and send confirmation. And if you read them, you can see I'm just logging out a message that says I'm trying to do this.
Um but I've also built in a little bomb. Inside of reserve inventory, I'm going to fail the first two times this is run. On the third time, it should succeed. And so, this is going to let us demonstrate what it kind of looks like in the real world when uh payment processor fails or an inventory API fails or something like that. All right, let's run this thing.
As it runs, you see the logs streaming in. We're validating the order, we're charging the payment, and we're trying to reserve the inventory, but it fails because of that bomb that we put in there. Now, if this was it, we'd be in trouble because we actually charged the customer and we didn't reserve the inventory. This is a bad outcome. So, how do we get around that?
We go to edit, settings, and we turn on auto retry. And this is going to try up to two more times, and it's going to wait 10 seconds between each try. All right, let's run it with the retry. Okay, we see order validation, charge payment. Uh, what happened?
Well, we know it failed. It immediately bumped us to attempt two. We can go back and see attempt one, but it's going to sit here in attempt two for 10 seconds, the back-off time, and then it's going to run again. This time, you can see that it failed very quickly again. But, you can see that the validate order was skipped, and the charge payment was skipped.
Uh, again, it failed on the reserve inventory. However, the third attempt, when it runs, succeeds. And so, in this case, we can see that all four steps got completed. Each one got run exactly once. That's precisely the outcome we want, and that's what lets us sleep at night because we know that this thing is going to succeed when it runs, even in the background, when no one's watching.