Test-driven development means writing the tests before the code. The objection to it has always been the cost: writing tests by hand is slow, and on a short script it can take longer than the script. When the editor generates the code, that cost drops enough to change the calculation. A suite you got for free still records what the script is supposed to do, and you can run it whenever you need to check that it still does it.
What's covered
- Adding a test file to the Package before you write a prompt, which is what tells the builder it can write and run tests
- Building a new script tests-first: failing tests, then an implementation, then a passing run
- Extending the script to take a parameter with Iterate with AI, again tests-first
- Running the suite by hand, then editing the code to watch the suite fail
Add the test file first
The builder writes tests when a test file is present. In the side panel, Package holds the files in the task, and the + menu adds a New test file alongside main.py. Adding it before the first prompt is the whole setup. From there the builder can write and run tests as part of its own loop rather than only at the end. The mechanics are in Testing tasks.
Square root of two
The first ask is small on purpose: calculate the square root of two, using TDD. The builder starts by writing the tests, runs them against an empty script, and they fail, because there is no implementation yet. It reads the failures, writes the script, and runs the suite again. The second run passes.
What comes out is a task with two files: the script, and a test file with two cases in it. Neither was written by hand.
Taking n as a parameter
The same loop works on code that already exists. Iterate with AI hands the current script to the builder with a description of the change, so the next ask is to improve it to take n as a parameter, using TDD again.
The builder writes a failing test for the state it is aiming at, then runs the full suite rather than only the new case. Four of the five tests fail, as expected: the new cases fail because n does not exist yet, and the old cases fail because their assumption about a fixed input no longer holds. Then it edits the script until all five pass.
Taking n as a parameter means n has to be declared as an input on the task, or nothing will pass a value in at run time. The prompt did not say that. The editor registers the input itself and carries on, and Deploying & sharing covers how inputs become the run form. The finished script is a little longer than the first version, and the test file has more cases in it, because there is a parameter to cover.
Running the suite by hand
Tests are not limited to the builder's loop. Run tests at the top of the editor runs the suite on demand, and the results stream into the console below the code: each case as passed or failed, and each failure expanded to its assertion error.
The suite is a record of the script's behaviour that somebody can run six months from now, without reading the script first to work out what it was for.
Catching a regression
In the square root call, n becomes n + 1, and running the suite reports several failures. The script's behaviour changed, the assertions still describe the old behaviour, and the difference shows up in the console. A square root is a small case, and the mechanism is the same on a longer script: when another developer changes what the script does, the suite reports it.
Video transcript
I'm going to show you how you can do test-driven development using tasks. Test-driven development is a way of building software where you write tests before you write code. Now, the knock on test-driven development has always been that it's labor-intensive and it's slow. But, in the age of AI, when we can generate code, it's actually much more practical and effective than it ever has been. If you're working on a team where many people are touching the same code, or even if you're a solo developer and you're coming back to a piece of code after a long time, the test suite gives you a record of what the code is really supposed to do, and it gives you something you can run to validate that it still does it.
Here's a brand new task. Now, before I start working with the AI editor, I'm going to go to package and I'm going to add a new test file. If a test file is present, the AI knows that it can create and run tests. Now, I can ask for what I want. Let's calculate the square root of two.
And I'm going to tell it to use TDD. Okay, step one is write failing tests. It's going to write tests and it's going to run them, and they are going to fail because we have no application code yet. Now, it sees that the tests fail, and it implements the script. Finally, it runs the tests afterwards and sees that they pass.
So, now I have a working task, and it contains both a script and a test file with two test cases in it. So, now we've seen TDD in action when writing brand new code. Uh let's do the same thing, but for iterating on existing code. So, I'm going to go to iterate with AI, and I'm going to say, uh let's improve this to take n as a parameter. Use TDD again.
This time, it's going to write a failing test for the desired state, and it will run the full test suite again, and we'll see some will pass and some will fail. Okay, here we go. Four of five failing as expected. That's good. Now, it's going to make the changes to the script so that the new test pass.
The editor has noticed that the input N was not registered, and it's adding it. Finally, we run the tests, and they all pass. And here's our script. It's just slightly more complex than last time because it's taking N as a parameter. Similarly, our test file slightly more complex because there are more cases.
So, up here I can run the tests manually. When I do this, I see the results below. And now, let's catch a regression. I'm going to make a change to the code. I'm going to run the tests again and see that they fail.
Let's change N to N + 1 in the square root call. When I run the tests, we see that a number of them fail. This is a very simple case, but it demonstrates how we can catch regressions when, say, another developer makes a change to the script that changes its functionality.