How qbash Tasks work: a simple model test

Last week's build was a model test suite that ran 15 tests across eight models on a schedule, scored them with a judge, and diffed the results week over week. The AI Task Builder wrote most of it, and the result was long enough that the mechanics were hard to see. This video is the same idea cut down to one model, two inline prompts, three tests, and 24 lines of Python.

What's covered

• Creating two inline prompts from the package panel: a judge that returns pass or fail, and a runner that carries whatever test you hand it
• Why the runner's message is a single variable, so one prompt serves every test
• The script: a dictionary of runners, a list of tests, and a nested loop that runs each test through the runner and then through the judge
• A test run and the chat it produces, with a three-row table of answers and verdicts

Two inline prompts

Both prompts are created with the plus button next to Package in the editor, which opens the inline prompt editor.

The judge runs on Sonnet 4.6 with default settings. It takes three variables, question, expected, and answer, and is told to reply pass or fail with one sentence explaining why.

The runner is the prompt under test. This one is called runner-opus and runs on Opus 5. Its whole user message is one variable, instruction. Every test question gets injected there, so one runner prompt covers every test for its model. In the eight-model suite there were eight of these, one per model, and the judge and change detector alongside them.

The script

RUNNERS = {
    "claude-opus-5": "runner-opus",
}

TESTS = [
    {"question": "What is 1 + 2? Reply with just the number.", "expected": "3"},
    {"question": "What is the capital of France? Reply with just the city.", "expected": "Paris"},
    {"question": "Spell the word qbash backwards. Reply with just the word.", "expected": "hsabq"},
]

rows = ["| Model | Question | Answer | Verdict |", "|---|---|---|---|"]

for model, runner in RUNNERS.items():
    for test in TESTS:
        answer = qbash.ai.run_prompt(runner, {"instruction": test["question"]})
        verdict = qbash.ai.run_prompt("judge", {
            "question": test["question"],
            "expected": test["expected"],
            "answer": answer,
        })
        rows.append(f"| {model} | {test['question']} | {answer} | {verdict} |")

qbash.create_chat("Model bench", "\n".join(rows))

The video uses the shorter qb alias for the same object.

The dictionary at the top maps a model name to the slug of the runner prompt that calls it. Adding a model to the suite means adding a runner prompt and one line here. The tests are a list of question and expected answer pairs. The rows list starts with a markdown table header, because the output is going to be read by a person in a chat.

For every model and runner pair, and for every test, the script runs the runner with the question as its instruction, then runs the judge with the question, the expected answer, and the answer that came back. The verdict and the answer become one row. When the loop finishes, create_chat posts the table as a chat.

The run

A test run fires three runner calls, one per test, then three judge calls to grade them. The chat holds a three-row table: 1 + 2 is 3, the capital of France is Paris, qbash backwards is hsabq, and each one is marked as a pass.

Scaling it back up

The larger suite is this loop with more in it: eight entries in the runners dictionary, 15 tests, the runners fanned out with qbash.parallel rather than run in sequence, and a change detector that compares this week's verdicts against the last run before the report is written to a memory folder. The structure is the same. A Task is the plain Python around the model calls, and the model calls are function calls with variables.

Open qbash →

Video transcript

Hey guys, Mitch from Qubyte here. I wanted to follow up to my previous video where we created quite a complex test suite for model performance. Basically, we put together a task that compared the performance of eight models going through a rigorous set of tests that you design for your business so that you can track that over time and start to develop your own intuitions around when you should update a model, maybe different model strengths for different areas of your business, just develop some conviction there. I wanted to zoom in maybe on the mechanics of how that works. It was quite a complex test case that we put together, and so I thought by doing something quite simple, we can really highlight what the task is doing, and then you can extrapolate from there.

So, uh blankish task here. I've created two inline prompts, which you can do by going next to package, new inline prompt. The first one is a judge prompt, right, running on Sonnet 4-6. Very simple, default settings, and we are defining a couple of variable inputs here, right, so question as question, expected answer as expected, answer given as answer, and then we're telling it to reply pass or fail with a sentence why. The next inline prompt is the runner prompt.

These are the prompts that you will use to inject all of your tests into, and they run on the models that you want to test. For this one, we've created a runner Opus prompt. The model is Opus 5, and then the message is just a variable, which is instruction. So, for every test that we have, we will inject it here. So, if we come back to the task, and I put this in here, you'll see this is a very small task, and it's quite straightforward.

So, these are the runners as we're going to define them. We just have one. The model is called Opus 5, and the runner slug, the prompt slug is runner-opus. We have three tests. What is 1 + 2?

What is the capital of France? Spell the word qbash backwards. And then we're defining rows for the outputs, right? This is important when you have multiple here, you'll have multiple models doing multiple tests. So, basically you want to create some sort of data structure.

These are the rows. Right. Now the meat of the task. It's essentially nested four statements. So, for every model and runner pair in runners, right?

So, these are the items, the model and runner pairs. So, this is our runners. In the previous video that I made, there was eight in total. For every test in tests, right? So, these are tests.

Each line here is a test, right? Now we are defining all of the different items. So, the answer is running the prompt runner opus in this case with the instruction variable being the test question, right? So, this is how you get the tests into the prompt. And then you take that output and you run the prompt judge, right?

With these variables, the question, the expected, and the answer. And then for every result you create a row. Finally, we finish off this task by creating a chat that has the overall output of the task. So, if we run a test here, you'll see it starts off by running a prompt, right? There's three of them for every test.

It's going to grade them all. And there's the final output. It would be a chat that contains a table with these three rows showing that 1 + 2 = 3. This is a pass. The capital of France is Paris.

This is a pass. And qbash backwards is H S A B Q. This is a pass. So, hopefully by simplifying it like that, you can kind of see the mechanics of how a task is working in this case. But you can extrapolate from there how you might use it.

Let me know if you have any questions. Thanks.