Fable 5.1 shipped last week, and GPT-6 Astra followed a day later. Each launch raises the same question for anyone with prompts in production: switch to the new model, or stay where you are? We keep a lot of our own Tasks on older Sonnet models, and we wanted a better basis for that call than the launch benchmarks. So this build is a Task that runs 15 tests of our own across eight models every Monday at 9 a.m., scores the outputs with a judge prompt, and reports what changed since the previous week.
What's covered
- What a Playgrounds comparison settles, and why it does not track model quality over months
- The brief: eight models, Sonnet 5 as judge, five test suites, a weekly schedule, and a report written to a memory folder
- What the AI Task Builder produced from that brief in about eight minutes, and the two things it left for a person to do
- The first run: 120 test executions, which runners failed and why, and what the report looked like with no baseline to diff against
Playgrounds answers one question at a time
Playgrounds already covers the quick version of this. Clone a prompt, switch the clone to a different model, run both, and compare the outputs and the token counts side by side. That is how we checked Opus 4.8 against Opus 5 on an error-monitoring prompt in the video.
Those comparisons do not accumulate. Each one is judged by whoever is looking at the screen and gone when the tab closes. To develop conviction that a new model should take over in production, you want the same tests run the same way, on a schedule, with the scores kept.
The brief
The spec that went into the builder was a markdown document with these parts:
- Eight models, chosen because they are already enabled on our account.
- Sonnet 5 as the judge that scores every output.
- Fifteen tests across five suites: administrative, content marketing, data and analysis, operations and finance, and general instruction following and judgment.
- A weekly run, Monday at 9 a.m.
- Scores compared against the previous run, then compiled into a report stored in a memory folder so other Tasks can read it later.
The tests in the demo are broad on purpose. The version worth running for your company would be narrow. If you have a data transformation or entity resolution step where you already know the correct output, that belongs in the suite, because it tests the thing you would be trusting a new model to do. Over enough weeks the memory folder becomes a dataset on model quality built from your own jobs.
What the builder produced
The brief went into the AI Task Builder as-is, through the Build with AI option on a new Task. If the sidebar looks different from earlier videos, that is a recent change: chats, projects, and connectors stay on the main menu, and a Build button opens the builder surfaces, where Tasks, Memories, MCP servers, execution servers, Playgrounds, Prompts, and model enablement live.
The build ran for around eight minutes and came back with:
- Ten inline prompts: one runner per model, plus a judge and a change detector. Each runner is a container, and the test instructions are injected as variables at run time, so one runner handles all fifteen tests for its model.
- A script in six phases: run every test, compare with previous runs, save outputs to the cache, compile, generate the report, store it.
- All fifteen test cases written into the script.
- A 15-minute timeout and retries on the model calls.
Asked how much of that runs in parallel, the builder confirmed the runners fan out at a concurrency of eight and the judge calls run in sequence.
Two pieces needed a person. The schedule had to be set in the Task settings, and the memory folder the script writes to did not exist yet, so it was created by hand as "model bench reports" and attached to the Task.
The first run
Running outside test mode gives a better view of the logs. The run fired 120 test executions, eight models by fifteen cases, and most of them returned quickly. A couple of the GPT runners and the Grok runner errored out, which points at model configuration on the account and is a separate fix. There were a couple of timeouts as well. None of that stopped the run. The failures were recorded and the remaining results went to the judge.
The judge prompt looked at every test's outputs across all models and scored them. The scores were logged, so the next run has a previous result to compare against, and the change detector runs only when that history exists. The report went into the memory folder, and a chat was created with the summary in it.
What came out
The report opened with a leaderboard. Opus 5 scored highest overall, with Sonnet 5, Haiku, and the GPT models behind it, and every model had a score per category underneath. It listed the failed runners for debugging. The analysis section was mostly caveats, and correctly so: with one run there is no baseline, and the change detector had nothing to diff. That part improves once there is a previous run to compare with.
Why build it this way
The whole thing, brief to first report, took about 25 minutes. That changes what a model upgrade decision looks like. You have a Task on a runtime that runs on a schedule, judges outputs against tests you wrote, and flags anything that moves. When a new model appears, add it to the list, and the next Monday's report shows whether it beats what you run today.
For the thinking behind treating the model as a swappable input, see The Model Is an Input.
Video transcript
Hey guys, Mitch from qbash here. Previously Aisle, now officially qbash. Wanted to do one of these again, where we're going to build something together. I'm sure you've all followed the news this week. It's been a hectic couple of days in model land.
So, 2 days ago, I think it was, Fable 5.1, Mythos 5.1 came out. The branding of all these things is always so interesting, but yeah, another surprise-ish model drop. And then yesterday, much more of a surprise, GPT-6 Astra came out. Got me thinking about something that we talk about a bit at qbash, which is when should you, for most business applications, update the model? And I think you'll If you've watched any of my videos or the team's videos before, you'll see that we're a stickler for for older Sonic models for a lot of the things that we do.
And we have this feature in the platform called Playgrounds, right, which allows you to take a prompt and basically duplicate that prompt. I'll show you duplicate that prompt. So, go here, got this one, right? I don't even know what this is. Error monitoring, right?
But we can copy this, clone it, change it. So, you've got 4.8 and 5 that we're comparing here. We can duplicate this again, and we'll change this and run it on, you know, GPT-54, right? Shoot these off against each other, compare the outputs, compare the token usage. We think about this all the time, like when a new model is released, when should you upgrade an existing task or prompts or anything like that?
Or when should you just leave it, and how can you develop conviction around like, yes, this makes sense for our business to, you know, adopt this new model or whatever. And so, Playgrounds is one way that you can do that. But I've been thinking, and I think this is something that we can try and build together, is there a way that you can use tasks to do this more programmatically, go a little bit deeper, have something that runs on a schedule, and basically build your own kind of model testing suite and tracker, where you can view output result quality over time? So, that's the idea, right? You can see here I have a bit of a markdown brief going and we're going to throw this in to the builder and see if we can build a task.
The idea is like here are eight models. I've just picked these eight because I know that we have them enabled for this account and we're going to specify Sonic 5 as a judge, right? Weekly at 9:00 a.m. We are going to run a series of tests that get scored and all of that output gets compiled down to a memory document that's going to live in a memory folder so that we can view that. We can develop supplementary tasks that are doing analysis over time, looking at changes in scores and all that kind of stuff.
As I said, develop like your own company data set that allows you to look at model quality. The interesting idea here is that the tests are things that you can design that are very specific for your company. So, if you have things that you know you want done in a certain way, like if it's any kind of like I'm thinking about data, data transformation, entity resolution, you have base understanding of like what the result should be, um you can bake that into the test. I've gone quite broad. This is all, you know, relatively generated, but um you know, we have a couple of different test suites here.
So, you have an administrative suite, you have a content marketing suite, you have a data analysis suite, you have an operations and finance suite, and you have a general instruction following and judgment uh suite. We're going to throw this in and we're going to see how we go. One thing I know that we're going to have to do is create uh a memory space. We'll get to that when we get to that uh in qbash. One thing the eagle eye amongst you might have noticed is that pretty significant UX change.
I've kind of bifurcated the experience uh between uh builders and, you know, chatters or consumers. Um so, if you see here uh I have a lot of chats, so forgive the messiness of this menu, but you have this area, which is, you know, raw chats, projects, connectors, and then you have a build button, which takes you to a different menu where you have tasks, memories, uh MCP servers, execution servers, playgrounds, prompts, connectors, model enablement. Anyway, that's a digression. Task new task, right? and we are going to iterate with AI.
Um [clears throat] and we're going to dump in that entire spec like we have before. Let's end. See how we go. I might pause it here, let it run, come back when we have an output. Um >> [clears throat] >> yeah, cool.
Okay, we're back. So, I mean, probably ran for 8 minutes. Pretty hectic build. In summary, what was built? So, the way it's started to tackle problem is [clears throat] 10 inline prompts.
One is a judge, one is a change detector. Right, leave those to the side. The other are the eight model specific runners. So, it's using inline prompts as the container, and then all of the tests are instructions that are dynamically inserted. So, it is going to, for example, Haiku 45, it's just going to throw in the instruction for every task that it runs, right?
It's created a long script with six execution phases, where it's running through every test, comparing to previous runs, saving outputs to the cache, and then obviously compiling everything, generating report, and storing that. It has put in all of the test cases. It's figured out the timeout, 15 minutes should be enough. Uh retry, sweep version. I need to set the schedule task.
I think I'll need to set up the memories database as well. I'm going to accept these changes. I'm going to ask it how much of this runs in parallel? Ideally, it should be able to break these down and run everything in parallel. Judge calls, everything else is sequential.
Concurrency of eight. Right. Cool. Okay, so if we leave AI mode, >> [sighs] >> everything here seems to be pretty good. Look at the model bench.
Look in here. Doesn't seem to be wanting to store anything to a memory folder. Let's create the task and just see if I come back in. Memory folder is a model bench reports. Integrations memory search here.
Model There isn't one. So, I'm going to create a model bench reports. Create that. That's added there. I wonder if this will run.
I want to run it out of test mode first because then we can look at the logs in a bit of a better way. Run task. You can see it is firing off tests in the different prompt launchers. A lot of them in parallel. A lot of them are returning quickly.
Some of the tests are going to take longer than others. Some of them are failing. Looks like the GPT 5.5 ones are failing, which might be a model settings issue on my side. GPT 5 mini is working. I'm going to let this run again, and I will come back when this is completed and we can go over the output.
Okay, back once again. So, it is finished. We can open the chat to review the output. I just want to go over the logs and eyeball what happened. So, we kicked off a new session.
We kicked off 120 unique tests. Eight different models all testing 15 cases across different categories, and it ran them all through these launcher prompts that we created. You can see here that some series didn't work, which to me is like a config issue on the model side. I need to dig into that separately. There are a couple of timeouts that I think have been isolated.
The Grok prompts didn't work, which I'll have to investigate separately, but none of that stopped the task from running. And then it got through. It ran all the results through this bench judge prompt where it was looking at for every test that ran all the results across all the different and making a judgment, right? And it's logging that, so at a future run it'll have the previous runs results in the task. Then, if that is the case, it will do a comparison prompt, right?
And so, in the output it'll show increase or decrease in output quality, right? It generates a report, stores it to a memory, logs everything that happened, and then creates a chat as the output. Right? So, if we do chats and look here, right? Here is our output.
Uh summary, best overall model, Chronopus 5.3 out of 10, drastic changes, there's no baseline. Little board, Sonic 5, Haiku, GPTs, etc. You can see each of them has been scored here for each of their categories, and then it gives you an overall. It's dictating some failures, which I think we'll have to debug and then work through. Analysis.
Cool. Two observations produce exactly one observed change, linear absolute. Q3 estimate. Uh these two methods converge, which looks reassuring but isn't with two points. There is some interesting stuff happening in here that I want to read through.
This is all analysis output. So, I think this will do better over time once it has more runs. However, from a framework perspective, we did this really quickly, and I think like spinning up something like this is so much more possible than it used to be, right? The prospect of being able to very [clears throat] quickly put together a test suite based on the things that you or your organization cares about, and have that be attached to a runtime, scheduled, running automatically, compiling and analyzing the results, and then flagging you of anything critical, being able to put that together in like 25 minutes is pretty cool. So, yeah, that was today's build.
Let me know what you think, if there's anything that bubbles up for you, or if there's any other idea that you'd like to see me try and build, let me know. Thanks.