When I'm building a Task, it surprises me how little I'm thinking about the AI step.
I ran an experiment last week. Built a competitive intelligence automation. Six phases, seven prompts, scraping, web search, social search, three analysis passes, a summary, a memory write at the end. Complex bit of kit.
The point was to push our AI Task Builder. Feed it a spec (requirements, outcomes, context), let it compile the script, write the prompts, wire the whole thing together. Hands off.
I recorded the session if you want to watch it play out:
The builder is really good at scaffolding complex operations
I've built enough Tasks by hand to know how long the scaffolding usually takes. It's not the interesting part. It's the plumbing. Setting up the scrape, deciding which pages to hit, writing the fail-fast condition when pages come back empty, shaping the intermediate data so the downstream analysis can actually use it, placing each prompt in the pipeline, writing the memory write so the output is findable later.
That scaffolding is the majority of my time, and it can eat up hours if I'm being really careful.
The builder produced a working version in a few minutes. Six phases with clean hand-offs. Seven inline prompts with system and user messages that fit their spots in the pipeline. Fail-fast where it belonged. A summary before the memory write. Checkpoints and retries on request.
Not every decision matched mine. A couple of the phase boundaries felt off and I'll probably restructure them. But the shape was right. And the shape used to be most of the work.
Watching it produce a competent pipeline in the time it takes to make coffee is when it clicked: most of building a Task is structural, not creative. If a builder can produce that structure from a spec, the structure isn't where the value is.
The value's somewhere else.
The AI step is the smallest part of the work
Seven prompts. Out of roughly forty distinct operations in the script.
I sat with that ratio for a while, because it shows up in every Task I've built.
Maybe 10% of my time goes to the prompts. Another 10% to picking models and tuning. The other 80% is everything around the model.
Formatting the input. Deciding what context to inject where. Shaping the output so the next step can consume it without another parsing pass. Choosing where to fail fast, where to retry, where to checkpoint. Working out which parts deserve their own prompt and which should just be a Python function.
I don't think that's a limitation of the tools. I think it's the whole thing.
A model is a function
Text in, text out. The quality is bounded by the shape of what you hand it. Feed it a raw HTML dump, you get raw-HTML-dump answers. Feed it three clean variables and a well-labeled context object, you get clean structured answers doing the reasoning you actually wanted.
Every good step comes down to two questions, over and over:
What does this model need to see to do its job well? What shape does its output need to be in for the next step to work?
Those aren't prompt questions. They're structural. They shape the pipeline underneath the prompts, and that's where most of the reliability lives.
The 80% compounds
An input transformation you got right once gets reused across six Tasks. A memory doc you populated last month gets read by three prompts this month. A retry policy you tuned in one place becomes the default everywhere.
Prompts don't compound. Prompts are the part you re-tune when the model changes.
That's the inverse of most AI writing out there. Prompts get all the attention, and the infrastructure gets treated as plumbing. In real systems the ratio flips. The plumbing is where the reliability comes from. Prompts are the last-mile tuning on top.
Why they feel different
This answers a question worth asking: why does a well-built automation feel different from a chatbot doing the same job?
The chatbot is 90% model, 10% wrapping. The automation is 10% prompt, 10% model, 80% everything else.
The AI is the smallest, most impressive-looking part. Everything around it is what makes it worth having.
Video transcript
Hey guys, Mitch from Aisle here. I was just in the middle of running a bit of an experiment with our tasks feature, specifically the AI builder that is a part of the tasks feature, and I thought I'd start recording and take you on the journey with me. I wanted to put together a competitor deep dive research task, something that I can run with the inputs of a company name and a company website, and start to create context files that I can use in my other automations to understand the product positioning and marketing and feature set and pricing of other tools out there. And so, in one of my projects, I've gone back and forth until I've landed on quite a complex spec that I liked. You can see here that it runs through product positioning, feature overlap with Aisle, pricing and packaging, ICP and GTM, market signals including social's deep scan, strengths and weaknesses, and positioning wedge, and really pulling it apart.
And the output here is a synthesized chat, and also writing a context file that I can use elsewhere. And instead of me trying to build this, I thought it'd be a cool experiment just copy and paste this entire spec into the AI builder, which is something I hadn't done before. And so, I started doing that, as you can see here on the screen, we're kind of halfway through this run. So, on the right-hand side, if I scroll to the top here, you can see literally copied and pasted in that entire spec, right? And the AI builder has refreshed itself on the Aisle SDK.
It's looked at a bunch of my other tasks and inline prompts that have been built before, and now it has gone and it has written seven prompts that do the kinds of analysis that I want to do. And now it's written the main Python script and put that in there and fully built it. The summary of what it's built, says it's built a competitive intelligence analyzer complete build, six-phase pipeline, seven inline analysis prompts. So, pre-flight, it looks context from the competitive intelligence memory folder. >> >> Phase one, it does a scrape.
So, it's looking for a bunch of pages. If it doesn't find those pages, fail fast. Phase two, it's doing a web search and a Twitter search for news. Phase three to seven, >> >> there are a bunch of pillars that I have in in context about Aisle that it's looking for information in regards to. Phase four, it's kind of starting to summarize.
>> >> Phase five, it's creating the chat that has the output. And phase six, it's writing that context file. These are the prompts that it's created. This is how it wants to handle errors, right? These are the inputs that it requires from me.
>> >> And that's where I landed. This looked to my eye to be quite a long-running task, and so I just said add in some checkpoints and retries, >> >> right? It's done that, updated the script. Cool. So, we're going to hit accept.
>> >> Right. And this is the entire task here. I'm going to get out of the AI assistant. Okay. Memory folder competitive intelligence is not bound to this task.
You can see on the right here, it is pretty proactive in in finding errors. Memory folder competitive intelligence is not bound to this task. Okay. I actually don't think that memory folder exists. Let me check.
Right. So, if we come in here, it'll go to the tools tab. You can go to memory, we can look for competitive intelligence. It does not exist. So, we can create one.
I'm going to open this, right? And I create a new folder, competitive intelligence. Create that folder. Right, that exists. >> >> Come back in here.
Let's see if it shows up. Probably going to need a refresh, which means I'm going to have to save it, which is okay. If I go create task, saves it, >> >> come back to edit. Let's fix this, competitive intelligence. >> >> Bang.
Save. And add a note here. Added folder. >> >> Excellent. Just to go through the overall package that's been created here, right?
So, as you can see, you have the main script that we were looking at before. We have all of these inline prompts. What you'll find is the AI builder has a a pretty good a way of figuring out what models should go with each prompt. One way to think about it is it's really looking at the sizing of a prompt in the job, and it will determine like the appropriate model capabilities based on what it thinks it needs to do. But, you can see here for this product positioning prompt where it's it's written a system message and the user message.
It's >> >> looking at this. These are variables, by the way. So, it's saying, "This is the competitor." And then per run, the company name and the company website. This is the content that was scraped. >> >> This is our context that it's injecting.
And running it through this prompt. That all looks pretty good to me. You can see that it has put together a bunch of different things here. It has the Aisle product documentation memory folder as a tool. So, that's all the context that it needs.
You know what? Let's just try and run it. See how that goes, right? I'm going to hit run. It needs these two inputs, so that's great.
Try and hit enter. And we'll try https colon slash runtask. Right, this is the test run mode, so we kind of have a bit of a live debugger here. This is running. Don't know how long this is going to take.
Uh so, I will probably pause the video here and then come back with the end um the end results. Okay. So, I think we're back. All in all, it took about 8 minutes to run through the entire pipeline for this competitor. Let me see.
So, it says here down the bottom it's created a new knowledge base document competitor N8N. Great. It says that it quoted a chat. So, let's start by looking in the inbox. If I open this in a new tab, go to my inbox.
Cuz we're in test mode, I don't think it's going to be there. Look in memories. Should have the same output. All right, competitive intelligence one competitor N8N. Great.
It has their one-line positioning, their elevator pitch, claim differentiators, marketing tone and audience, primary audience technical practitioners developers DevOps engineers, automation model, classification canvas with code escape hatches, their primitives map to our primitives, key product pages found. It's great. Analyst notes. Uh very detailed feature overlap. Some gaps.
We win. We have strengths and they have strengths and all that kind of stuff. Pricing analysis. Usage metering. This is a very very fullsome document that I'm actually going to have to go through with a fine-tooth comb.
But, I think the interesting thing to think through is that the experiment kind of worked. Putting in a very very complex pipeline spec to the builder has worked. Where I'd usually go from here, where my brain's kind of going is like >> >> what I've proven is like structurally this thing kind of works and it works in the way that I want the automation to work, meaning like I'm giving a simple output and there is a a long quite fullsome pipeline >> >> that is producing very very in-depth document at the end. I think my next step would actually be to start manually maybe tuning these prompts. So, going through the output document, reading that, noting what I like and what I don't like, and then going through these prompts and shooting them manually, and maybe running again and again, and doing some other different outputs.
>> >> But anyway, fun experiment. If you haven't tried Tasks yet, please do. It's really really good at quite simple kind of automations, and obviously it's it's very adept at quite complex automations as well. Thanks.