When a company comes up on a call, we want the transcript linked to that company's record in our database. The name in a transcript often differs from the one on the record, because people say short names and speech-to-text misspells names it hasn't heard. We built a qbash Task to compare Jev and OpenAI Decisions on this match. The first version sends each model the whole company catalog. The second sends a shortlist of candidates picked by phonetic encoding, which cut the estimated cost by about 89% and left the two models tied at 83%.
What's covered
- The test data: a 60-company catalog, 30 transcript snippets, and an answer key
- Building the Task from a written brief with Iterate with AI, including one prompt for Jev and one for OpenAI Decisions
- Results with the full catalog in every call, and why that version can't scale
- Double Metaphone encodings, and a second brief that adds a shortlist of up to 10 candidates
- The two reports side by side, and where to take the Task next
The test data
The test set is three JSON files stored in qbash Memories. synthetic-company-catalog.json stands in for the company database, with 60 fictional companies. Each record has an id, a name, a short_name, and a list of domains. Company 1 is Cedarvane Technologies, with the short name Cedarvane and two domains.
synthetic-transcript-cases.json contains 30 cases, each with the mention and a context snippet from a made-up call, such as "I spoke with Mosswick this morning. They are sending the updated contract after lunch." Some mentions use the full name and some the short name. Others look like transcription errors: case 7 mentions "brindal key systems", and the catalog has a Brindlequay Systems.
synthetic-answer-key.json gives each case an expected decision (MATCH, NONE, or DEFER), the expected company id for a match, and a short rationale such as "Unique short name in this catalog." Twenty-four cases expect a match, four expect no match, and two are ambiguous enough that the right answer is to defer to a person. The Task uses the key for scoring and never sends it to a model.
Version one: the whole catalog in every call
The Task starts empty. We wrote the brief ahead of time and gave it to Iterate with AI, the AI builder in the Task IDE. Its goal reads: "Resolve each transcript mention of a company against the whole catalog. Jev and OpenAI Decisions each answer every case, and the task scores both against an answer key and writes a Markdown report." The brief also names the three memory folders and the fields in each file.
The builder attached the Company Catalog, Fixtures, and Outputs folders with read and write access, wrote the package, and ran a test:
main.pyruns the strategy chosen in thestrategyinput and scores the answers.raw.pysends every case to both models along with the full catalog.jev-company-resolver.promptruns on Jev.decisions-company-resolver.promptruns on GPT-6 Luna in Decisions mode.
Both inline prompts evaluate the same text:
Catalog records:
{{catalog}}
Transcript mention: {{mention}}
Transcript snippet: {{context}}
Each asks one question, company, whose answer is a catalog id, NONE, or DEFER. The scorer gives every miss a type, such as wrong_company or deferred_on_clear_match. The builder added two types beyond the three in the brief: none_on_clear_match and none_when_ambiguous. Errors, refusals, and invalid answers each count as a type of their own.
Full catalog: Jev 93%, OpenAI Decisions 80%
The Task saves its report as a Markdown file in the Outputs folder. The Raw comparison resolved all 30 mentions against the 60-company catalog:
| Metric | Jev | OpenAI Decisions |
|---|---|---|
| Overall accuracy | 93% (28/30) | 80% (24/30) |
| Match cases | 23 of 24 | 21 of 24 |
| No-match cases | 4 of 4 | 2 of 4 |
| Defer cases | 1 of 2 | 1 of 2 |
| Average latency | 1,210 ms | 1,283 ms |
| Estimated tokens per call | 2,588 | 2,588 |
| Estimated cost per 1,000 cases | $0.1087 | $0.2588 |
We aim for the high 90s on this job, so Jev's 93% was a good first result and OpenAI Decisions had further to go. OpenAI Decisions struggled with the no-match cases, getting two of the four right. Both models got one of the two defer cases, which is too few to judge either on. Latency was about 1.2 seconds a call for both. OpenAI Decisions cost about 2.4 times as much as Jev.
Sixty companies fit easily in a prompt, but a production database has tens of thousands of company names, or millions. Sending all of them with every mention would be slow and expensive, and before long the catalog wouldn't fit in a model call.
Version two: a phonetic shortlist
The second version picks a shortlist of candidates before either model sees the mention, so every call carries at most 10 companies however large the database grows. The shortlist comes from phonetic encoding, here with Double Metaphone. It turns a word into a code for how it sounds, with an alternate code when there is a second likely pronunciation, so names that sound alike tend to share a code even when they're spelled differently.
synthetic-phonetic-index.json, a new file in the Company Catalog folder, maps codes for whole names and for single words to the ids of the companies that produce them. Cedarvane Technologies produces codes such as STRFN and STRFNTKNLJS, and a code can point to more than one company, as TPN does for companies 25 and 26. synthetic-transcript-encodings.json, a new file in the Fixtures folder, has the encoded form of each mention:
{
"id": "case_001",
"mention": "Cedarvane Technologies",
"context": "We received the revised proposal from Cedarvane Technologies yesterday. Can you attach it to their company record?",
"phonetics": {
"normalized": "cedarvane technologies",
"primary": "STRFNTKNLJS",
"alternate": "STRFNTKNLKS",
"numbers": [],
"tokens": [
{"token": "cedarvane", "primary": "STRFN", "alternate": ""},
{"token": "technologies", "primary": "TKNLJS", "alternate": "TKNLKS"}
]
}
}
The goal in the second brief reads: "Build a shortlist of up to 10 candidates from a phonetic index before calling either model, then run the same comparison on the shortlist instead of the full catalog." It names the index's two keys, whole_code_to_company_ids and token_key_to_company_ids, and has the script raise an error if either is missing. The builder added phonetic.py, a shortlist prompt for each model, and a Phonetic option on the strategy input. When a mention's shortlist comes back empty, the Task answers NONE without calling a model.
Shortlist: a tie at 83%
| Metric | Jev | OpenAI Decisions |
|---|---|---|
| System accuracy | 83% (25/30) | 83% (25/30) |
| Model accuracy, cases asked | 88% (22/25) | 88% (22/25) |
| Match cases | 20 of 24 | 20 of 24 |
| No-match cases | 4 of 4 | 3 of 4 |
| Defer cases | 1 of 2 | 2 of 2 |
| Average latency | 1,145 ms | 1,121 ms |
| Estimated tokens per call | 352 | 352 |
| Estimated cost per 1,000 cases | $0.0123 | $0.0293 |
The shortlists averaged 1.3 candidates, and the right company was on the list for 22 of the 24 cases that expect a match. Five mentions got an empty shortlist and went to neither model. Three of those expected no match, so the Task got them right. The other two expected a match the shortlist had left out.
System accuracy in the report covers all 30 cases. Model accuracy covers the 25 cases where a model was asked, and it is the fairer score for the models, since neither can pick a company that isn't on its list. By that measure both models got 88%. Jev scored five points below its full-catalog 93%, and OpenAI Decisions scored eight points above its 80%. OpenAI Decisions got both defer cases right this time. On the one no-match case a model was asked about, Jev was right and OpenAI Decisions was wrong. Latency dropped slightly, to about 1.1 seconds.
Tokens per call fell from 2,588 to 352. Estimated cost per 1,000 cases fell from $0.1087 to $0.0123 for Jev and from $0.2588 to $0.0293 for OpenAI Decisions, a cut of about 89% for each. Both columns use the same token count, priced at $0.042 per million input tokens for Jev and $0.10 for GPT-6 Luna. Independent tests have found Jev billing two to three and a half times as many input tokens as OpenAI's models for the same text, so the gap between the columns could shrink or reverse on a bill. Who the hell is Jev, and why is everyone talking about decisions? covers those tests.
Where to take it next
The next iteration could win back accuracy, cut cost further, speed up responses, or send ambiguous cases to a person for review. On the shortlist run, right answers averaged 0.92 confidence for Jev and 0.91 for OpenAI Decisions, and wrong answers averaged 0.49 and 0.58. A threshold could route low-confidence answers to a reviewer, though three wrong answers per model is too few to set one.
Use Jev with qbash Tasks covers setting up Jev. The Jev decision prompts and OpenAI Decisions sections of the inline prompt docs cover each model's question format. OpenAI Decisions is available to organizations with the feature enabled. Entity Resolution in Python for Contact Matching builds its candidate list with vector search.
Video transcript
Hey, Jack here with Qubesh. I've recently been investigating entity resolution methods to solve a technical challenge we've been facing. Um the challenge being when a company gets mentioned on a call transcript, we need to match that company to a company in our database. So, with Jev being released and now OpenAI's decision API also being released, I thought now was a good time as they're both available in the Qubesh platform to compare the two and see how they do. We'll start with a brute force technique where we hand them both the full database and see how they do as a baseline.
And then we'll see if we could come up with a more scalable solution and evaluate the pros and cons. So, we're starting with a blank task right now. Um but before we get into writing the actual task with our AI editor, let's take a look at the test data that we're dealing with here. So, we have our synthetic company catalog that represents our database. Um obviously, these are fictional companies.
Um but you can see here, we have Cedar Vein Technologies. We have our synthetic transcript cases. So, these are uh synthesized uh transcript snippets. You can see this is kind of representing what a human would be talking about on the transcript. Um in this example, it's a very easy example.
Cedar Vein and Cedar Vein, that's a direct string match. And we also have an answer key. So, in case one, we expect a match. And the reason being is that the exact full name is included in the transcript snippet. So, I've generated 60 companies and 30 cases.
That seems like a good enough size that we can kind of get a feel for how this is looking. Um so, let's get back to actually writing the task. To save us some time here, I've pre-filled the instructions with uh Qubesh's AI editor here. Um so, let's start with this. Let's let it run and uh we're going to start with the brute force version.
Uh we're going to give each model, um Jev and OpenAI Decisions API, the entire company catalog and the transcription snippet. We're going to ask it to choose a company or return no match or flag ambiguous cases for human review. So, we'll let the task editor run here. It's going to build out the comparison logic. We're going to ask it to report on accuracy, latency, and estimated cost.
Okay, so our task is all written up. We have a couple of scripts and prompts. We have our memories are all hooked up, no issues there. The AI editor understood that we wanted the Jev prompt and a decisions prompt. It all looks good.
Let's go take a look at an output over here. So, we're outputting to a memory file in markdown. So, let's take a look at some of the results here. So, Jev performing pretty well, it only got two wrong. Good 93% accuracy.
OpenAI a little behind. It's dropped about six. Normally, we're looking for high 90s, so I'd say Jev is doing pretty good as a an initial shot. OpenAI leaving a little bit to be be desired. We'll see if we can bump those numbers up potentially.
You can see that OpenAI is having a bit of a hard time with the non cases where there's no matches. They're both struggling a little bit with the defer cases, but that's a pretty small sample size to be making any judgments from. Our latency is good, I have no problems with that. Our cost is okay. It's 10 cents per thousand cases for Jev, about 26 cents for OpenAI, about 2.5 times the cost.
So, that's all right. It could probably be much much better. But the kind of big thing here, the elephant in the room, is that this method is not going to scale, right? As we said at the top, we're giving these models the entire database. Now, it works here in our contrived example because we're giving them 60 models and we're only looking at 30 cases.
It's a tiny database. In real life, a database is going to house tens of thousands, hundreds of thousands, maybe millions of company names. So, this isn't going to scale. We can't pass them the entire database. It's going to be slow.
It's going to be expensive. And frankly, we're just going to run out of tokens. So, let's see if we can improve the scalability of this by using our phonetic encoding method that we mentioned at the top. We'll get into the details a little bit, but at a high level basically all we're trying to do is generate a short list of candidates for these models. Let's say 10 to 20 to 25, maybe 50, maybe 100.
We'll see what the results look like. But that way they're picking from a static amount of company names no matter what, which is going to let us scale, it's going to let us keep our token cost down etc. So let's get into that next. Before we write the task, again let's look at our data. So remember we have this list of company records here.
We have our transcript cases and we have our answer key. Now what phonetic encoding does is if we take a look at this split screen here, on the left is again our company catalog with Cedar Vein Technologies. A phonetic encoder I'm using double metaphone in this example. What it's going to do is match the company ID to a string and you can see it gives us some aliases. So Cedar Vein Technologies, you can see this is kind of Cedar Vein right there and this will be represented by Technologies.
And that lets us pull what we can do is come into our transcript encodings. And if we look here you can see we have our snippet that's mentioning the Cedar Vein Technologies. We have our mention. We have our normalized name. We have our primary encoding as well as separate encodings right here.
So we're going to be able to cross check those encodings and generate our short list. So let's go back to our task and go write up step number two there. We're back in our task. Again I'm going to open our AI editor. I have my instructions already pre-ready to go here for step two.
So let's give it that. Let's let it chug away. We'll double check things and then we'll take a look at another output. Okay, so we're back at our task. Our AI editor has finished making its changes.
You can see we now have a phonetic script Python script here. We also have our new short list prompt both Jev and Decisions. So once again I've reviewed all the work here. Everything looks good to me. Let's go take a look at an output.
So, here we're looking at our phonetic comparison. Um you'll see the scores are a little bit different here. Um they're not quite as bad as they seem at first glance. So, the difference here is because we're generating a short list, there are cases where the model is actually given zero candidates. So, that's where you see this difference between 25 out of 30 and 22 out of 25.
So, this is the one we care about, the model accuracy cases asked, because in five cases we couldn't actually generate uh a suitable candidate from um our database. So, we can't hold the model accountable for that cuz we gave it no data. In the cases where we did have a uh a candidate in a short list, at least one of them, you can see now they're performing pretty well. I think Jev actually came down a couple points there. Um OpenAI is actually doing better.
Um so, the performance again, pretty good. We could probably get it better. I'd probably want to get it a little bit better there. Um up into the high 90s once again. Um in this case, OpenAI did a good job with its defer cases.
Um Jev did a very good job with the non cases. See here, our latency stayed about the same, so that's good. We could probably get that down, but I mean, honestly, that's pretty small anyway. However, the big thing, if you remember, um we've taken our cost down by uh 90% in both cases. Um so, let's take a little quick look at a side-by-side comparison between the phonetic and the raw cases.
Okay, so here's our phonetic on the left, here's our raw original on the right. You can see Jev 28 out of 30. You can see OpenAI got 24 out of 30. And over here, we're at 22 out of 25, 22 out of 25. Um and just to circle it once again, let's look at the cost difference here.
So, 10 cents, 11 cents if we round up per thousand cases for Jev, that's down to 1 cent per thousand cases. And 26 cents per thousand, that's now down to 3 cents. Um so, huge savings. But again, the biggest thing here is the scalability, right? That we know that this raw comparison was not going to scale well.
And now we know that this phonetic comparison, yes, we have lost a couple points in accuracy. We can get that back up. However, generating a short list, it doesn't matter how big our database grows, we're still going to be able to keep that scalable. Okay, so there you have it. We got a working task.
Um we kind of started with the brute force iteration. Um obviously, things were sped up, but the iteration to go to a step two, quite quick uh in natural time without the magic of video editing. Um and so, from here we can keep iterating. We can either attack that accuracy. We could lower costs even further if we want to.
Um we could try and speed our response time up. Um or we could add some support for adding a human review step for some of the more ambiguous cases. So, hopefully that was useful. Um I think that's a pretty cool uh use case of of Jev or OpenAI decisions. Um I'm going to dive into this uh later today and try and get it a little bit better.
Um so, stay tuned. Maybe I'll post some further iterations. And thanks for watching.