Parents know that a smooth ride depends on more than a powerful engine. You want to know how a vehicle handles when the road gets rough, and whether its safety systems work when pressure builds. Businesses face a similar question as they give AI more responsibility: how will it act when the week goes wrong?
Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
Firmulate puts that question to a live, watchable experiment. Its AI models run a small software company through crises, customer demands and attempts at manipulation. The aim is to see how they manage, not just how convincingly they talk.
A shared worst week
In the final Crucible League, held in July 2026, each frontier model faced the same company, customers, crises and temptations. Every decision was versioned and auditable. The results: gpt-5.6-sol scored 95, Kimi K3 93, Sonnet 5 88, Fable 5 77 and Opus 4.8 73. The do-nothing baseline scored 26. The league’s trust standard is plain: “no amount of good work outweighs a breach of trust.”
The striking result was not simply the ranking. Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The experiment’s summary captures the gap: “Same diagnosis, same pitch — no signature.” Recognizing the right move and carrying it through are different tests.
The clue was in the files
The deal hinged on a competitor weakness hidden two document references deep in the company’s own files. It was not included in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The detail makes the exercise feel less like a quiz with a tidy answer and more like a test of whether an AI can connect information scattered across the work it has been given.
Firmulate also staged fake CEO messages that escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” For a business considering AI in customer service or operations, that kind of boundary matters alongside speed and competence.
More effort did not guarantee the close
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. It left the close on the table and its discipline slipped: it attempted to write into a locked department instead of escalating. The same weakness appeared, more weakly, in all four models. Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh—a relevant caveat when comparing results.
The live company adds a human-readable frame to those decisions. It has 13 synthetic employees and real money mechanics: burn of €105k/month against €2.3k MRR, with a public cash countdown. Its playbooks include 680+ self-learned rules, and every workday is versioned. Readers can watch the experiment at firmulate.com. A separate quiz draws on 242 real, unedited management decisions and invites visitors to guess which model made them.
From watching to trying it on your company
A leaderboard can show who did well in one exercise. The more practical question for a company is what its own AI would do with its own customers, procedures and pressure points. Firmulate says enterprises can run the same kind of wargame against a read-only export of their business, producing a board report with model rankings and weaknesses in their playbooks. Nothing writes back to real systems.
To explore a pilot for your organization, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
