(../_next/static/media/797e433ab948586e-s.p.29207c2f.woff2)  (../_next/static/media/CursorGothic_Bold-s.p.95169710.woff2)  (../_next/static/media/CursorGothic_BoldItalic-s.p.036859cb.woff2)  (../_next/static/media/CursorGothic_Italic-s.p.00bb2606.woff2)  (../_next/static/media/CursorGothic_Regular-s.p.a361088d.woff2)   (../_next/static/chunks/d096e46d614aab4c.css@dpl=dpl_ABGtodWDkmm7BqTikqe2cpVTakLw.css)  (../_next/static/chunks/51baa5b0968bf3ce.css@dpl=dpl_ABGtodWDkmm7BqTikqe2cpVTakLw.css)  (../_next/static/chunks/614d29df71c5dc25.js@dpl=dpl_ABGtodWDkmm7BqTikqe2cpVTakLw)  (https://snap.licdn.com/li.lms-analytics/insight.min.js)   Real-SWE Benchmark — Specific Labs  (real-swe.html)  (../icon@70317f5ff00e501d)    (https://withspecific.com) (Specific)   (https://withspecific.com/benchmarks) ← Benchmarks  September 2026  Introducing Real-SWE Benchmarking frontier AI models on private, real-world, enterprise codebases.  (real-swe.html#leaderboard) Results (real-swe.html#task-by-task) Analysis (real-swe.html#effort) Effort (real-swe.html#evaluation-setup) Setup  .                         
;+;+;:;+;+;.#                                                                                           
;           ;                                                                                           
; .   . . . ;                                    .                                                      
+           ;                                                                                           
; .     .   : @;;+;+#                            +;+;+;+;:;+:#                                          
:           : ;     .       ;                    ;           ;              @;+;+.+;:;+;+*              
; . .       : ; . . :       ;       @++:;:;+.;;@ ;. . .   . .:     .        ; .     . . .;  @++;;;;+;:;@
;           : ;     ; @;+++:@;++;:+ ;          : :           . :+.+;;;+:;@  ;            :  +          .
; . .   .   ; ;   . ; : . . . .     ; . . . . .; ;  . . . . .+ +  . . . .;  ; . .   . . .;  ;     . .  :
;           : ;     ; ;           ; :          ; .           ; .         ;  ;            ;  ;          :
;       . . %   . . ; : . .   . . ; ; ... . . .; +. .   . . .: ;. . :.. .:  : . . ::. .  .  ; .   . . .;
:        *. @ ;     ; +.@     .   ; : +:    *+ : ;    :.     %+;    +.   ;  *.    *#     @: :.*.       ;
; .   . .;::@;; . . ; :#@;. ..@:. . ;.@%: .:@%:: :. . #+.   :@%;. .:@*. .:  @+  . #*. . ;@#.;.*   . . .:
.::: ..:%@@+@+#... :+:%@@+..:@@@::*.:;@@:..+@@:; *.::#@@#:..:@@#...+@#: :*:%@@+ .:.+::..*@@:@@@#;.. ::.#
        +@#.@         .*@:  .*@%:     *:   :@%.      ;@@:   .@%.    @     .;@@:   .:    :@#..#@*.       
        +@#.@          .@   .*@%:.    *:    %+       ;@@:    #;     +#    .;@@:   ..     @: .#@*.       
         @. ..         ;@   .*@%:     *:    @@       ;@@:    %#    * #    .;@@:          @+  .@         
        :*#..          @::    @.       .   ::;.       @@     ..     .       @%          .*@  ;@.        
        %.@            ..    .@;            .         @@     ..            .@@.          ..  @:#        
        ...             .    ;%#             .        @@                   +;#:          .. .@ @        
         .                   ...                      ...                   ..               ..         
                              ...                     ..                   ...                .   01 Introduction Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product. Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet. Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services. Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.  Can a coding agent actually do the work of a software engineer in the real world?   1 ()  Fable 5.1 Claude Code  Resolution rate:  38.8%        2 ()  GPT-6 Astra Codex CLI  Resolution rate:  33.8%        3 ()  Gemini 3.8 Flash Gemini CLI  Resolution rate:  31.2%        4 ()  GLM 5.3 Claude Code  Resolution rate:  28.8%        =5 ()  Grok 4.6 Grok Build  Resolution rate:  23.8%        =5 ()  Muse Spark 1.3 Muse Code  Resolution rate:  23.8%        7 ()  Kimi K3 Kimi Code  Resolution rate:  18.8%        8 ()  GPT-5.6 Sol Codex CLI  Resolution rate:  16.2%         # Model Harness Resolution rate   1 ()  Fable 5.1  Claude Code     38.8%    2 ()  GPT-6 Astra  Codex CLI     33.8%    3 ()  Gemini 3.8 Flash  Gemini CLI     31.2%    4 ()  GLM 5.3  Claude Code     28.8%    =5 ()  Grok 4.6  Grok Build     23.8%    =5 ()  Muse Spark 1.3  Muse Code     23.8%    7 ()  Kimi K3  Kimi Code     18.8%    8 ()  GPT-5.6 Sol  Codex CLI     16.2%       Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown.   Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models. We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.  Real company tasks require company-specific context Correct billing depends on business rules and external services Fix invoice billing so each business charges the right tax and exempt customers aren't taxed.  View full instruction Hide full instruction ▾  Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL , PROD_TAX_JAR_URL and INFLUX_URL .  Services in the sandbox TJ TaxJar sandbox TJ TaxJar production   InfluxDB ledger   NestJS service   TypeScript     Agents work across code, infrastructure, and business tools Tools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.   AWS emulator   Docker   Kubernetes   GitHub   Linear MCP   PostgreSQL   MySQL   MongoDB Ge Gel   Redis   Go   Python   Node.js   Vitest   Slack   Intercom   Google Drive    Email   ClickUp    Codebase Selection We selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The sample tasks analyzed below come from these codebases, including: A Luma/Partiful competitor with 200K+ users and atop 100 App Store ranking  A consumer fintech platform processing100K+ bank statements  Enterprise AI sales platforms supporting complex business workflows  We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.  Brief instructions can require changes across many files Our tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions. The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.  Prompt length · median A typical Real-SWE instruction is 1,742 characters. FrontierCode 2,056 chars     DeepSWE 1,975 chars     Terminal-Bench 3 1,584 chars     FrontierSWE v2 992 chars     Real-SWE 1,742 chars       Files edited by the reference solution · median 11 files in Real-SWE, compared with 6 in FrontierCode and DeepSWE. FrontierCode 6     DeepSWE 6     Real-SWE 11        All figures are medians. FrontierCode and DeepSWE use(https://cognition.com/blog/frontier-code) Cognition's published comparison ; FrontierCode includes task descriptions and codebase guidelines. We measured instruction files from(https://github.com/harbor-framework/terminal-bench/tree/2b0442c3c583b710ca8da14c8e601b99f2f1f244/tasks) Terminal-Bench 3's 74 tasks ,(https://github.com/Proximal-Labs/frontier-swe-v2/tree/9e3f71cac38ef3d7e14a41b361c7b2b54c59899b/tasks) FrontierSWE v2's 34 tasks , and Real-SWE's eight repository-backed sample tasks. Character counts are rounded to the nearest whole character. No comparable files-edited figure is included for Terminal-Bench 3 or FrontierSWE v2.  Models fail even in short rollouts. 71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts. Triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns is difficult. Under 10 min Under 10 min: 70 failed (71.4%) and 28 passed (28.6%), out of 98 rollouts.   71.4% 28.6%   70/98failed  10 min or longer 10 min or longer: 398 failed (73.4%) and 144 passed (26.6%), out of 542 rollouts.   73.4% 26.6%   398/542failed    Failed  Passed    Every task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons: Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models. These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world. Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.     02 Analysis Here's an analysis of a small sample of tasks from our benchmark. If you're interested in the sample,(https://withspecific.com/benchmarks/real-swe/request-access) request access here . 6of 10tasks have resolution rates below 15% Select a task to view model results. Percentages show the overall resolution rate. Multi-region sweep 67.2% ⌄  ()  Fable 5.1 7/8passed  ()  GPT-6 Astra 8/8passed  ()  Gemini 3.8 Flash 8/8passed  ()  GLM 5.3 2/8passed  ()  Grok 4.6 3/8passed  ()  Muse Spark 1.3 8/8passed  ()  Kimi K3 2/8passed  ()  GPT-5.6 Sol 5/8passed    API keys & environments 65.6% ⌄  ()  Fable 5.1 8/8passed  ()  GPT-6 Astra 5/8passed  ()  Gemini 3.8 Flash 7/8passed  ()  GLM 5.3 5/8passed  ()  Grok 4.6 4/8passed  ()  Muse Spark 1.3 6/8passed  ()  Kimi K3 0/8passed  ()  GPT-5.6 Sol 7/8passed    Entitlement overage lines 50.0% ⌄  ()  Fable 5.1 8/8passed  ()  GPT-6 Astra 7/8passed  ()  Gemini 3.8 Flash 5/8passed  ()  GLM 5.3 3/8passed  ()  Grok 4.6 1/8passed  ()  Muse Spark 1.3 1/8passed  ()  Kimi K3 6/8passed  ()  GPT-5.6 Sol 1/8passed    Customer identity migration 40.6% ⌄  ()  Fable 5.1 3/8passed  ()  GPT-6 Astra 1/8passed  ()  Gemini 3.8 Flash 3/8passed  ()  GLM 5.3 4/8passed  ()  Grok 4.6 8/8passed  ()  Muse Spark 1.3 3/8passed  ()  Kimi K3 4/8passed  ()  GPT-5.6 Sol 0/8passed    Billing schedule migration 14.1% ⌄  ()  Fable 5.1 3/8passed  ()  GPT-6 Astra 1/8passed  ()  Gemini 3.8 Flash 2/8passed  ()  GLM 5.3 2/8passed  ()  Grok 4.6 0/8passed  ()  Muse Spark 1.3 0/8passed  ()  Kimi K3 1/8passed  ()  GPT-5.6 Sol 0/8passed    API token metering 12.5% ⌄  ()  Fable 5.1 1/8passed  ()  GPT-6 Astra 5/8passed  ()  Gemini 3.8 Flash 0/8passed  ()  GLM 5.3 1/8passed  ()  Grok 4.6 0/8passed  ()  Muse Spark 1.3 0/8passed  ()  Kimi K3 1/8passed  ()  GPT-5.6 Sol 0/8passed    S3 datastore measurement 10.9% ⌄  ()  Fable 5.1 0/8passed  ()  GPT-6 Astra 0/8passed  ()  Gemini 3.8 Flash 0/8passed  ()  GLM 5.3 3/8passed  ()  Grok 4.6 2/8passed  ()  Muse Spark 1.3 1/8passed  ()  Kimi K3 1/8passed  ()  GPT-5.6 Sol 0/8passed    Linearizable scan 4.7% ⌄  ()  Fable 5.1 0/8passed  ()  GPT-6 Astra 0/8passed  ()  Gemini 3.8 Flash 0/8passed  ()  GLM 5.3 2/8passed  ()  Grok 4.6 1/8passed  ()  Muse Spark 1.3 0/8passed  ()  Kimi K3 0/8passed  ()  GPT-5.6 Sol 0/8passed    Tax jurisdiction 3.1% ⌄  ()  Fable 5.1 1/8passed  ()  GPT-6 Astra 0/8passed  ()  Gemini 3.8 Flash 0/8passed  ()  GLM 5.3 1/8passed  ()  Grok 4.6 0/8passed  ()  Muse Spark 1.3 0/8passed  ()  Kimi K3 0/8passed  ()  GPT-5.6 Sol 0/8passed    Analytics stream reducer 0.0% ⌄  ()  Fable 5.1 0/8passed  ()  GPT-6 Astra 0/8passed  ()  Gemini 3.8 Flash 0/8passed  ()  GLM 5.3 0/8passed  ()  Grok 4.6 0/8passed  ()  Muse Spark 1.3 0/8passed  ()  Kimi K3 0/8passed  ()  GPT-5.6 Sol 0/8passed         Task ()  Fable 5.1   ()  GPT-6 Astra   ()  Gemini 3.8 Flash   ()  GLM 5.3   ()  Grok 4.6   ()  Muse Spark 1.3   ()  Kimi K3   ()  GPT-5.6 Sol   Resolution rate   Multi-region sweep (Multi-region sweep · Fable 5.1: 7/8) 7/8  (Multi-region sweep · GPT-6 Astra: 8/8) 8/8  (Multi-region sweep · Gemini 3.8 Flash: 8/8) 8/8  (Multi-region sweep · GLM 5.3: 2/8) 2/8  (Multi-region sweep · Grok 4.6: 3/8) 3/8  (Multi-region sweep · Muse Spark 1.3: 8/8) 8/8  (Multi-region sweep · Kimi K3: 2/8) 2/8  (Multi-region sweep · GPT-5.6 Sol: 5/8) 5/8    67.2%    API keys & environments (API keys & environments · Fable 5.1: 8/8) 8/8  (API keys & environments · GPT-6 Astra: 5/8) 5/8  (API keys & environments · Gemini 3.8 Flash: 7/8) 7/8  (API keys & environments · GLM 5.3: 5/8) 5/8  (API keys & environments · Grok 4.6: 4/8) 4/8  (API keys & environments · Muse Spark 1.3: 6/8) 6/8  (API keys & environments · Kimi K3: 0/8) 0/8  (API keys & environments · GPT-5.6 Sol: 7/8) 7/8    65.6%    Entitlement overage lines (Entitlement overage lines · Fable 5.1: 8/8) 8/8  (Entitlement overage lines · GPT-6 Astra: 7/8) 7/8  (Entitlement overage lines · Gemini 3.8 Flash: 5/8) 5/8  (Entitlement overage lines · GLM 5.3: 3/8) 3/8  (Entitlement overage lines · Grok 4.6: 1/8) 1/8  (Entitlement overage lines · Muse Spark 1.3: 1/8) 1/8  (Entitlement overage lines · Kimi K3: 6/8) 6/8  (Entitlement overage lines · GPT-5.6 Sol: 1/8) 1/8    50.0%    Customer identity migration (Customer identity migration · Fable 5.1: 3/8) 3/8  (Customer identity migration · GPT-6 Astra: 1/8) 1/8  (Customer identity migration · Gemini 3.8 Flash: 3/8) 3/8  (Customer identity migration · GLM 5.3: 4/8) 4/8  (Customer identity migration · Grok 4.6: 8/8) 8/8  (Customer identity migration · Muse Spark 1.3: 3/8) 3/8  (Customer identity migration · Kimi K3: 4/8) 4/8  (Customer identity migration · GPT-5.6 Sol: 0/8) 0/8    40.6%    Billing schedule migration (Billing schedule migration · Fable 5.1: 3/8) 3/8  (Billing schedule migration · GPT-6 Astra: 1/8) 1/8  (Billing schedule migration · Gemini 3.8 Flash: 2/8) 2/8  (Billing schedule migration · GLM 5.3: 2/8) 2/8  (Billing schedule migration · Grok 4.6: 0/8) 0/8  (Billing schedule migration · Muse Spark 1.3: 0/8) 0/8  (Billing schedule migration · Kimi K3: 1/8) 1/8  (Billing schedule migration · GPT-5.6 Sol: 0/8) 0/8    14.1%    API token metering (API token metering · Fable 5.1: 1/8) 1/8  (API token metering · GPT-6 Astra: 5/8) 5/8  (API token metering · Gemini 3.8 Flash: 0/8) 0/8  (API token metering · GLM 5.3: 1/8) 1/8  (API token metering · Grok 4.6: 0/8) 0/8  (API token metering · Muse Spark 1.3: 0/8) 0/8  (API token metering · Kimi K3: 1/8) 1/8  (API token metering · GPT-5.6 Sol: 0/8) 0/8    12.5%    S3 datastore measurement (S3 datastore measurement · Fable 5.1: 0/8) 0/8  (S3 datastore measurement · GPT-6 Astra: 0/8) 0/8  (S3 datastore measurement · Gemini 3.8 Flash: 0/8) 0/8  (S3 datastore measurement · GLM 5.3: 3/8) 3/8  (S3 datastore measurement · Grok 4.6: 2/8) 2/8  (S3 datastore measurement · Muse Spark 1.3: 1/8) 1/8  (S3 datastore measurement · Kimi K3: 1/8) 1/8  (S3 datastore measurement · GPT-5.6 Sol: 0/8) 0/8    10.9%    Linearizable scan (Linearizable scan · Fable 5.1: 0/8) 0/8  (Linearizable scan · GPT-6 Astra: 0/8) 0/8  (Linearizable scan · Gemini 3.8 Flash: 0/8) 0/8  (Linearizable scan · GLM 5.3: 2/8) 2/8  (Linearizable scan · Grok 4.6: 1/8) 1/8  (Linearizable scan · Muse Spark 1.3: 0/8) 0/8  (Linearizable scan · Kimi K3: 0/8) 0/8  (Linearizable scan · GPT-5.6 Sol: 0/8) 0/8    4.7%    Tax jurisdiction (Tax jurisdiction · Fable 5.1: 1/8) 1/8  (Tax jurisdiction · GPT-6 Astra: 0/8) 0/8  (Tax jurisdiction · Gemini 3.8 Flash: 0/8) 0/8  (Tax jurisdiction · GLM 5.3: 1/8) 1/8  (Tax jurisdiction · Grok 4.6: 0/8) 0/8  (Tax jurisdiction · Muse Spark 1.3: 0/8) 0/8  (Tax jurisdiction · Kimi K3: 0/8) 0/8  (Tax jurisdiction · GPT-5.6 Sol: 0/8) 0/8    3.1%    Analytics stream reducer (Analytics stream reducer · Fable 5.1: 0/8) 0/8  (Analytics stream reducer · GPT-6 Astra: 0/8) 0/8  (Analytics stream reducer · Gemini 3.8 Flash: 0/8) 0/8  (Analytics stream reducer · GLM 5.3: 0/8) 0/8  (Analytics stream reducer · Grok 4.6: 0/8) 0/8  (Analytics stream reducer · Muse Spark 1.3: 0/8) 0/8  (Analytics stream reducer · Kimi K3: 0/8) 0/8  (Analytics stream reducer · GPT-5.6 Sol: 0/8) 0/8    0.0%       Each task had 8 rollouts per model.  Missed requirements are the most common failure Failures are grouped by observed submission behavior using the same taxonomy across models, following(https://arxiv.org/html/2607.07946v1#A3) DeepSWE .  ()  Fable 5.1  (Unverified assumption: 24.5% of failures) 24.5%  (Missed requirement: 36.7% of failures) 36.7%  (Integration error: 34.7% of failures) 34.7%  (Regression: 4.1% of failures) 4.1%    ()  GPT-6 Astra  (Unverified assumption: 34.0% of failures) 34.0%  (Missed requirement: 28.3% of failures) 28.3%  (Integration error: 34.0% of failures) 34.0%  (Regression: 3.8% of failures) 3.8%    ()  Gemini 3.8 Flash  (Unverified assumption: 10.9% of failures) 10.9%  (Missed requirement: 29.1% of failures) 29.1%  (Integration error: 49.1% of failures) 49.1%  (Regression: 10.9% of failures) 10.9%    ()  GLM 5.3  (Unverified assumption: 28.1% of failures) 28.1%  (Missed requirement: 38.6% of failures) 38.6%  (Integration error: 26.3% of failures) 26.3%  (Wrong file: 7.0% of failures) 7.0%    ()  Grok 4.6  (Unverified assumption: 24.6% of failures) 24.6%  (Missed requirement: 67.2% of failures) 67.2%  (Integration error: 8.2% of failures) 8.2%    ()  Muse Spark 1.3  (Unverified assumption: 19.7% of failures) 19.7%  (Missed requirement: 36.1% of failures) 36.1%  (Integration error: 41.0% of failures) 41.0%  (Regression: 3.3% of failures) 3.3%    ()  Kimi K3  (Unverified assumption: 15.4% of failures) 15.4%  (Missed requirement: 53.8% of failures) 53.8%  (Integration error: 27.7% of failures) 27.7%  (Wrong file: 3.1% of failures) 3.1%    ()  GPT-5.6 Sol  (Unverified assumption: 43.3% of failures) 43.3%  (Missed requirement: 31.3% of failures) 31.3%  (Integration error: 16.4% of failures) 16.4%  (Regression: 9.0% of failures) 9.0%       Unverified assumption  Missed requirement  Integration error  Regression  Wrong file   No model solves every task One square per rollout: each row is a task, each column a trial, eight trials per task for every model. ()  Fable 5.1 01 (Fable 5.1 · Entitlement overage lines · trial 1: Pass)  (Fable 5.1 · Entitlement overage lines · trial 2: Pass)  (Fable 5.1 · Entitlement overage lines · trial 3: Pass)  (Fable 5.1 · Entitlement overage lines · trial 4: Pass)  (Fable 5.1 · Entitlement overage lines · trial 5: Pass)  (Fable 5.1 · Entitlement overage lines · trial 6: Pass)  (Fable 5.1 · Entitlement overage lines · trial 7: Pass)  (Fable 5.1 · Entitlement overage lines · trial 8: Pass)    02 (Fable 5.1 · Multi-region sweep · trial 1: Pass)  (Fable 5.1 · Multi-region sweep · trial 2: Pass)  (Fable 5.1 · Multi-region sweep · trial 3: Pass)  (Fable 5.1 · Multi-region sweep · trial 4: Pass)  (Fable 5.1 · Multi-region sweep · trial 5: Pass)  (Fable 5.1 · Multi-region sweep · trial 6: Missed requirement)  (Fable 5.1 · Multi-region sweep · trial 7: Pass)  (Fable 5.1 · Multi-region sweep · trial 8: Pass)    03 (Fable 5.1 · Tax jurisdiction · trial 1: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 2: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 3: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 4: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 5: Pass)  (Fable 5.1 · Tax jurisdiction · trial 6: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 7: Unverified assumption)  (Fable 5.1 · Tax jurisdiction · trial 8: Unverified assumption)    04 (Fable 5.1 · API token metering · trial 1: Missed requirement)  (Fable 5.1 · API token metering · trial 2: Integration error)  (Fable 5.1 · API token metering · trial 3: Integration error)  (Fable 5.1 · API token metering · trial 4: Pass)  (Fable 5.1 · API token metering · trial 5: Integration error)  (Fable 5.1 · API token metering · trial 6: Integration error)  (Fable 5.1 · API token metering · trial 7: Integration error)  (Fable 5.1 · API token metering · trial 8: Integration error)    05 (Fable 5.1 · API keys & environments · trial 1: Pass)  (Fable 5.1 · API keys & environments · trial 2: Pass)  (Fable 5.1 · API keys & environments · trial 3: Pass)  (Fable 5.1 · API keys & environments · trial 4: Pass)  (Fable 5.1 · API keys & environments · trial 5: Pass)  (Fable 5.1 · API keys & environments · trial 6: Pass)  (Fable 5.1 · API keys & environments · trial 7: Pass)  (Fable 5.1 · API keys & environments · trial 8: Pass)    06 (Fable 5.1 · S3 datastore measurement · trial 1: Integration error)  (Fable 5.1 · S3 datastore measurement · trial 2: Integration error)  (Fable 5.1 · S3 datastore measurement · trial 3: Integration error)  (Fable 5.1 · S3 datastore measurement · trial 4: Integration error)  (Fable 5.1 · S3 datastore measurement · trial 5: Integration error)  (Fable 5.1 · S3 datastore measurement · trial 6: Regression)  (Fable 5.1 · S3 datastore measurement · trial 7: Regression)  (Fable 5.1 · S3 datastore measurement · trial 8: Integration error)    07 (Fable 5.1 · Customer identity migration · trial 1: Integration error)  (Fable 5.1 · Customer identity migration · trial 2: Integration error)  (Fable 5.1 · Customer identity migration · trial 3: Integration error)  (Fable 5.1 · Customer identity migration · trial 4: Integration error)  (Fable 5.1 · Customer identity migration · trial 5: Pass)  (Fable 5.1 · Customer identity migration · trial 6: Pass)  (Fable 5.1 · Customer identity migration · trial 7: Integration error)  (Fable 5.1 · Customer identity migration · trial 8: Pass)    08 (Fable 5.1 · Billing schedule migration · trial 1: Pass)  (Fable 5.1 · Billing schedule migration · trial 2: Unverified assumption)  (Fable 5.1 · Billing schedule migration · trial 3: Pass)  (Fable 5.1 · Billing schedule migration · trial 4: Unverified assumption)  (Fable 5.1 · Billing schedule migration · trial 5: Pass)  (Fable 5.1 · Billing schedule migration · trial 6: Unverified assumption)  (Fable 5.1 · Billing schedule migration · trial 7: Unverified assumption)  (Fable 5.1 · Billing schedule migration · trial 8: Unverified assumption)    09 (Fable 5.1 · Linearizable scan · trial 1: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 2: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 3: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 4: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 5: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 6: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 7: Missed requirement)  (Fable 5.1 · Linearizable scan · trial 8: Missed requirement)    10 (Fable 5.1 · Analytics stream reducer · trial 1: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 2: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 3: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 4: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 5: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 6: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 7: Missed requirement)  (Fable 5.1 · Analytics stream reducer · trial 8: Missed requirement)      ()  GPT-6 Astra 01 (GPT-6 Astra · Entitlement overage lines · trial 1: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 2: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 3: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 4: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 5: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 6: Unverified assumption)  (GPT-6 Astra · Entitlement overage lines · trial 7: Pass)  (GPT-6 Astra · Entitlement overage lines · trial 8: Pass)    02 (GPT-6 Astra · Multi-region sweep · trial 1: Pass)  (GPT-6 Astra · Multi-region sweep · trial 2: Pass)  (GPT-6 Astra · Multi-region sweep · trial 3: Pass)  (GPT-6 Astra · Multi-region sweep · trial 4: Pass)  (GPT-6 Astra · Multi-region sweep · trial 5: Pass)  (GPT-6 Astra · Multi-region sweep · trial 6: Pass)  (GPT-6 Astra · Multi-region sweep · trial 7: Pass)  (GPT-6 Astra · Multi-region sweep · trial 8: Pass)    03 (GPT-6 Astra · Tax jurisdiction · trial 1: Missed requirement)  (GPT-6 Astra · Tax jurisdiction · trial 2: Unverified assumption)  (GPT-6 Astra · Tax jurisdiction · trial 3: Unverified assumption)  (GPT-6 Astra · Tax jurisdiction · trial 4: Unverified assumption)  (GPT-6 Astra · Tax jurisdiction · trial 5: Unverified assumption)  (GPT-6 Astra · Tax jurisdiction · trial 6: Missed requirement)  (GPT-6 Astra · Tax jurisdiction · trial 7: Unverified assumption)  (GPT-6 Astra · Tax jurisdiction · trial 8: Missed requirement)    04 (GPT-6 Astra · API token metering · trial 1: Pass)  (GPT-6 Astra · API token metering · trial 2: Pass)  (GPT-6 Astra · API token metering · trial 3: Integration error)  (GPT-6 Astra · API token metering · trial 4: Integration error)  (GPT-6 Astra · API token metering · trial 5: Pass)  (GPT-6 Astra · API token metering · trial 6: Pass)  (GPT-6 Astra · API token metering · trial 7: Pass)  (GPT-6 Astra · API token metering · trial 8: Integration error)    05 (GPT-6 Astra · API keys & environments · trial 1: Integration error)  (GPT-6 Astra · API keys & environments · trial 2: Integration error)  (GPT-6 Astra · API keys & environments · trial 3: Pass)  (GPT-6 Astra · API keys & environments · trial 4: Integration error)  (GPT-6 Astra · API keys & environments · trial 5: Pass)  (GPT-6 Astra · API keys & environments · trial 6: Pass)  (GPT-6 Astra · API keys & environments · trial 7: Pass)  (GPT-6 Astra · API keys & environments · trial 8: Pass)    06 (GPT-6 Astra · S3 datastore measurement · trial 1: Regression)  (GPT-6 Astra · S3 datastore measurement · trial 2: Integration error)  (GPT-6 Astra · S3 datastore measurement · trial 3: Integration error)  (GPT-6 Astra · S3 datastore measurement · trial 4: Regression)  (GPT-6 Astra · S3 datastore measurement · trial 5: Integration error)  (GPT-6 Astra · S3 datastore measurement · trial 6: Integration error)  (GPT-6 Astra · S3 datastore measurement · trial 7: Missed requirement)  (GPT-6 Astra · S3 datastore measurement · trial 8: Integration error)    07 (GPT-6 Astra · Customer identity migration · trial 1: Integration error)  (GPT-6 Astra · Customer identity migration · trial 2: Integration error)  (GPT-6 Astra · Customer identity migration · trial 3: Integration error)  (GPT-6 Astra · Customer identity migration · trial 4: Integration error)  (GPT-6 Astra · Customer identity migration · trial 5: Integration error)  (GPT-6 Astra · Customer identity migration · trial 6: Integration error)  (GPT-6 Astra · Customer identity migration · trial 7: Pass)  (GPT-6 Astra · Customer identity migration · trial 8: Integration error)    08 (GPT-6 Astra · Billing schedule migration · trial 1: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 2: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 3: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 4: Pass)  (GPT-6 Astra · Billing schedule migration · trial 5: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 6: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 7: Unverified assumption)  (GPT-6 Astra · Billing schedule migration · trial 8: Unverified assumption)    09 (GPT-6 Astra · Linearizable scan · trial 1: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 2: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 3: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 4: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 5: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 6: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 7: Missed requirement)  (GPT-6 Astra · Linearizable scan · trial 8: Missed requirement)    10 (GPT-6 Astra · Analytics stream reducer · trial 1: Unverified assumption)  (GPT-6 Astra · Analytics stream reducer · trial 2: Unverified assumption)  (GPT-6 Astra · Analytics stream reducer · trial 3: Unverified assumption)  (GPT-6 Astra · Analytics stream reducer · trial 4: Missed requirement)  (GPT-6 Astra · Analytics stream reducer · trial 5: Missed requirement)  (GPT-6 Astra · Analytics stream reducer · trial 6: Unverified assumption)  (GPT-6 Astra · Analytics stream reducer · trial 7: Missed requirement)  (GPT-6 Astra · Analytics stream reducer · trial 8: Unverified assumption)      ()  Gemini 3.8 Flash 01 (Gemini 3.8 Flash · Entitlement overage lines · trial 1: Pass)  (Gemini 3.8 Flash · Entitlement overage lines · trial 2: Pass)  (Gemini 3.8 Flash · Entitlement overage lines · trial 3: Missed requirement)  (Gemini 3.8 Flash · Entitlement overage lines · trial 4: Missed requirement)  (Gemini 3.8 Flash · Entitlement overage lines · trial 5: Pass)  (Gemini 3.8 Flash · Entitlement overage lines · trial 6: Pass)  (Gemini 3.8 Flash · Entitlement overage lines · trial 7: Pass)  (Gemini 3.8 Flash · Entitlement overage lines · trial 8: Missed requirement)    02 (Gemini 3.8 Flash · Multi-region sweep · trial 1: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 2: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 3: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 4: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 5: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 6: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 7: Pass)  (Gemini 3.8 Flash · Multi-region sweep · trial 8: Pass)    03 (Gemini 3.8 Flash · Tax jurisdiction · trial 1: Missed requirement)  (Gemini 3.8 Flash · Tax jurisdiction · trial 2: Integration error)  (Gemini 3.8 Flash · Tax jurisdiction · trial 3: Unverified assumption)  (Gemini 3.8 Flash · Tax jurisdiction · trial 4: Unverified assumption)  (Gemini 3.8 Flash · Tax jurisdiction · trial 5: Missed requirement)  (Gemini 3.8 Flash · Tax jurisdiction · trial 6: Unverified assumption)  (Gemini 3.8 Flash · Tax jurisdiction · trial 7: Integration error)  (Gemini 3.8 Flash · Tax jurisdiction · trial 8: Integration error)    04 (Gemini 3.8 Flash · API token metering · trial 1: Integration error)  (Gemini 3.8 Flash · API token metering · trial 2: Integration error)  (Gemini 3.8 Flash · API token metering · trial 3: Integration error)  (Gemini 3.8 Flash · API token metering · trial 4: Integration error)  (Gemini 3.8 Flash · API token metering · trial 5: Integration error)  (Gemini 3.8 Flash · API token metering · trial 6: Integration error)  (Gemini 3.8 Flash · API token metering · trial 7: Integration error)  (Gemini 3.8 Flash · API token metering · trial 8: Integration error)    05 (Gemini 3.8 Flash · API keys & environments · trial 1: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 2: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 3: Missed requirement)  (Gemini 3.8 Flash · API keys & environments · trial 4: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 5: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 6: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 7: Pass)  (Gemini 3.8 Flash · API keys & environments · trial 8: Pass)    06 (Gemini 3.8 Flash · S3 datastore measurement · trial 1: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 2: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 3: Missed requirement)  (Gemini 3.8 Flash · S3 datastore measurement · trial 4: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 5: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 6: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 7: Regression)  (Gemini 3.8 Flash · S3 datastore measurement · trial 8: Unverified assumption)    07 (Gemini 3.8 Flash · Customer identity migration · trial 1: Missed requirement)  (Gemini 3.8 Flash · Customer identity migration · trial 2: Integration error)  (Gemini 3.8 Flash · Customer identity migration · trial 3: Pass)  (Gemini 3.8 Flash · Customer identity migration · trial 4: Integration error)  (Gemini 3.8 Flash · Customer identity migration · trial 5: Integration error)  (Gemini 3.8 Flash · Customer identity migration · trial 6: Pass)  (Gemini 3.8 Flash · Customer identity migration · trial 7: Pass)  (Gemini 3.8 Flash · Customer identity migration · trial 8: Integration error)    08 (Gemini 3.8 Flash · Billing schedule migration · trial 1: Pass)  (Gemini 3.8 Flash · Billing schedule migration · trial 2: Integration error)  (Gemini 3.8 Flash · Billing schedule migration · trial 3: Integration error)  (Gemini 3.8 Flash · Billing schedule migration · trial 4: Integration error)  (Gemini 3.8 Flash · Billing schedule migration · trial 5: Integration error)  (Gemini 3.8 Flash · Billing schedule migration · trial 6: Integration error)  (Gemini 3.8 Flash · Billing schedule migration · trial 7: Pass)  (Gemini 3.8 Flash · Billing schedule migration · trial 8: Integration error)    09 (Gemini 3.8 Flash · Linearizable scan · trial 1: Missed requirement)  (Gemini 3.8 Flash · Linearizable scan · trial 2: Integration error)  (Gemini 3.8 Flash · Linearizable scan · trial 3: Integration error)  (Gemini 3.8 Flash · Linearizable scan · trial 4: Missed requirement)  (Gemini 3.8 Flash · Linearizable scan · trial 5: Missed requirement)  (Gemini 3.8 Flash · Linearizable scan · trial 6: Missed requirement)  (Gemini 3.8 Flash · Linearizable scan · trial 7: Missed requirement)  (Gemini 3.8 Flash · Linearizable scan · trial 8: Missed requirement)    10 (Gemini 3.8 Flash · Analytics stream reducer · trial 1: Integration error)  (Gemini 3.8 Flash · Analytics stream reducer · trial 2: Unverified assumption)  (Gemini 3.8 Flash · Analytics stream reducer · trial 3: Unverified assumption)  (Gemini 3.8 Flash · Analytics stream reducer · trial 4: Missed requirement)  (Gemini 3.8 Flash · Analytics stream reducer · trial 5: Missed requirement)  (Gemini 3.8 Flash · Analytics stream reducer · trial 6: Integration error)  (Gemini 3.8 Flash · Analytics stream reducer · trial 7: Integration error)  (Gemini 3.8 Flash · Analytics stream reducer · trial 8: Integration error)      ()  GLM 5.3 01 (GLM 5.3 · Entitlement overage lines · trial 1: Missed requirement)  (GLM 5.3 · Entitlement overage lines · trial 2: Pass)  (GLM 5.3 · Entitlement overage lines · trial 3: Pass)  (GLM 5.3 · Entitlement overage lines · trial 4: Missed requirement)  (GLM 5.3 · Entitlement overage lines · trial 5: Pass)  (GLM 5.3 · Entitlement overage lines · trial 6: Missed requirement)  (GLM 5.3 · Entitlement overage lines · trial 7: Missed requirement)  (GLM 5.3 · Entitlement overage lines · trial 8: Missed requirement)    02 (GLM 5.3 · Multi-region sweep · trial 1: Wrong file)  (GLM 5.3 · Multi-region sweep · trial 2: Pass)  (GLM 5.3 · Multi-region sweep · trial 3: Wrong file)  (GLM 5.3 · Multi-region sweep · trial 4: Wrong file)  (GLM 5.3 · Multi-region sweep · trial 5: Pass)  (GLM 5.3 · Multi-region sweep · trial 6: Wrong file)  (GLM 5.3 · Multi-region sweep · trial 7: Integration error)  (GLM 5.3 · Multi-region sweep · trial 8: Integration error)    03 (GLM 5.3 · Tax jurisdiction · trial 1: Unverified assumption)  (GLM 5.3 · Tax jurisdiction · trial 2: Unverified assumption)  (GLM 5.3 · Tax jurisdiction · trial 3: Unverified assumption)  (GLM 5.3 · Tax jurisdiction · trial 4: Pass)  (GLM 5.3 · Tax jurisdiction · trial 5: Missed requirement)  (GLM 5.3 · Tax jurisdiction · trial 6: Unverified assumption)  (GLM 5.3 · Tax jurisdiction · trial 7: Missed requirement)  (GLM 5.3 · Tax jurisdiction · trial 8: Unverified assumption)    04 (GLM 5.3 · API token metering · trial 1: Integration error)  (GLM 5.3 · API token metering · trial 2: Integration error)  (GLM 5.3 · API token metering · trial 3: Integration error)  (GLM 5.3 · API token metering · trial 4: Pass)  (GLM 5.3 · API token metering · trial 5: Unverified assumption)  (GLM 5.3 · API token metering · trial 6: Integration error)  (GLM 5.3 · API token metering · trial 7: Integration error)  (GLM 5.3 · API token metering · trial 8: Integration error)    05 (GLM 5.3 · API keys & environments · trial 1: Pass)  (GLM 5.3 · API keys & environments · trial 2: Pass)  (GLM 5.3 · API keys & environments · trial 3: Integration error)  (GLM 5.3 · API keys & environments · trial 4: Pass)  (GLM 5.3 · API keys & environments · trial 5: Integration error)  (GLM 5.3 · API keys & environments · trial 6: Unverified assumption)  (GLM 5.3 · API keys & environments · trial 7: Pass)  (GLM 5.3 · API keys & environments · trial 8: Pass)    06 (GLM 5.3 · S3 datastore measurement · trial 1: Integration error)  (GLM 5.3 · S3 datastore measurement · trial 2: Missed requirement)  (GLM 5.3 · S3 datastore measurement · trial 3: Pass)  (GLM 5.3 · S3 datastore measurement · trial 4: Missed requirement)  (GLM 5.3 · S3 datastore measurement · trial 5: Unverified assumption)  (GLM 5.3 · S3 datastore measurement · trial 6: Pass)  (GLM 5.3 · S3 datastore measurement · trial 7: Pass)  (GLM 5.3 · S3 datastore measurement · trial 8: Unverified assumption)    07 (GLM 5.3 · Customer identity migration · trial 1: Pass)  (GLM 5.3 · Customer identity migration · trial 2: Missed requirement)  (GLM 5.3 · Customer identity migration · trial 3: Integration error)  (GLM 5.3 · Customer identity migration · trial 4: Pass)  (GLM 5.3 · Customer identity migration · trial 5: Pass)  (GLM 5.3 · Customer identity migration · trial 6: Pass)  (GLM 5.3 · Customer identity migration · trial 7: Integration error)  (GLM 5.3 · Customer identity migration · trial 8: Integration error)    08 (GLM 5.3 · Billing schedule migration · trial 1: Pass)  (GLM 5.3 · Billing schedule migration · trial 2: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 3: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 4: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 5: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 6: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 7: Unverified assumption)  (GLM 5.3 · Billing schedule migration · trial 8: Pass)    09 (GLM 5.3 · Linearizable scan · trial 1: Pass)  (GLM 5.3 · Linearizable scan · trial 2: Integration error)  (GLM 5.3 · Linearizable scan · trial 3: Pass)  (GLM 5.3 · Linearizable scan · trial 4: Missed requirement)  (GLM 5.3 · Linearizable scan · trial 5: Missed requirement)  (GLM 5.3 · Linearizable scan · trial 6: Missed requirement)  (GLM 5.3 · Linearizable scan · trial 7: Missed requirement)  (GLM 5.3 · Linearizable scan · trial 8: Missed requirement)    10 (GLM 5.3 · Analytics stream reducer · trial 1: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 2: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 3: Unverified assumption)  (GLM 5.3 · Analytics stream reducer · trial 4: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 5: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 6: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 7: Missed requirement)  (GLM 5.3 · Analytics stream reducer · trial 8: Missed requirement)      ()  Grok 4.6 01 (Grok 4.6 · Entitlement overage lines · trial 1: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 2: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 3: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 4: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 5: Pass)  (Grok 4.6 · Entitlement overage lines · trial 6: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 7: Missed requirement)  (Grok 4.6 · Entitlement overage lines · trial 8: Missed requirement)    02 (Grok 4.6 · Multi-region sweep · trial 1: Missed requirement)  (Grok 4.6 · Multi-region sweep · trial 2: Missed requirement)  (Grok 4.6 · Multi-region sweep · trial 3: Pass)  (Grok 4.6 · Multi-region sweep · trial 4: Missed requirement)  (Grok 4.6 · Multi-region sweep · trial 5: Missed requirement)  (Grok 4.6 · Multi-region sweep · trial 6: Missed requirement)  (Grok 4.6 · Multi-region sweep · trial 7: Pass)  (Grok 4.6 · Multi-region sweep · trial 8: Pass)    03 (Grok 4.6 · Tax jurisdiction · trial 1: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 2: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 3: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 4: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 5: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 6: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 7: Unverified assumption)  (Grok 4.6 · Tax jurisdiction · trial 8: Unverified assumption)    04 (Grok 4.6 · API token metering · trial 1: Unverified assumption)  (Grok 4.6 · API token metering · trial 2: Unverified assumption)  (Grok 4.6 · API token metering · trial 3: Unverified assumption)  (Grok 4.6 · API token metering · trial 4: Integration error)  (Grok 4.6 · API token metering · trial 5: Integration error)  (Grok 4.6 · API token metering · trial 6: Missed requirement)  (Grok 4.6 · API token metering · trial 7: Missed requirement)  (Grok 4.6 · API token metering · trial 8: Missed requirement)    05 (Grok 4.6 · API keys & environments · trial 1: Pass)  (Grok 4.6 · API keys & environments · trial 2: Missed requirement)  (Grok 4.6 · API keys & environments · trial 3: Pass)  (Grok 4.6 · API keys & environments · trial 4: Pass)  (Grok 4.6 · API keys & environments · trial 5: Integration error)  (Grok 4.6 · API keys & environments · trial 6: Missed requirement)  (Grok 4.6 · API keys & environments · trial 7: Pass)  (Grok 4.6 · API keys & environments · trial 8: Missed requirement)    06 (Grok 4.6 · S3 datastore measurement · trial 1: Missed requirement)  (Grok 4.6 · S3 datastore measurement · trial 2: Pass)  (Grok 4.6 · S3 datastore measurement · trial 3: Missed requirement)  (Grok 4.6 · S3 datastore measurement · trial 4: Pass)  (Grok 4.6 · S3 datastore measurement · trial 5: Missed requirement)  (Grok 4.6 · S3 datastore measurement · trial 6: Missed requirement)  (Grok 4.6 · S3 datastore measurement · trial 7: Missed requirement)  (Grok 4.6 · S3 datastore measurement · trial 8: Integration error)    07 (Grok 4.6 · Customer identity migration · trial 1: Pass)  (Grok 4.6 · Customer identity migration · trial 2: Pass)  (Grok 4.6 · Customer identity migration · trial 3: Pass)  (Grok 4.6 · Customer identity migration · trial 4: Pass)  (Grok 4.6 · Customer identity migration · trial 5: Pass)  (Grok 4.6 · Customer identity migration · trial 6: Pass)  (Grok 4.6 · Customer identity migration · trial 7: Pass)  (Grok 4.6 · Customer identity migration · trial 8: Pass)    08 (Grok 4.6 · Billing schedule migration · trial 1: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 2: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 3: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 4: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 5: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 6: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 7: Missed requirement)  (Grok 4.6 · Billing schedule migration · trial 8: Integration error)    09 (Grok 4.6 · Linearizable scan · trial 1: Missed requirement)  (Grok 4.6 · Linearizable scan · trial 2: Missed requirement)  (Grok 4.6 · Linearizable scan · trial 3: Missed requirement)  (Grok 4.6 · Linearizable scan · trial 4: Pass)  (Grok 4.6 · Linearizable scan · trial 5: Unverified assumption)  (Grok 4.6 · Linearizable scan · trial 6: Missed requirement)  (Grok 4.6 · Linearizable scan · trial 7: Missed requirement)  (Grok 4.6 · Linearizable scan · trial 8: Missed requirement)    10 (Grok 4.6 · Analytics stream reducer · trial 1: Missed requirement)  (Grok 4.6 · Analytics stream reducer · trial 2: Unverified assumption)  (Grok 4.6 · Analytics stream reducer · trial 3: Missed requirement)  (Grok 4.6 · Analytics stream reducer · trial 4: Missed requirement)  (Grok 4.6 · Analytics stream reducer · trial 5: Missed requirement)  (Grok 4.6 · Analytics stream reducer · trial 6: Unverified assumption)  (Grok 4.6 · Analytics stream reducer · trial 7: Missed requirement)  (Grok 4.6 · Analytics stream reducer · trial 8: Unverified assumption)      ()  Muse Spark 1.3 01 (Muse Spark 1.3 · Entitlement overage lines · trial 1: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 2: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 3: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 4: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 5: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 6: Missed requirement)  (Muse Spark 1.3 · Entitlement overage lines · trial 7: Pass)  (Muse Spark 1.3 · Entitlement overage lines · trial 8: Missed requirement)    02 (Muse Spark 1.3 · Multi-region sweep · trial 1: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 2: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 3: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 4: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 5: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 6: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 7: Pass)  (Muse Spark 1.3 · Multi-region sweep · trial 8: Pass)    03 (Muse Spark 1.3 · Tax jurisdiction · trial 1: Missed requirement)  (Muse Spark 1.3 · Tax jurisdiction · trial 2: Unverified assumption)  (Muse Spark 1.3 · Tax jurisdiction · trial 3: Missed requirement)  (Muse Spark 1.3 · Tax jurisdiction · trial 4: Missed requirement)  (Muse Spark 1.3 · Tax jurisdiction · trial 5: Unverified assumption)  (Muse Spark 1.3 · Tax jurisdiction · trial 6: Missed requirement)  (Muse Spark 1.3 · Tax jurisdiction · trial 7: Unverified assumption)  (Muse Spark 1.3 · Tax jurisdiction · trial 8: Unverified assumption)    04 (Muse Spark 1.3 · API token metering · trial 1: Integration error)  (Muse Spark 1.3 · API token metering · trial 2: Integration error)  (Muse Spark 1.3 · API token metering · trial 3: Integration error)  (Muse Spark 1.3 · API token metering · trial 4: Integration error)  (Muse Spark 1.3 · API token metering · trial 5: Integration error)  (Muse Spark 1.3 · API token metering · trial 6: Unverified assumption)  (Muse Spark 1.3 · API token metering · trial 7: Unverified assumption)  (Muse Spark 1.3 · API token metering · trial 8: Unverified assumption)    05 (Muse Spark 1.3 · API keys & environments · trial 1: Pass)  (Muse Spark 1.3 · API keys & environments · trial 2: Pass)  (Muse Spark 1.3 · API keys & environments · trial 3: Pass)  (Muse Spark 1.3 · API keys & environments · trial 4: Pass)  (Muse Spark 1.3 · API keys & environments · trial 5: Pass)  (Muse Spark 1.3 · API keys & environments · trial 6: Integration error)  (Muse Spark 1.3 · API keys & environments · trial 7: Pass)  (Muse Spark 1.3 · API keys & environments · trial 8: Integration error)    06 (Muse Spark 1.3 · S3 datastore measurement · trial 1: Regression)  (Muse Spark 1.3 · S3 datastore measurement · trial 2: Missed requirement)  (Muse Spark 1.3 · S3 datastore measurement · trial 3: Missed requirement)  (Muse Spark 1.3 · S3 datastore measurement · trial 4: Missed requirement)  (Muse Spark 1.3 · S3 datastore measurement · trial 5: Unverified assumption)  (Muse Spark 1.3 · S3 datastore measurement · trial 6: Pass)  (Muse Spark 1.3 · S3 datastore measurement · trial 7: Regression)  (Muse Spark 1.3 · S3 datastore measurement · trial 8: Missed requirement)    07 (Muse Spark 1.3 · Customer identity migration · trial 1: Pass)  (Muse Spark 1.3 · Customer identity migration · trial 2: Pass)  (Muse Spark 1.3 · Customer identity migration · trial 3: Integration error)  (Muse Spark 1.3 · Customer identity migration · trial 4: Integration error)  (Muse Spark 1.3 · Customer identity migration · trial 5: Integration error)  (Muse Spark 1.3 · Customer identity migration · trial 6: Pass)  (Muse Spark 1.3 · Customer identity migration · trial 7: Integration error)  (Muse Spark 1.3 · Customer identity migration · trial 8: Integration error)    08 (Muse Spark 1.3 · Billing schedule migration · trial 1: Unverified assumption)  (Muse Spark 1.3 · Billing schedule migration · trial 2: Integration error)  (Muse Spark 1.3 · Billing schedule migration · trial 3: Integration error)  (Muse Spark 1.3 · Billing schedule migration · trial 4: Unverified assumption)  (Muse Spark 1.3 · Billing schedule migration · trial 5: Integration error)  (Muse Spark 1.3 · Billing schedule migration · trial 6: Unverified assumption)  (Muse Spark 1.3 · Billing schedule migration · trial 7: Integration error)  (Muse Spark 1.3 · Billing schedule migration · trial 8: Unverified assumption)    09 (Muse Spark 1.3 · Linearizable scan · trial 1: Integration error)  (Muse Spark 1.3 · Linearizable scan · trial 2: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 3: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 4: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 5: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 6: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 7: Missed requirement)  (Muse Spark 1.3 · Linearizable scan · trial 8: Missed requirement)    10 (Muse Spark 1.3 · Analytics stream reducer · trial 1: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 2: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 3: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 4: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 5: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 6: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 7: Integration error)  (Muse Spark 1.3 · Analytics stream reducer · trial 8: Integration error)      ()  Kimi K3 01 (Kimi K3 · Entitlement overage lines · trial 1: Missed requirement)  (Kimi K3 · Entitlement overage lines · trial 2: Pass)  (Kimi K3 · Entitlement overage lines · trial 3: Pass)  (Kimi K3 · Entitlement overage lines · trial 4: Pass)  (Kimi K3 · Entitlement overage lines · trial 5: Pass)  (Kimi K3 · Entitlement overage lines · trial 6: Missed requirement)  (Kimi K3 · Entitlement overage lines · trial 7: Pass)  (Kimi K3 · Entitlement overage lines · trial 8: Pass)    02 (Kimi K3 · Multi-region sweep · trial 1: Pass)  (Kimi K3 · Multi-region sweep · trial 2: Missed requirement)  (Kimi K3 · Multi-region sweep · trial 3: Wrong file)  (Kimi K3 · Multi-region sweep · trial 4: Integration error)  (Kimi K3 · Multi-region sweep · trial 5: Wrong file)  (Kimi K3 · Multi-region sweep · trial 6: Pass)  (Kimi K3 · Multi-region sweep · trial 7: Missed requirement)  (Kimi K3 · Multi-region sweep · trial 8: Missed requirement)    03 (Kimi K3 · Tax jurisdiction · trial 1: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 2: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 3: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 4: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 5: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 6: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 7: Unverified assumption)  (Kimi K3 · Tax jurisdiction · trial 8: Unverified assumption)    04 (Kimi K3 · API token metering · trial 1: Pass)  (Kimi K3 · API token metering · trial 2: Integration error)  (Kimi K3 · API token metering · trial 3: Integration error)  (Kimi K3 · API token metering · trial 4: Integration error)  (Kimi K3 · API token metering · trial 5: Integration error)  (Kimi K3 · API token metering · trial 6: Integration error)  (Kimi K3 · API token metering · trial 7: Integration error)  (Kimi K3 · API token metering · trial 8: Integration error)    05 (Kimi K3 · API keys & environments · trial 1: Integration error)  (Kimi K3 · API keys & environments · trial 2: Integration error)  (Kimi K3 · API keys & environments · trial 3: Integration error)  (Kimi K3 · API keys & environments · trial 4: Integration error)  (Kimi K3 · API keys & environments · trial 5: Missed requirement)  (Kimi K3 · API keys & environments · trial 6: Integration error)  (Kimi K3 · API keys & environments · trial 7: Missed requirement)  (Kimi K3 · API keys & environments · trial 8: Integration error)    06 (Kimi K3 · S3 datastore measurement · trial 1: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 2: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 3: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 4: Pass)  (Kimi K3 · S3 datastore measurement · trial 5: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 6: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 7: Missed requirement)  (Kimi K3 · S3 datastore measurement · trial 8: Missed requirement)    07 (Kimi K3 · Customer identity migration · trial 1: Integration error)  (Kimi K3 · Customer identity migration · trial 2: Integration error)  (Kimi K3 · Customer identity migration · trial 3: Pass)  (Kimi K3 · Customer identity migration · trial 4: Integration error)  (Kimi K3 · Customer identity migration · trial 5: Pass)  (Kimi K3 · Customer identity migration · trial 6: Pass)  (Kimi K3 · Customer identity migration · trial 7: Missed requirement)  (Kimi K3 · Customer identity migration · trial 8: Pass)    08 (Kimi K3 · Billing schedule migration · trial 1: Unverified assumption)  (Kimi K3 · Billing schedule migration · trial 2: Missed requirement)  (Kimi K3 · Billing schedule migration · trial 3: Missed requirement)  (Kimi K3 · Billing schedule migration · trial 4: Missed requirement)  (Kimi K3 · Billing schedule migration · trial 5: Pass)  (Kimi K3 · Billing schedule migration · trial 6: Missed requirement)  (Kimi K3 · Billing schedule migration · trial 7: Missed requirement)  (Kimi K3 · Billing schedule migration · trial 8: Missed requirement)    09 (Kimi K3 · Linearizable scan · trial 1: Integration error)  (Kimi K3 · Linearizable scan · trial 2: Missed requirement)  (Kimi K3 · Linearizable scan · trial 3: Missed requirement)  (Kimi K3 · Linearizable scan · trial 4: Missed requirement)  (Kimi K3 · Linearizable scan · trial 5: Missed requirement)  (Kimi K3 · Linearizable scan · trial 6: Missed requirement)  (Kimi K3 · Linearizable scan · trial 7: Missed requirement)  (Kimi K3 · Linearizable scan · trial 8: Missed requirement)    10 (Kimi K3 · Analytics stream reducer · trial 1: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 2: Unverified assumption)  (Kimi K3 · Analytics stream reducer · trial 3: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 4: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 5: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 6: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 7: Missed requirement)  (Kimi K3 · Analytics stream reducer · trial 8: Missed requirement)      ()  GPT-5.6 Sol 01 (GPT-5.6 Sol · Entitlement overage lines · trial 1: Missed requirement)  (GPT-5.6 Sol · Entitlement overage lines · trial 2: Missed requirement)  (GPT-5.6 Sol · Entitlement overage lines · trial 3: Pass)  (GPT-5.6 Sol · Entitlement overage lines · trial 4: Integration error)  (GPT-5.6 Sol · Entitlement overage lines · trial 5: Missed requirement)  (GPT-5.6 Sol · Entitlement overage lines · trial 6: Missed requirement)  (GPT-5.6 Sol · Entitlement overage lines · trial 7: Missed requirement)  (GPT-5.6 Sol · Entitlement overage lines · trial 8: Missed requirement)    02 (GPT-5.6 Sol · Multi-region sweep · trial 1: Missed requirement)  (GPT-5.6 Sol · Multi-region sweep · trial 2: Missed requirement)  (GPT-5.6 Sol · Multi-region sweep · trial 3: Pass)  (GPT-5.6 Sol · Multi-region sweep · trial 4: Pass)  (GPT-5.6 Sol · Multi-region sweep · trial 5: Pass)  (GPT-5.6 Sol · Multi-region sweep · trial 6: Pass)  (GPT-5.6 Sol · Multi-region sweep · trial 7: Pass)  (GPT-5.6 Sol · Multi-region sweep · trial 8: Missed requirement)    03 (GPT-5.6 Sol · Tax jurisdiction · trial 1: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 2: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 3: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 4: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 5: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 6: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 7: Unverified assumption)  (GPT-5.6 Sol · Tax jurisdiction · trial 8: Unverified assumption)    04 (GPT-5.6 Sol · API token metering · trial 1: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 2: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 3: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 4: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 5: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 6: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 7: Unverified assumption)  (GPT-5.6 Sol · API token metering · trial 8: Unverified assumption)    05 (GPT-5.6 Sol · API keys & environments · trial 1: Integration error)  (GPT-5.6 Sol · API keys & environments · trial 2: Pass)  (GPT-5.6 Sol · API keys & environments · trial 3: Pass)  (GPT-5.6 Sol · API keys & environments · trial 4: Pass)  (GPT-5.6 Sol · API keys & environments · trial 5: Pass)  (GPT-5.6 Sol · API keys & environments · trial 6: Pass)  (GPT-5.6 Sol · API keys & environments · trial 7: Pass)  (GPT-5.6 Sol · API keys & environments · trial 8: Pass)    06 (GPT-5.6 Sol · S3 datastore measurement · trial 1: Missed requirement)  (GPT-5.6 Sol · S3 datastore measurement · trial 2: Integration error)  (GPT-5.6 Sol · S3 datastore measurement · trial 3: Regression)  (GPT-5.6 Sol · S3 datastore measurement · trial 4: Regression)  (GPT-5.6 Sol · S3 datastore measurement · trial 5: Regression)  (GPT-5.6 Sol · S3 datastore measurement · trial 6: Regression)  (GPT-5.6 Sol · S3 datastore measurement · trial 7: Regression)  (GPT-5.6 Sol · S3 datastore measurement · trial 8: Regression)    07 (GPT-5.6 Sol · Customer identity migration · trial 1: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 2: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 3: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 4: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 5: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 6: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 7: Integration error)  (GPT-5.6 Sol · Customer identity migration · trial 8: Integration error)    08 (GPT-5.6 Sol · Billing schedule migration · trial 1: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 2: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 3: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 4: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 5: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 6: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 7: Unverified assumption)  (GPT-5.6 Sol · Billing schedule migration · trial 8: Unverified assumption)    09 (GPT-5.6 Sol · Linearizable scan · trial 1: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 2: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 3: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 4: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 5: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 6: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 7: Missed requirement)  (GPT-5.6 Sol · Linearizable scan · trial 8: Missed requirement)    10 (GPT-5.6 Sol · Analytics stream reducer · trial 1: Unverified assumption)  (GPT-5.6 Sol · Analytics stream reducer · trial 2: Missed requirement)  (GPT-5.6 Sol · Analytics stream reducer · trial 3: Unverified assumption)  (GPT-5.6 Sol · Analytics stream reducer · trial 4: Missed requirement)  (GPT-5.6 Sol · Analytics stream reducer · trial 5: Unverified assumption)  (GPT-5.6 Sol · Analytics stream reducer · trial 6: Unverified assumption)  (GPT-5.6 Sol · Analytics stream reducer · trial 7: Unverified assumption)  (GPT-5.6 Sol · Analytics stream reducer · trial 8: Missed requirement)       Pass  Unverified assumption  Missed requirement  Integration error  Regression  Wrong file    Different models fail in different ways Percentages are out of each model's failed runs, not all runs. Unverified assumption Builds on a guess about the system instead of checking it in the workspace.  (GPT-5.6 Sol: 29 of 67 failed runs (43.3%) were classified as unverified assumption.) ()  GPT-5.6 Sol   43.3%: 29of 67failed runs   (GPT-6 Astra: 18 of 53 failed runs (34.0%) were classified as unverified assumption.) ()  GPT-6 Astra   34.0%: 18of 53failed runs   (GLM 5.3: 16 of 57 failed runs (28.1%) were classified as unverified assumption.) ()  GLM 5.3   28.1%: 16of 57failed runs   (Grok 4.6: 15 of 61 failed runs (24.6%) were classified as unverified assumption.) ()  Grok 4.6   24.6%: 15of 61failed runs   (Fable 5.1: 12 of 49 failed runs (24.5%) were classified as unverified assumption.) ()  Fable 5.1   24.5%: 12of 49failed runs   (Muse Spark 1.3: 12 of 61 failed runs (19.7%) were classified as unverified assumption.) ()  Muse Spark 1.3   19.7%: 12of 61failed runs   (Kimi K3: 10 of 65 failed runs (15.4%) were classified as unverified assumption.) ()  Kimi K3   15.4%: 10of 65failed runs   (Gemini 3.8 Flash: 6 of 55 failed runs (10.9%) were classified as unverified assumption.) ()  Gemini 3.8 Flash   10.9%: 6of 55failed runs     Missed requirement Leaves out behavior the instruction requires.  (Grok 4.6: 41 of 61 failed runs (67.2%) were classified as missed requirement.) ()  Grok 4.6   67.2%: 41of 61failed runs   (Kimi K3: 35 of 65 failed runs (53.8%) were classified as missed requirement.) ()  Kimi K3   53.8%: 35of 65failed runs   (GLM 5.3: 22 of 57 failed runs (38.6%) were classified as missed requirement.) ()  GLM 5.3   38.6%: 22of 57failed runs   (Fable 5.1: 18 of 49 failed runs (36.7%) were classified as missed requirement.) ()  Fable 5.1   36.7%: 18of 49failed runs   (Muse Spark 1.3: 22 of 61 failed runs (36.1%) were classified as missed requirement.) ()  Muse Spark 1.3   36.1%: 22of 61failed runs   (GPT-5.6 Sol: 21 of 67 failed runs (31.3%) were classified as missed requirement.) ()  GPT-5.6 Sol   31.3%: 21of 67failed runs   (Gemini 3.8 Flash: 16 of 55 failed runs (29.1%) were classified as missed requirement.) ()  Gemini 3.8 Flash   29.1%: 16of 55failed runs   (GPT-6 Astra: 15 of 53 failed runs (28.3%) were classified as missed requirement.) ()  GPT-6 Astra   28.3%: 15of 53failed runs     Integration error Right idea, wired into the surrounding system incorrectly.  (Gemini 3.8 Flash: 27 of 55 failed runs (49.1%) were classified as integration error.) ()  Gemini 3.8 Flash   49.1%: 27of 55failed runs   (Muse Spark 1.3: 25 of 61 failed runs (41.0%) were classified as integration error.) ()  Muse Spark 1.3   41.0%: 25of 61failed runs   (Fable 5.1: 17 of 49 failed runs (34.7%) were classified as integration error.) ()  Fable 5.1   34.7%: 17of 49failed runs   (GPT-6 Astra: 18 of 53 failed runs (34.0%) were classified as integration error.) ()  GPT-6 Astra   34.0%: 18of 53failed runs   (Kimi K3: 18 of 65 failed runs (27.7%) were classified as integration error.) ()  Kimi K3   27.7%: 18of 65failed runs   (GLM 5.3: 15 of 57 failed runs (26.3%) were classified as integration error.) ()  GLM 5.3   26.3%: 15of 57failed runs   (GPT-5.6 Sol: 11 of 67 failed runs (16.4%) were classified as integration error.) ()  GPT-5.6 Sol   16.4%: 11of 67failed runs   (Grok 4.6: 5 of 61 failed runs (8.2%) were classified as integration error.) ()  Grok 4.6   8.2%: 5of 61failed runs     Regression Breaks existing behavior while making the change.  (Gemini 3.8 Flash: 6 of 55 failed runs (10.9%) were classified as regression.) ()  Gemini 3.8 Flash   10.9%: 6of 55failed runs   (GPT-5.6 Sol: 6 of 67 failed runs (9.0%) were classified as regression.) ()  GPT-5.6 Sol   9.0%: 6of 67failed runs   (Fable 5.1: 2 of 49 failed runs (4.1%) were classified as regression.) ()  Fable 5.1   4.1%: 2of 49failed runs   (GPT-6 Astra: 2 of 53 failed runs (3.8%) were classified as regression.) ()  GPT-6 Astra   3.8%: 2of 53failed runs   (Muse Spark 1.3: 2 of 61 failed runs (3.3%) were classified as regression.) ()  Muse Spark 1.3   3.3%: 2of 61failed runs   (GLM 5.3: 0 of 57 failed runs (0%) were classified as regression.) ()  GLM 5.3   0%: 0of 57failed runs   (Grok 4.6: 0 of 61 failed runs (0%) were classified as regression.) ()  Grok 4.6   0%: 0of 61failed runs   (Kimi K3: 0 of 65 failed runs (0%) were classified as regression.) ()  Kimi K3   0%: 0of 65failed runs     Wrong file Delivers the change somewhere the running application never calls, such as a one-off script.  (GLM 5.3: 4 of 57 failed runs (7.0%) were classified as wrong file.) ()  GLM 5.3   7.0%: 4of 57failed runs   (Kimi K3: 2 of 65 failed runs (3.1%) were classified as wrong file.) ()  Kimi K3   3.1%: 2of 65failed runs   (Fable 5.1: 0 of 49 failed runs (0%) were classified as wrong file.) ()  Fable 5.1   0%: 0of 49failed runs   (GPT-6 Astra: 0 of 53 failed runs (0%) were classified as wrong file.) ()  GPT-6 Astra   0%: 0of 53failed runs   (Gemini 3.8 Flash: 0 of 55 failed runs (0%) were classified as wrong file.) ()  Gemini 3.8 Flash   0%: 0of 55failed runs   (Grok 4.6: 0 of 61 failed runs (0%) were classified as wrong file.) ()  Grok 4.6   0%: 0of 61failed runs   (Muse Spark 1.3: 0 of 61 failed runs (0%) were classified as wrong file.) ()  Muse Spark 1.3   0%: 0of 61failed runs   (GPT-5.6 Sol: 0 of 67 failed runs (0%) were classified as wrong file.) ()  GPT-5.6 Sol   0%: 0of 67failed runs        03 Effort & the Frontier Higher cost does not guarantee a higher resolution rate Cost (USD) Output tokens    Estimated frontier  Resolution rate (%)  10   15   20   25   30   35   40   45   $2   $3   $5   $10  Cost per rollout (USD, log scale)  Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLI  Gemini 3.8 Flash 31.2% · $2.50   GPT-5.6 Sol: 16.2% · $2.65; Codex CLI  GPT-5.6 Sol 16.2% · $2.65   Muse Spark 1.3: 23.8% · $2.74; Muse Code  Muse Spark 1.3 23.8% · $2.74   Grok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higher  Grok 4.6 23.8% · $3.44   Kimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higher  Kimi K3 18.8% · $3.90   GPT-6 Astra: 33.8% · $4.67; Codex CLI  GPT-6 Astra 33.8% · $4.67   GLM 5.3: 28.8% · $5.12; Claude Code  GLM 5.3 28.8% · $5.12   Fable 5.1: 38.8% · $6.96; Claude Code  Fable 5.1 38.8% · $6.96     Resolution rate (%)  10   15   20   25   30   35   40   45   $2   $3   $5   $10  Cost / rollout ($, log scale)  Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLI  3  GPT-5.6 Sol: 16.2% · $2.65; Codex CLI  8  Muse Spark 1.3: 23.8% · $2.74; Muse Code  6  Grok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higher  5  Kimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higher  7  GPT-6 Astra: 33.8% · $4.67; Codex CLI  2  GLM 5.3: 28.8% · $5.12; Claude Code  4  Fable 5.1: 38.8% · $6.96; Claude Code  1   1 Fable 5.1 38.8%· $6.96  2 GPT-6 Astra 33.8%· $4.67  3 Gemini 3.8 Flash 31.2%· $2.50  4 GLM 5.3 28.8%· $5.12  5 Grok 4.6 23.8%· $3.44  6 Muse Spark 1.3 23.8%· $2.74  7 Kimi K3 18.8%· $3.90  8 GPT-5.6 Sol 16.2%· $2.65     Estimated rollout costs range from $2.50to $6.96 Rank Model Estimated cost (USD)   1 Gemini 3.8 Flash $2.50  2 GPT-5.6 Sol $2.65  3 Muse Spark 1.3 $2.74  4 Grok 4.6 $3.44  5 Kimi K3 $3.90  6 GPT-6 Astra $4.67  7 GLM 5.3 $5.12  8 Fable 5.1 $6.96     Output tokens Tool calls Wall-clock minutes mean per rollout, by task  Swipe the chart to see all tasks.  0   100k   200k   300k   400k  01 02 03 04 05 06 07 08 09 10 task  Entitlement overage lines · Fable 5.1: 34k  Multi-region sweep · Fable 5.1: 30k  Tax jurisdiction · Fable 5.1: 78k  API token metering · Fable 5.1: 95k  API keys & environments · Fable 5.1: 71k  S3 datastore measurement · Fable 5.1: 62k  Customer identity migration · Fable 5.1: 67k  Billing schedule migration · Fable 5.1: 26k  Linearizable scan · Fable 5.1: 86k  Analytics stream reducer · Fable 5.1: 88k    Entitlement overage lines · GPT-6 Astra: 13k  Multi-region sweep · GPT-6 Astra: 13k  Tax jurisdiction · GPT-6 Astra: 24k  API token metering · GPT-6 Astra: 31k  API keys & environments · GPT-6 Astra: 32k  S3 datastore measurement · GPT-6 Astra: 22k  Customer identity migration · GPT-6 Astra: 25k  Billing schedule migration · GPT-6 Astra: 15k  Linearizable scan · GPT-6 Astra: 33k  Analytics stream reducer · GPT-6 Astra: 29k    Entitlement overage lines · Gemini 3.8 Flash: 78k  Multi-region sweep · Gemini 3.8 Flash: 67k  Tax jurisdiction · Gemini 3.8 Flash: 95k  API token metering · Gemini 3.8 Flash: 134k  API keys & environments · Gemini 3.8 Flash: 102k  S3 datastore measurement · Gemini 3.8 Flash: 106k  Customer identity migration · Gemini 3.8 Flash: 97k  Billing schedule migration · Gemini 3.8 Flash: 70k  Linearizable scan · Gemini 3.8 Flash: 106k  Analytics stream reducer · Gemini 3.8 Flash: 88k    Entitlement overage lines · GLM 5.3: 68k  Multi-region sweep · GLM 5.3: 53k  Tax jurisdiction · GLM 5.3: 141k  API token metering · GLM 5.3: 177k  API keys & environments · GLM 5.3: 125k  S3 datastore measurement · GLM 5.3: 121k  Customer identity migration · GLM 5.3: 90k  Billing schedule migration · GLM 5.3: 58k  Linearizable scan · GLM 5.3: 172k  Analytics stream reducer · GLM 5.3: 169k    Entitlement overage lines · Grok 4.6: 7k  Multi-region sweep · Grok 4.6: 3k  Tax jurisdiction · Grok 4.6: 12k  API token metering · Grok 4.6: 15k  API keys & environments · Grok 4.6: 16k  S3 datastore measurement · Grok 4.6: 13k  Customer identity migration · Grok 4.6: 20k  Billing schedule migration · Grok 4.6: 6k  Linearizable scan · Grok 4.6: 261k  Analytics stream reducer · Grok 4.6: 315k    Entitlement overage lines · Muse Spark 1.3: 36k  Multi-region sweep · Muse Spark 1.3: 43k  Tax jurisdiction · Muse Spark 1.3: 67k  API token metering · Muse Spark 1.3: 152k  API keys & environments · Muse Spark 1.3: 104k  S3 datastore measurement · Muse Spark 1.3: 71k  Customer identity migration · Muse Spark 1.3: 76k  Billing schedule migration · Muse Spark 1.3: 38k  Linearizable scan · Muse Spark 1.3: 141k  Analytics stream reducer · Muse Spark 1.3: 137k    Entitlement overage lines · Kimi K3: 30k  Multi-region sweep · Kimi K3: 9k  Tax jurisdiction · Kimi K3: 39k  API token metering · Kimi K3: 69k  API keys & environments · Kimi K3: 44k  S3 datastore measurement · Kimi K3: 32k  Customer identity migration · Kimi K3: 66k  Billing schedule migration · Kimi K3: 19k  Linearizable scan · Kimi K3: 71k  Analytics stream reducer · Kimi K3: 55k    Entitlement overage lines · GPT-5.6 Sol: 12k  Multi-region sweep · GPT-5.6 Sol: 8k  Tax jurisdiction · GPT-5.6 Sol: 22k  API token metering · GPT-5.6 Sol: 31k  API keys & environments · GPT-5.6 Sol: 25k  S3 datastore measurement · GPT-5.6 Sol: 25k  Customer identity migration · GPT-5.6 Sol: 24k  Billing schedule migration · GPT-5.6 Sol: 13k  Linearizable scan · GPT-5.6 Sol: 37k  Analytics stream reducer · GPT-5.6 Sol: 30k      Fable 5.1·64koverall  GPT-6 Astra·24koverall  Gemini 3.8 Flash·94koverall  GLM 5.3·117koverall  Grok 4.6·67koverall  Muse Spark 1.3·87koverall  Kimi K3·43koverall  GPT-5.6 Sol·23koverall  View task values Task (entitlement-overage-lines) Entitlement overage lines (multi-region-sweep) Multi-region sweep (tax-jurisdiction) Tax jurisdiction (api-token-metering) API token metering (api-keys-and-environments) API keys & environments (s3-datastore-measurement) S3 datastore measurement (customer-identity-migration) Customer identity migration (customer-billing-schedule-migration) Billing schedule migration (linearizable-scan) Linearizable scan (analytics-stream-reducer) Analytics stream reducer  Fable 5.1 34k  GPT-6 Astra 13k  Gemini 3.8 Flash 78k  GLM 5.3 68k  Grok 4.6 7k  Muse Spark 1.3 36k  Kimi K3 30k  GPT-5.6 Sol 12k      04 Evaluation Setup Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.     © 2026 Specific (https://withspecific.com/about) About (https://withspecific.com/privacy) Privacy   ()