# MacroscopeBench MacroscopeBench evaluates AI models on real code review tasks. It measures known-bug recall and raw bug-detection precision using a fixed prompt, tool set, and evaluation harness. Canonical page: https://macroscope.com/benchmark Historical tool-comparison benchmark (separate dataset and methodology): https://macroscope.com/blog/code-review-benchmark ## Dataset - Source: benchmark_tiny.csv - Records: 195 - Catalogued known bugs: 144 - Records without a catalogued bug: 51 - Replication: all 41 final benchmark exports declare k=3 - Review trials: 585 billed reviews per configuration - Export revisions: the current benchmark revision is recorded under two dataset-ID content hashes; both are published. 32 configurations use dataset ID cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859; 9 use cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72. The two share the same 195 records, 144 catalogued bugs, severity split, and k; only the hashed bytes differ. Any other dataset ID is filtered off the page. ## Metric definitions - Overall Score: the final benchmark export's supplied f1Score, converted to a percentage for display - Severity Score: the F1 recorded for that severity bucket, converted to a percentage for display. Blank where the dataset carries none. - Severity Recall: detected catalogued bugs divided by total catalogued bugs in that severity and rollout - Severity Precision: valid runtime findings divided by all graded findings in that severity and rollout; the denominator includes valid, invalid, and opinion outcomes - Recall: average share of catalogued known bugs found per review trial - Any-of-k Recall: share of catalogued known bugs found at least once across the replicated review trials - Precision: share of validator-graded findings marked valid - Cost: actual billed review-pipeline cost per task, excluding benchmark-only matching - Latency: median review-pipeline duration in seconds - Signal-to-noise: benchmark signal divided by benchmark noise - Coverage: completed review trials divided by expected review trials Precision is directional because the benchmark validator can assign contradictory invalid outcomes to findings that match catalogued bugs. ## Configuration data Columns are tab-separated: model, configuration, k, score, recall_any_of_k_pct, precision_pct, cost_usd_per_task, median_latency_seconds, recall_average_trial_pct, p90_latency_seconds, signal_to_noise, error_rate_pct, coverage, benchmark_id, dataset_id, harness_version, api_version DeepSeek V4.1 Flash max effort 3 71.99 90.97 63.49 0.75 977.285 83.10 1818.966 1.27 52ccc297-28aa-58d1-914f-9b8e9ba00e16 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 DeepSeek V4.1 Flash high effort 3 70.50 85.42 64.63 0.38 460.21 77.55 927.771 1.33 4b6f1bfc-9d2d-5124-bfa4-6f0295c67ad7 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 DeepSeek V4.1 Flash low effort 3 70.72 86.81 65.00 0.38 489.42 77.55 1009.848 1.37 814e5ec4-8c99-5ac9-959d-35dc4362caec cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GLM 5.3 max effort 3 77.38 87.50 78.40 3.86 750.345 76.39 1367.982 2.24 f8ed8869-c28f-5bba-a5c5-6e366685d145 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GLM 5.3 high effort 3 69.41 77.08 75.01 2.02 328.949 64.58 787.528 1.94 3541ae6b-327d-535e-a954-f08bf733938e cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GLM 5.3 low effort 3 54.56 61.81 60.04 0.41 82.365 50.00 299.573 1.17 2895e1db-dd2a-5461-b20e-3ec66671926a cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5.5 max effort 3 83.92 91.67 84.04 11.58 607.828 83.80 1104.165 3.01 f7463661-a438-561c-96ac-843c7939d71b cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5.5 xhigh effort 3 78.88 83.33 83.19 3.51 172.191 75.00 406.076 2.95 c9b7bdbd-39c8-5140-84e4-a0fff12bb1ea cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5.5 high effort 3 73.08 76.39 78.89 1.30 62.947 68.06 154.841 2.35 3eab60f0-c8d8-5f3e-bd67-7da5b90edf9d cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5.5 medium effort 3 70.06 74.31 78.26 0.97 44.468 63.43 98.627 2.35 12df91d3-3f0c-5dd3-bb48-02f22c7c6c08 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5.5 low effort 3 61.40 60.42 74.31 0.68 29.423 52.31 60.184 2.21 449da42a-2057-5574-9de3-56bcd79e7a94 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5 high effort 3 69.87 82.64 64.22 6.50 257.922 76.62 504.313 1.20 78cb4dc8-3927-5060-8d0c-14ae662c19ba cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5 medium effort 3 65.50 76.39 62.54 3.50 131.671 68.75 310.143 1.18 3e9b4fb2-1fcd-505c-9bce-90f7111fd25e cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Claude Opus 5 low effort 3 59.38 68.06 61.48 1.40 56.006 57.41 137.597 1.12 e8feb853-cb76-5b43-afa4-ccb4d06477d9 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Grok 4.6 high effort 3 75.09 81.25 81.10 3.51 929.989 69.91 2577.037 3.17 6c6bf706-ce17-5d5b-beb5-14e1ca274c77 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Grok 4.6 xhigh effort 3 75.87 78.47 83.28 4.68 880.604 69.68 2735.451 3.67 a46493ee-207c-5a4e-acc6-041c5a21cab4 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Grok 4.6 medium effort 3 72.80 75.69 79.52 2.56 518.544 67.13 2088.625 3.11 cb59e9a8-c128-56c1-ace9-2f291b18f118 cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Grok 4.6 low effort 3 57.10 59.72 62.52 0.30 54.939 52.55 113.728 1.30 19b1e774-2306-5d68-a71a-1476e94ffb01 cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-5.6 Sol high effort 3 77.77 80.56 86.21 3.90 180.398 70.83 330.454 3.73 909a01d6-c340-5b97-9367-68493498325a cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-5.6 Sol low effort 3 66.11 61.81 82.63 0.81 63.909 55.09 114.25 3.17 1e0dc077-2b51-5853-a08a-3300116ae4ce cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Kimi K3 max effort 3 72.86 77.08 80.32 2.82 493.238 66.67 1334.473 2.18 5b0f70ad-7c7d-5627-882b-1951e67fb1c1 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Kimi K3 high effort 3 68.56 71.53 76.63 1.90 260.902 62.04 669.326 1.89 082ea61d-001a-5fd3-ab7a-9bbb4c8847d8 cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Kimi K3 low effort 3 53.52 55.56 65.73 0.48 63.179 45.14 179.884 1.37 d37f6c88-f0ff-54b2-ba8d-35f44a196366 cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Astra max effort 3 77.96 74.31 91.66 7.87 228.86 67.82 478.725 6.26 a44b3d47-5e28-5fc1-8b85-1425b991ae62 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Sol max effort 3 80.30 81.25 88.01 3.68 297.459 73.84 624.074 3.94 23bb69f6-adb5-5f3e-8215-d714b4e47dd2 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Sol xhigh effort 3 77.19 75.00 87.62 2.00 186.158 68.98 348.14 3.86 08236e89-43dc-5064-a50f-051bb3e5ef45 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Sol high effort 3 74.32 75.69 85.87 1.23 123.404 65.51 228.895 3.62 cef20cac-e997-5ea5-8a9e-eaa15d1176a5 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Sol medium effort 3 69.18 67.36 82.18 0.69 91.198 59.72 161.816 3.08 f5c89c7f-1e3d-5fd4-92b5-21c6bc59f61d cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Sol low effort 3 61.76 59.72 75.85 0.36 57.882 52.08 121.146 2.35 d4878386-3e39-5cd0-9bdd-7ce093f99d9d cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6.1 Sol max effort 3 78.52 72.22 90.71 1.05 238.664 69.21 556.124 6.42 ea5dd947-714e-5bee-85fb-ece01fb061a9 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6.1 Sol xhigh effort 3 77.25 69.44 91.38 0.79 163.379 66.90 334.502 6.74 dfeb649e-809e-5c63-b795-15dced32f18e cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6.1 Sol high effort 3 76.74 71.53 91.28 0.56 111.798 66.20 243.215 6.93 6daffe2f-8a96-50bf-8a0e-765caf2dd865 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6.1 Sol medium effort 3 73.54 64.58 91.26 0.28 65.697 61.57 139.682 7.47 64a4d50a-6494-57a3-bf57-f76dfd0f9223 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6.1 Sol low effort 3 70.57 61.11 89.86 0.16 45.592 58.10 109.114 6.63 36153d87-267e-55c8-bed6-6eac40ffdbc3 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Luna max effort 3 72.50 70.14 89.10 0.13 241.482 61.11 704.647 4.28 901bbfa8-49f3-5f65-9a40-0e0dde982666 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Luna xhigh effort 3 68.21 65.97 86.63 0.06 121.581 56.25 267.831 3.61 fbc652a2-5ba6-57e4-aeec-a552eaaf445f cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Luna high effort 3 65.60 63.89 86.03 0.05 99.784 53.01 212.998 3.50 a78cc3e7-05b6-537f-a27d-e4ebd863dae7 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Luna medium effort 3 62.39 59.72 80.52 0.03 74.398 50.93 147.144 2.60 a9f5b61b-29e9-5fda-a437-00afe8fe2aa4 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 GPT-6 Luna low effort 3 44.74 52.78 54.46 0.02 41.652 37.96 90.004 1.00 36397071-f3e5-58ac-af6e-bce5c1286993 cd4560f98b6c769f336bd75e8783f983e4ce8f5ffdb534fcf05f797f08c30859 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Gemini 3.8 Flash medium effort 3 70.73 70.83 78.86 3.38 663.127 64.12 1589.509 2.34 99c74d2e-77c4-5df2-a3fe-d67a6b5acbfd cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 Gemini 3.8 Flash low effort 3 51.33 50.69 60.75 0.90 158.429 44.44 566.957 1.11 2e48442d-1b30-51cd-b40c-41012b8e9168 cf1c76e384da0db1180a88c48f437bb80559576b08a87e257eb44be5762e5c72 4d954bee2fb72fe444118c86d5add75af41936170ba8db1ce797b2981ca161d4 2026-09-11 ## Severity data Columns are tab-separated: model, configuration, severity, score, recall_pct, precision_pct, benchmark_id DeepSeek V4.1 Flash max effort critical 81.36 71.43 94.51 52ccc297-28aa-58d1-914f-9b8e9ba00e16 DeepSeek V4.1 Flash high effort critical 70.90 64.29 79.03 4b6f1bfc-9d2d-5124-bfa4-6f0295c67ad7 DeepSeek V4.1 Flash low effort critical 74.89 69.05 81.82 814e5ec4-8c99-5ac9-959d-35dc4362caec GLM 5.3 max effort critical 79.12 69.05 92.63 f8ed8869-c28f-5bba-a5c5-6e366685d145 GLM 5.3 high effort critical 65.07 50.00 93.15 3541ae6b-327d-535e-a954-f08bf733938e GLM 5.3 low effort critical 64.33 52.38 83.33 2895e1db-dd2a-5461-b20e-3ec66671926a Claude Opus 5.5 max effort critical 80.02 76.19 84.26 f7463661-a438-561c-96ac-843c7939d71b Claude Opus 5.5 xhigh effort critical 76.56 69.05 85.92 c9b7bdbd-39c8-5140-84e4-a0fff12bb1ea Claude Opus 5.5 high effort critical 71.43 57.14 95.24 3eab60f0-c8d8-5f3e-bd67-7da5b90edf9d Claude Opus 5.5 medium effort critical 67.21 52.38 93.75 12df91d3-3f0c-5dd3-bb48-02f22c7c6c08 Claude Opus 5.5 low effort critical 63.74 50.00 87.88 449da42a-2057-5574-9de3-56bcd79e7a94 Claude Opus 5 high effort critical 67.16 57.14 81.43 78cb4dc8-3927-5060-8d0c-14ae662c19ba Claude Opus 5 medium effort critical 60.94 47.62 84.62 3e9b4fb2-1fcd-505c-9bce-90f7111fd25e Claude Opus 5 low effort critical 55.12 40.48 86.36 e8feb853-cb76-5b43-afa4-ccb4d06477d9 Grok 4.6 high effort critical 65.17 50.00 93.55 6c6bf706-ce17-5d5b-beb5-14e1ca274c77 Grok 4.6 xhigh effort critical 55.74 40.48 89.47 a46493ee-207c-5a4e-acc6-041c5a21cab4 Grok 4.6 medium effort critical 64.34 50.00 90.20 cb59e9a8-c128-56c1-ace9-2f291b18f118 Grok 4.6 low effort critical 54.10 42.86 73.33 19b1e774-2306-5d68-a71a-1476e94ffb01 GPT-5.6 Sol high effort critical 66.67 50.00 100.00 909a01d6-c340-5b97-9367-68493498325a GPT-5.6 Sol low effort critical 51.53 35.71 92.50 1e0dc077-2b51-5853-a08a-3300116ae4ce Kimi K3 max effort critical 61.98 47.62 88.73 5b0f70ad-7c7d-5627-882b-1951e67fb1c1 Kimi K3 high effort critical 62.53 47.62 91.03 082ea61d-001a-5fd3-ab7a-9bbb4c8847d8 Kimi K3 low effort critical 53.53 38.10 90.00 d37f6c88-f0ff-54b2-ba8d-35f44a196366 GPT-6 Astra max effort critical 61.30 45.24 95.06 a44b3d47-5e28-5fc1-8b85-1425b991ae62 GPT-6 Sol max effort critical 74.55 61.90 93.69 23bb69f6-adb5-5f3e-8215-d714b4e47dd2 GPT-6 Sol xhigh effort critical 61.67 47.62 87.50 08236e89-43dc-5064-a50f-051bb3e5ef45 GPT-6 Sol high effort critical 67.15 52.38 93.51 cef20cac-e997-5ea5-8a9e-eaa15d1176a5 GPT-6 Sol medium effort critical 55.53 40.48 88.41 f5c89c7f-1e3d-5fd4-92b5-21c6bc59f61d GPT-6 Sol low effort critical 50.20 35.71 84.44 d4878386-3e39-5cd0-9bdd-7ce093f99d9d GPT-6.1 Sol max effort critical 56.74 40.48 94.87 ea5dd947-714e-5bee-85fb-ece01fb061a9 GPT-6.1 Sol xhigh effort critical 51.64 35.71 93.22 dfeb649e-809e-5c63-b795-15dced32f18e GPT-6.1 Sol high effort critical 51.91 35.71 95.00 6daffe2f-8a96-50bf-8a0e-765caf2dd865 GPT-6.1 Sol medium effort critical 44.16 28.57 97.14 64a4d50a-6494-57a3-bf57-f76dfd0f9223 GPT-6.1 Sol low effort critical 46.90 30.95 96.77 36153d87-267e-55c8-bed6-6eac40ffdbc3 GPT-6 Luna max effort critical 59.17 42.86 95.56 901bbfa8-49f3-5f65-9a40-0e0dde982666 GPT-6 Luna xhigh effort critical 54.78 38.10 97.50 fbc652a2-5ba6-57e4-aeec-a552eaaf445f GPT-6 Luna high effort critical 54.72 38.10 97.06 a78cc3e7-05b6-537f-a27d-e4ebd863dae7 GPT-6 Luna medium effort critical 50.00 33.33 100.00 a9f5b61b-29e9-5fda-a437-00afe8fe2aa4 GPT-6 Luna low effort critical 47.06 33.33 80.00 36397071-f3e5-58ac-af6e-bce5c1286993 Gemini 3.8 Flash medium effort critical 54.15 38.10 93.62 99c74d2e-77c4-5df2-a3fe-d67a6b5acbfd Gemini 3.8 Flash low effort critical 25.88 16.67 57.89 2e48442d-1b30-51cd-b40c-41012b8e9168 DeepSeek V4.1 Flash max effort high 82.39 90.15 75.87 52ccc297-28aa-58d1-914f-9b8e9ba00e16 DeepSeek V4.1 Flash high effort high 77.36 79.55 75.30 4b6f1bfc-9d2d-5124-bfa4-6f0295c67ad7 DeepSeek V4.1 Flash low effort high 75.29 76.52 74.10 814e5ec4-8c99-5ac9-959d-35dc4362caec GLM 5.3 max effort high 77.91 75.00 81.06 f8ed8869-c28f-5bba-a5c5-6e366685d145 GLM 5.3 high effort high 72.12 65.15 80.75 3541ae6b-327d-535e-a954-f08bf733938e GLM 5.3 low effort high 58.56 50.00 70.67 2895e1db-dd2a-5461-b20e-3ec66671926a Claude Opus 5.5 max effort high 86.38 85.61 87.17 f7463661-a438-561c-96ac-843c7939d71b Claude Opus 5.5 xhigh effort high 79.62 73.48 86.87 c9b7bdbd-39c8-5140-84e4-a0fff12bb1ea Claude Opus 5.5 high effort high 74.25 67.42 82.61 3eab60f0-c8d8-5f3e-bd67-7da5b90edf9d Claude Opus 5.5 medium effort high 73.14 65.15 83.37 12df91d3-3f0c-5dd3-bb48-02f22c7c6c08 Claude Opus 5.5 low effort high 64.46 53.79 80.41 449da42a-2057-5574-9de3-56bcd79e7a94 Claude Opus 5 high effort high 76.74 74.24 79.41 78cb4dc8-3927-5060-8d0c-14ae662c19ba Claude Opus 5 medium effort high 72.93 70.45 75.59 3e9b4fb2-1fcd-505c-9bce-90f7111fd25e Claude Opus 5 low effort high 66.47 59.09 75.97 e8feb853-cb76-5b43-afa4-ccb4d06477d9 Grok 4.6 high effort high 75.43 68.94 83.27 6c6bf706-ce17-5d5b-beb5-14e1ca274c77 Grok 4.6 xhigh effort high 76.54 68.94 86.04 a46493ee-207c-5a4e-acc6-041c5a21cab4 Grok 4.6 medium effort high 73.49 65.91 83.04 cb59e9a8-c128-56c1-ace9-2f291b18f118 Grok 4.6 low effort high 60.01 51.52 71.87 19b1e774-2306-5d68-a71a-1476e94ffb01 GPT-5.6 Sol high effort high 78.02 69.70 88.60 909a01d6-c340-5b97-9367-68493498325a GPT-5.6 Sol low effort high 68.21 57.58 83.66 1e0dc077-2b51-5853-a08a-3300116ae4ce Kimi K3 max effort high 74.52 65.91 85.73 5b0f70ad-7c7d-5627-882b-1951e67fb1c1 Kimi K3 high effort high 71.14 62.12 83.23 082ea61d-001a-5fd3-ab7a-9bbb4c8847d8 Kimi K3 low effort high 51.44 39.39 74.11 d37f6c88-f0ff-54b2-ba8d-35f44a196366 GPT-6 Astra max effort high 79.52 70.45 91.25 a44b3d47-5e28-5fc1-8b85-1425b991ae62 GPT-6 Sol max effort high 80.83 74.24 88.71 23bb69f6-adb5-5f3e-8215-d714b4e47dd2 GPT-6 Sol xhigh effort high 77.36 68.94 88.13 08236e89-43dc-5064-a50f-051bb3e5ef45 GPT-6 Sol high effort high 72.56 62.88 85.78 cef20cac-e997-5ea5-8a9e-eaa15d1176a5 GPT-6 Sol medium effort high 69.58 59.09 84.59 f5c89c7f-1e3d-5fd4-92b5-21c6bc59f61d GPT-6 Sol low effort high 60.07 48.48 78.93 d4878386-3e39-5cd0-9bdd-7ce093f99d9d GPT-6.1 Sol max effort high 80.15 71.97 90.42 ea5dd947-714e-5bee-85fb-ece01fb061a9 GPT-6.1 Sol xhigh effort high 78.24 68.18 91.77 dfeb649e-809e-5c63-b795-15dced32f18e GPT-6.1 Sol high effort high 77.57 67.42 91.31 6daffe2f-8a96-50bf-8a0e-765caf2dd865 GPT-6.1 Sol medium effort high 75.88 65.15 90.83 64a4d50a-6494-57a3-bf57-f76dfd0f9223 GPT-6.1 Sol low effort high 74.89 64.39 89.49 36153d87-267e-55c8-bed6-6eac40ffdbc3 GPT-6 Luna max effort high 72.46 60.61 90.09 901bbfa8-49f3-5f65-9a40-0e0dde982666 GPT-6 Luna xhigh effort high 68.08 56.06 86.66 fbc652a2-5ba6-57e4-aeec-a552eaaf445f GPT-6 Luna high effort high 62.97 49.24 87.30 a78cc3e7-05b6-537f-a27d-e4ebd863dae7 GPT-6 Luna medium effort high 60.59 48.48 80.76 a9f5b61b-29e9-5fda-a437-00afe8fe2aa4 GPT-6 Luna low effort high 44.93 32.58 72.37 36397071-f3e5-58ac-af6e-bce5c1286993 Gemini 3.8 Flash medium effort high 72.49 65.15 81.68 99c74d2e-77c4-5df2-a3fe-d67a6b5acbfd Gemini 3.8 Flash low effort high 58.87 48.48 74.90 2e48442d-1b30-51cd-b40c-41012b8e9168 DeepSeek V4.1 Flash max effort medium 77.95 84.24 72.54 52ccc297-28aa-58d1-914f-9b8e9ba00e16 DeepSeek V4.1 Flash high effort medium 75.99 82.42 70.49 4b6f1bfc-9d2d-5124-bfa4-6f0295c67ad7 DeepSeek V4.1 Flash low effort medium 76.51 81.82 71.84 814e5ec4-8c99-5ac9-959d-35dc4362caec GLM 5.3 max effort medium 82.49 83.03 81.96 f8ed8869-c28f-5bba-a5c5-6e366685d145 GLM 5.3 high effort medium 75.15 72.12 78.43 3541ae6b-327d-535e-a954-f08bf733938e GLM 5.3 low effort medium 62.13 57.58 67.48 2895e1db-dd2a-5461-b20e-3ec66671926a Claude Opus 5.5 max effort medium 86.59 87.88 85.34 f7463661-a438-561c-96ac-843c7939d71b Claude Opus 5.5 xhigh effort medium 85.14 84.24 86.05 c9b7bdbd-39c8-5140-84e4-a0fff12bb1ea Claude Opus 5.5 high effort medium 78.89 77.58 80.25 3eab60f0-c8d8-5f3e-bd67-7da5b90edf9d Claude Opus 5.5 medium effort medium 75.08 72.12 78.29 12df91d3-3f0c-5dd3-bb48-02f22c7c6c08 Claude Opus 5.5 low effort medium 67.02 59.39 76.90 449da42a-2057-5574-9de3-56bcd79e7a94 Claude Opus 5 high effort medium 78.12 87.88 70.31 78cb4dc8-3927-5060-8d0c-14ae662c19ba Claude Opus 5 medium effort medium 74.79 76.36 73.27 3e9b4fb2-1fcd-505c-9bce-90f7111fd25e Claude Opus 5 low effort medium 65.92 64.24 67.69 e8feb853-cb76-5b43-afa4-ccb4d06477d9 Grok 4.6 high effort medium 83.60 83.03 84.18 6c6bf706-ce17-5d5b-beb5-14e1ca274c77 Grok 4.6 xhigh effort medium 84.43 84.85 84.01 a46493ee-207c-5a4e-acc6-041c5a21cab4 Grok 4.6 medium effort medium 81.63 83.03 80.28 cb59e9a8-c128-56c1-ace9-2f291b18f118 Grok 4.6 low effort medium 65.43 64.24 66.67 19b1e774-2306-5d68-a71a-1476e94ffb01 GPT-5.6 Sol high effort medium 81.51 77.58 85.88 909a01d6-c340-5b97-9367-68493498325a GPT-5.6 Sol low effort medium 73.37 64.24 85.52 1e0dc077-2b51-5853-a08a-3300116ae4ce Kimi K3 max effort medium 80.89 78.79 83.11 5b0f70ad-7c7d-5627-882b-1951e67fb1c1 Kimi K3 high effort medium 76.10 72.12 80.53 082ea61d-001a-5fd3-ab7a-9bbb4c8847d8 Kimi K3 low effort medium 63.07 55.15 73.64 d37f6c88-f0ff-54b2-ba8d-35f44a196366 GPT-6 Astra max effort medium 84.54 77.58 92.87 a44b3d47-5e28-5fc1-8b85-1425b991ae62 GPT-6 Sol max effort medium 84.81 82.42 87.34 23bb69f6-adb5-5f3e-8215-d714b4e47dd2 GPT-6 Sol xhigh effort medium 82.49 76.97 88.87 08236e89-43dc-5064-a50f-051bb3e5ef45 GPT-6 Sol high effort medium 80.22 74.55 86.82 cef20cac-e997-5ea5-8a9e-eaa15d1176a5 GPT-6 Sol medium effort medium 75.22 69.09 82.55 f5c89c7f-1e3d-5fd4-92b5-21c6bc59f61d GPT-6 Sol low effort medium 71.73 64.85 80.24 d4878386-3e39-5cd0-9bdd-7ce093f99d9d GPT-6.1 Sol max effort medium 84.89 79.39 91.21 ea5dd947-714e-5bee-85fb-ece01fb061a9 GPT-6.1 Sol xhigh effort medium 84.70 78.79 91.58 dfeb649e-809e-5c63-b795-15dced32f18e GPT-6.1 Sol high effort medium 84.45 78.18 91.81 6daffe2f-8a96-50bf-8a0e-765caf2dd865 GPT-6.1 Sol medium effort medium 83.29 75.76 92.50 64a4d50a-6494-57a3-bf57-f76dfd0f9223 GPT-6.1 Sol low effort medium 80.03 71.52 90.84 36153d87-267e-55c8-bed6-6eac40ffdbc3 GPT-6 Luna max effort medium 80.05 72.73 89.02 901bbfa8-49f3-5f65-9a40-0e0dde982666 GPT-6 Luna xhigh effort medium 75.25 65.45 88.48 fbc652a2-5ba6-57e4-aeec-a552eaaf445f GPT-6 Luna high effort medium 74.58 65.45 86.66 a78cc3e7-05b6-537f-a27d-e4ebd863dae7 GPT-6 Luna medium effort medium 73.12 64.85 83.81 a9f5b61b-29e9-5fda-a437-00afe8fe2aa4 GPT-6 Luna low effort medium 59.29 50.30 72.18 36397071-f3e5-58ac-af6e-bce5c1286993 Gemini 3.8 Flash medium effort medium 78.13 73.94 82.83 99c74d2e-77c4-5df2-a3fe-d67a6b5acbfd Gemini 3.8 Flash low effort medium 61.38 53.94 71.20 2e48442d-1b30-51cd-b40c-41012b8e9168 DeepSeek V4.1 Flash max effort low 65.48 76.34 57.32 52ccc297-28aa-58d1-914f-9b8e9ba00e16 DeepSeek V4.1 Flash high effort low 65.57 72.04 60.17 4b6f1bfc-9d2d-5124-bfa4-6f0295c67ad7 DeepSeek V4.1 Flash low effort low 67.40 75.27 61.02 814e5ec4-8c99-5ac9-959d-35dc4362caec GLM 5.3 max effort low 71.34 69.89 72.86 f8ed8869-c28f-5bba-a5c5-6e366685d145 GLM 5.3 high effort low 61.40 56.99 66.56 3541ae6b-327d-535e-a954-f08bf733938e GLM 5.3 low effort low 44.42 35.48 59.38 2895e1db-dd2a-5461-b20e-3ec66671926a Claude Opus 5.5 max effort low 77.93 77.42 78.44 f7463661-a438-561c-96ac-843c7939d71b Claude Opus 5.5 xhigh effort low 69.64 63.44 77.18 c9b7bdbd-39c8-5140-84e4-a0fff12bb1ea Claude Opus 5.5 high effort low 65.49 56.99 76.97 3eab60f0-c8d8-5f3e-bd67-7da5b90edf9d Claude Opus 5.5 medium effort low 61.70 50.54 79.19 12df91d3-3f0c-5dd3-bb48-02f22c7c6c08 Claude Opus 5.5 low effort low 50.80 38.71 73.86 449da42a-2057-5574-9de3-56bcd79e7a94 Claude Opus 5 high effort low 62.51 68.82 57.26 78cb4dc8-3927-5060-8d0c-14ae662c19ba Claude Opus 5 medium effort low 58.98 62.37 55.95 3e9b4fb2-1fcd-505c-9bce-90f7111fd25e Claude Opus 5 low effort low 56.34 50.54 63.64 e8feb853-cb76-5b43-afa4-ccb4d06477d9 Grok 4.6 high effort low 64.11 56.99 73.26 6c6bf706-ce17-5d5b-beb5-14e1ca274c77 Grok 4.6 xhigh effort low 65.88 56.99 78.05 a46493ee-207c-5a4e-acc6-041c5a21cab4 Grok 4.6 medium effort low 58.91 48.39 75.29 cb59e9a8-c128-56c1-ace9-2f291b18f118 Grok 4.6 low effort low 45.75 37.63 58.33 19b1e774-2306-5d68-a71a-1476e94ffb01 GPT-5.6 Sol high effort low 76.09 69.89 83.49 909a01d6-c340-5b97-9367-68493498325a GPT-5.6 Sol low effort low 59.30 44.09 90.54 1e0dc077-2b51-5853-a08a-3300116ae4ce Kimi K3 max effort low 63.70 54.84 75.98 5b0f70ad-7c7d-5627-882b-1951e67fb1c1 Kimi K3 high effort low 58.58 50.54 69.66 082ea61d-001a-5fd3-ab7a-9bbb4c8847d8 Kimi K3 low effort low 46.94 38.71 59.61 d37f6c88-f0ff-54b2-ba8d-35f44a196366 GPT-6 Astra max effort low 68.85 56.99 86.96 a44b3d47-5e28-5fc1-8b85-1425b991ae62 GPT-6 Sol max effort low 74.06 63.44 88.95 23bb69f6-adb5-5f3e-8215-d714b4e47dd2 GPT-6 Sol xhigh effort low 73.76 64.52 86.10 08236e89-43dc-5064-a50f-051bb3e5ef45 GPT-6 Sol high effort low 71.22 59.14 89.50 cef20cac-e997-5ea5-8a9e-eaa15d1176a5 GPT-6 Sol medium effort low 64.56 52.69 83.33 f5c89c7f-1e3d-5fd4-92b5-21c6bc59f61d GPT-6 Sol low effort low 55.70 41.94 82.93 d4878386-3e39-5cd0-9bdd-7ce093f99d9d GPT-6.1 Sol max effort low 72.42 60.22 90.85 ea5dd947-714e-5bee-85fb-ece01fb061a9 GPT-6.1 Sol xhigh effort low 71.52 58.06 93.10 dfeb649e-809e-5c63-b795-15dced32f18e GPT-6.1 Sol high effort low 69.79 56.99 90.00 6daffe2f-8a96-50bf-8a0e-765caf2dd865 GPT-6.1 Sol medium effort low 60.15 46.24 86.05 64a4d50a-6494-57a3-bf57-f76dfd0f9223 GPT-6.1 Sol low effort low 54.69 37.63 100.00 36153d87-267e-55c8-bed6-6eac40ffdbc3 GPT-6 Luna max effort low 63.04 49.46 86.90 901bbfa8-49f3-5f65-9a40-0e0dde982666 GPT-6 Luna xhigh effort low 60.98 48.39 82.45 fbc652a2-5ba6-57e4-aeec-a552eaaf445f GPT-6 Luna high effort low 56.28 43.01 81.40 a78cc3e7-05b6-537f-a27d-e4ebd863dae7 GPT-6 Luna medium effort low 51.40 37.63 81.05 a9f5b61b-29e9-5fda-a437-00afe8fe2aa4 GPT-6 Luna low effort low 37.85 25.81 70.97 36397071-f3e5-58ac-af6e-bce5c1286993 Gemini 3.8 Flash medium effort low 68.15 56.99 84.75 99c74d2e-77c4-5df2-a3fe-d67a6b5acbfd Gemini 3.8 Flash low effort low 46.99 34.41 74.07 2e48442d-1b30-51cd-b40c-41012b8e9168