Three new LLMs tested: Haiku 5.5, DeepSeek 4.1 Flash 1017 and Gemini 4 Argon
Summary
This week several new models were released, so I tested how well they perform and if they are worth upgrading to. Haiku 5.5 performed as I expected, but I was surprised by the results from DeepSeek 4.1 Flash 1017 and Gemini 4 Argon.
Haiku 5.5: better, but still behind Luna 6
Predictably, Haiku 5.5 is an upgrade from Haiku 4.5. It is better and cheaper, with the clearest improvement in extraction from noisy data. Haiku 5.5 is a compelling specialist when the job is to make one bounded decision quickly and abstain when the evidence is weak.
But Luna 6 remains the better general default. It is cheaper and better in most categories.
Three shared non-coding workloads
| model | mean quality | cost / task | latency |
|---|---|---|---|
| Claude Haiku 4.5 | 0.811 | $0.0051 | 3.6 s |
| Claude Haiku 5.5 | 0.898 | $0.0009 | 4.4 s |
| Luna 6 | 0.945 | $0.0005 | 6.9 s |
Luna 6 produced notably shorter responses than Haiku 5.5 in these tests, despite its higher latency.
DeepSeek 4.1 Flash 1017: faster, but not fast enough
Newly released DeepSeek 4.1 Flash 1017 shows interesting results. It seems to be optimized for speed. In our testing it was faster than DeepSeek 4.1 Flash, with roughly the same quality.
But it was still slower than Luna 6 on average and more expensive. Not recommended.
Three shared non-coding workloads
| model | mean quality | cost / task | latency |
|---|---|---|---|
| DeepSeek 4.1 Flash | 0.933 | $0.0008 | 12.9 s |
| DeepSeek 4.1 Flash 1017 | 0.915 | $0.0007 | 10.4 s |
| Luna 6 | 0.945 | $0.0005 | 6.9 s |
Gemini 4 Argon: a solid upgrade over Gemini 3.8 Flash
Gemini 4 Argon was also a surprise. I expected it to compete in the Astra 6 or Opus 5.5 tier, but it did not reach that tier. It is instead a nice upgrade over Gemini 3.8 Flash.
It is still far behind the frontier models on the most challenging design tasks. But if you are using Google models, it is a solid upgrade over Gemini 3.8 Flash. Argon was higher in quality and faster at roughly the same cost.
All five workloads
| model | mean quality | cost / task | latency |
|---|---|---|---|
| Gemini 3.8 Flash | 0.789 | $0.040 | 50.1 s |
| Gemini 4 Argon | 0.822 | $0.040 | 27.1 s |
| Astra 6 | 0.892 | $0.172 | 28.0 s |
Testing results
Full results
| model | mean quality | cost / task | latency | Architecture | Repository implementation | Noisy extraction | Semantic matching | Grounded research |
|---|---|---|---|---|---|---|---|---|
| Luna 6 | 0.828 | $0.0013 | 14.1 s | 0.479 | 0.737 | 0.922 | 0.912 | 1.000 |
| Claude Haiku 5.5 | 0.777 | $0.0026 | 15.6 s | 0.414 | — | 0.859 | 0.941 | 0.895 |
| DeepSeek 4.1 Flash 1017 | 0.785 | $0.0017 | 31.7 s | 0.394 | 0.780 | 0.898 | 0.883 | 0.965 |
| Gemini 4 Argon | 0.812 | $0.027 | 19.5 s | 0.382 | 0.863 | 0.938 | 0.939 | 0.988 |
| Gemini 3.8 Flash | 0.779 | $0.016 | 20.6 s | 0.318 | 0.832 | 0.938 | 0.883 | 0.977 |
| Astra 6 | 0.900 | $0.113 | 26.0 s | 0.607 | 0.861 | 0.992 | 1.000 | 1.000 |
| Claude Sonnet 5.5 | 0.811 | $0.057 | 28.9 s | 0.471 | 0.760 | 0.938 | 0.930 | 0.906 |
| GLM 5.3 Flash | 0.740 | $0.0019 | 17.2 s | 0.340 | 0.804 | 0.918 | 0.818 | 0.883 |
| Grok 4.7 | 0.834 | $0.049 | 67.2 s | 0.536 | — | 0.957 | 0.854 | 0.988 |
Check the next article, which compares GPT-6.1 Terra and Kimi K3.1 Pro on the same five workloads.
Methodology
We tested architecture under ambiguity, repository implementation, noisy document extraction, semantic matching with abstention, and grounded research. Each model received the same eight cases per workload, replayed twice. GPT-6.1 Sol graded the outputs against workload-specific rubrics.
The charts and summary metrics use the four workloads completed by every model. Repository implementation is shown separately in the full table. Earlier Haiku and DeepSeek versions were rerun only on the three shared non-coding workloads, so they appear only in the section tables.