We test models on synthetic tasks and production workloads, then publish what we measure: what each one costs, how fast it is, and whether the output does the job.
14 frontier and high-efficiency models measured on six software factory workloads. Frontier models lead on design; on coding, review, CI triage and pull requests, high-efficiency models match them at a fraction of the cost.
13 low-cost models measured on production workloads: quality, cost and latency against the field, and the format failures that rule a model out.
How to choose where to run an LLM. The same open-weight model can cost up to 14x more depending on the provider, and the cheapest endpoint may run a lower-precision copy. Built from 1,317 live provider endpoints.
Five AI models answered the same startup-studio tasks. Pick your preferred responses blindly, then see what each cost. An interactive case study with Pioneer Square Labs.
We used libfx to give 11 models the same business tasks, check their answers, and compare cost per successful task. Here is what the bills showed.
Ten inexpensive models, three production workloads, identical inputs. We scored quality, latency and cost, then ranked each model against its peers on the two ratios that decide a deployment.
