Kimi K3 vs Claude Sonnet 5 for frontend design agents
Main takeaways
Kimi K3 matched Sonnet on completed designs, but slower runs and failed outputs changed the result across the full evaluation.
Kimi produced similar quality as Sonnet when runs finished
On completed designs, Kimi reached 0.739 visual similarity and Sonnet reached 0.698.
Reliability changed the full-run result
Kimi timed out 14 times and returned 12 blank outputs across 172 runs. Sonnet had neither outcome across 162 runs.
Kimi cost less, but ran more slowly
Kimi averaged $1.79 per run versus Sonnet's $2.27, while its median run took 711 seconds to Sonnet's 462.
A recent independent eval compared the Paper MCP server and the Figma MCP server as design tools for a coding agent, all on claude-sonnet-5.
Then Kimi K3 came out to claims that it's a genuinely good, frontier-level model for a fraction of the price. It's open-weights, so you can run it yourself instead of renting it from a lab. A model that is cheaper, open, and good enough to drive a real coding agent would be a major shift for the industry.
The harness already existed, so this eval reuses it and swaps the agent model from Sonnet to Kimi K3, holding everything else constant.
The original eval, in one minute
The setup gives a coding agent a screenshot of a webpage and asks it to recreate the page as a single self-contained HTML file, with a design MCP server as the only variable. One agent got Figma's MCP server, the other got Paper's. It ran across two datasets. One held 40 ordinary webpages from an academic benchmark, the other 27 more complex, design-heavy pages selected for textures that stress a CSS-first approach, like glassmorphism, 3D-rendered elements, volumetric glow, dense dashboards, and big display type. Every recreation was scored three ways. render_success checks that the HTML renders to a non-blank page in headless Chrome. visual_similarity uses CLIP, an OpenAI model that maps images into a space where similar-looking ones sit close together, to measure how close the rendered output is to the reference screenshot. recreation_faithfulness is an LLM judge making a yes-or-no call on whether someone who knew the original page would recognize the recreation. That judge ran on gpt-4o, a model from neither the Sonnet nor the Kimi family, so neither agent under test grades its own output and no self-preference can creep into the score.
In the original eval, the design tool made no difference on ordinary pages. Paper and Figma tied at 0.741 and 0.744 on visual similarity, a 0-to-1 score of how closely the recreation resembles the original where 1.0 is identical. On the complex pages they were statistically indistinguishable as well, Paper at 0.716 and Figma at 0.679. The paired difference of roughly 0.04 was not statistically significant (paired test across the 27 designs, p = 0.21). Where the tools separated was on consistency and price. Figma's run-to-run variance ran about 1.9 times Paper's, and it cost 32% more per point of visual quality.
Those results all ran on Sonnet. The sections below report the same eval with Kimi K3 as the agent.
When Kimi finished, it was good
The table reports completed-run scores for Kimi K3 and Sonnet.
| Completed runs only | Sonnet | Kimi K3 |
|---|---|---|
| visual similarity, Paper | 0.716 | 0.738 |
| visual similarity, Figma | 0.679 | 0.741 |
| recreation faithfulness | 0.87 | 0.93 |
On the designs Kimi got through, its output was close to Sonnet's. In paired per-design comparisons, neither model difference was statistically significant. The Paper difference favored Sonnet (p = 0.11) and the Figma difference favored Kimi, also not significant, both consistent with no real difference at this sample size. If all you did was look at the finished pages, you'd conclude an open model is a perfectly reasonable agent for this task.
The catch is that these numbers are for Kimi's completed runs only.
The catch was finishing
The eval stops any run that goes past 25 minutes. Sonnet never came close, since its runs finished in about 8 minutes on average and none timed out. Kimi ran so much slower that its slowest runs crossed the 25-minute line 14 times, and once a run crosses it the harness cuts it off wherever it is. Kimi wasn't crashing or erroring out. It was taking far longer than Sonnet and hitting a limit that the faster model never reached.
A timeout did not always mean a wasted run. A few of the timed-out runs had built a good page before the cutoff, so they scored normally and were only flagged. Separately, 12 runs returned nothing renderable at all, either a blank page or no HTML, and those score zero.
The failures clustered where the original eval said the hard designs were, the texture-heavy and illustration-heavy ones that CSS can't express. The spatial-warped-type-poster design timed out on all three Paper trials. The flat park illustration came back blank under both tools. Where Sonnet would flatten an un-CSS-able texture into a valid page and move on, Kimi was more likely to keep iterating until it timed out.
Some runs never converged. One Figma run emitted 297,000 characters of HTML before timing out, roughly ten times a normal page, repeating itself rather than settling. The trace shows the turn count climbing well past what a normal page needs, with no sign of stopping. This is the failure mode that doesn't show up in an average until you look at what the timed-out runs were doing.
Counting these failures in, Kimi's score drops from 0.739 to 0.685.
Cost per run
Kimi came in cheaper, $1.79 a run versus $2.27, and the gap was widest on Figma, where Kimi ran at $1.77 against Sonnet's $2.53.
The saving does not come from cheaper tokens. Kimi K3 lists at $3 per million input tokens and $15 per million output, the same as Sonnet 5's standard rate and more than the $2 and $10 Sonnet is charging right now. K3 is a 2.8-trillion-parameter flagship and is priced like one, well above what the "open model" label suggests. It came out cheaper per run only because it used fewer turns, so it billed fewer tokens even at a higher rate.
K3 is often reported to use more tokens than other models, but here it billed fewer than Sonnet on the same designs. The exception is the worst runs. When K3 spiraled it spent far more, and the run in the trace above is one of those, so the lower average comes with much heavier spending on the runs that go wrong.
Comparing the traces between Kimi and Sonnet
Paper still pulled the agent into a tight, granular loop, lots of write_html, get_screenshot, and update_styles calls, with the agent reading back the canvas as JSX to build its final file. Figma still funneled almost everything through one coarse use_figma call. Kimi adopted both grammars the same way Sonnet did, so the tool surface shapes how an agent works more than the model does.
Where Kimi differed was intensity. It made fewer MCP calls per run than Sonnet, about 14 on average versus 21, and it self-corrected less. Both models screenshot their own canvas unprompted to compare against the reference, but Kimi did it about 2.5 times a run to Sonnet's 4.3, so it checked its own work noticeably less often. The original eval found that self-correction frequency doesn't correlate with quality, so that difference isn't worth much on its own, though it's consistent and worth noting.
The traces also explain where Paper's efficiency edge from the first eval came from, and why it disappeared here. Under Sonnet, Figma runs took about 25% more turns than Paper, roughly 51 against 41, and the extra turns were mostly ceremony. Figma sessions averaged around five project-management calls each, TaskCreate and TaskUpdate, three times as many Bash calls, and a fixed setup cost of whoami plus create_new_file on every run, while Paper agents mostly wrote HTML to the canvas and screenshotted it to check their work. Paper made more design calls but fewer turns, because its canvas is already HTML and each edit is direct, where Figma's single use_figma tool makes a heavier round-trip through its own format.
Under Kimi that gap disappears. Kimi runs at roughly 22 to 29 seconds per turn against Sonnet's ten, so the model's own per-turn latency outweighs any difference in how many turns each tool needs, and both variants land near 700 seconds regardless of tool. A slow enough model flattens the tool-level efficiency differences that a fast one shows.
You can see the whole pattern in a single design. Below are the two agents building the same CRM dashboard in Paper, Sonnet on top and Kimi underneath. Sonnet's trace is a dense stack of short spans, dozens of small model turns and tool calls a few seconds each. Kimi's trace has far fewer spans, but several of its model calls run 40 to 80 seconds on their own, and the whole run takes 854 seconds against Sonnet's 383. Fewer, slower steps, side by side.
What to take away
On raw capability, the hype around Kimi K3 is real. Its finished pages matched Sonnet's. The claim that it's cheaper because it's open doesn't hold. Kimi K3 lists at a higher per-token price than Sonnet and came out cheaper here only by taking fewer turns. And a good number of its runs never finished at all.
Swapping in an open model didn't change the conclusion of the original eval. Paper and Figma are still a tie.
What it changed was the kind of risk. With Sonnet, the eval measured design quality, since reliability was a given. With Kimi, quality is largely solved and the open question is operational. Will the run finish inside your budget, and can you absorb the runs that don't?
If you're building an eval like this one, Braintrust is the active observability platform for agents in production, bringing tracing and evals together so you can go from experiment scores down to each agent's individual tool calls. Get started →
Appendix
The gallery opens each selected design in a three-way comparison of the reference, Sonnet, and Kimi K3. Select an image to focus it, then use the arrows or model selector to move between outputs. The score badges show each model's mean visual similarity across all runs for the selected design. "Best" means the single highest-scoring attempt that model produced across both design tools and all three trials. The mean scores include timed-out and blank runs, so they will usually sit below the best-attempt image.
agentcard
Dark hero for an AI-agent debit card