Independent eval research. Open methodology, published datasets, and the tools to run your own studies.
Original, open-source studies
Model comparisons, agent setups, cost, and modalities, measured on real tasks. Read the methodology and results, or rerun a study on your own data.
Models answered 175 MathTutorBench cases covering mathematical problem solving and tutoring. Deterministic scorers checked correctness, and GLM 5.3 Flash judged writing quality.
125 objectively scored caseshigher is better
GLM 5.3 judgehigher is better
Jev was compared with DeepSeek V4 Flash, GPT-5.6 Luna, GPT-4.1 Mini, and Gemma 4B on 616 answer-correctness pairs from JudgeBench and 1,086 groundedness claims from LLM-AggreFact. Each judgment was scored against an objectively verified or human-authored reference label.
1,086 claimshigher is better
616 independent pairshigher is better
Kimi K3 ran through Moonshot and Fireworks in 30 direct latency requests per provider and 60 planned Figma-to-HTML runs per provider. Paired intervals compare matched designs.
55% faster
Run evals like us
Each skill packages best practices from leading AI teams and researchers into a plain SKILL.md file your coding agent can read. Use them to get started running evals on your data.
Point your agent at a skill
Ask it to do what the skill describes, like validating a scorer against human labels or finding hidden failure modes.
- 1.Name the reference tier before computing anything.
- 2.Verify alignment: scorer outputs and reference labels must line up at the item and criterion level.
- 3.Report agreement (κ or α) with uncertainty, not raw accuracy.
- 4.Lead with the most decision-relevant false acceptance before any aggregate, and enumerate the dangerous cells case by case.
- 5.Break errors down by class and severity.
- 6.Test sensitivity and shortcuts: inject known regressions and improvements and confirm the scorer moves; probe whether length, confidence, or polish raise the score independent of quality; probe whether text addressed to the judge moves it; slice agreement by subgroup.