How well can AI agents build a compiler end-to-end? Results from our production-grounded evaluation framework.
| # | Model | Workload | T0 | T1 | T2 | T3 | T4 | T5 | PL | MR | Cost $ | Time |
|---|
GCPS (Granular Capability Profiling) — Each task scored 0–100 independently. Relaxed, incremental evaluation.
E2E-PRS (End-to-End Production Readiness) — Tasks 0–3 must achieve perfect scores. Any failure cascades to zero for subsequent tasks.
Guided (Incremental) — New context per task. Scaffolded starter code provided.
Semi-Guided (Progressive) — Persistent context across all 6 tasks. Same agent carries the full project.
Autonomous (RAMP-Pro) — No code framework. Build the full compiler from scratch.
Last updated: | Powered by RAMP