Benchmarks are Not Enough.
RAMP measures AI agents in a real compiler construction pipeline — from lexer to code generation — under persistent production conditions with automatic failure diagnosis via resurrection.
Semi-Guided Progressive · GCPS scoring
| # | Model | PL | MR |
|---|---|---|---|
| Loading... | |||
Current benchmarks test isolated tasks in sterile environments. RAMP evaluates models under realistic production conditions.
Six compiler construction tasks form a strict causal pipeline. Task 2 needs Task 1's output — just like real engineering.
When an agent fails, we inject the correct intermediate artifact and continue — decomposing cascading failure into diagnostic components.
GCPS measures incremental capability. E2E-PRS measures production readiness. Same traces, complementary insights.
Track token usage, wall-clock time, and API cost per task — not just outcome scores. Process metrics as first-class signals.
OpenHands SDK · Claude Code · Codex CLI · Kimi CLI — unified API through AIhub gateway for fair comparisons.
Built on YatCC, a full-semester compiler course platform used by hundreds of students. Real workload, real evaluation.
Select from OpenHands SDK, Claude Code, Codex, Kimi CLI, or any OpenAI-compatible API via AIhub.
ramp run --backend openhands --model your-model
Agent reads docs, writes code, compiles, tests — under persistent context. Resurrection kicks in on failure.
Detailed reports with outcome scores and process metrics: cost, tokens, rounds, elapsed time.
pip install ramp
ramp run --backend openhands --model your-model --tasks 0-5