# v0.0.8 frozen blind result: Qwen2.5-Coder-7B

## Outcome

The v0.0.8 candidate fails the proof gate on untouched HumanEval 60–79.

| Metric | LIR v0.0.8 | Python |
|---|---:|---:|
| Tasks | 20 | 20 |
| Pass@1 | 20.0% (4/20) | 85.0% (17/20) |
| Final pass rate after one repair | 25.0% (5/20) | 90.0% (18/20) |
| Generated o200k tokens | 1,961 | 1,165 |
| Repair-prompt o200k tokens | 693 | 71 |
| Session-amortized total tokens | 5,985 | 4,293 |
| Session-amortized tokens per correct | 1,197.0 | 238.5 |
| Cold-request tokens per correct | 2,291.4 | 253.3 |

LIR is 65 percentage points below Python at Pass@1 and uses about 5.02 times as many session-amortized tokens per correct solution. The required quality gap was no more than two points and the required token saving was at least 25%; v0.0.8 meets neither condition.

No breakthrough, no-quality-loss, or token-saving claim is supported.

## Frozen-evaluation configuration

- Model: `Qwen/Qwen2.5-Coder-7B-Instruct`
- Tasks: official HumanEval 60–79, prepared without `canonical_solution`
- Decoding: temperature 0, maximum 1,200 output tokens
- Repair budget: one for both languages
- Serving: OpenAI-compatible vLLM `v0.11.0`, FP16, eager mode, 4,096-token context
- GPU class for completed runs: RTX 4000 Ada
- Frozen language version: `0.0.8`
- Frozen language SHA-256: `2d7f1e21e02d7873b594ce35407ac6e880f27a78599683b67ac0b0641b98f1ba`
- Prepared blind-task SHA-256: `3aaa71a3217218002250e4b908b250995f47543fa197ec394592b513fd5e211b`
- Compact specification: 288 o200k tokens

The LIR run completed before a transient network-tunnel failure interrupted the initial Python run after task 62. The pod was explicitly terminated. Python was then rerun from the beginning on a fresh RTX 4000 Ada pod with the same frozen tasks, model, image, decoding, and repair settings. Transport retries were added without changing the frozen language or model inputs. The second pod terminated automatically after the result files and comparison were written.

## What improved from v0.0.7

v0.0.7 rejected all 20 first generations at the line parser. In the v0.0.8 blind run:

- 4/20 first generations passed official tests.
- 2/20 first generations failed during compilation.
- 7/20 reached runtime but failed from unsupported or mismatched operations.
- 7/20 reached official assertions but contained incorrect logic.

The whitespace/brace parser catastrophe was substantially repaired. The remaining failure is broader: the 7B model still does not reliably map tasks into the unfamiliar language and its compact specification.

## Repair behavior

- 16 tasks received one repair attempt.
- 9/16 repair generations repeated the first code exactly.
- Only HumanEval/62 changed from failing to passing.
- Final failures consisted of compiler, runtime/semantic, and algorithmic errors.

## Decision

Preserve the v0.0.8 language, task artifact, and raw results as a failed frozen evaluation. HumanEval 60–79 is now development data and must not be reused as headline evidence for a later version.

A scientifically valid next revision should be v0.0.9 or later, developed on HumanEval 0–79 and other non-headline data, then frozen before a new untouched range is opened. Likely work includes a more model-familiar function grammar, a smaller and more coherent standard library, stronger diagnostic repairs, and testing with a stronger model before claiming general token efficiency.

Raw files:

- `benchmarks/external/blind_v008/results/qwen25coder7b_lir.json`
- `benchmarks/external/blind_v008/results/qwen25coder7b_python.json`
- `benchmarks/external/blind_v008/results/qwen25coder7b_compare.md`