# Goal 3A Usefulness Benchmark Result

Status: **COMPLETE — Phase 3A only**. Phase 3B cost-aware planning was not run.

## Central question

> Does the existing mechanism show enough repeatable leverage to justify a usefulness claim beyond the Twin-World demo?

**Recommendation: `KEEP EXISTING OBJECTIVE`**

This is a deterministic synthetic benchmark of the existing equal-cost objective. It is not PMF, warehouse ROI, or a Cheekky pilot result.

## Reproduction record

- Master seed: `20260906`
- Base models per bin/transfer dimension: `6`
- Historical variants per base model: `10`
- Intentional fail-closed cases: `240`
- Supported cases: `1944`
- Total generated cases including fail-closed cases: `2184`
- Generator: `scripts/benchmark_usefulness.py`
- Raw results: `reports/final-sprint/goal3a-usefulness-benchmark.json`
- Runtime: `14.829 seconds`
- Production solver files changed by benchmark: **no**

The generator records every batch's truth assignment and historical observation payload in the raw JSON. Re-running the script with the same configuration reproduces the case IDs and result fields; wall-clock runtime is the only expected non-deterministic field.

## What was compared

For each supported case, no-memory uses the production reconstruction/planner with no historical count. Memory-aware uses the same production code after adding a truthful count at one or more generated movement boundaries. The planner objective remains its existing lexicographic `(worst-case depth, total path length, canonical signature)` objective.

- Full physical recount reference: `n (all declared bins) counts`, one direct count per bin.
- Realized additional counts: policy path depth for every feasible final vector in that model, plus the generated truth path depth.
- Strict advantage: memory-aware `(worst, total)` is lexicographically lower than no-memory `(worst, total)`.
- No advantage: those objective fields are equal, regardless of whether a tie-broken first action changes.

## Overall distribution

| Supported cases | Strict advantage | No advantage | First action changed | Worst depth improved | Total improved |
| --- | --- | --- | --- | --- | --- |
| 1944 | 1110 (57.1%) | 834 (42.9%) | 465 (23.9%) | 720 (37.0%) | 1110 (57.1%) |


Worst-depth delta means `no-memory worst - memory-aware worst`; positive is better for memory. Total delta is defined the same way. Exact distributions are in the raw JSON.

### By bin count

| Bins | Cases | Strict advantage | No advantage | Action changed | Worst improved | Total improved |
| --- | --- | --- | --- | --- | --- | --- |
| 3 | 486 | 315 | 171 | 150 | 188 | 315 |
| 4 | 486 | 297 | 189 | 116 | 207 | 297 |
| 5 | 486 | 262 | 224 | 109 | 164 | 262 |
| 6 | 486 | 236 | 250 | 90 | 161 | 236 |


### By uncertain-transfer count

| Transfers | Cases | Strict advantage | No advantage | Action changed | Worst improved | Total improved |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | 24 | 0 | 24 | 0 | 0 | 0 |
| 1 | 240 | 71 | 169 | 71 | 71 | 71 |
| 2 | 240 | 104 | 136 | 54 | 67 | 104 |
| 3 | 240 | 134 | 106 | 69 | 99 | 134 |
| 4 | 240 | 142 | 98 | 43 | 77 | 142 |
| 5 | 240 | 134 | 106 | 40 | 80 | 134 |
| 6 | 240 | 181 | 59 | 64 | 109 | 181 |
| 7 | 240 | 161 | 79 | 63 | 102 | 161 |
| 8 | 240 | 183 | 57 | 61 | 115 | 183 |


## Representative cases

- Strong win: `case-803abdb4e5ca` (5 bins, 7 transfers, 2 historical observations): no-memory worst/total 4/388; memory-aware 1/4; action changed=True.
- No win: `case-000ee9d4a9bc` (5 bins, 1 transfers, 1 historical observations): no-memory worst/total 1/2; memory-aware 1/2; action changed=False.
- Action changed but worst-case depth did not: `case-5f23e0e7a0a0` (5 bins, 8 transfers, 2 historical observations): no-memory worst/total 3/760; memory-aware 3/80; action changed=True.

## Fail-closed benchmark

The intentionally generated fail-closed set contains `240` cases. `240` returned an expected fail-closed status, for a rate of `100.0%`. Status counts: `{'INSUFFICIENT_OBSERVABILITY': 120, 'MODEL_CONTRADICTION': 120}`. Categories and exact cases are in the raw JSON. These are control cases, not silently included as successful benchmark cases.

## Interpretation

The existing mechanism shows **repeatable leverage** when historical boundary observations reduce the feasible execution model enough to lower the existing equal-cost objective. The result is useful evidence beyond the Twin-World demo only at the level of algorithmic usefulness: it demonstrates distributional, reproducible behavior across the supported contract rather than one fixture.

It does **not** establish warehouse frequency, monetary savings, PMF, or ROI. The benchmark has no real warehouse timing, count-error distribution, labor rates, throughput mix, or customer outcome data. The Cheekky document therefore separates validated operational pain/access from this synthetic quantitative result and leaves future pilot evidence unclaimed.

## Limitations

- Generated cases are model-valid synthetic cases, not observations sampled from Cheekky or a warehouse population.
- All physical locations have equal verification cost because that is the existing planner objective.
- The benchmark uses one SKU and the current binary, full-once, ordered-transfer contract; it does not validate omitted movements or multi-SKU interactions.
- Historical observations are generated from truth, so observation-entry error and boundary misclassification are represented only by the existing fail-closed tests, not by a measured field-error rate.
- The strict-advantage rate depends on this disclosed generator/configuration and must not be relabeled as PMF or ROI.

## Regression and provenance

- Full regression command: `python -m pytest`
- Full regression result: `91 passed, 2 warnings in 281.15 seconds`
- Git HEAD before evidence commit: `46cdc4c82ca605efef643f286536e0dcb3e785db`
- Benchmark evidence commit: recorded in the delivery response; pre-existing unrelated working-tree files were left untouched.
- Final HEAD: recorded in the delivery response.

## Phase boundary

Phase 3B was **not authorized or implemented**. No weighted verification costs, cost-aware policy fields, or changes to solver behavior were made.