Evals
How often a claim verdict matches a hand-assigned label, given fixed evidence. This scores the verdict step only; it does not score web retrieval or the memo's other sections.
Dataset
venturevision-claims v1
updated 2026-11-15
Cases
36
Labels
supported 12
unsupported 12
contradicted 12
Origin
seed-real 14
seed-constructed 22
0 user-feedback candidates awaiting review
Latest scores
| Verifier | Overall | supported | unsupported | contradicted | Run · cost |
|---|---|---|---|---|---|
Heuristic baseline No model: word overlap, number and direction checks | 83.3% (30/36) | 100% (12/12) | 91.7% (11/12) | 58.3% (7/12) | Nov 15, 2026, 22:00 UTC $0 |
LLM verifier | Not run yet. python evals/run_evals.py --mode llm --publish | ||||
CI fails a pull request if heuristic accuracy drops more than 3 points below the stored baseline (83.3% on v1).
Run history
| Run | Verifier | Dataset | Overall |
|---|---|---|---|
| Nov 15, 2026, 22:00 UTC | Heuristic baseline | v1 · 36 cases | 83.3% |
Heuristic baseline misses (6)
Latest run, Nov 15, 2026, 22:00 UTC. Click a case for the evidence.
con-01Meal-planning apps are a $3B global market growing over 12% a year.contradicted expected · got unsupported
- Pitch
- Market: Meal-planning apps are a $3B global market growing over 12% a year, alongside a $100B+ US online grocery market.
- Evidence
“The Meal Planning App Market was valued at USD 1.47 Billion in 2025 and is expected to reach USD 3.87 Billion by 2035, growing at a CAGR of 10.18%.” www.snsinsider.com
- Verifier said
- evidence is off-topic (word overlap 0.33)
- Why the label
- Same market and metric: evidence gives $1.47B (2025) and 10.18% CAGR, incompatible with $3B and over 12%. The demo run labelled this weak_claim.
- Case origin
- seed-real
con-02Thermalith is the first company to commercialize a sand-based thermal battery.contradicted expected · got unsupported
- Pitch
- Thermalith is the first company to commercialize a sand-based thermal battery.
- Evidence
“Manufacturer of sand battery based thermal energy storage systems.” tracxn.com
- Verifier said
- related but not confirming (word overlap 0.43)
- Why the label
- Another company (Polar Night Energy) already manufactures sand-battery thermal storage, so 'first to commercialize' cannot hold. The demo run labelled this unsubstantiated.
- Case origin
- seed-real
con-05Flaky tests cost teams only a few minutes of rerun time per day.contradicted expected · got supported
- Pitch
- Engineers at companies with large CI pipelines lose up to 20% of their week to flaky tests, and teams learn to ignore red builds, which is how real regressions ship.
- Evidence
“Flaky tests cost teams over 4,200 hours of lost productivity per day when considering just the rerun time.” viral-patel.kit.com
- Verifier said
- high word overlap (word overlap 0.78)
- Why the label
- Evidence: over 4,200 hours per day of rerun time.
- Case origin
- seed-constructed
con-11CircleCI has never raised outside funding.contradicted expected · got unsupported
- Pitch
- Product: Flakeguard is a GitHub App and CI plugin for GitHub Actions, CircleCI, and Buildkite.
- Evidence
“CircleCI Reaches $1.7 Billion Valuation with $100 Million in New Financing” www.businesswire.com
- Verifier said
- evidence is off-topic (word overlap 0.20)
- Why the label
- Evidence reports a $100M financing round.
- Case origin
- seed-constructed
con-12Lithium-ion batteries are cheaper than thermal storage for durations of 8 hours or more.contradicted expected · got supported
- Pitch
- Technology: Sand is abundant, non-toxic, and costs under $40 per ton. Our storage medium costs under $15 per kWh of capacity, a fraction of lithium-ion batteries.
- Evidence
“Thermal and compressed air storage cheaper than lithium-ion batteries for 8-plus hour durations: BNEF” www.utilitydive.com
- Verifier said
- numbers agree (word overlap 1.00)
- Why the label
- Reverses the comparison in the evidence for the same duration range. Word overlap is almost total, so this tests direction, not vocabulary.
- Case origin
- seed-constructed
uns-05The average US household throws away about 30% of the food it buys, roughly $1,500 a year.unsupported expected · got contradicted
- Pitch
- Problem: The average US household throws away about 30% of the food it buys, roughly $1,500 a year.
- Evidence
“The average American spends more than $762 per year on food that ends up wasted.” copiaresources.com
- Verifier said
- numbers disagree (word overlap 0.38)
- Why the label
- Evidence is per person, the claim is per household, and the evidence has no share-of-food figure. Different unit, so not a contradiction. The demo run marked this strong.
- Case origin
- seed-real
The dataset comes from the three demo runs: every evidence quote is verbatim from a source the Scout retrieved. Some claims are copied from the demo pitches; others were written to test one verdict boundary against a real quote. All labels were assigned by hand, by the same person who wrote the heuristic, so treat the heuristic score as a floor rather than a fair competitor.
Data generated Nov 15, 2026, 22:00 UTC by evals/run_evals.py.