← VentureVision

Evals

How often a claim verdict matches a hand-assigned label, given fixed evidence. This scores the verdict step only; it does not score web retrieval or the memo's other sections.

Dataset

venturevision-claims v1

updated 2026-11-15

Cases

36

Labels

supported 12

unsupported 12

contradicted 12

Origin

seed-real 14

seed-constructed 22

0 user-feedback candidates awaiting review

Latest scores

VerifierOverallsupportedunsupportedcontradictedRun · cost

Heuristic baseline

No model: word overlap, number and direction checks

83.3% (30/36)100% (12/12)91.7% (11/12)58.3% (7/12)Nov 15, 2026, 22:00 UTC
$0

LLM verifier

Not run yet. python evals/run_evals.py --mode llm --publish

CI fails a pull request if heuristic accuracy drops more than 3 points below the stored baseline (83.3% on v1).

Run history

RunVerifierDatasetOverall
Nov 15, 2026, 22:00 UTCHeuristic baselinev1 · 36 cases
83.3%

Heuristic baseline misses (6)

Latest run, Nov 15, 2026, 22:00 UTC. Click a case for the evidence.

con-01Meal-planning apps are a $3B global market growing over 12% a year.contradicted expected · got unsupported
Pitch
Market: Meal-planning apps are a $3B global market growing over 12% a year, alongside a $100B+ US online grocery market.
Evidence

“The Meal Planning App Market was valued at USD 1.47 Billion in 2025 and is expected to reach USD 3.87 Billion by 2035, growing at a CAGR of 10.18%.” www.snsinsider.com

Verifier said
evidence is off-topic (word overlap 0.33)
Why the label
Same market and metric: evidence gives $1.47B (2025) and 10.18% CAGR, incompatible with $3B and over 12%. The demo run labelled this weak_claim.
Case origin
seed-real
con-02Thermalith is the first company to commercialize a sand-based thermal battery.contradicted expected · got unsupported
Pitch
Thermalith is the first company to commercialize a sand-based thermal battery.
Evidence

“Manufacturer of sand battery based thermal energy storage systems.” tracxn.com

Verifier said
related but not confirming (word overlap 0.43)
Why the label
Another company (Polar Night Energy) already manufactures sand-battery thermal storage, so 'first to commercialize' cannot hold. The demo run labelled this unsubstantiated.
Case origin
seed-real
con-05Flaky tests cost teams only a few minutes of rerun time per day.contradicted expected · got supported
Pitch
Engineers at companies with large CI pipelines lose up to 20% of their week to flaky tests, and teams learn to ignore red builds, which is how real regressions ship.
Evidence

“Flaky tests cost teams over 4,200 hours of lost productivity per day when considering just the rerun time.” viral-patel.kit.com

Verifier said
high word overlap (word overlap 0.78)
Why the label
Evidence: over 4,200 hours per day of rerun time.
Case origin
seed-constructed
con-11CircleCI has never raised outside funding.contradicted expected · got unsupported
Pitch
Product: Flakeguard is a GitHub App and CI plugin for GitHub Actions, CircleCI, and Buildkite.
Evidence

“CircleCI Reaches $1.7 Billion Valuation with $100 Million in New Financing” www.businesswire.com

Verifier said
evidence is off-topic (word overlap 0.20)
Why the label
Evidence reports a $100M financing round.
Case origin
seed-constructed
con-12Lithium-ion batteries are cheaper than thermal storage for durations of 8 hours or more.contradicted expected · got supported
Pitch
Technology: Sand is abundant, non-toxic, and costs under $40 per ton. Our storage medium costs under $15 per kWh of capacity, a fraction of lithium-ion batteries.
Evidence

“Thermal and compressed air storage cheaper than lithium-ion batteries for 8-plus hour durations: BNEF” www.utilitydive.com

Verifier said
numbers agree (word overlap 1.00)
Why the label
Reverses the comparison in the evidence for the same duration range. Word overlap is almost total, so this tests direction, not vocabulary.
Case origin
seed-constructed
uns-05The average US household throws away about 30% of the food it buys, roughly $1,500 a year.unsupported expected · got contradicted
Pitch
Problem: The average US household throws away about 30% of the food it buys, roughly $1,500 a year.
Evidence

“The average American spends more than $762 per year on food that ends up wasted.” copiaresources.com

Verifier said
numbers disagree (word overlap 0.38)
Why the label
Evidence is per person, the claim is per household, and the evidence has no share-of-food figure. Different unit, so not a contradiction. The demo run marked this strong.
Case origin
seed-real

The dataset comes from the three demo runs: every evidence quote is verbatim from a source the Scout retrieved. Some claims are copied from the demo pitches; others were written to test one verdict boundary against a real quote. All labels were assigned by hand, by the same person who wrote the heuristic, so treat the heuristic score as a floor rather than a fair competitor.

Data generated Nov 15, 2026, 22:00 UTC by evals/run_evals.py.