Method · OutFigure

Which evidence counts.

Each lesson comes with a plan: which checks to wait for, when, and whether help is allowed. The linker, linker-v1, gives every answer that comes in a place in that plan and a reason, and lists it on the receipt whether it counted or not.

Anything from up to 30 days before the lesson sets the starting point. Inside a window, direct evidence counts when the learner had no help. If a role only prefers unaided work, evidence with no record of help still counts, and says so. If a role insists on it, that evidence is left out. A transfer role counts only a question the learner had never seen, or a transfer task. Engagement, behavior, model estimates and answers scored by a model only ever count as proxies.

Every reason the linker gives, in its own words
Reason	Meaning
`eligible`	Counted
`eligible_assistance_unknown`	Counted; help not recorded
`before_delivery`	Before the lesson (the starting point)
`outside_window`	Outside every checking time
`assisted`	Left out: done with help
`assistance_unknown`	Left out: help not recorded
`item_seen_before`	Not counted as new: seen before
`item_novelty_unknown`	Not counted as new: not known to be new
`not_external`	Left out: not an outside test
`proxy_evidence`	Only an indirect sign
`model_scored`	Only an indirect sign: scored by a model

How far a result reaches.

It depends only on the checks that counted. Calling a check “later” isn’t enough: it has to happen at least 24 hours after the lesson.

Each measurement depth and what it requires
Depth	What it requires
MD0	Nothing usable came after the lesson.
MD1	Only indirect signs came after it: activity, behaviour, or a model’s guess or score.
MD2	The learner was checked on the skill soon after the lesson.
MD3	The learner was checked on the skill a day or more after the lesson.
MD4	The learner was checked later on a question they hadn’t seen, a new kind of task, or an outside test.

How firmly it can be pinned on the lesson.

This is scored separately. A result can last a long time and still be hard to pin on the lesson, and one learner’s receipt never says more than “it came after the lesson”.

Each causal strength and what it means
Strength	What it means
CS0	Can’t say: this only describes what was recorded.
CS1	It came after the lesson, and that’s all. Nothing rules out another cause.
CS2	A pattern with other causes allowed for, under stated assumptions. OutFigure doesn’t produce this yet.
CS3	A fair test: chance decided who got which version, and the analysis was fixed before anyone saw the result.
CS4	A fair test, repeated on separate groups of learners. Never given automatically.

How a receipt changes as evidence comes in.

The evaluator, outfigure.learning_delta.v1@1.0.0, writes a receipt as evidence arrives and again as each check’s time runs out.

Provisional. Some checks are in, but a later one is still due. The receipt can still change.
Evaluated. Every check is in, or its time has run out.
Insufficient evidence. Every check’s time ran out and none came back usable. Missing doesn’t mean failed.
Every change is a new version. Late evidence gets a new version next to the old one, and says so. Checking for a lesson stops 3 days after its last check was due, and its latest receipt stands.

How experiment groups are compared.

The analyzer, outfigure.experiment_difference.v1@1.1.0, compares learners who got the version they were assigned, using their receipts.

Assignment comes first. The plan is sealed when the test starts, and each learner’s group is worked out from their ID, the same way every time, so nobody can be moved afterwards.
Enough learners, and the clock run out. No difference is estimated until every group has 20 learners with a result and every window has closed.
Always a margin of error. Each difference from the comparison group comes with a 95% margin of error, worked out by reshuffling the data 2,000 times (a bootstrap). The reshuffling starts from the sealed plan, so the same data always gives the same answer.
The split gets checked. A standard test (chi-square) compares the group sizes with the planned split. Below p = 0.001 the split counts as broken, and the result drops to “it came after, that’s all”.
Who saw what, and who is missing. Both are reported next to the comparison, and flagged when they differ between groups, because either can skew it.

Predictions, and the rules behind them.

A prediction is locked in, under the name of the rule that made it, before its outcome exists. Each rule has a card that says what it reads, when it refuses to guess and what it must never be used for. Every rule runs in the background only, so nothing reaches a learner, and none has been tested against real outcomes yet.

Each prediction model, its kind and its status, with a link to its card
Model	Kind	Status
baseline-v1	smoothed_learner_accuracy	shadow, unvalidated. Never promoted, never shown to a learner.
recent-v1	smoothed_recent_accuracy	candidate, shadow, unvalidated. Never promoted, never shown to a learner.

What the evidence shows so far.

Each stage below counts only once its result exists. Nothing is claimed ahead of that.

What each stage would show and where this product stands
Stage	What it shows
Implemented	The software does what this page says, under tests that try the ways it could fail.	reached
Synthetic recovery	On invented data with planted effects, the intervals contain the planted effects. Recomputed at every build and shown on the demo.	reached
Partner data	Receipts on a design partner’s own logs, with coverage reported.	not started
Controlled comparison	A randomized experiment on real learners, analyzed as declared.	not started
Replication	The same comparison holding in an independent cohort.	not started

What the tests check.

Each statement below is checked by automated tests, with what they don’t cover written next to it.

Behaviour the test suite checks, what it does not cover, and where
Behaviour	What it does not cover	Where
Retrying or duplicating an event submission does not change learner-concept counts.	Local Postgres only; no hosted deployment has been tested. Network delivery is at least once; the guarantee applies to stored effects.	tests/integration/ingestion.test.ts (A04, A05, A07)
One project cannot read or modify another project's learner data through the API, the console or the runtime database role.	Does not protect against a stolen runtime database credential. No independent security review has been performed.	tests/isolation/tenant-isolation.test.ts (A10, A11)
Given the same stored events and rule versions, observed-v0 produces the same summary regardless of arrival order.	Determinism is not evidence that the summary is useful or predictive.	packages/engine/src/observed-v0.test.ts; tests/replay/replay-and-recovery.test.ts (A12)
Responses without reported assistance are stored but excluded from observed summaries.	Depends on partners reporting assistance accurately, which is not verified.	packages/engine/src/observed-v0.test.ts (eligibility-v2)
The console's key screens have no serious or critical findings in automated accessibility scans at 1440 px and 360 px.	Automated scans cover part of WCAG 2.2 AA. No manual audit or assistive-technology testing has been done, and no conformance is claimed.	tests/e2e/console.spec.ts; tests/e2e/mobile.spec.ts
On the synthetic export, the readiness report's counts of learner-concept pairs, statuses, flags and eligible responses match what API ingestion and processing produce from the same events.	Verified on the synthetic export only; real exports may contain structures the fixture lacks. The test compares totals, not each learner-concept state individually.	tests/integration/audit-parity.test.ts
The readiness report makes no network calls, and its output contains no learner identifiers. Validation problems are reported as codes and line numbers, never as submitted values.	The output names item versions with their counts and, when at least 10 learners answered them, their statistics, and it names the input files. Aggregate counts and item identifiers can still identify people in small cohorts.	packages/audit/src/analyze.test.ts; packages/audit/src/items.test.ts; apps/cli/src/cli.test.ts; eslint.config.js rules for packages/audit and apps/cli