Synthetic demo data
What won on the day, and what lasted.
DemoTutor is a pretend tutor with 5,000 made-up learners, ten skills and three teaching styles, handed out at random. Every number below comes from the engine the API runs. We planted the effects in the data on purpose, so you can check the estimates against the truth.
One learner’s receipt.
On the standard plan: one check right after the lesson, then retention and transfer checks four to ten days later, done without help.
Receipt for dt_int_88
Status evaluated every check is in, or its time ran out
How far MD4 later, on something new
How sure CS1 it came after, that’s all
- Before the lesson wrong
- Straight after wrong
- A week on, new item right
- 2 days before Before the lesson Starting point
- 13 minutes after Right after Counted
- 7 days after Later, same kind of question Counted
- 7 days after Later, new question Counted
- Nobody knows how reliable the test was
- It came after the lesson, and that’s all
- One person’s change can’t be pinned on the lesson
- Worked example52% the comparison
- Socratic question59% +6.5 points (+2.1 to +11.1)
- Contrast case65% +12.9 points (+8.4 to +17.3)
Wrong before the lesson, still wrong right after it, then right a week later on a question they had never seen. Judged on the check right after the lesson, this would have looked like a failure. It is still one learner, though. The receipt says the result came after the lesson, and that nothing here rules out some other cause.
-
Nobody knows how reliable the test was
Nobody has recorded how reliable the test behind this result is.
-
It came after the lesson, and that’s all
The result came after the lesson. Nothing here rules out another cause.
-
One person’s change can’t be pinned on the lesson
This learner was in a fair test. The test compares groups; it never explains one learner’s change.
Across every learner.
Who got counted is part of the answer. A number worked out on the learners whose checks came back says something different from one worked out on everyone.
- checked afterwards
- 87% of lessons whose checking time is over (MD2 or deeper)
- checked a day or more later
- 53% of lessons whose checking time is over (MD3 or MD4)
| Depth | Meaning | Receipts |
|---|---|---|
| MD0 | Nothing usable came after the lesson. | 650 |
| MD1 | Only indirect signs came after it: activity, behaviour, or a model’s guess or score. | 0 |
| MD2 | The learner was checked on the skill soon after the lesson. | 1,707 |
| MD3 | The learner was checked on the skill a day or more after the lesson. | 1,367 |
| MD4 | The learner was checked later on a question they hadn’t seen, a new kind of task, or an outside test. | 1,276 |
The head-to-head.
Each learner’s teaching style was picked at random before any lesson, so the gap between styles a week later is a fair test (CS3 on the scale). It tells you about the styles, never about any one learner.
| Strategy | Straight after | A week later | Difference, points | Built in |
|---|---|---|---|---|
| Worked example | 79% | 52% | control | control |
| Socratic question | 70% | 59% | +6.5 (+2.1 to +11.1) | +6.0 |
| Contrast case | 71% | 65% | +12.9 (+8.4 to +17.3) | +12.0 |
Learners with a week-later result for each style: worked example 915, socratic question 885, contrast case 843. The split between styles was checked against the plan: it held (p = 0.50).
The same comparison, written up the way a team would pass it on: the evidence report, in Markdown, exactly as the API returns it. It is a sandbox report on invented data, and it says so.
What this example can’t tell you.
It shows the engine finding effects we planted. It says nothing about your learners.
-
Real teaching
DemoTutor’s learners and their results are made up.
-
Your data
Real logs miss checks, record help unevenly and reuse questions. Running the audit on your own export shows how far yours reach.
-
Your product
The engine finds the effects it was given, under the conditions it was given them.