# Evidence report: Three teaching strategies (synthetic demo data)

**Randomized experiment.** Experiment `exp_demotutor_strategies`. Report `experiment-report-v2`, built from the result computed 24 September 2026, 12:00 UTC.

> **Sandbox project.** This report is computed from test data. It says nothing about learning.

## Summary

- **Hypothesis, as declared before the start:** Contrast cases and Socratic questions produce better 7-day retention than worked examples, even if worked examples look better straight after the lesson.
- **Status:** Ready. Every taught learner’s checks are in or their time has passed, and each group has enough results.
- **Design:** causal strength CS3. A fair test: chance decided who got which version, and the analysis was fixed before anyone saw the result.
- **Primary outcome:** Later, same kind of question (MD3).
  - `socratic_question` minus `worked_example`: +6.5 points (95% interval +2.1 to +11.1); the interval lies above zero; the interval reaches both inside and outside the minimum meaningful difference declared before the start, 5.0 points either way.
  - `contrast_case` minus `worked_example`: +12.9 points (95% interval +8.4 to +17.3); the interval lies above zero; the whole interval is at or above the minimum meaningful difference declared before the start, +5.0 points.

## Method

Learners were assigned to an arm by a keyed hash of their identifier, in the proportions declared before the start: `worked_example` 33.3% (control), `socratic_question` 33.3%, `contrast_case` 33.3%. A learner keeps the arm first assigned. The randomization unit is the learner.

Arms are compared on the Outcome Receipts of learners who received the arm they were assigned. For each outcome, a learner contributes the mean normalized score of their receipts, between 0 and 1, and each arm's mean is compared with the control's. Differences are given in points of that scale, with 95% bootstrap percentile intervals that resample learners (2,000 resamples, seeded, so the same records give the same interval).

A difference is estimated only once every arm has at least 20 learners with the outcome and every exposed learner's window has closed, so the result is read once, at a fixed horizon. A chi-square test compares the assignment counts with the declared allocation and is flagged below p = 0.001. Receipts on every capability the exposed interventions taught are compared.

## Period

- Started: 1 September 2026, 08:01 UTC
- Stopped: still running
- Latest receipt read: 24 September 2026, 12:00 UTC
- Result computed: 24 September 2026, 12:00 UTC

## Arms

| Arm | Allocation | Policy release |
| --- | --- | --- |
| `worked_example` (control) | 33.3% | `demotutor-worked` |
| `socratic_question` | 33.3% | `demotutor-socratic` |
| `contrast_case` | 33.3% | `demotutor-contrast` |

## Who was compared

Counts for the primary outcome, Later, same kind of question.

| Arm | Assigned | Received this arm | Received another arm | With the outcome | Windows closed |
| --- | ---: | ---: | ---: | ---: | ---: |
| `worked_example` | 1704 | 1704 | 0 | 915 | 1704 |
| `socratic_question` | 1659 | 1659 | 0 | 885 | 1659 |
| `contrast_case` | 1637 | 1637 | 0 | 843 | 1637 |

## What was measured

**Later, same kind of question** (primary; MD3: The learner was checked on the skill a day or more after the lesson)

- 4 to 10 days after delivery, unassisted only, at least 1 observation: 5000 contracts.

**Right after** (secondary; MD2: The learner was checked on the skill soon after the lesson)

- 1 to 30 minutes after delivery, unassisted, or of unknown assistance and flagged, at least 1 observation: 5000 contracts.

Each plan was fixed when its contract opened. Only direct evidence that meets a plan counts toward its outcome.

## Results

### Later, same kind of question (primary, MD3)

| Arm | Learners with the outcome | Windows closed | Mean |
| --- | ---: | ---: | ---: |
| `worked_example` (control) | 915 | 1704 | 52.1% |
| `socratic_question` | 885 | 1659 | 58.6% |
| `contrast_case` | 843 | 1637 | 65.0% |

| Difference | Estimate | 95% interval | Reading |
| --- | --- | --- | --- |
| `socratic_question` minus `worked_example` | +6.5 points | +2.1 to +11.1 | Above zero; across the declared minimum |
| `contrast_case` minus `worked_example` | +12.9 points | +8.4 to +17.3 | Above zero; at or above the declared minimum |

`socratic_question` minus `worked_example`: +6.5 points (95% interval +2.1 to +11.1); the interval lies above zero; the interval reaches both inside and outside the minimum meaningful difference declared before the start, 5.0 points either way.

`contrast_case` minus `worked_example`: +12.9 points (95% interval +8.4 to +17.3); the interval lies above zero; the whole interval is at or above the minimum meaningful difference declared before the start, +5.0 points.

### Right after (secondary, MD2)

| Arm | Learners with the outcome | Windows closed | Mean |
| --- | ---: | ---: | ---: |
| `worked_example` (control) | 1226 | 1704 | 79.2% |
| `socratic_question` | 1178 | 1659 | 70.1% |
| `contrast_case` | 1196 | 1637 | 71.0% |

| Difference | Estimate | 95% interval | Reading |
| --- | --- | --- | --- |
| `socratic_question` minus `worked_example` | −9.1 points | −12.7 to −5.6 | Below zero |
| `contrast_case` minus `worked_example` | −8.2 points | −11.7 to −4.7 | Below zero |

### Straight after and later

- `socratic_question` against `worked_example`. Right after: the interval lies below zero. Later, same kind of question: the interval lies above zero.
- `contrast_case` against `worked_example`. Right after: the interval lies below zero. Later, same kind of question: the interval lies above zero.

## Guardrails

No guardrail was declared.

## Checks

- **Sample ratio.** Chi-square 1.38 on 2 degrees of freedom, p = 0.502. Not flagged.
- **Interventions naming this experiment for learners it never assigned:** 0.

Missing primary outcomes (Later, same kind of question), by arm:

| Arm | Received this arm | Without the outcome | Share |
| --- | ---: | ---: | ---: |
| `worked_example` | 1704 | 789 | 46.3% |
| `socratic_question` | 1659 | 774 | 46.7% |
| `contrast_case` | 1637 | 794 | 48.5% |

No warnings.

## Limitations

- This is a sandbox project. Its records are test data, and this report says nothing about learning.
- A comparison of arms describes groups. It does not explain why any one learner did well or badly.
- Only learners who received the arm they were assigned are compared. This is not an intention-to-treat analysis: learners who received nothing, or another arm, are counted and left out.
- A learner with no outcome in a window is missing, never counted as a failure. Missing outcomes are reported by arm.
- The outcome measures are the partner's own checks. Their reliability and validity are not assessed here.

## Versions

- Report: `experiment-report-v2`
- Analyzer: `outfigure.experiment_difference.v1`, version `1.1.0`
- Declaration hash: `2c66032b1ce826d2aabaf2b06d4998e52a6330671323432e54c63c87ac984896`
- Population: `exposed_in_assigned_arm`
- Result: computed 24 September 2026, 12:00 UTC, current