A session report, on made-up markers.
Eight made-up markers, 360 made-up scripts from an accountancy mock paper. Every script was read a second time, the gaps went to a senior marker, and the report says how each marker did. The numbers come from the same code the API runs.
What the second reading found.
Allowed gaps: deferred tax on a machine, 1 of 8; going-concern indicators, 1 of 6; lease liability at the year end, 2 of 10. A flag the senior marker kept means the second reading was the one that was wrong.
- Scripts
- 360, every one read twice
- Flagged
- 61, the two marks further apart than allowed
- Mark changed
- 49, by a senior marker
- Mark kept
- 12
Each marker, against the second reading.
Severity is how far a marker’s marks sit from the second reading on average, spread is how much that varies, and drift is how much it moved from the first half of their pile to the second, each as a share of the question’s marks.
| Marker | Read twice | Severity | Spread | Drift | Flags | Pattern |
|---|---|---|---|---|---|---|
| Marker 01 | 45 of 45 | −2% | 12% | −1% | 4 | in line |
| Marker 02 | 45 of 45 | 0% | 7% | +2% | 0 | in line |
| Marker 03 | 45 of 45 | +1% | 7% | +2% | 0 | in line |
| Marker 04 | 45 of 45 | 0% | 8% | +2% | 1 | in line |
| Marker 05 | 45 of 45 | −12% | 17% | +4% | 12 | harsh |
| Marker 06 | 45 of 45 | +15% | 17% | +4% | 11 | lenient |
| Marker 07 | 45 of 45 | −4% | 25% | −10% | 20 | inconsistent |
| Marker 08 | 45 of 45 | +10% | 19% | +19% | 13 | drifting |
- Marker 01
- Marks well. The report found: in line.
- Marker 02
- Marks well. The report found: in line.
- Marker 03
- Marks well. The report found: in line.
- Marker 04
- Marks well. The report found: in line.
- Marker 05
- Harsh: often a mark or two under. The report found: harsh.
- Marker 06
- Lenient: often a mark or two over. The report found: lenient.
- Marker 07
- Inconsistent: sometimes well over, sometimes well under. The report found: inconsistent.
- Marker 08
- Drifts: marks well, then grows lenient late in the session. The report found: drifting.
The largest gaps.
Every one of these went to a senior marker, who kept the mark or changed it.
- marker-05/014 marked 1, read again as 5 mark changed
- marker-06/032 marked 4, read again as 0 mark changed
- marker-07/013 marked 4, read again as 0 mark changed
- marker-07/021 marked 7, read again as 3 mark changed
- marker-08/041 marked 5, read again as 1 mark changed
- marker-01/004 marked 2, read again as 5 mark kept
- marker-01/027 marked 4, read again as 1 mark kept
- marker-01/039 marked 3, read again as 6 mark kept
What a second reading would have caught.
The same made-up paper as if its results were already out, second-marked on two scripts in five, as many organisations do. Some marks changed later, at moderation, a review or an appeal. Here is which of those a flag would have caught before results.
- Read twice
- 144 of 360 scripts
- Would have been flagged
- 21
- Changed later
- 28 of the 58 scripts whose later mark is known
- Flagged first
- 6
- Nobody read it twice
- 20
- Appeals
- 12 changed a mark; a flag had caught 5 of them first
- marker-07/015 marked 2, an appeal made it 5 never read twice
- marker-07/031 marked 0, an appeal made it 3 flagged before results
- marker-07/034 marked 1, an appeal made it 4 never read twice
- marker-07/037 marked 1, an appeal made it 4 never read twice
- marker-07/043 marked 1, an appeal made it 4 flagged before results
- marker-05/001 marked 2, an appeal made it 4 flagged before results
- marker-05/007 marked 2, an appeal made it 4 never read twice
- marker-05/010 marked 1, moderation made it 3 never read twice
A script with no later outcome is counted as not known, never as unchanged. Most of the changes nothing caught were on scripts the sample never reached. That’s the case for reading every script.
Reading the report.
- Called out
- A marker is called out as harsh or lenient at 10% severity or more either way, inconsistent at 20% spread or more, and drifting at 12% drift or more either way. These are starting points an organisation can change, not a standard.
- Too few scripts
- A marker with fewer than 20 scripts read twice gets counts, never a pattern.
- The second reading
- Here, a second marker’s. It can be wrong, which is why a person decides every flag and the report counts the flags they kept.
- Made-up data
- Every script, marker and mark on this page is invented, so the report can be checked against the habits written into it. It says nothing about any real marker or organisation.
This report, on a session you’ve already marked.
Send a past session with the marks that changed later. We’ll show you what a second reading would have flagged.