# Building a VCE exam analysis tool — the method

**Upload this file into your project.** It is the written method behind the kit
at <https://almanac.jotty.au>. With it in your project's knowledge, Claude has
both halves of the job: the method for building the tool, and your study's
official documents to build it from.

You don't have to read it front to back. It's here so that when you go off the
script — and you will — Claude already knows what you're doing and why, instead
of improvising a plausible alternative.

> **The kit page:** <https://almanac.jotty.au>
> The interactive version of this document. It runs the diagnostic, holds every
> prompt with your study name already filled in, remembers where you got to
> between sittings, and releases a download pack at the end of Phase 2.
>
> **The Tag Checker:** <https://almanac.jotty.au/tagcheck/>
> Where the Phase 2 verification actually happens. Takes your dataset files,
> finds the mechanical errors, and lets you correct the tagging in place.
> Neither page needs a Claude account — which is what makes the checking
> shareable with colleagues who don't have one.

At the end of this you'll have a tool that maps every past exam question against
your study design, shows where the marks actually sit, ranks what's most likely
to come up next, and builds practice papers to a blueprint.

You gather the files and check the coding. Claude does the reading and the
building. About two hours, spread across a few sittings. Most of it is
downloading and verifying.

---

## First: does this kit fit your study?

**One minute, three questions.** This approach works by finding patterns across
a large number of individually-marked questions. Some VCE studies give you that;
some don't. Answer these before you download anything.

**1. Open a recent exam. How many separately-marked questions are there in the whole paper?**
- **8 or more** → yes
- **5 to 7** → partly
- **4 or fewer** → no

**2. Open the matching examiner report. Does it give you a number for each question — an average mark, or a distribution of how many students scored 0, 1, 2 and so on?**
- **Yes, per question** → yes
- **Only for some questions or sections** → partly
- **No — it discusses criteria and sample responses instead** → no

**3. Open the study design. Could you list what's examinable as discrete content points a single question could target?**
- **Yes, it reads as a list of content** → yes
- **A mix of content and skills** → partly
- **No — it's skills applied to texts or works that change year to year** → no

### Reading your answers

**Three yeses — use this kit as written.** Your study is the shape this kit was
built for. That covers most of VCE: the sciences, Psychology, Legal Studies,
Business Management, Accounting, Health and Human Development, Applied
Computing, Economics, Geography, and the design and technology studies.

**A mix — use it, but drop what your data won't support.** Build Phases 1 and 2,
then leave out any view your dataset can't honestly feed. If your reports have no
per-question figures, the difficulty signal in the prediction model has nothing
to run on: drop the model and the blueprint, and keep the coverage matrix,
question explorer, command glossary and exam builder. Those alone are worth the
afternoon. Studies with both short-answer and essay components often land here —
code the short-answer questions properly and describe the essay tasks separately
rather than forcing them into the same shape.

**Two or three noes — stop, and don't use this kit.** You'd get a tool that
looks authoritative and says almost nothing, which is worse than no tool.
English, Literature, EAL and Languages sit here.

### If your study doesn't fit

The problem is arithmetic. Three years of English gives about nine tasks to
analyse, and the skills assessed are the same every year — so a coverage matrix
shows near-total coverage annually and tells you nothing you didn't know. What
actually varies is the text, the topic and the framework, and students don't even
sit a common paper, since schools choose texts from the approved list.

A genuinely useful tool for these studies would measure different things:

- **Topic and prompt archetypes** across years — what kinds of propositions students are asked to argue, and how they're phrased
- **Assessment criteria against report commentary** — which criterion the reports keep returning to, and what the described high-scoring responses did differently
- **Stimulus and framework patterns** — what forms the unseen material takes, and which themes have appeared
- **Text list movement** — what's rotated on and off, and what the reports say about texts your school teaches

That's a real tool, and it's buildable. It just isn't this one, and it needs its
own worked exemplar before a kit like this can hand it to someone. If you teach
one of these studies and want to build the first one, start from your own reports
rather than adapting the schema below — the unit of analysis is the task and the
criterion, not the question.

---

## What you're actually building

Three things, in order. Each depends on the one before it.

| Phase | What it is | Why it has to be first |
|---|---|---|
| **1. The project** | A Claude Project holding your study's official documents | Everything downstream is read out of these files |
| **2. The dataset** | Every past question coded to key knowledge, marks, command terms and state-wide performance | This is the engine. The tool is just a window onto it |
| **3. The tool** | A single self-contained HTML file your faculty can open, share and print from | Only as good as the dataset underneath |

The temptation is to jump to Phase 3. Don't. A beautiful interface over a badly
coded dataset produces confident, wrong answers about where the marks sit — and
nobody will notice, because it looks authoritative.

**The reference tool.** A finished exam analysis tool for a different VCE study,
which Claude reads in Phase 3 as the structural model for yours. This is the
single most reliable thing you can do — a working exemplar beats any amount of
description in a prompt. You don't need it until Step 7, and the kit page hands
it to you in a download pack once Phase 2 is finished.

---

# Phase 1 — The project

## Step 1: Gather the source files

This part can't be automated, because it needs judgement about what's current.

On the VCAA site: **Study Designs → [your study]**, and the **Past examinations**
page.

| File | Why it matters |
|---|---|
| **This method document** | So Claude knows the method, not just the individual prompt in front of it |
| **Study design (current)** | Becomes the key knowledge index — the spine of the whole tool |
| **Support materials / FAQs** | Clarifies grey areas. Often answers exactly what students get wrong |
| **Specifications, conventions, glossaries** | Subject-specific rules a question writer must respect |
| **Sample exam + answers** | The intended shape of the paper before any live exam existed |
| **Past exam papers — every year under the current study design** | The question corpus. Every one gets coded |
| **Insert / resource / stimulus books** | Essential if your exam supplies stimulus. Without them the papers don't parse |
| **Examiner reports — every matching year** | Where the performance data comes from. Skip one and that paper's questions have no `avg` |

**Current study design only.** Older papers were written against a syllabus that
no longer applies, and mixing them in corrupts the coverage analysis — you'll get
key knowledge points that look well-assessed but were assessed under different
wording.

If you've only got one or two compliant papers, that's still workable. It means
your tool leans harder on the sample exam and reports honestly that the evidence
base is thin. Say so rather than padding it out with superseded papers.

**Two file-prep gotchas:**

- **Scanned PDFs won't work.** Open each one and try to select text. If you
  can't, run it through OCR (Acrobat, or free tools like OCRmyPDF) and upload
  that version. This is the most common reason these builds fail, and it fails
  silently — Claude just doesn't see those questions.
- **Name files so the year and type are obvious**: `2025-Biology-exam.pdf`,
  `2025-Biology-report.pdf`. Claude cites by filename, so this makes its coding
  checkable.

### Before you gather thirty files, test one

Don't take the selectable-text test on trust. Start any Claude chat, attach a
single exam paper to the message, and run this:

```
I've attached one VCE [STUDY NAME] exam paper. Don't analyse it yet.

Tell me only this:
1. Can you read the text of this PDF, or is it a scanned image?
2. Quote the exact wording of the last separately-marked question in the paper,
   and its mark allocation.
3. How many separately-marked questions are in the whole paper, and what do the
   section totals add to?

If you can't read any part of it, say which pages.
```

If it can't quote the last question back to you accurately, the file is unusable
— and so is every other file that looks like it. Fix them all before Step 2. A
scan found here costs five minutes; found in Phase 2 it costs the afternoon.

Any chat will do for this. You don't need a project yet, and you shouldn't code
anything in this chat — it's a readability test, nothing more.

## Step 2: Create the project and upload

**Projects → Create project.** Name it for your study. Upload everything from
Step 1, including this document.

You don't need the reference tool yet — it arrives in the handover pack at the
end of Phase 2, and it's only read in Step 7.

**One subject per project.** Don't share a project with a colleague teaching a
different study, and don't add a second study's papers to your own. Projects
don't partition: if Biology papers and Legal Studies papers sit in the same
project, Claude reads both at once with no way to tell which study it's working
on — and it will happily code a question from the wrong subject.

## Step 3: Paste in the project instructions

This is the block that holds the accuracy line for every message that follows.
The kit page has it with your study name already filled in.

```
ROLE
You are an experienced VCE [STUDY NAME] teacher, exam assessor and data analyst,
working with a teacher who is building an exam analysis tool for their faculty.
The method you are both following is in this project as
vce-exam-analysis-method.md. Follow it. Where a request and the method disagree,
say so rather than quietly picking one.

YOUR SOURCES
Everything you produce must be grounded in the files in this project:
- The current study design is the authority on what is examinable and on
  correct terminology. Never introduce content outside it.
- The examiner reports are the authority on state-wide performance and on what
  loses marks. Draw on them heavily.
- Past exam papers are the corpus. Quote question stems accurately.
Cite the file and year for every specific claim. If the files don't tell you
something, write UNKNOWN rather than guessing.

ACCURACY RULES — these matter more than fluency
- Never invent a statistic, an average mark, or a question that doesn't exist.
- Never estimate a figure that could be read from a file. Read it.
- When you tag a question against key knowledge, you must be able to point to
  the words in the question stem that justify the tag.
- Flag your own uncertainty inline rather than presenting a guess cleanly.

WHAT NOT TO DO
- Don't reproduce an entire past exam paper on request.
- Don't state that a future exam will contain something. Rotation patterns are
  hypotheses about likelihood, never forecasts, and the whole study design
  remains examinable every year.

HOLD ME TO THE VERIFICATION
Phase 2 is where this either becomes analysis or decoration, and it is the part
most likely to be rushed — checking tags is slower and duller than watching a
tool appear. Every error caught there is an error that would otherwise have
shipped invisibly.
- Tell me to save the key knowledge index and each coded paper to its own .txt
  file as I go, and to run the lot through the Tag Checker at
  https://almanac.jotty.au/tagcheck/ before reading tags by eye.
- If I say the coding is done, don't congratulate me and move on. Ask which of
  the four checks I ran, and ask me to name one tag I disagreed with and
  changed. If I can't name one, I haven't read the justifications — say so, and
  send me back.
- Don't reassure me that a thin dataset will be fine. If I have one compliant
  paper, say what that limits and what the tool must state in its own overview.

RETURNED TAG CHECKS
A file headed "TAG CHECK RETURNED BY A TEACHER" is a verification pass done
outside this project. Follow the instructions at the top of it. In particular:
check each change against the source paper before applying it, and query rather
than silently apply anything that contradicts the paper. A teacher's correction
can be wrong too.

THIS IS A TEACHER'S TOOL
There is no student view of it and there won't be. Given a ranked likelihood
list, students revise the top few and drop the rest, which is the opposite of
what the analysis is for. What reaches students is my teaching and revision
advice, filtered through my own judgement. If I ask for a student-facing
version, say plainly why not.
```

The rule about future exams matters. Your own examiner reports warn students
against relying on past papers — the tool has to hold that line, or it teaches
exactly the behaviour VCAA is cautioning against.

---

# Phase 2 — The dataset

This is the real work, and it's where teacher expertise earns its keep. Three
passes, each verified before the next.

## Two terms this kit never lets drift

**The state cohort** is the VCAA examiner report and every candidate who sat the
paper. **The school cohort** is one school's own students.

Those two phrases are the standard terms — in this document, in every prompt, and
in the tool you build. The word *cohort* never appears on its own, because on its
own it means either.

They are never the same figure, and they never share a field. `avg` and `dist`
are the state cohort, always. School cohort figures live in a separate file with
its own object name, produced on a separate page, and loaded by the built tool at
run time.

A number in the wrong field is the one error nothing downstream can detect, and
an ambiguous word is how it gets there.

## Step 4: Build the key knowledge index

```
Read the study design and produce a complete index of examinable key knowledge
for Units 3 and 4, as a JavaScript object.

Format each entry as:
  "CODE": { o:"U3 AoS1", label:"short plain-language description" }

Use a code scheme of [Unit][Outcome]-[number], e.g. "U3O1-1". Add separate
codes for any cross-cutting terminology the study design defines separately
and that questions could target directly.

Rules:
- Cover every key knowledge point in Units 3 and 4. Completeness matters more
  than brevity — this index is the spine of everything else.
- Keep labels under about 15 words, in language a student would recognise.
- Cite the study design page number for each entry.
```

**Check before moving on:** does the count look right against your study design?
Is anything examinable missing? A gap here becomes an invisible blind spot —
content that never shows as under-assessed because the tool doesn't know it
exists.

Save the index to a file — a plain `.txt` is fine. You'll load it into the Tag
Checker at Step 5, which is what turns a code like `U3O2-4` back into words you
can check a question against.

## Step 5: Code every question

Run this **one paper per answer**. Several papers in one answer produces
shallower tagging, and unlike a truncated file it costs you that invisibly: what
comes back looks complete, the `why` justifications are just thinner and the `kk`
lists lazier. Nothing downstream can detect it, because there is no correct
version to compare against.

You can still queue a run of papers so you're not re-pasting the prompt six
times. The kit page builds one for you: tick the papers, and it writes a prompt
that lists them in order with each one's paper code, codes the first, and then
waits for you to say **next**. Say it only once you've saved the paper in front
of you — the queue saves the re-pasting, not the reading.

Ask for a count line at the end of each paper either way:

```
// <paper code> — N questions, section A x marks, section B y marks, z total
```

It takes five seconds to check against the paper before you save the file, and
it's the cheapest way to notice a reply that ran out of room. A `//` comment is
valid JavaScript, so the Tag Checker reads the file as normal.

On the kit page there's a picker above this prompt — choose the year, tick **NHT
exam** if it's a Northern Hemisphere paper, and the prompt rewrites itself. If
you're working from this document instead, change **both** the year in the
sentence and the `paper` code in the example, together. The code convention is
`E25` for a 2025 exam, `NHT25` for the 2025 NHT paper, `SAMPLE` for the sample
exam.

Getting that code wrong is the quiet one. Claude copies whatever the example
shows, so every question in the file ends up stamped with the wrong paper — the
coverage matrix then attributes one year's questions to another, and no check
anywhere can tell, because a wrong-but-valid paper code looks exactly like a
right one.

```
Code every question in [the 2025 exam] as JavaScript objects in this shape:

{ paper:"E25", sec:"A", q:"1", marks:4, cmd:["Identify","Discuss"],
  kk:["CODE1","CODE2"], avg:2.3, dist:[19,9,17,35,20],
  stem:"the question stem, quoted accurately",
  why:{ "CODE1":"the words in the stem that justify this tag",
        "CODE2":"the words in the stem that justify this tag" },
  examiner:"what the report says about this question specifically",
  choice:"what the student gets to choose, if anything" }

- cmd: every command term in the question, in the order they appear.
- kk: the key knowledge codes from the index that the question actually
  assesses. Usually one to three. Only tag what the question truly requires,
  not everything it touches — over-tagging flattens the coverage analysis.
- avg: the average mark from the examiner report for that question. Read it
  from the mark distribution table. If the report gives a distribution rather
  than an average, calculate it and show your working. Omit avg entirely if
  the report doesn't cover this paper.
- dist: the mark distribution from that same table, as an array running from 0
  marks up to full marks — so a 4-mark question has 5 numbers. Use the report's
  own figures, usually percentages of the state cohort. Omit dist if the report gives
  only an average. You are already reading this table for avg, so capturing it
  costs nothing and it is what makes avg checkable later.
- why: one entry per kk code, quoting the words in the stem that justify that
  tag. Quote the stem, don't paraphrase it. Every code in kk needs an entry.
- examiner: what the report says about THIS question — what students got wrong,
  what a full-mark answer did. One or two sentences, close to the report's own
  words. You are already on that page reading avg and dist. Omit it where the
  report says nothing about the question; don't pad it out with generic advice.

Use exactly these field names. Don't add fields I haven't asked for.
```

### Which model, and how to tell whether it's coping

**Use the most capable model your plan offers, with extended thinking on if
it's offered.** This step asks for the hardest thing in the kit — reading a mark
distribution table out of a report, matching it to the right question, quoting a
stem exactly, then defending a tag against the study design. Step 7 wants the
same, though that failure at least announces itself: a weak build produces a tool
that visibly doesn't work, where weak coding produces a file that looks fine.

The model is a per-conversation choice rather than something the project
remembers, so it's worth a glance each time you start a paper rather than once at
the beginning.

No model name appears here on purpose. Names and tiers change, this document
sits in your project and won't update itself, and a stale instruction to use
something that no longer exists is worse than none. What doesn't change is the
check:

**Three things in the first paper that comes back will tell you**, and the Tag
Checker reports all three.

- Do the `why` entries **quote** words from the stem, or only paraphrase it?
- Does `dist` reconcile with `avg`?
- Is nearly every question carrying four or more tags?

If those look poor, change something — the model, a conversation that has gone
long, a paper you tried to rush — rather than only fixing the file in front of
you. Thin coding has causes, and it produces the same confident wrong answers
however it got there.

The `why` field is the one that makes verification possible. Without it the
justifications live in Claude's prose and vanish when you scroll; with it they
travel with the data, and the Tag Checker can put each one beside its tag.

`dist` is what lets the average be checked rather than trusted. A distribution
and an average that disagree is the single most common transcription error in
this whole process, and it is invisible without both.

Save each paper's coded questions to its own `.txt` file as you go.

`examiner` is what makes the question explorer worth reading: the report's comment
sits with the question rather than in a list somewhere else. `phase` — which stage
of a staged Section B a question belongs to — is recognised too but not asked for.
Neither is required, and a dataset without them isn't wrong.

**If you ever lose these files**, they aren't gone: a finished tool carries the
index and the questions inline, and they can be read back out of it. Don't feed
the whole HTML to the Tag Checker, though — it will pick up every other array in
the file as questions and give you a plausible count that is simply wrong.

### Run it through the Tag Checker

**<https://almanac.jotty.au/tagcheck/>** — drop in the index and every question
file. It works out which is which, and turns the dataset into a proof sheet:
codes resolved to their labels, justifications beside the stems, and the
mechanical errors found for you. It flags; it never fixes.

What it catches:

- a `kk` code that isn't in the index — that question's coverage lands nowhere at all, and nothing else would ever show you
- an average higher than the question's own mark allocation — impossible, so the distribution table was misread
- section totals that don't match what the paper prints — you type in the printed total
- four or more tags on one question, the over-tagging signal
- questions with nothing tagged, duplicates, truncated stems, field names that have drifted from the schema
- index entries no question touches — a finding worth confirming, and sometimes a mistyped code in disguise
- a justification missing for a tag, **or left behind for a tag the question no longer carries** — the second reads as evidence for a claim nobody is making
- a `dist` with the wrong number of buckets, or one that doesn't total about 100 as percentages
- a `dist` that implies a different average from the one recorded, beyond what rounding explains

You can also **correct the tagging in place**: codes are picked from your own
index, so a code that doesn't exist can't be created, and every tag change needs
a reason — which then becomes the stored justification. Marks, averages, command
terms and stems are editable too, and there's a tick per question so you can stop
at 23 and come back.

It marks which questions **can't be judged without the exam paper** — anything
referring to a diagram, a data set, a source, or a lead-in stem that lives above
a sub-part — and filters to just those, so you do them in one pass with the paper
open. A question it doesn't mark is *checkable from what's here*, which is not
the same as verified: every stem was transcribed by someone, and only the paper
confirms it was transcribed faithfully.

When you're done it writes a corrections file: what you changed and why, what's
still outstanding, and the corrected dataset. Paste that into your project and
Claude checks each change against the source paper before applying it.

### Share the checking out

**Nobody needs a Claude account to use the Tag Checker.** That makes this step
delegable, which matters more than it sounds. Send a colleague the dataset files,
the exam papers and the link; they send back a corrections file. Two people
checking a paper each is closer to how you'd moderate anything else — and it's
better than the person who did the coding being the only one who ever reads it.

### Then check every tag yourself

The checker clears the arithmetic. It cannot tell you whether a tag is right, and
that's the part that matters:

- **Do the marks total correctly?** Section totals must match the paper. If not, a question was misread or missed.
- **Are the `avg` figures real?** Spot-check three against the report. An invented average silently corrupts the difficulty signal in the prediction model.
- **Is the tagging defensible?** Read the justifications. This is where your subject knowledge does work no model can do — you know that a question *mentioning* a concept isn't the same as *assessing* it.
- **Is anything over-tagged?** Four or five codes on one question usually means Claude tagged topic mentions rather than assessment focus.

A paper isn't done when it's coded. It's done when it's verified — the kit tracks
those two states separately for a reason.

## Step 6: Build the command term glossary and the report findings

```
1. From the coded questions, produce a command term glossary:
   Term: { marks:[min,max], def:"generic definition",
           note:"what the examiner reports specifically say about how this
                 term is handled in THIS study, with the year cited" }

2. From the examiner reports, list every weakness they name across all years,
   with the year and the exact question it relates to. Mark which ones recur
   in more than one report — those are the persistent ones.

3. List the terminology confusions the reports name directly. Search for
   "confused", "incorrectly", "misunderstood", "uncertainty around".
```

The `note` field is what makes the glossary worth having. A generic definition of
*Explain* is available anywhere; what a full-mark *Explain* looks like **in your
study, according to your assessors** is not. If the note reads as a restated
generic definition, push back and ask for the report wording and the year.

## The handover pack

Once Steps 1 to 6 are ticked on the kit page, a download appears: a `.zip`
holding the reference tool, your project instructions with your study name filled
in, every prompt in order, and a note explaining each.

It's deliberately locked until then. The reference tool makes Phase 3 look
finished fast, which is exactly why it's the wrong thing to have in hand while
the dataset is still unverified — a convincing interface over unchecked tagging
is the failure this whole kit exists to prevent.

**One thing not to do with it.** Don't ask Claude to copy its data. It holds
another study's questions, key knowledge codes and state figures, and none of
them belong in your subject's dataset. Step 7's prompt is worded to take the
structure and derive the visual treatment from your own study. Keep it that way.

---

## Optional: your own school's results

Everything above works without this, and most of the value is there already. But
if your school can give you the VASS **External Assessments — Questions/Criteria**
page for a paper you have coded, the tool can show how your students did on the
same questions, against the state.

### Two places to type it, one file either way

**In the tool you build.** Step 7 includes a **School figures** view. Pick the
paper and it lays out a row per coded question, with the section, question number
and marks already filled in from your own dataset. That is worth more than the
convenience: a row for a question your tool hasn't coded cannot be created, so a
file for the wrong subject cannot be built in the first place. Prevention at the
point of entry, rather than a check afterwards.

**Or at <https://almanac.jotty.au/school/>**, a page of its own, which needs
neither a Claude account nor a built tool. Use it before Phase 3 exists, when a
colleague or an admin is doing the typing, or to repair a file. Both write the
same file and each reads the other's.

Four numbers a row, in the order the VASS page prints them: school mean, state
mean, school %, state %.

Do it that way rather than uploading the report or a photo of it. The page carries
a Privacy and Data Protection Act notice, your school's name and a teacher's name,
and none of that needs to go anywhere. What comes out is a small file of aggregate
figures in which no student appears.

It is also the safer route. The report prints each mean **and** its percentage,
and the percentage is just the mean over `Max` — so every row checks itself, and a
row only turns green when the two agree. A digit misread from a scan stays hidden;
a mistyped one does not.

### Why the tool leaves the state column blank

It could fill it in. It holds the examiner report average for every one of those
questions, and the state column is that same figure on the report's scale. It
stays blank on purpose, and this is the one design decision in the feature worth
understanding.

The state column is a **second transcription, from a different document**.
Comparing it with `avg` is the only check in this whole method that reads two
sources against each other — everything else checks the dataset against itself.
Pre-fill that column and the check compares a number with itself: still green,
now meaningless.

So the tool holds its own figure back until yours is typed, then says whether the
two agree. It costs two more numbers a row, and it is the difference between a
check and a decoration.

Guard against the softer version of the same mistake too. Showing you its figure
and asking you to confirm looks like checking and isn't: read 5.5 off the page
with 5.6 on the screen and most people tick.

### What the report is, and the one trap in it

`Max` is normally **twice** the question's marks, because external assessments are
marked at least twice, and the means are on that doubled scale. A 5.5 against a
4-mark question is not a mark and a half over — it is 2.75. Store what the report
prints and derive the marks-scale figure when you need it.

The `Statistical Difference` column is VCAA's own judgement — *Higher*,
*Significantly Higher* and so on. Quote it verbatim and never recompute it.

### The export is the save file

There is one file, and it does both jobs. Rows that fully reconcile go in `rows`.
Anything half-typed goes in a separate `draft` array that no view ever reads, so a
part-finished row can never reach the cohort table and be counted as data.

That means you can stop. Export part-way, load the file back, and entry carries on
where it left off. It also means you type a page once: keep the file and load it
next year rather than typing it again.

Nothing is stored in the browser at either end. What you type lives in memory
until the page is closed, and the export is the only way out — which is what keeps
the tool safe to email around, since it never holds anyone's results.

Both ends read the file back for correction, so a file a colleague sends you
reopens for checking rather than retyping.

### The schema

```
const SCHOOL_E25 = {
  meta: { study:"VCE Biology Units 3 & 4", paper:"E25", label:"2025 exam",
          scope:"school", classes:"several", n:22,
          source:"VASS External Assessments — Questions/Criteria",
          entered:"2026-02-14" },
  rows: [
    { sec:"A", q:"1", marks:4, max:8,
      school:5.5, schoolPct:69, state:4.5, statePct:56, diff:"Higher" }
  ]
};
```

Keep it as its own object. School figures must never end up in `avg`, which means
the examiner report and nothing else.

### Why the study name is in there

Every VCE study numbers its papers the same way. `E25` means the 2025 exam in
Biology, in Legal Studies, in Visual Communication Design — the code identifies a
year, not a subject. So a tool that accepts a school file on the strength of its
paper code will happily load another study's figures, join whatever rows happen to
share a section letter and a question number, and draw a cohort performance view
out of the coincidence. The numbers are real; they are simply about a different
exam. Nothing on the face of the view says so.

That is the same silent-failure shape as the rest of this kit: well formed, right
field names, plausible magnitudes, and no correct version to compare against. So
the tool is told to check before it accepts, and to refuse rather than partly
load:

1. **The paper is one this tool has coded.** Otherwise there is nothing to join to.
2. **The study name matches**, compared loosely enough that punctuation and
   capitals don't matter.
3. **Every row joins a coded question** on section and question number. Rows are a
   subset of the questions — a file may cover part of a paper, but it may never
   contain a question the paper doesn't.
4. **The marks agree** on every row. The figures are keyed to a mark allocation
   and `max` is derived from it, so a disagreement means the two documents
   describe different papers.
5. **The state column lands near `avg`** — within 0.5 on the marks scale, where
   the question has an `avg` at all. This is the identity test rather than the
   transcription test, and another study's file fails it almost everywhere.

A file that fails any of these loads nothing at all, not even the rows that
passed, and the tool says which check failed. The repair happens back at
almanac.jotty.au/school/, which reads its own output back for correction.

Files written before the study stamp existed have no name to check. Those pass on
structure alone, and the tool should say so rather than reporting a name it never
verified.

### What it buys you, and what it doesn't

**A second source for `avg`.** The state column is the same figure as the examiner
report average, differently scaled. Converted back it should agree to about 0.1.
Until now `avg` could only be checked against `dist`, and only where a
distribution exists — this checks it against another document.

**Where your teaching is strong and weak relative to itself.** Not the raw gap: a
school whose students are stronger sits above the state on nearly everything, and reading that as a
result tells you nothing. The finding is where a question departs from your own
average gap.

**Not a prediction input.** How your students did has no bearing on what VCAA
chooses to examine. The school cohort figures feed the cohort performance and revision priority views, and are kept out of the prediction model on purpose.

### Two things to be clear about before you circulate anything

**The School column is every student taking the subject at your school**, not your
class. Where a school runs several classes it is a faculty signal and not a
verdict on any one teacher. Where it runs one, school and class are the same
people — decide deliberately who that comparison is for.

**The tool should read the school file at runtime, not carry it.** Step 7's prompt
asks for a control that loads a `SCHOOL_` file from disk into memory for the
session, and for an entry view that writes one. Neither stores anything: one tool
file circulates with no school results in it, and each teacher points it at their
own or types their own in.

---

## One failure, five disguises

Five things in this kit can go wrong. They look like five different problems and
they are one.

**A stale paper code.** You run Step 5 for the 2023 paper but the example still
says `paper:"E25"`. Claude copies it, and every question in the file is stamped
with the wrong year. The coverage matrix then attributes one year's questions to
another.

**Several papers in one answer.** The reply runs out of room and the tagging
thins — `why` entries get terser, `kk` lists lazier. Nothing is missing. It is
just weaker than it should be.

**A weaker model, or a conversation that has gone long.** Exactly the same
thinning, from a different cause.

**A school figure landing in `avg`.** `avg` means the examiner report and nothing
else. A school figure there is a real number from the wrong document.

**A digit misread off a scanned marks table.** 5.5 becomes 5.6. An invented
statistic, perfectly plausible.

### Why they are one failure

In every case the output is **well formed**. Right shape, right field names,
right types, sensible magnitudes. Nothing about it is detectably wrong from
inside the dataset, because there is no correct version to compare it against.

That is the whole difficulty. A truncated file is loud — it fails to parse, or the
section totals don't reconcile. A missing question is loud. These are silent, and
they stay silent for as long as the tool exists, and everything built on top of
them looks exactly as authoritative as it would if they weren't there.

**You cannot catch this class of error by looking at the result.** There is
nothing to see. You can only catch it two ways: against a second source, or by
preventing it at the point of entry.

That is why the kit is arranged the way it is.

| The failure | What stands in its way |
|---|---|
| Stale paper code | The picker rewrites the sentence and the `paper` code together, so they cannot drift apart |
| Batched papers | The run queue gives one paper per reply; the count line at the end of each |
| Thin coding | Three signals in the first paper back — quoted justifications, `dist` against `avg`, tag counts |
| School figures in `avg` | Two pages, two files, and check 13 against the examiner report |
| Misread digits | Typed entry with the printed percentage as a check digit, and an honest "on trust" where the arithmetic can't see |

Only two of those are checks in the ordinary sense. The rest are arrangements
that make the error hard to commit in the first place, which is the better kind
of defence when the error is invisible after the fact.

### The rule that follows

**A check that could not run must never look like a check that passed.**

With no distributions in a dataset, audit checks 8 and 9 have nothing to run on.
With no school file, 11 to 14 don't either. A green tick in either case is a lie
of exactly the kind this section is about — a well-formed, plausible, entirely
unfounded reassurance. Every one of them must say *not run*, and say why.

---

# Phase 3 — The tool

## Step 7: Build it in stages

Upload `reference-exam-analysis-tool.html` from the handover pack into your
project's knowledge first — the prompt below tells Claude to read it as the
structural model, and without it there this step produces something generic.

Then don't ask for the whole thing at once. Build the shell and two views, check
them, then add views one at a time.

```
Using reference-exam-analysis-tool.html as the structural model,
build the equivalent tool for VCE [STUDY NAME] as a single self-contained
HTML file — no external dependencies, no browser storage.

Use the verified dataset we built: the key knowledge index, the coded
questions, the command glossary and the report findings. Do not add data
that isn't in it, and do not smooth over gaps.

Start with the shell plus two views only:
  - Overview: headline findings, papers included, what's been assessed in
    every live paper and what's never been assessed at all.
  - Question explorer: every coded question, filterable by paper, section,
    command term and key knowledge.

Match the reference tool's approach to layout and interaction, but derive the
visual treatment from [STUDY NAME] rather than copying the reference's palette.
```

Then add the remaining views **one per message**:

- **Curriculum coverage matrix** — every key knowledge point against every paper, showing marks carried and gaps
- **Prediction model** — visible, adjustable weightings, no black box. In the reference these are core status, rotation pressure, mark weight and difficulty signal. State the reasoning for each ranking
- **Paper blueprint** — the predicted shape of next year's paper, section by section
- **Cohort performance** — how the state cohort actually did, and this school's cohort beside it where you have its figures
- **School figures** — where a teacher types their own school's VASS page in, one row per coded question, with the state column typed blind. Optional, and only worth building if you have those figures
- **Revision priorities** — what this implies for teaching sequence, as a planning list you can work through
- **Exam builder** — assemble a practice paper against the blueprint, with checks on section totals and coverage, and a print view
- **Dataset audit** — fourteen numbered checks built in permanently. Build this one regardless of your diagnostic result, so next year's paper gets verified the same way from inside the tool

**Each of these has its own copy-ready prompt on the kit page**, folded into the
ledger the diagnostic produces. A view the diagnostic rules out has no prompt at
all; a view it restricts has that restriction written into its prompt, so you
don't have to remember to add it.

You can tick several views and copy one combined prompt, and the page will let
you — with the cost stated. Each view comes back more generic, a long reply can
be cut off part-way and a truncated view looks finished, and one wrong view means
regenerating the lot. The one that matters most: **you can't test a view you
didn't look at**, which is the whole premise of Step 8.

Build only the views your diagnostic result supports. Drop a view rather than
feeding it weak data — a view running on inference looks identical to one running
on evidence.

### The Dataset audit view, and why it has fourteen checks

Six of them are the Tag Checker's, run permanently from inside the tool. The
other four came out of a real build rather than the design, and they were the
ones that caught bugs:

7. **Justifications that don't line up with the tags** — in both directions. A code with no justification, and a justification left behind for a code the question isn't tagged against any more.
8. **Distribution bucket counts** — a `dist` must hold exactly `marks + 1` numbers, and total about 100 where the figures are percentages.
9. **Average reconciliation** — the average the distribution implies, against the one recorded. Flagged past 0.15 marks, because rounded report figures account for about 0.1 on their own.
10. **Averages with no report behind them** — a figure whose paper has no examiner report is an invented figure, and this is the only check that can catch one. The Tag Checker can't run it, because knowing which papers have reports is something only the built tool holds.

The last four run only if you have loaded your own school's cohort figures
(see below): rows that match no coded question, the row arithmetic on the report,
the state column against the averages already in the dataset, and figures that
cannot be right.

The prompt on the kit page carries all fourteen, worded. It also carries one rule
that matters as much as the checks: **never let a check pass silently because the
data it needs is absent.** With no distributions in a dataset, checks 8 and 9 have
nothing to run on; with no school cohort file, 11 to 14 don't either. "Not run" has to
look different from "passed".

**This is a teacher's tool.** Don't build a student view of it, and don't hand
the file to a class. Students given a ranked likelihood list will study to it —
they'll revise the top five and drop the rest, which is the opposite of what the
analysis is for. Use it to decide what *you* teach and in what order; what
reaches students is your teaching, and revision advice you've already filtered
through your own judgement.

## Step 8: Test it against things you already know

Open it and check four things:

1. **Pick a question you know well.** Is it coded correctly? Does the explorer show the right marks and command terms?
2. **Pick a topic you know was hammered last year.** Does the coverage matrix reflect that?
3. **Look at the prediction rankings.** Do they match your instinct? Where they don't, is the tool seeing something you missed, or is the coding wrong? Both are useful findings.
4. **Build a practice paper.** Do the section totals come out right?

**If rankings look arbitrary** → the model weights need tuning, or coverage
tagging is uneven.
**If a topic looks never-assessed and you know it was** → a tagging error in
Step 5.
**If the tool states a figure you can't trace** → ask it which file and page.
Anything untraceable comes out.

## Step 9: Share it, and keep it honest

It's a single HTML file. Email it, drop it in a shared drive, or share the
project so colleagues can regenerate it — with colleagues, not with classes.

**Each year, when VCAA publishes the new exam and report:** add both to the
project, code the new paper (Step 5), verify it, rebuild. Then compare the new
coverage matrix against last year's — what rotated in and out is the most useful
thing the tool tells you.

The annual re-verification is the step most likely to get skipped, because by
then the build feels finished. It's also the easiest to hand to someone else: the
new paper's dataset, the exam, and the Tag Checker link is everything a colleague
needs, with no account to create. Half an hour a year is what keeps this honest.

**Two things to say out loud when you hand it round:**

- The predictions are pattern-reading, not forecasting. The whole study design is
  examinable every year, and every paper is unique. The tool is for prioritising
  revision when time is short, not for narrowing what gets taught.
- The analysis is only as good as the coding. If a colleague disagrees with a
  tag, they're probably right to check — send them back to the dataset rather
  than defending the interface.

---

## What generalises and what doesn't

The structure transfers to any study that passed the diagnostic at the top. Some
content won't:

- **Scenario generation** — the reference tool builds fictional briefs for its own subject. Your equivalent might be case scenarios (Legal Studies), stimulus datasets (Biology), or a business situation (Business Management). Same idea, different shape.
- **Student choice** — some studies let students pick which context, case or option they answer on. If your exam has a choice like that, code it: it changes what the coverage matrix means. If not, drop the field rather than leaving it empty.
- **Practical components** — if your study has a SAT, SAC-weighted performance or folio component, the exam only covers part of the story. Say so in the overview rather than letting the tool imply it's the whole picture.

---

## Checklist

The kit page tracks all of this for you, and will tell you if you tick a step
ahead of the ones it rests on. Ticks are self-reported either way — the list is a
record, not a lock.

**Before you start**

- [ ] Diagnostic answered — your study fits, or you know which views to drop

**Phase 1 — Project**

- [ ] One paper tested for readable text before gathering the rest
- [ ] Study design, support materials, FAQs, specifications
- [ ] Every exam paper under the current study design
- [ ] Resource / stimulus books for those years
- [ ] Matching examiner report for every paper
- [ ] All PDFs checked for selectable text; scans OCR'd
- [ ] This method document uploaded into the project
- [ ] Project instructions pasted in

**Phase 2 — Dataset**

- [ ] Key knowledge index built, checked for completeness, saved to a file
- [ ] Every paper coded, one per answer, with a `why` entry for every tag
- [ ] First paper checked for signs of thin coding — quoted justifications, `dist` against `avg`, tag counts
- [ ] `dist` captured wherever the report gives a distribution table
- [ ] Dataset run through the Tag Checker; every critical flag resolved
- [ ] Questions marked as needing the paper checked with the paper open
- [ ] Section mark totals reconcile against each paper
- [ ] `avg` figures spot-checked against the reports
- [ ] Every kk tag read and defensible; over-tagging pruned
- [ ] Command glossary with study-specific notes
- [ ] Persistent weaknesses and terminology confusions listed
- [ ] Handover pack downloaded
- [ ] *(optional)* School cohort figures typed at almanac.jotty.au/school/, every row reconciling

**Phase 3 — Tool**

- [ ] Reference tool uploaded to the project
- [ ] Shell plus overview and question explorer built and checked
- [ ] Remaining views added one at a time
- [ ] Dataset audit view included, with all fourteen checks
- [ ] No student view built — this stays a teacher's tool
- [ ] Tested against four things you already knew
- [ ] Untraceable figures removed
- [ ] Shared with colleagues, with the caveats stated