Jinovo/language-conditioned-psychometric-harmonization-model
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(stage1): add a step-by-step adjudication guide for a first-time adjudicator

Browse Source 
Twelve sections covering the single editable file, the two signed files that
must never be touched, safe ways to open a UTF-8 CSV, every one of the 14
columns, the exact attestation string, a worked example, the five decision
groups, the validation command, and what done looks like.

Two prohibitions are stated explicitly: do not edit the signed reviewer files,
and do not run build_human_review_packets.py, which overwrites them with blank
templates. Also flags that adjudicators cannot use needs_source even though
52 rows carry it from reviewer 1.

Every hardcoded string was verified against source: the attestation matches the
validator byte for byte, the allowed final_decision set matches, all 14 columns
are covered with no phantom or missing ones, and all six referenced paths exist.

Corrects an earlier claim in the record. The review_date problem is not one
problem. Reviewer 2 has 2026/9/20 on all 84 rows, a pure formatting issue.
Reviewer 1 has 84 distinct dates incrementing daily to 2026/12/12 - an Excel
drag-fill, mostly future-dated. That column cannot be fixed by reformatting,
since the values themselves are wrong; only reviewer 1 can supply the real date.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jinotech
2026-09-21 00:23:31 +12:00
co-authored by Claude Opus 5
parent ab175ce253
commit 911d534407
2 changed files with 299 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
research/stage1/adjudication_instructions.md
+279
View File
@@ -0,0 +1,279 @@
# Gate 1 adjudication instructions
This document assumes you have never opened this project before. Follow it in order.
## 1. What you are being asked to do
Two people already reviewed the same 84 survey questions independently. They recorded a
decision for each question. They disagreed on 63 of them, and on the other 21 they agreed but
did not give a plain approval.
You are the **adjudicator**. For each of the 84 rows you read both reviewers' reasoning and
record one final decision. You are the tie-breaker and the final word.
Nothing in the project moves forward until this is done. Gate 1 is blocked on it, and every
later stage is blocked on Gate 1.
You must be a **real person**, and you must not be either of the two reviewers.
## 2. The one file you edit
```
research/stage1/primary_outcome_review_adjudication.csv
```
That is the only file you change. Do not edit anything else.
You will also **read** these two files, because the adjudication file does not contain the
reviewers' reasoning:
```
research/stage1/primary_outcome_review_reviewer_1.csv
research/stage1/primary_outcome_review_reviewer_2.csv
```
> **Never edit, re-save, or "clean up" those two files.** They carry signatures. If they
> change at all, the review has to be done again from scratch.
> **Never run `build_human_review_packets.py`.** It overwrites both reviewer files with blank
> templates and destroys all of their work permanently.
## 3. Opening the file without corrupting it
The file is a UTF-8 CSV. Excel damages this kind of file in two specific ways, and it has
already happened once on this project.
**Danger 1 - dates.** If you type `2026-09-21` into Excel, it may silently rewrite it as
`2026/9/21`. That format is rejected. This exact bug is currently sitting in the reviewer
files and is blocking validation.
**Danger 2 - non-English text.** The reviewer notes contain Chinese. If Excel saves as plain
`CSV`, that text turns into garbage permanently.
Pick one of these three options.
**Option A - a plain text editor (safest).** Open the file in VS Code, Notepad++, or similar.
No auto-formatting, no encoding surprises. Harder to read, impossible to corrupt.
**Option B - Excel, done carefully.**
1. Open Excel first, with no file loaded.
2. `Data` -> `From Text/CSV`, pick the file.
3. Set `File Origin` to **65001: Unicode (UTF-8)**.
4. Click `Transform Data`, set **every** column type to **Text**, then `Close & Load`.
5. When saving, choose **CSV UTF-8 (Comma delimited)**, not plain `CSV`.
**Option C - Google Sheets.** `File` -> `Import` -> `Upload`, and set every date cell's format
to `Plain text` before typing. Download as `.csv` when finished.
If you are not confident, use Option A.
## 4. What the columns mean
**Already filled in for you. Do not change these.**
| Column | Meaning |
|---|---|
| `item_version_id` | The unique ID of the survey question. Your key to look things up. |
| `reviewer_1_decision` | What reviewer 1 concluded. |
| `reviewer_2_decision` | What reviewer 2 concluded. |
| `conflict_reason` | Why this row needs you. |
**You fill these in. Required on every row.**
| Column | What to put |
|---|---|
| `final_decision` | One of exactly three values. See section 5. |
| `adjudicator_name` | Your real full name. Identical on all 84 rows. |
| `adjudication_date` | The date you decided, as `YYYY-MM-DD`, e.g. `2026-09-21`. |
| `adjudicator_attestation` | One exact sentence. See section 6. |
**You fill these in only when your decision is `approve_with_correction`.** Put the corrected
value in whichever ones you are fixing, and leave the rest blank.
| Column | Use when you are correcting... |
|---|---|
| `final_question_text` | the wording of the question |
| `final_construct` | what the question is measuring |
| `final_population` | who was asked |
| `final_language` | the language evidence |
| `final_response_or_missing_rule` | the answer codes or missing-data rules |
**Optional but strongly encouraged.**
| Column | What to put |
|---|---|
| `adjudication_notes` | One or two sentences on *why* you decided this way. |
Write notes especially where you sided against a reviewer. Someone will read this in a year
and needs to understand your reasoning without you in the room.
## 5. The three values you may use
`final_decision` must be **exactly** one of these, lowercase, no spaces:
| Value | Meaning | Consequence |
|---|---|---|
| `approve` | The row is correct as recorded. | The question enters the analysis as-is. |
| `approve_with_correction` | Usable, but something must be fixed first. | You must fill at least one `final_*` column. |
| `reject` | Not usable. | The question is excluded from the analysis. |
> **The trap.** Reviewers were allowed a fourth value, `needs_source`, meaning "I cannot
> verify this yet." **You are not.** 52 rows say `needs_source` from reviewer 1, and you
> cannot pass that forward. For each one you must decide: is the evidence good enough
> (`approve`), good enough once fixed (`approve_with_correction`), or not good enough
> (`reject`)?
>
> `reject` is a legitimate, expected outcome. The project explicitly allows negative findings.
> Excluding a question you cannot verify is better science than admitting one you cannot.
## 6. The attestation sentence
Copy this into `adjudicator_attestation` on every row, character for character:
```
I am a human adjudicator and resolved this item after reviewing both assessments.
```
No extra spaces, no changed punctuation, no rewording. The checker compares it exactly and
rejects anything else.
## 7. How to do one row
1. Take the `item_version_id`, for example `YRBS::2013::Q27`.
2. Find that same ID in `primary_outcome_review_reviewer_1.csv` and read its `review_notes`.
3. Find the same ID in `primary_outcome_review_reviewer_2.csv` and read its `review_notes`.
4. Decide which reviewer is right, or chart a middle course.
5. In the adjudication file, on that row, fill `final_decision`, `adjudicator_name`,
`adjudication_date`, `adjudicator_attestation`, plus any `final_*` columns if correcting,
plus `adjudication_notes`.
6. Repeat 83 more times.
Worked example. On `YRBS::2013::Q27`, reviewer 1 said `needs_source` because the numeric
answer codes for that year were never checked against an official source. Reviewer 2 said
`approve` because the annual Data Users Guide lists `1=Yes, 2=No`. You decide whether the
guide counts as sufficient. If yes, write `approve`. If you want the official SAS format
program checked first, write `reject` or `approve_with_correction` and say so in the notes.
## 8. Do it in groups, not one at a time
The 84 rows are not 84 separate puzzles. They fall into five blocks, and every row in a block
turns on the same question. Decide the block, then apply it.
### Group A - 27 rows - reviewer 1 `needs_source`, reviewer 2 `approve`
`YRBS 2005 Q24-Q26 | 2007 Q24-Q26 | 2009 Q24-Q26 | 2011 Q25-Q27 | 2013 Q27-Q29 | 2015 Q27-Q29 | 2017 Q26-Q28 | 2019 Q26-Q28 | 2023 Q27-Q29`
**The question:** is the annual Data Users Guide codebook enough to establish the numeric
answer codes, or must the year's official SAS format program be checked?
**Worth knowing:** for YRBS **2023** the official format program is already on disk at
`Dataset/YRBS_National_1991_2023/source_archive/2023_cdc_download/2023XXH-Formats-Program.sas`,
and it defines `1=Yes`, `2=No`, blank = `Missing`. Those 3 rows can be settled by looking, not
by judging. The other 24 rows cover years with no such file stored locally.
### Group B - 18 rows - reviewer 1 `needs_source`, reviewer 2 `approve_with_correction`
All 6 sourced GSHS components (Morocco, Ghana, India, Jamaica, Mongolia, Wallis and Futuna)
times 3 questions each.
**The question:** reviewer 1 wants the sampling ages/grades, the actual language used, and a
direct letter-to-number code mapping. Reviewer 2 verified the codes against the country
codebook *and* the observed data distribution, and argues the fact sheet in the same folder
already answers the population question. Is that enough?
### Group C - 18 rows - reviewer 1 `approve_with_correction`, reviewer 2 `approve`
`YRBS 1991 Q19-Q21 | 1993 Q24-Q26 | 1999 Q23-Q25 | 2001 Q24-Q26 | 2003 Q24-Q26 | 2021 Q26-Q28`
**The question:** here the codes *were* verified. The only gap left is that the language is
recorded as "English source text accepted, actual language of administration unproven." Does
that count as a defect needing correction, or is it acceptable as recorded?
### Group D - 14 rows - both reviewers `approve_with_correction`
`YRBS 1995 Q22-Q24 | 1997 Q22-Q24 | NSDUH 2021-2024 YUSUITHK and YUSUIPLN`
Both agree a fix is needed, so this is mostly confirmation. **Several fixes here are already
applied** - the YRBS 1995/1997 Louisiana exclusion, and the removal of a codebook note that
had been wrongly stored as question wording. Check what is already fixed before asking for it
again.
**Still outstanding in this group:** the 2021 NSDUH ideation question is missing the sentence
that the 2022-2024 versions carry, telling respondents they may answer "I'm not sure" or
"I don't want to answer" - even though 2021 still offers those two answer codes. Both
reviewers asked for it back. It was deliberately not added, because writing official question
wording is your call, not an automated one. If you want it restored, put the exact sentence in
`final_question_text`.
### Group E - 7 rows - both reviewers `needs_source`
`GSHS Solomon Islands x 3 | NSDUH 2021-2024 YUSUICTRY x 4`
Both reviewers independently found no usable source. For Solomon Islands there is no country
questionnaire, codebook, or fact sheet anywhere in the dataset. For the NSDUH attempt
question, the variable is absent from the public-use data entirely.
Remember you cannot write `needs_source`. These rows need `approve`,
`approve_with_correction`, or `reject`.
## 9. Checking your work
Run this from the project root, `E:\Model`:
```powershell
.\.venv-research\Scripts\python.exe research/stage1/validate_human_reviews.py research/stage1/primary_outcome_review_reviewer_1.csv research/stage1/primary_outcome_review_reviewer_2.csv --adjudication research/stage1/primary_outcome_review_adjudication.csv
```
It prints a list of problems. Each line names the file, the row number, and what is wrong.
Fix them and run it again. Repeat until the adjudication errors are gone.
Common messages and what they mean:
| Message | Fix |
|---|---|
| `valid final_decision required` | That row's `final_decision` is blank or misspelled. Use exactly `approve`, `approve_with_correction`, or `reject`. |
| `human adjudicator identity and attestation required` | `adjudicator_name` is blank, or the attestation sentence does not match exactly. |
| `review_date must be YYYY-MM-DD` | **Not yours.** See section 10. |
## 10. One thing you cannot fix
You will see **168 errors reading `review_date must be YYYY-MM-DD`**. These are not your rows
and not your fault. They come from the two signed reviewer files, 84 rows each.
The two files are wrong in different ways, and the difference matters:
- **Reviewer 2** has `2026/9/20` on all 84 rows. Only the separator is wrong; the date itself
is consistent. That is purely a formatting problem.
- **Reviewer 1** has **84 different dates**, one per row, incrementing by a day from
`2026/9/20` all the way to `2026/12/12`. This is the signature of an Excel drag-fill: a date
was typed once and dragged down the column, and Excel counted upward. Most of those dates
are in the future and none of them after the first can be the real review date.
So reviewer 1's column cannot simply be reformatted - the *values* are wrong, not just their
shape. Only reviewer 1 can say what the true date or dates were.
Both live in signed files. Only the reviewers themselves can authorize changing them, because
those are signature fields. Leave them alone and report them.
**This means the validator will still say `failed` even after you finish perfectly.** That is
expected. Your work is complete when every `final_decision`, adjudicator identity, and
attestation error is gone and only the 168 `review_date` errors remain.
## 11. Rules that invalidate the work
- You must be a real person, not an AI, and not either reviewer.
- Do not copy a reviewer's decision without reading both sides.
- Do not edit the two reviewer files, for any reason.
- Do not run `build_human_review_packets.py`.
- Do not leave `final_decision` blank on any row.
- Your name, date, and attestation must be identical on all 84 rows.
## 12. When you are finished
Tell the project maintainer that adjudication is complete. They will re-run the Gate 1
acceptance audit and, once the remaining `review_date` issue is resolved, Gate 1 can close and
Stage 2 can begin.
The completed file contains personal attestations. Have it checked before it is committed or
shared.