docs(stage1): add a step-by-step adjudication guide for a first-time adjudicator
Twelve sections covering the single editable file, the two signed files that must never be touched, safe ways to open a UTF-8 CSV, every one of the 14 columns, the exact attestation string, a worked example, the five decision groups, the validation command, and what done looks like. Two prohibitions are stated explicitly: do not edit the signed reviewer files, and do not run build_human_review_packets.py, which overwrites them with blank templates. Also flags that adjudicators cannot use needs_source even though 52 rows carry it from reviewer 1. Every hardcoded string was verified against source: the attestation matches the validator byte for byte, the allowed final_decision set matches, all 14 columns are covered with no phantom or missing ones, and all six referenced paths exist. Corrects an earlier claim in the record. The review_date problem is not one problem. Reviewer 2 has 2026/9/20 on all 84 rows, a pure formatting issue. Reviewer 1 has 84 distinct dates incrementing daily to 2026/12/12 - an Excel drag-fill, mostly future-dated. That column cannot be fixed by reformatting, since the values themselves are wrong; only reviewer 1 can supply the real date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
ab175ce253
commit
911d534407
+20
-1
@@ -1,6 +1,6 @@
|
|||||||
# 项目进度档案
|
# 项目进度档案
|
||||||
|
|
||||||
版本:1.25
|
版本:1.26
|
||||||
建立日期:2026-09-20
|
建立日期:2026-09-20
|
||||||
当前总状态:`in_progress`
|
当前总状态:`in_progress`
|
||||||
当前阶段:阶段 1 `in_progress`;题库、回答字典、919,863 行回答长表及三调查设计字段/方法均已核验;Gate 1 仅剩主要结局的两名实际人工审核者签署
|
当前阶段:阶段 1 `in_progress`;题库、回答字典、919,863 行回答长表及三调查设计字段/方法均已核验;Gate 1 仅剩主要结局的两名实际人工审核者签署
|
||||||
@@ -1055,3 +1055,22 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
|
|||||||
1. 建议为 `build_human_review_packets.py` 增加 fail-closed 防护,禁止覆盖已签署的审核表(本次未授权,未实施);
|
1. 建议为 `build_human_review_packets.py` 增加 fail-closed 防护,禁止覆盖已签署的审核表(本次未授权,未实施);
|
||||||
2. 由裁决人按 A-20260921-072 的三个政策问题成组填写 `final_decision`、身份、日期与声明;
|
2. 由裁决人按 A-20260921-072 的三个政策问题成组填写 `final_decision`、身份、日期与声明;
|
||||||
3. 完成后重跑验证器与 Gate 1 验收。
|
3. 完成后重跑验证器与 Gate 1 验收。
|
||||||
|
|
||||||
|
|
||||||
|
### A-20260921-074:编写裁决操作指南并发现 reviewer 1 日期为拖拽填充伪值
|
||||||
|
|
||||||
|
- 时间:2026-09-21T00:22:59+12:00;阶段 1,Gate 1 保持 `in_progress`。
|
||||||
|
- 动作:行动前完整读取工作流与本档案并核对干净 Git 状态。按用户要求编写面向零基础裁决人的分步指南;撰写过程中逐项核对指南内所有硬编码字符串与路径。未修改任何审核表或生成物,未替任何人裁决。
|
||||||
|
- 输出:新增 `research/stage1/adjudication_instructions.md`(12 节),与既有 `human_review_instructions.md` 同目录同语言同风格。
|
||||||
|
- 指南内容要点:唯一可编辑文件与两份只读签署文件的边界;Excel 破坏 UTF-8 与日期的两类风险及三种安全打开方式;14 个列的逐列说明(4 列已填禁改、4 列必填、5 列仅在 `approve_with_correction` 时填、1 列建议填);`final_decision` 仅允许 `approve`/`approve_with_correction`/`reject`;逐行操作步骤与实例;按五个分组批量裁决;验证命令与错误对照表;完成标准。
|
||||||
|
- **明确写入指南的两条禁令:** 不得编辑两份签署审核表;不得运行 `build_human_review_packets.py`(该脚本会以空白模板覆盖两份签署表)。
|
||||||
|
- **裁决人不得使用 `needs_source`。** 验证器第 82 行仅接受三值,而 reviewer 1 有 52 行为 `needs_source`。指南明确说明这 52 行必须收敛为三值之一,并说明 `reject` 是工作流允许的正当结果。
|
||||||
|
- 事实核验(全部通过):裁决人声明串与验证器第 84 行逐字节一致;`final_decision` 允许值集与验证器一致;指南提及的列与 CSV 的 14 列**完全互相覆盖**(无虚构列、无遗漏列);6 个引用路径全部存在;五个分组行数 27+18+18+14+7=84。
|
||||||
|
- **本次新发现(更正此前记录):** 此前记录与说明将 `review_date` 问题一律描述为“`2026/9/20` 非 ISO 格式”。逐行核对后该描述**仅对 reviewer 2 成立**(84 行同为 `2026/9/20`)。**reviewer 1 的 84 行是 84 个互不相同的日期**,自 `2026/9/20` 起逐日递增至 `2026/12/12`,为 Excel 拖拽填充(drag-fill)特征;其中绝大多数晚于当前日期 2026-09-21,属未来日期,首行之后不可能是真实审核日期。
|
||||||
|
- **该发现改变修复性质:** reviewer 1 的日期**不能以格式规范化修复**,因为值本身错误而非仅形状错误。此前提出的“将 `review_date` 改为 ISO”对 reviewer 1 并不适用——那会把拖拽产生的伪日期固化为合法 ISO 日期。真实日期只能由 reviewer 1 本人提供。指南第 10 节已按此改写。
|
||||||
|
- 检查:指南所有断言经脚本核对;未运行任何构建脚本;两份签署审核表未读写改动。Gate 1 状态未变。
|
||||||
|
- 未解决问题:84 项裁决待人工执行;reviewer 1 日期需本人提供真实值、reviewer 2 日期需本人授权改格式;`build_human_review_packets.py` 覆盖风险未加防护;13 项未关闭冲突;2021 提示句缺口。
|
||||||
|
- 下一步:
|
||||||
|
1. 将本指南与裁决表交付裁决人;
|
||||||
|
2. 请 reviewer 1 提供真实审核日期,请 reviewer 2 授权日期格式更正;
|
||||||
|
3. 裁决完成后重跑验证器与 Gate 1 验收。
|
||||||
|
|||||||
@@ -0,0 +1,279 @@
|
|||||||
|
# Gate 1 adjudication instructions
|
||||||
|
|
||||||
|
This document assumes you have never opened this project before. Follow it in order.
|
||||||
|
|
||||||
|
## 1. What you are being asked to do
|
||||||
|
|
||||||
|
Two people already reviewed the same 84 survey questions independently. They recorded a
|
||||||
|
decision for each question. They disagreed on 63 of them, and on the other 21 they agreed but
|
||||||
|
did not give a plain approval.
|
||||||
|
|
||||||
|
You are the **adjudicator**. For each of the 84 rows you read both reviewers' reasoning and
|
||||||
|
record one final decision. You are the tie-breaker and the final word.
|
||||||
|
|
||||||
|
Nothing in the project moves forward until this is done. Gate 1 is blocked on it, and every
|
||||||
|
later stage is blocked on Gate 1.
|
||||||
|
|
||||||
|
You must be a **real person**, and you must not be either of the two reviewers.
|
||||||
|
|
||||||
|
## 2. The one file you edit
|
||||||
|
|
||||||
|
```
|
||||||
|
research/stage1/primary_outcome_review_adjudication.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the only file you change. Do not edit anything else.
|
||||||
|
|
||||||
|
You will also **read** these two files, because the adjudication file does not contain the
|
||||||
|
reviewers' reasoning:
|
||||||
|
|
||||||
|
```
|
||||||
|
research/stage1/primary_outcome_review_reviewer_1.csv
|
||||||
|
research/stage1/primary_outcome_review_reviewer_2.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Never edit, re-save, or "clean up" those two files.** They carry signatures. If they
|
||||||
|
> change at all, the review has to be done again from scratch.
|
||||||
|
|
||||||
|
> **Never run `build_human_review_packets.py`.** It overwrites both reviewer files with blank
|
||||||
|
> templates and destroys all of their work permanently.
|
||||||
|
|
||||||
|
## 3. Opening the file without corrupting it
|
||||||
|
|
||||||
|
The file is a UTF-8 CSV. Excel damages this kind of file in two specific ways, and it has
|
||||||
|
already happened once on this project.
|
||||||
|
|
||||||
|
**Danger 1 - dates.** If you type `2026-09-21` into Excel, it may silently rewrite it as
|
||||||
|
`2026/9/21`. That format is rejected. This exact bug is currently sitting in the reviewer
|
||||||
|
files and is blocking validation.
|
||||||
|
|
||||||
|
**Danger 2 - non-English text.** The reviewer notes contain Chinese. If Excel saves as plain
|
||||||
|
`CSV`, that text turns into garbage permanently.
|
||||||
|
|
||||||
|
Pick one of these three options.
|
||||||
|
|
||||||
|
**Option A - a plain text editor (safest).** Open the file in VS Code, Notepad++, or similar.
|
||||||
|
No auto-formatting, no encoding surprises. Harder to read, impossible to corrupt.
|
||||||
|
|
||||||
|
**Option B - Excel, done carefully.**
|
||||||
|
|
||||||
|
1. Open Excel first, with no file loaded.
|
||||||
|
2. `Data` -> `From Text/CSV`, pick the file.
|
||||||
|
3. Set `File Origin` to **65001: Unicode (UTF-8)**.
|
||||||
|
4. Click `Transform Data`, set **every** column type to **Text**, then `Close & Load`.
|
||||||
|
5. When saving, choose **CSV UTF-8 (Comma delimited)**, not plain `CSV`.
|
||||||
|
|
||||||
|
**Option C - Google Sheets.** `File` -> `Import` -> `Upload`, and set every date cell's format
|
||||||
|
to `Plain text` before typing. Download as `.csv` when finished.
|
||||||
|
|
||||||
|
If you are not confident, use Option A.
|
||||||
|
|
||||||
|
## 4. What the columns mean
|
||||||
|
|
||||||
|
**Already filled in for you. Do not change these.**
|
||||||
|
|
||||||
|
| Column | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `item_version_id` | The unique ID of the survey question. Your key to look things up. |
|
||||||
|
| `reviewer_1_decision` | What reviewer 1 concluded. |
|
||||||
|
| `reviewer_2_decision` | What reviewer 2 concluded. |
|
||||||
|
| `conflict_reason` | Why this row needs you. |
|
||||||
|
|
||||||
|
**You fill these in. Required on every row.**
|
||||||
|
|
||||||
|
| Column | What to put |
|
||||||
|
|---|---|
|
||||||
|
| `final_decision` | One of exactly three values. See section 5. |
|
||||||
|
| `adjudicator_name` | Your real full name. Identical on all 84 rows. |
|
||||||
|
| `adjudication_date` | The date you decided, as `YYYY-MM-DD`, e.g. `2026-09-21`. |
|
||||||
|
| `adjudicator_attestation` | One exact sentence. See section 6. |
|
||||||
|
|
||||||
|
**You fill these in only when your decision is `approve_with_correction`.** Put the corrected
|
||||||
|
value in whichever ones you are fixing, and leave the rest blank.
|
||||||
|
|
||||||
|
| Column | Use when you are correcting... |
|
||||||
|
|---|---|
|
||||||
|
| `final_question_text` | the wording of the question |
|
||||||
|
| `final_construct` | what the question is measuring |
|
||||||
|
| `final_population` | who was asked |
|
||||||
|
| `final_language` | the language evidence |
|
||||||
|
| `final_response_or_missing_rule` | the answer codes or missing-data rules |
|
||||||
|
|
||||||
|
**Optional but strongly encouraged.**
|
||||||
|
|
||||||
|
| Column | What to put |
|
||||||
|
|---|---|
|
||||||
|
| `adjudication_notes` | One or two sentences on *why* you decided this way. |
|
||||||
|
|
||||||
|
Write notes especially where you sided against a reviewer. Someone will read this in a year
|
||||||
|
and needs to understand your reasoning without you in the room.
|
||||||
|
|
||||||
|
## 5. The three values you may use
|
||||||
|
|
||||||
|
`final_decision` must be **exactly** one of these, lowercase, no spaces:
|
||||||
|
|
||||||
|
| Value | Meaning | Consequence |
|
||||||
|
|---|---|---|
|
||||||
|
| `approve` | The row is correct as recorded. | The question enters the analysis as-is. |
|
||||||
|
| `approve_with_correction` | Usable, but something must be fixed first. | You must fill at least one `final_*` column. |
|
||||||
|
| `reject` | Not usable. | The question is excluded from the analysis. |
|
||||||
|
|
||||||
|
> **The trap.** Reviewers were allowed a fourth value, `needs_source`, meaning "I cannot
|
||||||
|
> verify this yet." **You are not.** 52 rows say `needs_source` from reviewer 1, and you
|
||||||
|
> cannot pass that forward. For each one you must decide: is the evidence good enough
|
||||||
|
> (`approve`), good enough once fixed (`approve_with_correction`), or not good enough
|
||||||
|
> (`reject`)?
|
||||||
|
>
|
||||||
|
> `reject` is a legitimate, expected outcome. The project explicitly allows negative findings.
|
||||||
|
> Excluding a question you cannot verify is better science than admitting one you cannot.
|
||||||
|
|
||||||
|
## 6. The attestation sentence
|
||||||
|
|
||||||
|
Copy this into `adjudicator_attestation` on every row, character for character:
|
||||||
|
|
||||||
|
```
|
||||||
|
I am a human adjudicator and resolved this item after reviewing both assessments.
|
||||||
|
```
|
||||||
|
|
||||||
|
No extra spaces, no changed punctuation, no rewording. The checker compares it exactly and
|
||||||
|
rejects anything else.
|
||||||
|
|
||||||
|
## 7. How to do one row
|
||||||
|
|
||||||
|
1. Take the `item_version_id`, for example `YRBS::2013::Q27`.
|
||||||
|
2. Find that same ID in `primary_outcome_review_reviewer_1.csv` and read its `review_notes`.
|
||||||
|
3. Find the same ID in `primary_outcome_review_reviewer_2.csv` and read its `review_notes`.
|
||||||
|
4. Decide which reviewer is right, or chart a middle course.
|
||||||
|
5. In the adjudication file, on that row, fill `final_decision`, `adjudicator_name`,
|
||||||
|
`adjudication_date`, `adjudicator_attestation`, plus any `final_*` columns if correcting,
|
||||||
|
plus `adjudication_notes`.
|
||||||
|
6. Repeat 83 more times.
|
||||||
|
|
||||||
|
Worked example. On `YRBS::2013::Q27`, reviewer 1 said `needs_source` because the numeric
|
||||||
|
answer codes for that year were never checked against an official source. Reviewer 2 said
|
||||||
|
`approve` because the annual Data Users Guide lists `1=Yes, 2=No`. You decide whether the
|
||||||
|
guide counts as sufficient. If yes, write `approve`. If you want the official SAS format
|
||||||
|
program checked first, write `reject` or `approve_with_correction` and say so in the notes.
|
||||||
|
|
||||||
|
## 8. Do it in groups, not one at a time
|
||||||
|
|
||||||
|
The 84 rows are not 84 separate puzzles. They fall into five blocks, and every row in a block
|
||||||
|
turns on the same question. Decide the block, then apply it.
|
||||||
|
|
||||||
|
### Group A - 27 rows - reviewer 1 `needs_source`, reviewer 2 `approve`
|
||||||
|
|
||||||
|
`YRBS 2005 Q24-Q26 | 2007 Q24-Q26 | 2009 Q24-Q26 | 2011 Q25-Q27 | 2013 Q27-Q29 | 2015 Q27-Q29 | 2017 Q26-Q28 | 2019 Q26-Q28 | 2023 Q27-Q29`
|
||||||
|
|
||||||
|
**The question:** is the annual Data Users Guide codebook enough to establish the numeric
|
||||||
|
answer codes, or must the year's official SAS format program be checked?
|
||||||
|
|
||||||
|
**Worth knowing:** for YRBS **2023** the official format program is already on disk at
|
||||||
|
`Dataset/YRBS_National_1991_2023/source_archive/2023_cdc_download/2023XXH-Formats-Program.sas`,
|
||||||
|
and it defines `1=Yes`, `2=No`, blank = `Missing`. Those 3 rows can be settled by looking, not
|
||||||
|
by judging. The other 24 rows cover years with no such file stored locally.
|
||||||
|
|
||||||
|
### Group B - 18 rows - reviewer 1 `needs_source`, reviewer 2 `approve_with_correction`
|
||||||
|
|
||||||
|
All 6 sourced GSHS components (Morocco, Ghana, India, Jamaica, Mongolia, Wallis and Futuna)
|
||||||
|
times 3 questions each.
|
||||||
|
|
||||||
|
**The question:** reviewer 1 wants the sampling ages/grades, the actual language used, and a
|
||||||
|
direct letter-to-number code mapping. Reviewer 2 verified the codes against the country
|
||||||
|
codebook *and* the observed data distribution, and argues the fact sheet in the same folder
|
||||||
|
already answers the population question. Is that enough?
|
||||||
|
|
||||||
|
### Group C - 18 rows - reviewer 1 `approve_with_correction`, reviewer 2 `approve`
|
||||||
|
|
||||||
|
`YRBS 1991 Q19-Q21 | 1993 Q24-Q26 | 1999 Q23-Q25 | 2001 Q24-Q26 | 2003 Q24-Q26 | 2021 Q26-Q28`
|
||||||
|
|
||||||
|
**The question:** here the codes *were* verified. The only gap left is that the language is
|
||||||
|
recorded as "English source text accepted, actual language of administration unproven." Does
|
||||||
|
that count as a defect needing correction, or is it acceptable as recorded?
|
||||||
|
|
||||||
|
### Group D - 14 rows - both reviewers `approve_with_correction`
|
||||||
|
|
||||||
|
`YRBS 1995 Q22-Q24 | 1997 Q22-Q24 | NSDUH 2021-2024 YUSUITHK and YUSUIPLN`
|
||||||
|
|
||||||
|
Both agree a fix is needed, so this is mostly confirmation. **Several fixes here are already
|
||||||
|
applied** - the YRBS 1995/1997 Louisiana exclusion, and the removal of a codebook note that
|
||||||
|
had been wrongly stored as question wording. Check what is already fixed before asking for it
|
||||||
|
again.
|
||||||
|
|
||||||
|
**Still outstanding in this group:** the 2021 NSDUH ideation question is missing the sentence
|
||||||
|
that the 2022-2024 versions carry, telling respondents they may answer "I'm not sure" or
|
||||||
|
"I don't want to answer" - even though 2021 still offers those two answer codes. Both
|
||||||
|
reviewers asked for it back. It was deliberately not added, because writing official question
|
||||||
|
wording is your call, not an automated one. If you want it restored, put the exact sentence in
|
||||||
|
`final_question_text`.
|
||||||
|
|
||||||
|
### Group E - 7 rows - both reviewers `needs_source`
|
||||||
|
|
||||||
|
`GSHS Solomon Islands x 3 | NSDUH 2021-2024 YUSUICTRY x 4`
|
||||||
|
|
||||||
|
Both reviewers independently found no usable source. For Solomon Islands there is no country
|
||||||
|
questionnaire, codebook, or fact sheet anywhere in the dataset. For the NSDUH attempt
|
||||||
|
question, the variable is absent from the public-use data entirely.
|
||||||
|
|
||||||
|
Remember you cannot write `needs_source`. These rows need `approve`,
|
||||||
|
`approve_with_correction`, or `reject`.
|
||||||
|
|
||||||
|
## 9. Checking your work
|
||||||
|
|
||||||
|
Run this from the project root, `E:\Model`:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\.venv-research\Scripts\python.exe research/stage1/validate_human_reviews.py research/stage1/primary_outcome_review_reviewer_1.csv research/stage1/primary_outcome_review_reviewer_2.csv --adjudication research/stage1/primary_outcome_review_adjudication.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
It prints a list of problems. Each line names the file, the row number, and what is wrong.
|
||||||
|
Fix them and run it again. Repeat until the adjudication errors are gone.
|
||||||
|
|
||||||
|
Common messages and what they mean:
|
||||||
|
|
||||||
|
| Message | Fix |
|
||||||
|
|---|---|
|
||||||
|
| `valid final_decision required` | That row's `final_decision` is blank or misspelled. Use exactly `approve`, `approve_with_correction`, or `reject`. |
|
||||||
|
| `human adjudicator identity and attestation required` | `adjudicator_name` is blank, or the attestation sentence does not match exactly. |
|
||||||
|
| `review_date must be YYYY-MM-DD` | **Not yours.** See section 10. |
|
||||||
|
|
||||||
|
## 10. One thing you cannot fix
|
||||||
|
|
||||||
|
You will see **168 errors reading `review_date must be YYYY-MM-DD`**. These are not your rows
|
||||||
|
and not your fault. They come from the two signed reviewer files, 84 rows each.
|
||||||
|
|
||||||
|
The two files are wrong in different ways, and the difference matters:
|
||||||
|
|
||||||
|
- **Reviewer 2** has `2026/9/20` on all 84 rows. Only the separator is wrong; the date itself
|
||||||
|
is consistent. That is purely a formatting problem.
|
||||||
|
- **Reviewer 1** has **84 different dates**, one per row, incrementing by a day from
|
||||||
|
`2026/9/20` all the way to `2026/12/12`. This is the signature of an Excel drag-fill: a date
|
||||||
|
was typed once and dragged down the column, and Excel counted upward. Most of those dates
|
||||||
|
are in the future and none of them after the first can be the real review date.
|
||||||
|
|
||||||
|
So reviewer 1's column cannot simply be reformatted - the *values* are wrong, not just their
|
||||||
|
shape. Only reviewer 1 can say what the true date or dates were.
|
||||||
|
|
||||||
|
Both live in signed files. Only the reviewers themselves can authorize changing them, because
|
||||||
|
those are signature fields. Leave them alone and report them.
|
||||||
|
|
||||||
|
**This means the validator will still say `failed` even after you finish perfectly.** That is
|
||||||
|
expected. Your work is complete when every `final_decision`, adjudicator identity, and
|
||||||
|
attestation error is gone and only the 168 `review_date` errors remain.
|
||||||
|
|
||||||
|
## 11. Rules that invalidate the work
|
||||||
|
|
||||||
|
- You must be a real person, not an AI, and not either reviewer.
|
||||||
|
- Do not copy a reviewer's decision without reading both sides.
|
||||||
|
- Do not edit the two reviewer files, for any reason.
|
||||||
|
- Do not run `build_human_review_packets.py`.
|
||||||
|
- Do not leave `final_decision` blank on any row.
|
||||||
|
- Your name, date, and attestation must be identical on all 84 rows.
|
||||||
|
|
||||||
|
## 12. When you are finished
|
||||||
|
|
||||||
|
Tell the project maintainer that adjudication is complete. They will re-run the Gate 1
|
||||||
|
acceptance audit and, once the remaining `review_date` issue is resolved, Gate 1 can close and
|
||||||
|
Stage 2 can begin.
|
||||||
|
|
||||||
|
The completed file contains personal attestations. Have it checked before it is committed or
|
||||||
|
shared.
|
||||||
Reference in New Issue
Block a user