diff --git a/Progression.md b/Progression.md index fa61336..c4f1f7b 100644 --- a/Progression.md +++ b/Progression.md @@ -1,6 +1,6 @@ # 项目进度档案 -版本:1.25 +版本:1.26 建立日期:2026-09-20 当前总状态:`in_progress` 当前阶段:阶段 1 `in_progress`;题库、回答字典、919,863 行回答长表及三调查设计字段/方法均已核验;Gate 1 仅剩主要结局的两名实际人工审核者签署 @@ -1055,3 +1055,22 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、 1. 建议为 `build_human_review_packets.py` 增加 fail-closed 防护,禁止覆盖已签署的审核表(本次未授权,未实施); 2. 由裁决人按 A-20260921-072 的三个政策问题成组填写 `final_decision`、身份、日期与声明; 3. 完成后重跑验证器与 Gate 1 验收。 + + +### A-20260921-074:编写裁决操作指南并发现 reviewer 1 日期为拖拽填充伪值 + +- 时间:2026-09-21T00:22:59+12:00;阶段 1,Gate 1 保持 `in_progress`。 +- 动作:行动前完整读取工作流与本档案并核对干净 Git 状态。按用户要求编写面向零基础裁决人的分步指南;撰写过程中逐项核对指南内所有硬编码字符串与路径。未修改任何审核表或生成物,未替任何人裁决。 +- 输出:新增 `research/stage1/adjudication_instructions.md`(12 节),与既有 `human_review_instructions.md` 同目录同语言同风格。 +- 指南内容要点:唯一可编辑文件与两份只读签署文件的边界;Excel 破坏 UTF-8 与日期的两类风险及三种安全打开方式;14 个列的逐列说明(4 列已填禁改、4 列必填、5 列仅在 `approve_with_correction` 时填、1 列建议填);`final_decision` 仅允许 `approve`/`approve_with_correction`/`reject`;逐行操作步骤与实例;按五个分组批量裁决;验证命令与错误对照表;完成标准。 +- **明确写入指南的两条禁令:** 不得编辑两份签署审核表;不得运行 `build_human_review_packets.py`(该脚本会以空白模板覆盖两份签署表)。 +- **裁决人不得使用 `needs_source`。** 验证器第 82 行仅接受三值,而 reviewer 1 有 52 行为 `needs_source`。指南明确说明这 52 行必须收敛为三值之一,并说明 `reject` 是工作流允许的正当结果。 +- 事实核验(全部通过):裁决人声明串与验证器第 84 行逐字节一致;`final_decision` 允许值集与验证器一致;指南提及的列与 CSV 的 14 列**完全互相覆盖**(无虚构列、无遗漏列);6 个引用路径全部存在;五个分组行数 27+18+18+14+7=84。 +- **本次新发现(更正此前记录):** 此前记录与说明将 `review_date` 问题一律描述为“`2026/9/20` 非 ISO 格式”。逐行核对后该描述**仅对 reviewer 2 成立**(84 行同为 `2026/9/20`)。**reviewer 1 的 84 行是 84 个互不相同的日期**,自 `2026/9/20` 起逐日递增至 `2026/12/12`,为 Excel 拖拽填充(drag-fill)特征;其中绝大多数晚于当前日期 2026-09-21,属未来日期,首行之后不可能是真实审核日期。 +- **该发现改变修复性质:** reviewer 1 的日期**不能以格式规范化修复**,因为值本身错误而非仅形状错误。此前提出的“将 `review_date` 改为 ISO”对 reviewer 1 并不适用——那会把拖拽产生的伪日期固化为合法 ISO 日期。真实日期只能由 reviewer 1 本人提供。指南第 10 节已按此改写。 +- 检查:指南所有断言经脚本核对;未运行任何构建脚本;两份签署审核表未读写改动。Gate 1 状态未变。 +- 未解决问题:84 项裁决待人工执行;reviewer 1 日期需本人提供真实值、reviewer 2 日期需本人授权改格式;`build_human_review_packets.py` 覆盖风险未加防护;13 项未关闭冲突;2021 提示句缺口。 +- 下一步: + 1. 将本指南与裁决表交付裁决人; + 2. 请 reviewer 1 提供真实审核日期,请 reviewer 2 授权日期格式更正; + 3. 裁决完成后重跑验证器与 Gate 1 验收。 diff --git a/research/stage1/adjudication_instructions.md b/research/stage1/adjudication_instructions.md new file mode 100644 index 0000000..8ddf587 --- /dev/null +++ b/research/stage1/adjudication_instructions.md @@ -0,0 +1,279 @@ +# Gate 1 adjudication instructions + +This document assumes you have never opened this project before. Follow it in order. + +## 1. What you are being asked to do + +Two people already reviewed the same 84 survey questions independently. They recorded a +decision for each question. They disagreed on 63 of them, and on the other 21 they agreed but +did not give a plain approval. + +You are the **adjudicator**. For each of the 84 rows you read both reviewers' reasoning and +record one final decision. You are the tie-breaker and the final word. + +Nothing in the project moves forward until this is done. Gate 1 is blocked on it, and every +later stage is blocked on Gate 1. + +You must be a **real person**, and you must not be either of the two reviewers. + +## 2. The one file you edit + +``` +research/stage1/primary_outcome_review_adjudication.csv +``` + +That is the only file you change. Do not edit anything else. + +You will also **read** these two files, because the adjudication file does not contain the +reviewers' reasoning: + +``` +research/stage1/primary_outcome_review_reviewer_1.csv +research/stage1/primary_outcome_review_reviewer_2.csv +``` + +> **Never edit, re-save, or "clean up" those two files.** They carry signatures. If they +> change at all, the review has to be done again from scratch. + +> **Never run `build_human_review_packets.py`.** It overwrites both reviewer files with blank +> templates and destroys all of their work permanently. + +## 3. Opening the file without corrupting it + +The file is a UTF-8 CSV. Excel damages this kind of file in two specific ways, and it has +already happened once on this project. + +**Danger 1 - dates.** If you type `2026-09-21` into Excel, it may silently rewrite it as +`2026/9/21`. That format is rejected. This exact bug is currently sitting in the reviewer +files and is blocking validation. + +**Danger 2 - non-English text.** The reviewer notes contain Chinese. If Excel saves as plain +`CSV`, that text turns into garbage permanently. + +Pick one of these three options. + +**Option A - a plain text editor (safest).** Open the file in VS Code, Notepad++, or similar. +No auto-formatting, no encoding surprises. Harder to read, impossible to corrupt. + +**Option B - Excel, done carefully.** + +1. Open Excel first, with no file loaded. +2. `Data` -> `From Text/CSV`, pick the file. +3. Set `File Origin` to **65001: Unicode (UTF-8)**. +4. Click `Transform Data`, set **every** column type to **Text**, then `Close & Load`. +5. When saving, choose **CSV UTF-8 (Comma delimited)**, not plain `CSV`. + +**Option C - Google Sheets.** `File` -> `Import` -> `Upload`, and set every date cell's format +to `Plain text` before typing. Download as `.csv` when finished. + +If you are not confident, use Option A. + +## 4. What the columns mean + +**Already filled in for you. Do not change these.** + +| Column | Meaning | +|---|---| +| `item_version_id` | The unique ID of the survey question. Your key to look things up. | +| `reviewer_1_decision` | What reviewer 1 concluded. | +| `reviewer_2_decision` | What reviewer 2 concluded. | +| `conflict_reason` | Why this row needs you. | + +**You fill these in. Required on every row.** + +| Column | What to put | +|---|---| +| `final_decision` | One of exactly three values. See section 5. | +| `adjudicator_name` | Your real full name. Identical on all 84 rows. | +| `adjudication_date` | The date you decided, as `YYYY-MM-DD`, e.g. `2026-09-21`. | +| `adjudicator_attestation` | One exact sentence. See section 6. | + +**You fill these in only when your decision is `approve_with_correction`.** Put the corrected +value in whichever ones you are fixing, and leave the rest blank. + +| Column | Use when you are correcting... | +|---|---| +| `final_question_text` | the wording of the question | +| `final_construct` | what the question is measuring | +| `final_population` | who was asked | +| `final_language` | the language evidence | +| `final_response_or_missing_rule` | the answer codes or missing-data rules | + +**Optional but strongly encouraged.** + +| Column | What to put | +|---|---| +| `adjudication_notes` | One or two sentences on *why* you decided this way. | + +Write notes especially where you sided against a reviewer. Someone will read this in a year +and needs to understand your reasoning without you in the room. + +## 5. The three values you may use + +`final_decision` must be **exactly** one of these, lowercase, no spaces: + +| Value | Meaning | Consequence | +|---|---|---| +| `approve` | The row is correct as recorded. | The question enters the analysis as-is. | +| `approve_with_correction` | Usable, but something must be fixed first. | You must fill at least one `final_*` column. | +| `reject` | Not usable. | The question is excluded from the analysis. | + +> **The trap.** Reviewers were allowed a fourth value, `needs_source`, meaning "I cannot +> verify this yet." **You are not.** 52 rows say `needs_source` from reviewer 1, and you +> cannot pass that forward. For each one you must decide: is the evidence good enough +> (`approve`), good enough once fixed (`approve_with_correction`), or not good enough +> (`reject`)? +> +> `reject` is a legitimate, expected outcome. The project explicitly allows negative findings. +> Excluding a question you cannot verify is better science than admitting one you cannot. + +## 6. The attestation sentence + +Copy this into `adjudicator_attestation` on every row, character for character: + +``` +I am a human adjudicator and resolved this item after reviewing both assessments. +``` + +No extra spaces, no changed punctuation, no rewording. The checker compares it exactly and +rejects anything else. + +## 7. How to do one row + +1. Take the `item_version_id`, for example `YRBS::2013::Q27`. +2. Find that same ID in `primary_outcome_review_reviewer_1.csv` and read its `review_notes`. +3. Find the same ID in `primary_outcome_review_reviewer_2.csv` and read its `review_notes`. +4. Decide which reviewer is right, or chart a middle course. +5. In the adjudication file, on that row, fill `final_decision`, `adjudicator_name`, + `adjudication_date`, `adjudicator_attestation`, plus any `final_*` columns if correcting, + plus `adjudication_notes`. +6. Repeat 83 more times. + +Worked example. On `YRBS::2013::Q27`, reviewer 1 said `needs_source` because the numeric +answer codes for that year were never checked against an official source. Reviewer 2 said +`approve` because the annual Data Users Guide lists `1=Yes, 2=No`. You decide whether the +guide counts as sufficient. If yes, write `approve`. If you want the official SAS format +program checked first, write `reject` or `approve_with_correction` and say so in the notes. + +## 8. Do it in groups, not one at a time + +The 84 rows are not 84 separate puzzles. They fall into five blocks, and every row in a block +turns on the same question. Decide the block, then apply it. + +### Group A - 27 rows - reviewer 1 `needs_source`, reviewer 2 `approve` + +`YRBS 2005 Q24-Q26 | 2007 Q24-Q26 | 2009 Q24-Q26 | 2011 Q25-Q27 | 2013 Q27-Q29 | 2015 Q27-Q29 | 2017 Q26-Q28 | 2019 Q26-Q28 | 2023 Q27-Q29` + +**The question:** is the annual Data Users Guide codebook enough to establish the numeric +answer codes, or must the year's official SAS format program be checked? + +**Worth knowing:** for YRBS **2023** the official format program is already on disk at +`Dataset/YRBS_National_1991_2023/source_archive/2023_cdc_download/2023XXH-Formats-Program.sas`, +and it defines `1=Yes`, `2=No`, blank = `Missing`. Those 3 rows can be settled by looking, not +by judging. The other 24 rows cover years with no such file stored locally. + +### Group B - 18 rows - reviewer 1 `needs_source`, reviewer 2 `approve_with_correction` + +All 6 sourced GSHS components (Morocco, Ghana, India, Jamaica, Mongolia, Wallis and Futuna) +times 3 questions each. + +**The question:** reviewer 1 wants the sampling ages/grades, the actual language used, and a +direct letter-to-number code mapping. Reviewer 2 verified the codes against the country +codebook *and* the observed data distribution, and argues the fact sheet in the same folder +already answers the population question. Is that enough? + +### Group C - 18 rows - reviewer 1 `approve_with_correction`, reviewer 2 `approve` + +`YRBS 1991 Q19-Q21 | 1993 Q24-Q26 | 1999 Q23-Q25 | 2001 Q24-Q26 | 2003 Q24-Q26 | 2021 Q26-Q28` + +**The question:** here the codes *were* verified. The only gap left is that the language is +recorded as "English source text accepted, actual language of administration unproven." Does +that count as a defect needing correction, or is it acceptable as recorded? + +### Group D - 14 rows - both reviewers `approve_with_correction` + +`YRBS 1995 Q22-Q24 | 1997 Q22-Q24 | NSDUH 2021-2024 YUSUITHK and YUSUIPLN` + +Both agree a fix is needed, so this is mostly confirmation. **Several fixes here are already +applied** - the YRBS 1995/1997 Louisiana exclusion, and the removal of a codebook note that +had been wrongly stored as question wording. Check what is already fixed before asking for it +again. + +**Still outstanding in this group:** the 2021 NSDUH ideation question is missing the sentence +that the 2022-2024 versions carry, telling respondents they may answer "I'm not sure" or +"I don't want to answer" - even though 2021 still offers those two answer codes. Both +reviewers asked for it back. It was deliberately not added, because writing official question +wording is your call, not an automated one. If you want it restored, put the exact sentence in +`final_question_text`. + +### Group E - 7 rows - both reviewers `needs_source` + +`GSHS Solomon Islands x 3 | NSDUH 2021-2024 YUSUICTRY x 4` + +Both reviewers independently found no usable source. For Solomon Islands there is no country +questionnaire, codebook, or fact sheet anywhere in the dataset. For the NSDUH attempt +question, the variable is absent from the public-use data entirely. + +Remember you cannot write `needs_source`. These rows need `approve`, +`approve_with_correction`, or `reject`. + +## 9. Checking your work + +Run this from the project root, `E:\Model`: + +```powershell +.\.venv-research\Scripts\python.exe research/stage1/validate_human_reviews.py research/stage1/primary_outcome_review_reviewer_1.csv research/stage1/primary_outcome_review_reviewer_2.csv --adjudication research/stage1/primary_outcome_review_adjudication.csv +``` + +It prints a list of problems. Each line names the file, the row number, and what is wrong. +Fix them and run it again. Repeat until the adjudication errors are gone. + +Common messages and what they mean: + +| Message | Fix | +|---|---| +| `valid final_decision required` | That row's `final_decision` is blank or misspelled. Use exactly `approve`, `approve_with_correction`, or `reject`. | +| `human adjudicator identity and attestation required` | `adjudicator_name` is blank, or the attestation sentence does not match exactly. | +| `review_date must be YYYY-MM-DD` | **Not yours.** See section 10. | + +## 10. One thing you cannot fix + +You will see **168 errors reading `review_date must be YYYY-MM-DD`**. These are not your rows +and not your fault. They come from the two signed reviewer files, 84 rows each. + +The two files are wrong in different ways, and the difference matters: + +- **Reviewer 2** has `2026/9/20` on all 84 rows. Only the separator is wrong; the date itself + is consistent. That is purely a formatting problem. +- **Reviewer 1** has **84 different dates**, one per row, incrementing by a day from + `2026/9/20` all the way to `2026/12/12`. This is the signature of an Excel drag-fill: a date + was typed once and dragged down the column, and Excel counted upward. Most of those dates + are in the future and none of them after the first can be the real review date. + +So reviewer 1's column cannot simply be reformatted - the *values* are wrong, not just their +shape. Only reviewer 1 can say what the true date or dates were. + +Both live in signed files. Only the reviewers themselves can authorize changing them, because +those are signature fields. Leave them alone and report them. + +**This means the validator will still say `failed` even after you finish perfectly.** That is +expected. Your work is complete when every `final_decision`, adjudicator identity, and +attestation error is gone and only the 168 `review_date` errors remain. + +## 11. Rules that invalidate the work + +- You must be a real person, not an AI, and not either reviewer. +- Do not copy a reviewer's decision without reading both sides. +- Do not edit the two reviewer files, for any reason. +- Do not run `build_human_review_packets.py`. +- Do not leave `final_decision` blank on any row. +- Your name, date, and attestation must be identical on all 84 rows. + +## 12. When you are finished + +Tell the project maintainer that adjudication is complete. They will re-run the Gate 1 +acceptance audit and, once the remaining `review_date` issue is resolved, Gate 1 can close and +Stage 2 can begin. + +The completed file contains personal attestations. Have it checked before it is committed or +shared.