fix(stage1): correct YRBS outcome mappings

.learnings/ERRORS.md

@@ -2,6 +2,35 @@
 
 Command failures and integration errors.
 
+---
+## [ERR-20260920-036] pdf-text-console-encoding
+
+**Logged**: 2026-09-20
+**Priority**: low
+**Status**: resolved
+**Area**: docs
+
+### Summary
+An exploratory YRBS PDF text scan stopped when Windows console encoding could not print a soft-hyphen character.
+
+### Error
+`UnicodeEncodeError: 'gbk' codec can't encode character`.
+
+### Context
+PDF extraction itself succeeded; only diagnostic console output failed. Source files were unchanged.
+
+### Suggested Fix
+Configure diagnostic stdout as UTF-8 with replacement before printing extracted PDF text.
+
+### Metadata
+- Reproducible: yes
+- Related Files: `research/stage1/yrbs_primary_outcome_source_audit.json`
+- Pattern-Key: docs.console-encoding
+
+### Resolution
+- **Resolved**: 2026-09-20
+- **Notes**: The scan was rerun with UTF-8 stdout and all annual source pages were reviewed.
+
 ---
 ## [ERR-20260920-035] gshs-questionnaire-directory-assumption
 

Progression.md

@@ -1,9 +1,9 @@
 # 项目进度档案
 
-版本：1.10
+版本：1.11
 建立日期：2026-09-20  
 当前总状态：`in_progress`  
-当前阶段：阶段 1 `in_progress`；NSDUH 青少年尝试题及 GSHS 191 组件覆盖/原始与二元表示关系已核清，YRBS 直接来源、缺失规则和双人审核尚未完成
+当前阶段：阶段 1 `in_progress`；NSDUH、GSHS 与 YRBS 主要结局来源均已直接核验，YRBS 21 条错误映射已纠正；缺失规则、Solomon Islands 来源和双人审核尚未完成
 
 ## 1. 管理规则
 
@@ -28,7 +28,7 @@
 | 阶段 | 名称 | 状态 | 当前判定 |
 |---|---|---|---|
 | 0 | 冻结研究问题、范围与版本 | `passed` | 四项交付物齐备，Gate 0 已通过；研究工具链补强与锁定完成 |
-| 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约、入口审计、主要结局登记、NSDUH 尝试题来源核验及 GSHS 191 组件审计已完成；YRBS 直接来源、编码/缺失规则与人工双重审核尚未完成 |
+| 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约、入口审计及三套调查主要结局直接来源核验已完成；YRBS 错误列映射已纠正；缺失规则、Solomon Islands 来源、2021/2023 导入和人工双重审核尚未完成 |
 | 2 | 可行性、可识别性和连接审计 | `not_started` | 等待 Gate 1 |
 | 3 | 冻结验证协议与已知真值模拟 | `not_started` | 等待 Gate 2 |
 | 4 | 直接协调和传统基线 | `not_started` | 等待 Gate 3 |
@@ -87,7 +87,7 @@ Gate 0 当前判定：`passed`。四项交付物齐备；目标总体政策、
 | P1 | R、PyMC/ArviZ、心理测量包与 CUDA Toolkit 曾缺失 | 已解决：项目 Python/R 环境和 CUDA 端到端验收通过 | 后续仅按锁定文件维护；任何升级必须重跑三套烟雾测试 |
 | P1 | 文献初筛尚未形成可审计矩阵 | 研究定位和“贡献”表述仍不稳定 | 建立 literature matrix，记录 DOI、来源、证据层级和项目关系 |
 | P0 | NSDUH 的 `YSUI03/YUSUICTRY` 已确认施测，但尝试及严重度字段不在当前本地公开使用数据中；2024 指南明确记录派生尝试变量因披露风险排除 | 当前微数据只能外部验证意念/计划，不能验证尝试 | 不从其他题推导尝试；如核心设计必须使用尝试，评估受限数据申请或预先设计的聚合结果方案 |
-| P1 | GSHS 已证明 `_b_` 为派生/重复表示，但 Solomon Islands 2023 缺本地直接来源；YRBS 年度题目文本仍缺直接官方文档复核 | 不能批准全部题目家族或跨调查锚定 | 获取 Solomon Islands 来源；按年度复核 YRBS 并建立冲突清单 |
+| P1 | GSHS 已证明 `_b_` 为派生/重复表示，但 Solomon Islands 2023 缺本地直接来源；YRBS 2021/2023 仍处于导入隔离 | 不能批准全部题目家族或把后两年纳入回答层 | 获取 Solomon Islands 来源；按官方 SAS 布局验证 YRBS 2021/2023 导入 |
 
 ## 8. 行动日志
 
@@ -721,6 +721,20 @@ Gate 0 当前判定：`passed`。四项交付物齐备；目标总体政策、
 - 推送状态：`pushed`；研究增量提交 `f0a4538` 已推送至 `origin/main`。
 - 下一步：直接复核 YRBS 年度主要结局问卷/代码本，继续隔离 2021/2023 数据，并形成三套调查的回答编码与缺失规则草案。
 
+### A-20260920-057：完成 YRBS 年度主要结局直接来源审计并纠正列映射
+
+- 时间：2026-09-20
+- 阶段：1（有来源证据的题库和回答层）
+- 动作：行动前完整读取工作流与进度档案并核对干净 Git 状态；逐年读取 1991–2023 共 17 份国家高中问卷和对应数据用户指南/代码本；分开登记“问卷印刷题号”与“发布数据变量号”；对 1991–2019 年度完整 Parquet 的正确列执行值域检查，并视觉复核 1995 与 2023 的代表性来源页。
+- 输入：`documentation/Questionnaire` 的 17 份官方问卷、`documentation/UserGuide` 的 17 份官方指南、15 个 1991–2019 年度 full Parquet、现有 `yrbs_selected_variable_dictionary.csv`。
+- 输出：新增 `audit_yrbs_primary_outcome_sources.py`、`yrbs_primary_outcome_source_audit.json`、`yrbs_primary_outcome_source_audit.md`；主要结局登记改用直接来源 crosswalk；更新 `.learnings/ERRORS.md`。
+- 检查：51 条主要结局记录均有问卷和数据指南来源；15 个可用年度的正确列均存在，观测编码全部落在官方值域；2021/2023 保持隔离；脚本和 JSON 校验通过。
+- 新证据：现有登记在 1995、2001、2003、2005、2007、2009、2011 共 7 年把 21 条主要结局映射到错误数据列。原因是把印刷问卷题号、数据用户指南变量号或自动提取结果混用；例如 1995 正确列为 `Q22/Q23/Q24`，旧登记却为 `Q30/Q31/Q32`。正确列仍存在于年度完整数据，因此属于 crosswalk 缺陷而非原始数据丢失。登记现已改为用户指南变量，并单独保留印刷题号。
+- 质量边界：年度指南只把无值统一报告为 Missing，未区分拒答、合法跳题或数据错误；不能无依据细分。问卷为英语来源，不自动证明所有语言便利措施。两名实际审核者仍未签字，Gate 1 保持 `in_progress`。
+- 工程记录：PDF 诊断文本首次受 Windows 控制台编码影响中止，改用 UTF-8 输出后完成，记为已解决 `ERR-20260920-036`；Poppler 字体警告未影响关键页视觉内容。
+- 推送状态：`push_pending`；待本行动校验、提交并推送后回写提交号。
+- 下一步：建立三套调查主要结局的正式回答字典与保守缺失规则，生成双人审核队列；并定位 Solomon Islands 2023 GSHS 直接来源。
+
 ## 9. 剩余工作量估算（2026-09-20 基线）
 
 | 阶段 | 主要工作 | 预计人周 | 主要波动来源 |
@@ -743,6 +757,6 @@ Gate 0 当前判定：`passed`。四项交付物齐备；目标总体政策、
 
 阶段 1 下一行动：
 
-1. 直接复核 YRBS 年度主要结局题目，并继续隔离 2021/2023 数据；
+1. 形成三套调查主要结局的回答编码/缺失规则草案，并把需要两名实际审核者处理的记录导出为审核队列；
 2. 获取或定位 Solomon Islands 2023 GSHS 的国家问卷/代码本；
-3. 随后形成三套调查的回答编码/缺失规则草案，并把需要两名实际审核者处理的记录导出为审核队列。
+3. 按官方 SAS 布局验证 YRBS 2021/2023 导入，在验证前继续隔离。

research/stage1/audit_yrbs_primary_outcome_sources.py

@@ -0,0 +1,253 @@
+"""Cross-check YRBS primary outcomes against annual questionnaires and user guides."""
+
+from __future__ import annotations
+
+import csv
+import hashlib
+import json
+from collections import Counter
+from pathlib import Path
+
+import pyarrow.parquet as pq
+
+
+ROOT = Path(__file__).resolve().parents[2]
+STAGE_DIR = ROOT / "research" / "stage1"
+YRBS_ROOT = ROOT / "Dataset" / "YRBS_National_1991_2023"
+PACKAGE = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "YRBS"
+DICTIONARY = PACKAGE / "yrbs_selected_variable_dictionary.csv"
+OUT_JSON = STAGE_DIR / "yrbs_primary_outcome_source_audit.json"
+OUT_MD = STAGE_DIR / "yrbs_primary_outcome_source_audit.md"
+
+CONSTRUCTS = ["suicide_ideation", "suicide_plan", "suicide_attempt"]
+QUESTION_TEXT = {
+    "suicide_ideation": "During the past 12 months, did you ever seriously consider attempting suicide?",
+    "suicide_plan": "During the past 12 months, did you make a plan about how you would attempt suicide?",
+    "suicide_attempt": "During the past 12 months, how many times did you actually attempt suicide?",
+}
+OPTIONS = {
+    "suicide_ideation": {"1": "Yes", "2": "No"},
+    "suicide_plan": {"1": "Yes", "2": "No"},
+    "suicide_attempt": {
+        "1": "0 times",
+        "2": "1 time",
+        "3": "2 or 3 times",
+        "4": "4 or 5 times",
+        "5": "6 or more times",
+    },
+}
+
+# Data-variable numbers come from the annual user guides/codebooks. Printed
+# questionnaire numbers are retained separately because the two differ in some
+# years and are not interchangeable.
+YEAR_SOURCES = {
+    1991: (19, 19, "1991_hs_questionnaire.pdf", "page 5", "YRBS_1991_National_User_Guide.pdf", "page 9"),
+    1993: (24, 23, "1993_xxh_questionnaire.pdf", "page 6", "YRBS_1993_National_User_Guide.pdf", "page 11"),
+    1995: (22, 22, "1995_xxh_questionnaire.pdf", "page 6", "YRBS_1995_National_User_Guide.pdf", "pages 11-12"),
+    1997: (22, 21, "1997_xxh_questionnaire.pdf", "page 6", "YRBS_1997_National_User_Guide.pdf", "pages 11-12"),
+    1999: (23, 23, "1999_xxh_questionnaire.pdf", "page 7", "YRBS_1999_National_User_Guide.pdf", "page 7"),
+    2001: (24, 25, "2001_xxh_questionnaire.pdf", "page 7", "YRBS_2001_National_User_Guide.pdf", "pages 8-9"),
+    2003: (24, 25, "2003_xxh_questionnaire.pdf", "page 7", "YRBS_2003_National_User_Guide.pdf", "pages 9-10"),
+    2005: (24, 26, "2005_xxh_questionnaire.pdf", "page 7", "YRBS_2005_National_User_Guide.pdf", "pages 16-17"),
+    2007: (24, 25, "2007_xxh_questionnaire.pdf", "page 7", "YRBS_2007_National_User_Guide.pdf", "page 20"),
+    2009: (24, 25, "2009_xxh_questionnaire.pdf", "page 7", "YRBS_2009_National_User_Guide.pdf", "page 22"),
+    2011: (25, 27, "2011_xxh_questionnaire.pdf", "pages 7-8", "YRBS_2011_National_User_Guide.pdf", "page 20"),
+    2013: (27, 27, "2013_xxh_questionnaire.pdf", "page 8", "YRBS_2013_National_User_Guide.pdf", "page 28"),
+    2015: (27, 27, "2015_xxh_questionnaire.pdf", "pages 7-8", "2015_yrbs-data-users_guide_smy_combined.pdf", "page 29"),
+    2017: (26, 27, "2017_yrbs_national_hs_questionnaire.pdf", "page 8", "2017_YRBS_Data_Users_Guide.pdf", "page 29"),
+    2019: (26, 26, "2019_YRBS-National-HS-Questionnaire.pdf", "page 8", "2019_National_YRBS_Data_Users_Guide.pdf", "pages 27-28"),
+    2021: (26, 26, "2021-YRBS-National-HS-Questionnaire.pdf", "page 8", "2021_YRBS_Data_Users_Guide_508.pdf", "page 28"),
+    2023: (27, 31, "2023_YRBS_National_HS_Questionnaire.pdf", "page 9", "2023_National_YRBS_Data_Users_Guide508.pdf", "page 29"),
+}
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def dictionary_rows() -> dict[tuple[int, str], dict]:
+    with DICTIONARY.open(encoding="utf-8-sig", newline="") as handle:
+        rows = list(csv.DictReader(handle))
+    selected = {
+        (int(row["year"]), row["construct"]): row
+        for row in rows
+        if row["construct"] in CONSTRUCTS
+    }
+    if len(selected) != 51:
+        raise AssertionError(f"Expected 51 existing primary-outcome dictionary rows, found {len(selected)}")
+    return selected
+
+
+def value_counts(path: Path, variable: str) -> dict:
+    table = pq.read_table(path, columns=[variable])
+    values = table.column(variable).to_pylist()
+    counts = Counter("<missing>" if value is None or str(value).strip() == "" else str(value) for value in values)
+    return {key: counts[key] for key in sorted(counts)}
+
+
+def build_audit() -> dict:
+    existing = dictionary_rows()
+    items = []
+    source_files: dict[str, dict] = {}
+    mismatch_years = set()
+    questionnaire_data_number_differences = set()
+
+    for year, (data_start, questionnaire_start, q_name, q_section, g_name, g_section) in YEAR_SOURCES.items():
+        questionnaire = YRBS_ROOT / "documentation" / "Questionnaire" / q_name
+        guide = YRBS_ROOT / "documentation" / "UserGuide" / g_name
+        for path, role in [(questionnaire, "questionnaire"), (guide, "data_user_guide")]:
+            if not path.exists():
+                raise FileNotFoundError(path)
+            relative = str(path.relative_to(ROOT)).replace("\\", "/")
+            source_files.setdefault(relative, {"role": role, "sha256": sha256(path)})
+
+        full_path = PACKAGE / "full_by_year" / f"yrbs_{year}_full.parquet"
+        full_schema = set(pq.ParquetFile(full_path).schema_arrow.names) if year <= 2019 else set()
+        for offset, construct in enumerate(CONSTRUCTS):
+            official_variable = f"q{data_start + offset}"
+            questionnaire_number = questionnaire_start + offset
+            old = existing[(year, construct)]
+            old_variable = old["variable"].lower()
+            mapping_matches = old_variable == official_variable
+            if not mapping_matches:
+                mismatch_years.add(year)
+            if data_start != questionnaire_start:
+                questionnaire_data_number_differences.add(year)
+
+            if year <= 2019:
+                if official_variable not in full_schema:
+                    raise AssertionError(f"{year} full table lacks {official_variable}")
+                official_counts = value_counts(full_path, official_variable)
+                previous_counts = value_counts(full_path, old_variable) if old_variable in full_schema else None
+                observed_codes = {key for key in official_counts if key != "<missing>"}
+                expected_codes = set(OPTIONS[construct])
+                code_domain_status = "passed" if observed_codes <= expected_codes else "failed"
+                data_status = "available_full_table"
+            else:
+                official_counts = None
+                previous_counts = None
+                code_domain_status = "not_tested_quarantined"
+                data_status = "quarantined_pending_import_layout_validation"
+
+            items.append(
+                {
+                    "survey": "YRBS",
+                    "year": year,
+                    "construct": construct,
+                    "source_variable": official_variable.upper(),
+                    "previous_registry_variable": old["variable"],
+                    "mapping_status": "verified" if mapping_matches else "corrected_from_previous_registry",
+                    "questionnaire_question_number": questionnaire_number,
+                    "data_user_guide_variable_number": data_start + offset,
+                    "question_text": QUESTION_TEXT[construct],
+                    "time_window": "past 12 months",
+                    "response_type": "binary" if construct != "suicide_attempt" else "ordinal_frequency",
+                    "answer_options": OPTIONS[construct],
+                    "population": "students in grades 9-12 attending public and private schools in the United States; detailed eligibility remains subject to annual design review",
+                    "language": "English questionnaire source; actual accommodation/language coverage not established by this audit",
+                    "questionnaire_source_file": str(questionnaire.relative_to(ROOT)).replace("\\", "/"),
+                    "questionnaire_page_or_section": q_section,
+                    "data_guide_source_file": str(guide.relative_to(ROOT)).replace("\\", "/"),
+                    "data_guide_page_or_section": g_section,
+                    "data_status": data_status,
+                    "official_value_counts": official_counts,
+                    "previous_mapped_value_counts": previous_counts if not mapping_matches else official_counts,
+                    "code_domain_status": code_domain_status,
+                    "missing_rule_status": "blank/null retained as unresolved missing; annual guide reports Missing but does not distinguish reason",
+                    "review_status": "provisional",
+                    "reviewer_1": None,
+                    "reviewer_2": None,
+                }
+            )
+
+    return {
+        "schema_version": "1.0",
+        "generated_on": "2026-09-20",
+        "status": "provisional",
+        "purpose": "Gate 1 direct-source audit and question-number/data-variable crosswalk for YRBS primary outcomes.",
+        "inputs": {
+            "previous_dictionary": str(DICTIONARY.relative_to(ROOT)).replace("\\", "/"),
+            "previous_dictionary_sha256": sha256(DICTIONARY),
+            "source_files": source_files,
+        },
+        "summary": {
+            "years_reviewed": 17,
+            "items_reviewed": len(items),
+            "years_available_in_full_tables": 15,
+            "years_quarantined": 2,
+            "previous_mapping_mismatch_years": sorted(mismatch_years),
+            "previous_mapping_mismatch_items": sum(item["mapping_status"] != "verified" for item in items),
+            "questionnaire_data_number_difference_years": sorted(questionnaire_data_number_differences),
+            "code_domain_failures": sum(item["code_domain_status"] == "failed" for item in items),
+        },
+        "items": items,
+        "decision": {
+            "registry_action": "replace prior YRBS primary-outcome variable mappings with this direct-source crosswalk",
+            "data_action": "use verified data-user-guide variables from annual full tables; do not infer data columns from printed questionnaire numbers",
+            "quarantine": "retain 2021 and 2023 as metadata-only until their imports and layouts are independently validated",
+            "gate_1": "in_progress",
+        },
+    }
+
+
+def render_markdown(audit: dict) -> str:
+    summary = audit["summary"]
+    rows_by_year = {}
+    for item in audit["items"]:
+        rows_by_year.setdefault(item["year"], []).append(item)
+    lines = [
+        "# YRBS primary-outcome source audit",
+        "",
+        "Version: 1.0 | Status: provisional | Generated: 2026-09-20",
+        "",
+        "## Result",
+        "",
+        "All 17 national questionnaires and their annual data user guides were directly cross-checked for suicide ideation, plan, and attempt. The wording and answer options are stable across 1991-2023, but printed questionnaire numbers are not always the same as the variable numbers used in the released data.",
+        "",
+        f"The previous registry mapped {summary['previous_mapping_mismatch_items']} items in {len(summary['previous_mapping_mismatch_years'])} years to the wrong data variable: {', '.join(map(str, summary['previous_mapping_mismatch_years']))}. The correct columns are present in the annual 1991-2019 full Parquet files, and all observed codes stay within the official response domains. This is a registry/crosswalk defect, not loss of the underlying columns.",
+        "",
+        "## Annual crosswalk",
+        "",
+        "| Year | Data variables (ideation/plan/attempt) | Printed questionnaire numbers | Previous mapping | Data status |",
+        "|---:|---|---|---|---|",
+    ]
+    for year, items in sorted(rows_by_year.items()):
+        data_vars = "/".join(item["source_variable"] for item in items)
+        q_nums = "/".join(str(item["questionnaire_question_number"]) for item in items)
+        previous = "/".join(item["previous_registry_variable"] for item in items)
+        status = "verified" if all(item["mapping_status"] == "verified" for item in items) else "corrected"
+        lines.append(f"| {year} | `{data_vars}` | {q_nums} | {previous} ({status}) | {items[0]['data_status']} |")
+    lines += [
+        "",
+        "## Source interpretation",
+        "",
+        "- Data user guides/codebooks govern the released data-variable mapping; printed questionnaire numbers are retained as separate provenance fields.",
+        "- Ideation and plan use 1=Yes and 2=No. Attempt uses 1=0 times, 2=1 time, 3=2 or 3 times, 4=4 or 5 times, and 5=6 or more times.",
+        "- The 2023 printed questionnaire explicitly routes 0 attempts past the injury follow-up. That routing does not alter the primary attempt-frequency response itself.",
+        "- Annual guides report generic Missing counts but do not identify refusal, legitimate skip, data error, or other missing reasons for these primary items. Missing canonicalization therefore remains unresolved.",
+        "",
+        "## Gate 1 decision",
+        "",
+        "Gate 1 remains **in progress**. The 51 YRBS primary-outcome source records now have a direct-source crosswalk and clean response options, but 2021/2023 respondent imports remain quarantined, missing reasons require a conservative rule, and two actual human reviewers have not yet signed the primary outcomes.",
+        "",
+        "## Reproduction",
+        "",
+        "Run `research/stage1/audit_yrbs_primary_outcome_sources.py` in the project Python environment. It verifies source hashes, checks the 1991-2019 annual Parquet schemas, and compares official versus previous mappings.",
+        "",
+    ]
+    return "\n".join(lines)
+
+
+def main() -> None:
+    audit = build_audit()
+    OUT_JSON.write_text(json.dumps(audit, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+    OUT_MD.write_text(render_markdown(audit), encoding="utf-8")
+    print(json.dumps(audit["summary"], ensure_ascii=False))
+
+
+if __name__ == "__main__":
+    main()

research/stage1/build_primary_outcome_registry.py

@@ -19,6 +19,7 @@ ROOT = Path(__file__).resolve().parents[2]
 STAGE_DIR = ROOT / "research" / "stage1"
 GSHS_BOOK = ROOT / "Dataset" / "GSHS-全球学生健康调查数据" / "GSHS" / "02_documentation" / "GSHS_数据字典.xlsx"
 YRBS_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "YRBS"
+YRBS_AUDIT = STAGE_DIR / "yrbs_primary_outcome_source_audit.json"
 NSDUH_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "NSDUH"
 OUT_JSON = STAGE_DIR / "primary_outcome_registry.json"
 OUT_MD = STAGE_DIR / "primary_outcome_registry.md"
@@ -124,34 +125,37 @@ def build_gshs() -> tuple[list[dict], list[dict]]:
 
 
 def build_yrbs() -> list[dict]:
-    source = YRBS_DIR / "yrbs_selected_variable_dictionary.csv"
+    audit = json.loads(YRBS_AUDIT.read_text(encoding="utf-8"))
-    frame = pd.read_csv(source, dtype=str, keep_default_na=False)
-    frame = frame[frame["construct"].isin(PRIMARY_CONSTRUCTS)].copy()
     records = []
-    for row in frame.sort_values(["year", "variable"]).to_dict("records"):
+    for row in sorted(audit["items"], key=lambda item: (item["year"], item["construct"])):
-        year = int(row["year"])
+        year = row["year"]
         records.append(
             {
                 "survey": "YRBS",
                 "year": year,
-                "source_variable": row["variable"],
+                "source_variable": row["source_variable"],
                 "construct": row["construct"],
                 "record_type": "raw_candidate",
-                "population": "national high-school students; exact eligibility pending annual source review",
+                "population": row["population"],
-                "time_window": clean_text(row["time_window"]),
+                "time_window": row["time_window"],
-                "question_text": clean_text(row["original_question"]),
+                "question_text": row["question_text"],
-                "answer_options": parse_json_object(row["answer_options_json"]),
+                "answer_options": row["answer_options"],
-                "source_file": str(source.relative_to(ROOT)).replace("\\", "/"),
+                "source_file": row["questionnaire_source_file"],
-                "source_url": clean_text(row["official_url"]),
+                "data_guide_source_file": row["data_guide_source_file"],
-                "page_or_section": row["questionnaire_version"],
+                "source_url": "https://www.cdc.gov/yrbs/questionnaires/index.html",
+                "page_or_section": f"questionnaire {row['questionnaire_page_or_section']}; data guide {row['data_guide_page_or_section']}",
+                "questionnaire_question_number": row["questionnaire_question_number"],
+                "data_user_guide_variable_number": row["data_user_guide_variable_number"],
+                "previous_registry_variable": row["previous_registry_variable"],
+                "mapping_status": row["mapping_status"],
                 "present_in_local_data": year <= 2019,
-                "data_status": "available_selected_table" if year <= 2019 else "quarantined_pending_import_validation",
+                "data_status": row["data_status"],
-                "source_verification_status": "pending_direct_pdf_review",
+                "source_verification_status": "direct_questionnaire_and_data_user_guide_reviewed",
                 "review_status": "provisional",
                 "reviewer_1": None,
                 "reviewer_2": None,
                 "link_status": "not_evaluated",
-                "notes": "Repository dictionary candidate only. Stale source-machine absolute paths are intentionally not copied.",
+                "notes": "Direct-source crosswalk supersedes the previous automatic mapping. Printed questionnaire number and released data variable are retained separately; missing reasons and human review remain unresolved.",
             }
         )
     return records
@@ -266,7 +270,7 @@ def render_markdown(registry: dict) -> str:
     lines = [
         "# Primary outcome source registry",
         "",
-        "Version: 1.2 | Status: provisional | Generated: 2026-09-20",
+        "Version: 1.3 | Status: provisional | Generated: 2026-09-20",
         "",
         "## Scope and decision boundary",
         "",
@@ -280,7 +284,7 @@ def render_markdown(registry: dict) -> str:
     for survey in ["GSHS", "YRBS", "NSDUH"]:
         boundary = {
             "GSHS": "Three administered items occur in 7/191 components; local direct sources cover 6/7, with Solomon Islands pending.",
-            "YRBS": "51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review.",
+            "YRBS": "51 annual candidates directly cross-checked; 21 prior mappings corrected; 2021/2023 data remain quarantined.",
             "NSDUH": "Attempt is administered as YSUI03/YUSUICTRY, but the raw and recoded attempt fields are absent from the local public-use package.",
         }[survey]
         lines.append(
@@ -294,7 +298,7 @@ def render_markdown(registry: dict) -> str:
         "",
         "- GSHS ordinary codebooks identify the three unprefixed variables as administered items. Binary Codebooks plus exact row-level relations identify the three `_b_` columns as derived or duplicate representations, not independent items.",
         "- GSHS self-harm variables are not included as suicide-attempt items.",
-        "- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.",
+        "- Direct annual questionnaire/user-guide review found 21 wrong data-column mappings in 1995, 2001, 2003, 2005, 2007, 2009, and 2011. This registry now uses the corrected user-guide variables and retains printed questionnaire numbers separately.",
         "- YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.",
         "- NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.",
         "- Official restricted-use codebooks and the 2024 questionnaire verify `YSUI03`/`YUSUICTRY` as the youth attempt item. It is administered independently of the ideation response, but is absent from the local public-use data and cannot be analyzed from the current package.",
@@ -302,7 +306,7 @@ def render_markdown(registry: dict) -> str:
         "",
         "## Gate 1 status",
         "",
-        "Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct YRBS and Solomon Islands GSHS source review, exact population and skip-logic verification, canonical response/missing rules, and two actual human reviewers for primary outcomes.",
+        "Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs the Solomon Islands GSHS source, exact population and skip-logic verification, canonical response/missing rules, validated YRBS 2021/2023 imports, and two actual human reviewers for primary outcomes.",
         "",
         "## Reproduction",
         "",
@@ -317,7 +321,7 @@ def main() -> None:
     yrbs = build_yrbs()
     nsduh_raw, nsduh_derived, nsduh_excluded = build_nsduh()
     registry = {
-        "schema_version": "1.2",
+        "schema_version": "1.3",
         "generated_on": "2026-09-20",
         "status": "provisional",
         "allowed_uses": ["pipeline_development", "source_review_queue"],

research/stage1/primary_outcome_registry.md

@@ -1,6 +1,6 @@
 # Primary outcome source registry
 
-Version: 1.2 | Status: provisional | Generated: 2026-09-20
+Version: 1.3 | Status: provisional | Generated: 2026-09-20
 
 ## Scope and decision boundary
 
@@ -11,7 +11,7 @@ This registry captures source-linked candidates for suicide ideation, plan, and
 | Survey | Ideation | Plan | Attempt | Current evidence boundary |
 |---|---:|---:|---:|---|
 | GSHS | 1 | 1 | 1 | Three administered items occur in 7/191 components; local direct sources cover 6/7, with Solomon Islands pending. |
-| YRBS | 17 | 17 | 17 | 51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review. |
+| YRBS | 17 | 17 | 17 | 51 annual candidates directly cross-checked; 21 prior mappings corrected; 2021/2023 data remain quarantined. |
 | NSDUH | 4 | 4 | 4 | Attempt is administered as YSUI03/YUSUICTRY, but the raw and recoded attempt fields are absent from the local public-use package. |
 
 Counts are source-record counts, not distinct item families. YRBS counts include annual versions; NSDUH counts include 2021–2024 raw youth records only.
@@ -20,7 +20,7 @@ Counts are source-record counts, not distinct item families. YRBS counts include
 
 - GSHS ordinary codebooks identify the three unprefixed variables as administered items. Binary Codebooks plus exact row-level relations identify the three `_b_` columns as derived or duplicate representations, not independent items.
 - GSHS self-harm variables are not included as suicide-attempt items.
-- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.
+- Direct annual questionnaire/user-guide review found 21 wrong data-column mappings in 1995, 2001, 2003, 2005, 2007, 2009, and 2011. This registry now uses the corrected user-guide variables and retains printed questionnaire numbers separately.
 - YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.
 - NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.
 - Official restricted-use codebooks and the 2024 questionnaire verify `YSUI03`/`YUSUICTRY` as the youth attempt item. It is administered independently of the ideation response, but is absent from the local public-use data and cannot be analyzed from the current package.
@@ -28,7 +28,7 @@ Counts are source-record counts, not distinct item families. YRBS counts include
 
 ## Gate 1 status
 
-Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct YRBS and Solomon Islands GSHS source review, exact population and skip-logic verification, canonical response/missing rules, and two actual human reviewers for primary outcomes.
+Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs the Solomon Islands GSHS source, exact population and skip-logic verification, canonical response/missing rules, validated YRBS 2021/2023 imports, and two actual human reviewers for primary outcomes.
 
 ## Reproduction
 

research/stage1/yrbs_primary_outcome_source_audit.md

@@ -0,0 +1,46 @@
+# YRBS primary-outcome source audit
+
+Version: 1.0 | Status: provisional | Generated: 2026-09-20
+
+## Result
+
+All 17 national questionnaires and their annual data user guides were directly cross-checked for suicide ideation, plan, and attempt. The wording and answer options are stable across 1991-2023, but printed questionnaire numbers are not always the same as the variable numbers used in the released data.
+
+The previous registry mapped 21 items in 7 years to the wrong data variable: 1995, 2001, 2003, 2005, 2007, 2009, 2011. The correct columns are present in the annual 1991-2019 full Parquet files, and all observed codes stay within the official response domains. This is a registry/crosswalk defect, not loss of the underlying columns.
+
+## Annual crosswalk
+
+| Year | Data variables (ideation/plan/attempt) | Printed questionnaire numbers | Previous mapping | Data status |
+|---:|---|---|---|---|
+| 1991 | `Q19/Q20/Q21` | 19/20/21 | Q19/Q20/Q21 (verified) | available_full_table |
+| 1993 | `Q24/Q25/Q26` | 23/24/25 | Q24/Q25/Q26 (verified) | available_full_table |
+| 1995 | `Q22/Q23/Q24` | 22/23/24 | Q30/Q31/Q32 (corrected) | available_full_table |
+| 1997 | `Q22/Q23/Q24` | 21/22/23 | Q22/Q23/Q24 (verified) | available_full_table |
+| 1999 | `Q23/Q24/Q25` | 23/24/25 | Q23/Q24/Q25 (verified) | available_full_table |
+| 2001 | `Q24/Q25/Q26` | 25/26/27 | Q25/Q26/Q27 (corrected) | available_full_table |
+| 2003 | `Q24/Q25/Q26` | 25/26/27 | Q25/Q26/Q27 (corrected) | available_full_table |
+| 2005 | `Q24/Q25/Q26` | 26/27/28 | Q26/Q27/Q28 (corrected) | available_full_table |
+| 2007 | `Q24/Q25/Q26` | 25/26/27 | Q25/Q26/Q27 (corrected) | available_full_table |
+| 2009 | `Q24/Q25/Q26` | 25/26/27 | Q25/Q26/Q27 (corrected) | available_full_table |
+| 2011 | `Q25/Q26/Q27` | 27/28/29 | Q27/Q28/Q29 (corrected) | available_full_table |
+| 2013 | `Q27/Q28/Q29` | 27/28/29 | Q27/Q28/Q29 (verified) | available_full_table |
+| 2015 | `Q27/Q28/Q29` | 27/28/29 | Q27/Q28/Q29 (verified) | available_full_table |
+| 2017 | `Q26/Q27/Q28` | 27/28/29 | Q26/Q27/Q28 (verified) | available_full_table |
+| 2019 | `Q26/Q27/Q28` | 26/27/28 | Q26/Q27/Q28 (verified) | available_full_table |
+| 2021 | `Q26/Q27/Q28` | 26/27/28 | Q26/Q27/Q28 (verified) | quarantined_pending_import_layout_validation |
+| 2023 | `Q27/Q28/Q29` | 31/32/33 | Q27/Q28/Q29 (verified) | quarantined_pending_import_layout_validation |
+
+## Source interpretation
+
+- Data user guides/codebooks govern the released data-variable mapping; printed questionnaire numbers are retained as separate provenance fields.
+- Ideation and plan use 1=Yes and 2=No. Attempt uses 1=0 times, 2=1 time, 3=2 or 3 times, 4=4 or 5 times, and 5=6 or more times.
+- The 2023 printed questionnaire explicitly routes 0 attempts past the injury follow-up. That routing does not alter the primary attempt-frequency response itself.
+- Annual guides report generic Missing counts but do not identify refusal, legitimate skip, data error, or other missing reasons for these primary items. Missing canonicalization therefore remains unresolved.
+
+## Gate 1 decision
+
+Gate 1 remains **in progress**. The 51 YRBS primary-outcome source records now have a direct-source crosswalk and clean response options, but 2021/2023 respondent imports remain quarantined, missing reasons require a conservative rule, and two actual human reviewers have not yet signed the primary outcomes.
+
+## Reproduction
+
+Run `research/stage1/audit_yrbs_primary_outcome_sources.py` in the project Python environment. It verifies source hashes, checks the 1991-2019 annual Parquet schemas, and compares official versus previous mappings.