feat(stage1): register primary outcome candidates

This commit is contained in:
Jinotech
2026-09-20 08:05:50 +12:00
parent 5c31417935
commit fb30b60985
5 changed files with 3057 additions and 11 deletions
+29
View File
@@ -2,6 +2,35 @@
Command failures and integration errors. Command failures and integration errors.
---
## [ERR-20260920-030] gshs-workbook-column-name
**Logged**: 2026-09-20
**Priority**: low
**Status**: resolved
**Area**: docs
### Summary
An exploratory GSHS workbook filter referenced `column` instead of the actual `column_name` field.
### Error
`KeyError: 'column'`
### Context
The workbook schema was printed in the same command and showed the correct field name. The failed command was read-only and changed no data.
### Suggested Fix
Inspect the sheet columns first and filter on `column_name`.
### Metadata
- Reproducible: yes
- Related Files: `Dataset/GSHS-全球学生健康调查数据/GSHS/02_documentation/GSHS_数据字典.xlsx`
- Pattern-Key: runtime.python-exception
### Resolution
- **Resolved**: 2026-09-20
- **Notes**: Re-ran the read-only extraction using `column_name` and obtained the intended candidate rows.
--- ---
## [ERR-20260920-028] python-assignment-expression-comprehension ## [ERR-20260920-028] python-assignment-expression-comprehension
+26 -11
View File
@@ -1,9 +1,9 @@
# 项目进度档案 # 项目进度档案
版本:1.7 版本:1.8
建立日期:2026-09-20 建立日期:2026-09-20
当前总状态:`in_progress` 当前总状态:`in_progress`
当前阶段:阶段 1 `in_progress`输入结构与数据契约首轮审计完成 当前阶段:阶段 1 `in_progress`主要结局来源登记骨架已建立,直接来源复核与双人审核尚未完成
## 1. 管理规则 ## 1. 管理规则
@@ -28,7 +28,7 @@
| 阶段 | 名称 | 状态 | 当前判定 | | 阶段 | 名称 | 状态 | 当前判定 |
|---|---|---|---| |---|---|---|---|
| 0 | 冻结研究问题、范围与版本 | `passed` | 四项交付物齐备,Gate 0 已通过;研究工具链补强与锁定完成 | | 0 | 冻结研究问题、范围与版本 | `passed` | 四项交付物齐备,Gate 0 已通过;研究工具链补强与锁定完成 |
| 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约与三套候选入口结构审计完成;题目来源登记和人工复核尚未完成 | | 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约候选入口结构审计和主要结局来源登记骨架已完成;直接问卷核验、编码规则与人工双重审核尚未完成 |
| 2 | 可行性、可识别性和连接审计 | `not_started` | 等待 Gate 1 | | 2 | 可行性、可识别性和连接审计 | `not_started` | 等待 Gate 1 |
| 3 | 冻结验证协议与已知真值模拟 | `not_started` | 等待 Gate 2 | | 3 | 冻结验证协议与已知真值模拟 | `not_started` | 等待 Gate 2 |
| 4 | 直接协调和传统基线 | `not_started` | 等待 Gate 3 | | 4 | 直接协调和传统基线 | `not_started` | 等待 Gate 3 |
@@ -67,7 +67,7 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
- GSHS 现有说明记录主表 673,499 行、637 列、165 项调查、191 个非重叠组件;这些数字目前属于既有质量报告陈述,尚未由本轮执行重算。 - GSHS 现有说明记录主表 673,499 行、637 列、165 项调查、191 个非重叠组件;这些数字目前属于既有质量报告陈述,尚未由本轮执行重算。
- YRBS 原始库覆盖 19912023;现有整理包说明 19912019 已生成分析表,2021/2023 仍处于导入布局验证隔离状态。 - YRBS 原始库覆盖 19912023;现有整理包说明 19912019 已生成分析表,2021/2023 仍处于导入布局验证隔离状态。
- 系统 Python 3.13.7、Pandoc 3.11、31.82 GiB RAM 和 NVIDIA GeForce RTX 5060 Ti 16 GB 已核验;正式项目环境使用 CPython 3.12.14、PyMC 6.3.2、R 4.6.1、mirt/lavaan/survey 栈及 CUDA Toolkit 13.4.2,三套运行验收均通过。 - 系统 Python 3.13.7、Pandoc 3.11、31.82 GiB RAM 和 NVIDIA GeForce RTX 5060 Ti 16 GB 已核验;正式项目环境使用 CPython 3.12.14、PyMC 6.3.2、R 4.6.1、mirt/lavaan/survey 栈及 CUDA Toolkit 13.4.2,三套运行验收均通过。
- 当前根目录不是 Git 仓库;版本化与审计需使用明确的文件哈希和状态档案,或后续经用户授权建立版本控制 - 当前根目录已建立 Git 仓库`main` 跟踪 `origin/main`;远端为 `Jinovo/language-conditioned-psychometric-harmonization-model`,每项推进按强制协议独立提交并推送
## 6. 文献初筛结论(尚待矩阵固化) ## 6. 文献初筛结论(尚待矩阵固化)
@@ -81,12 +81,13 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
| 优先级 | 问题 | 影响 | 下一处理 | | 优先级 | 问题 | 影响 | 下一处理 |
|---|---|---|---| |---|---|---|---|
| P0 | 官方题目文本、回答标签、适用总体和跳题规则尚未逐项核验 | 阶段 1 确认性分析不能启动 | 在阶段 1 建立来源可追踪题库并保 provisional 状态 | | P0 | 官方题目文本、回答标签、适用总体和跳题规则尚未逐项核验 | Gate 1 确认性分析不能通过 | 来源登记骨架已建立;下一步直接复核官方问卷/代码本,并保 provisional 直到双人审核完成 |
| P0 | 题目共现、锚定图与可识别性尚未重算 | 不能确定 IRT/层级模型是否成立 | 完成阶段 1 后进入阶段 2 审计 | | P0 | 题目共现、锚定图与可识别性尚未重算 | 不能确定 IRT/层级模型是否成立 | 完成阶段 1 后进入阶段 2 审计 |
| P1 | 数据来源许可及可分发范围尚未逐来源完成最终复核 | 影响复现包与衍生数据交付 | 已在清单中显式标为 review required;阶段 1 按实际纳入来源逐项确认,不默认允许再分发 | | P1 | 数据来源许可及可分发范围尚未逐来源完成最终复核 | 影响复现包与衍生数据交付 | 已在清单中显式标为 review required;阶段 1 按实际纳入来源逐项确认,不默认允许再分发 |
| P1 | R、PyMC/ArviZ、心理测量包与 CUDA Toolkit 曾缺失 | 已解决:项目 Python/R 环境和 CUDA 端到端验收通过 | 后续仅按锁定文件维护;任何升级必须重跑三套烟雾测试 | | P1 | R、PyMC/ArviZ、心理测量包与 CUDA Toolkit 曾缺失 | 已解决:项目 Python/R 环境和 CUDA 端到端验收通过 | 后续仅按锁定文件维护;任何升级必须重跑三套烟雾测试 |
| P1 | 文献初筛尚未形成可审计矩阵 | 研究定位和“贡献”表述仍不稳定 | 建立 literature matrix,记录 DOI、来源、证据层级和项目关系 | | P1 | 文献初筛尚未形成可审计矩阵 | 研究定位和“贡献”表述仍不稳定 | 建立 literature matrix,记录 DOI、来源、证据层级和项目关系 |
| P2 | 根目录没有 Git 版本控制 | 变更追踪依赖手工哈希和档案 | 暂不擅自初始化;先维持文件哈希和行动日志 | | P0 | NSDUH 本地完整字典未暴露青少年自杀尝试变量,虽 2022–2024 模块说明提到 `YSUI03``YSUI05` | NSDUH 不能被假定覆盖三项主要结局;跨工具任务可能只支持意念/计划 | 直接核验年度青少年问卷、代码本与公开使用文件抑制/派生说明 |
| P1 | GSHS `_b_` 与无前缀题目版本、YRBS 年度题目文本仍缺直接官方文档复核 | 不能批准题目家族或跨调查锚定 | 按组件/年度建立直接来源证据和冲突清单 |
## 8. 行动日志 ## 8. 行动日志
@@ -673,9 +674,23 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
- 新证据:GSHS 三个设计候选字段各缺 20 行;YRBS 1997 有 1 行设计字段缺失,2017/2019 的 PSU/stratum 全缺且权重仅部分存在;NSDUH 四个权重列在全部年度均非空且为正,需要官方指南解释适用语义。 - 新证据:GSHS 三个设计候选字段各缺 20 行;YRBS 1997 有 1 行设计字段缺失,2017/2019 的 PSU/stratum 全缺且权重仅部分存在;NSDUH 四个权重列在全部年度均非空且为正,需要官方指南解释适用语义。
- 质量边界:所有审核状态默认 `provisional`;本次未批准题目等价、构念、缺失重编码、锚题或复杂抽样方差方法;Gate 1 未通过。 - 质量边界:所有审核状态默认 `provisional`;本次未批准题目等价、构念、缺失重编码、锚题或复杂抽样方差方法;Gate 1 未通过。
- 工程修复:内联聚合的 Python 语法失败和 pandas 内存耗尽分别记录为已解决的 `ERR-20260920-028``ERR-20260920-029`;最终脚本使用标准 CSV 流式扫描。 - 工程修复:内联聚合的 Python 语法失败和 pandas 内存耗尽分别记录为已解决的 `ERR-20260920-028``ERR-20260920-029`;最终脚本使用标准 CSV 流式扫描。
- 推送状态:提交后立即推送 - 推送状态:`pushed`;提交 `5c31417` 已推送至 `origin/main`
- 下一步:建立主要结局来源登记骨架;优先核验 GSHS/NSDUH 的意念、计划、尝试题与适用总体,并追查 YRBS 2017/2019 设计字段缺口。 - 下一步:建立主要结局来源登记骨架;优先核验 GSHS/NSDUH 的意念、计划、尝试题与适用总体,并追查 YRBS 2017/2019 设计字段缺口。
### A-20260920-054:建立主要结局来源登记骨架
- 时间:2026-09-20
- 阶段:1(有来源证据的题库和回答层)
- 动作:完整读取工作流与进度档案;从 GSHS 数据字典、YRBS selected 字典、NSDUH complete 字典及 full/selected Parquet schema 中提取自杀意念、计划、尝试候选;严格分开原始青少年题、派生变量、成人模块、COVID 条件题与复合指标。
- 输入:GSHS `GSHS_数据字典.xlsx``Master_Columns`YRBS `yrbs_selected_variable_dictionary.csv`NSDUH `nsduh_complete_variable_dictionary.csv` 及 20212024 full/selected Parquet schema。
- 输出:`research/stage1/build_primary_outcome_registry.py``primary_outcome_registry.json``primary_outcome_registry.md`;更新 `.learnings/ERRORS.md` 的已解决 `ERR-20260920-030`
- 检查:脚本通过 `py_compile`;JSON 可解析;65 条原始候选全部为 `provisional`、审核者为空、连接状态为 `not_evaluated`;分布为 GSHS 6 条、YRBS 51 条、NSDUH 8 条;另隔离 8 条 NSDUH 青少年派生记录和 40 条成人/COVID/派生或复合排除记录;语义断言和 `git diff --check` 通过。
- 新证据:GSHS 自杀尝试同时存在二分类与次数型版本,不能自动合并;YRBS 每个主要构念有 17 个年度候选,2021/2023 仍只保留元数据且数据隔离;NSDUH 本地完整字典只暴露青少年意念与计划,未找到青少年尝试变量,`YSUI03``YSUI05` 仅在模块迁移说明中出现。
- 质量边界:未直接核验年度 PDF/代码本页码;没有把 AI 检查登记为人类审核;没有批准任何构念等价、缺失重编码、题目家族或锚定关系。Gate 1 仍为 `in_progress`
- 工程修复:一次只读工作簿筛选错误使用 `column` 而非 `column_name`,无数据改动;修正后成功并记录为已解决的 `ERR-20260920-030`
- 推送状态:`push_pending`;完成本行动验证后提交并立即推送。
- 下一步:直接核验 NSDUH 年度青少年问卷/代码本中的 `YSUI03``YSUI05` 与公开数据字段关系,同时建立 GSHS 组件级题目版本来源映射。
## 9. 剩余工作量估算(2026-09-20 基线) ## 9. 剩余工作量估算(2026-09-20 基线)
| 阶段 | 主要工作 | 预计人周 | 主要波动来源 | | 阶段 | 主要工作 | 预计人周 | 主要波动来源 |
@@ -696,8 +711,8 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
下一次行动开始前重新读取 `Document/research-workflow.md` 与本文件。随后: 下一次行动开始前重新读取 `Document/research-workflow.md` 与本文件。随后:
环境补强已经完成。下一行动进入阶段 1 阶段 1 下一行动
1. 冻结阶段 1 数据实体 schema 与审核状态枚举 1. 直接核验 NSDUH 2021–2024 青少年问卷/代码本中的 `YSUI03``YSUI05`,解释其与公开使用字段、跳题和可能抑制规则的关系
2. 只读核验 GSHS、YRBS、NSDUH 候选入口的实际结构、行列数和设计字段 2. 建立 GSHS 组件级题目版本与官方问卷来源映射,保持 `_b_` 与无前缀版本分离
3. 建立主要结局的来源登记骨架,所有未完成实际双重核查的条目标为 `provisional` 3. 随后形成回答编码/缺失规则草案,并把需要两名实际审核者处理的记录导出为审核队列
@@ -0,0 +1,280 @@
"""Build a provisional, source-linked registry for the three primary outcomes.
This script deliberately does not approve cross-survey equivalence, recode
responses, or substitute AI review for the two required human reviewers.
"""
from __future__ import annotations
import html
import json
from collections import Counter
from pathlib import Path
import pandas as pd
import pyarrow.parquet as pq
ROOT = Path(__file__).resolve().parents[2]
STAGE_DIR = ROOT / "research" / "stage1"
GSHS_BOOK = ROOT / "Dataset" / "GSHS-全球学生健康调查数据" / "GSHS" / "02_documentation" / "GSHS_数据字典.xlsx"
YRBS_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "YRBS"
NSDUH_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "NSDUH"
OUT_JSON = STAGE_DIR / "primary_outcome_registry.json"
OUT_MD = STAGE_DIR / "primary_outcome_registry.md"
PRIMARY_CONSTRUCTS = {"suicide_ideation", "suicide_plan", "suicide_attempt"}
GSHS_CONSTRUCT = {
"raw_mh_considersui": "suicide_ideation",
"raw_mh_b_considersui": "suicide_ideation",
"raw_mh_plansui": "suicide_plan",
"raw_mh_b_plansui": "suicide_plan",
"raw_mh_attemptsui": "suicide_attempt",
"raw_mh_b_attemptsui": "suicide_attempt",
}
NSDUH_YOUTH_RAW = {"YUSUITHK": "suicide_ideation", "YUSUIPLN": "suicide_plan"}
NSDUH_YOUTH_DERIVED = {"YUSUITHKYR": "suicide_ideation", "YUSUIPLNYR": "suicide_plan"}
def clean_text(value: object) -> str:
return html.unescape(str(value or "")).strip()
def parse_json_object(value: object) -> object:
text = clean_text(value)
if not text:
return None
try:
return json.loads(text)
except json.JSONDecodeError:
return {"unparsed_source_text": text}
def parquet_columns(path: Path) -> set[str]:
return set(pq.ParquetFile(path).schema_arrow.names)
def build_gshs() -> list[dict]:
frame = pd.read_excel(GSHS_BOOK, sheet_name="Master_Columns", dtype=str, keep_default_na=False)
frame = frame[frame["column_name"].isin(GSHS_CONSTRUCT)].copy()
records = []
for row in frame.sort_values("ordinal").to_dict("records"):
records.append(
{
"survey": "GSHS",
"year": None,
"source_variable": row["column_name"],
"construct": GSHS_CONSTRUCT[row["column_name"]],
"record_type": "raw_candidate",
"population": "school-attending students; exact component ages pending source review",
"time_window": "past 12 months (from source label; questionnaire verification pending)",
"question_text": None,
"source_label": clean_text(row["label_variants"]),
"answer_options": parse_json_object(row["value_labels_example"]),
"source_file": str(GSHS_BOOK.relative_to(ROOT)).replace("\\", "/"),
"source_url": None,
"page_or_section": "Master_Columns",
"present_in_local_data": True,
"source_verification_status": "pending_questionnaire_review",
"review_status": "provisional",
"reviewer_1": None,
"reviewer_2": None,
"link_status": "not_evaluated",
"notes": "The `_b_` and unprefixed forms remain separate item versions; labels alone do not prove equivalence.",
}
)
return records
def build_yrbs() -> list[dict]:
source = YRBS_DIR / "yrbs_selected_variable_dictionary.csv"
frame = pd.read_csv(source, dtype=str, keep_default_na=False)
frame = frame[frame["construct"].isin(PRIMARY_CONSTRUCTS)].copy()
records = []
for row in frame.sort_values(["year", "variable"]).to_dict("records"):
year = int(row["year"])
records.append(
{
"survey": "YRBS",
"year": year,
"source_variable": row["variable"],
"construct": row["construct"],
"record_type": "raw_candidate",
"population": "national high-school students; exact eligibility pending annual source review",
"time_window": clean_text(row["time_window"]),
"question_text": clean_text(row["original_question"]),
"answer_options": parse_json_object(row["answer_options_json"]),
"source_file": str(source.relative_to(ROOT)).replace("\\", "/"),
"source_url": clean_text(row["official_url"]),
"page_or_section": row["questionnaire_version"],
"present_in_local_data": year <= 2019,
"data_status": "available_selected_table" if year <= 2019 else "quarantined_pending_import_validation",
"source_verification_status": "pending_direct_pdf_review",
"review_status": "provisional",
"reviewer_1": None,
"reviewer_2": None,
"link_status": "not_evaluated",
"notes": "Repository dictionary candidate only. Stale source-machine absolute paths are intentionally not copied.",
}
)
return records
def build_nsduh() -> tuple[list[dict], list[dict], list[dict]]:
source = NSDUH_DIR / "nsduh_complete_variable_dictionary.csv"
frame = pd.read_csv(source, dtype=str, keep_default_na=False)
full_cols = parquet_columns(NSDUH_DIR / "nsduh_2021_2024_full.parquet")
selected_cols = parquet_columns(NSDUH_DIR / "nsduh_2021_2024_selected.parquet")
raw_records: list[dict] = []
derived_records: list[dict] = []
excluded_records: list[dict] = []
def common(row: dict, construct: str, record_type: str) -> dict:
variable = row["variable"]
return {
"survey": "NSDUH",
"year": int(row["year"]),
"source_variable": variable,
"construct": construct,
"record_type": record_type,
"population": "youth module; exact age/eligibility and skip logic pending codebook review",
"time_window": "past 12 months",
"question_text": clean_text(row["original_question_or_definition"]) or None,
"source_label": clean_text(row["official_title"]),
"source_file": str(source.relative_to(ROOT)).replace("\\", "/"),
"source_url": clean_text(row["official_source"]),
"page_or_section": variable,
"present_in_full_table": variable in full_cols,
"present_in_selected_table": variable in selected_cols,
"source_verification_status": "official_api_metadata_reviewed_codebook_pending",
"review_status": "provisional",
"reviewer_1": None,
"reviewer_2": None,
"link_status": "not_evaluated",
}
for row in frame[frame["variable"].isin(NSDUH_YOUTH_RAW)].to_dict("records"):
item = common(row, NSDUH_YOUTH_RAW[row["variable"]], "raw_candidate")
item["notes"] = "Keep 'not sure' and refusal/nonresponse distinct until skip and missing rules are verified."
raw_records.append(item)
for row in frame[frame["variable"].isin(NSDUH_YOUTH_DERIVED)].to_dict("records"):
item = common(row, NSDUH_YOUTH_DERIVED[row["variable"]], "derived_youth_indicator")
item["question_text"] = None
item["notes"] = "Derived/recode variable; do not count as an independent administered item."
derived_records.append(item)
exclusion_mask = frame["variable"].str.match(r"^(SUICTHNK|SUIPLANYR|IRSUITRYYR|IRCOSUITRYYR|YUCOSUI|COSUI|ADSUIT)")
for row in frame[exclusion_mask].to_dict("records"):
variable = row["variable"]
reason = "adult_module"
if "COSUI" in variable:
reason = "covid_conditioned"
elif variable.startswith(("IR", "AD")):
reason = "adult_derived_or_composite"
excluded_records.append(
{
"survey": "NSDUH",
"year": int(row["year"]),
"source_variable": variable,
"source_label": clean_text(row["official_title"]),
"exclusion_reason": reason,
"status": "excluded_from_primary_youth_item_candidates",
}
)
return raw_records, derived_records, excluded_records
def render_markdown(registry: dict) -> str:
records = registry["candidate_records"]
counts = Counter((r["survey"], r["construct"]) for r in records)
lines = [
"# Primary outcome source registry",
"",
"Version: 1.0 | Status: provisional | Generated: 2026-09-20",
"",
"## Scope and decision boundary",
"",
"This registry captures source-linked candidates for suicide ideation, plan, and attempt. It is a development artifact, not an approved item bank. No row has completed the two-human-reviewer requirement, no cross-survey link is approved, and no missing or skip code has been canonically recoded.",
"",
"## Candidate coverage",
"",
"| Survey | Ideation | Plan | Attempt | Current evidence boundary |",
"|---|---:|---:|---:|---|",
]
for survey in ["GSHS", "YRBS", "NSDUH"]:
boundary = {
"GSHS": "Workbook labels/options only; component questionnaires still require direct review.",
"YRBS": "51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review.",
"NSDUH": "Official API metadata supports youth ideation/plan only; no youth attempt field appears in the local full dictionary.",
}[survey]
lines.append(
f"| {survey} | {counts[(survey, 'suicide_ideation')]} | {counts[(survey, 'suicide_plan')]} | {counts[(survey, 'suicide_attempt')]} | {boundary} |"
)
lines += [
"",
"Counts are source-record counts, not distinct item families. YRBS counts include annual versions; NSDUH counts include 20212024 raw youth records only.",
"",
"## Material conflicts and exclusions",
"",
"- GSHS exposes both `_b_` and unprefixed variants. They differ in naming and, for attempt, response form (binary versus frequency). They remain separate until questionnaire/component provenance is verified.",
"- GSHS self-harm variables are not included as suicide-attempt items.",
"- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.",
"- YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.",
"- NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.",
"- NSDUH youth attempt questions `YSUI03``YSUI05` are mentioned in notes, but no corresponding youth attempt variable appears in the local complete dictionary. Attempt coverage is therefore recorded as a source gap, not inferred from ideation wording.",
"",
"## Gate 1 status",
"",
"Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct official questionnaire/codebook review, exact population and skip-logic verification, canonical response/missing rules, component-specific GSHS provenance, and two actual human reviewers for primary outcomes.",
"",
"## Reproduction",
"",
"Run `research/stage1/build_primary_outcome_registry.py` in the locked research environment. The script reads repository-local dictionaries and Parquet schemas and rewrites this report plus `primary_outcome_registry.json`.",
"",
]
return "\n".join(lines)
def main() -> None:
gshs = build_gshs()
yrbs = build_yrbs()
nsduh_raw, nsduh_derived, nsduh_excluded = build_nsduh()
registry = {
"schema_version": "1.0",
"generated_on": "2026-09-20",
"status": "provisional",
"allowed_uses": ["pipeline_development", "source_review_queue"],
"prohibited_uses": ["confirmatory_analysis", "approved_anchor_links", "claim_of_human_double_review"],
"candidate_records": gshs + yrbs + nsduh_raw,
"derived_records_not_independent_items": nsduh_derived,
"excluded_nsduh_records": nsduh_excluded,
"source_gaps": [
{
"survey": "NSDUH",
"construct": "suicide_attempt",
"status": "not_found_in_local_complete_dictionary",
"evidence": "YSUI03YSUI05 are named in 20222024 module-move notes, but no youth attempt variable is exposed in the local 20212024 dictionary.",
"next_check": "Review annual youth questionnaires/codebooks and public-use suppression/derivation notes.",
}
],
"review_policy": {
"human_reviewers_required": 2,
"ai_review_is_human_review": False,
"default_review_status": "provisional",
"cross_survey_equivalence_default": "not_evaluated",
},
}
OUT_JSON.write_text(json.dumps(registry, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
OUT_MD.write_text(render_markdown(registry), encoding="utf-8")
print(json.dumps({
"candidate_records": len(registry["candidate_records"]),
"derived_records": len(nsduh_derived),
"excluded_nsduh_records": len(nsduh_excluded),
"json": str(OUT_JSON.relative_to(ROOT)),
"report": str(OUT_MD.relative_to(ROOT)),
}, ensure_ascii=False))
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,34 @@
# Primary outcome source registry
Version: 1.0 | Status: provisional | Generated: 2026-09-20
## Scope and decision boundary
This registry captures source-linked candidates for suicide ideation, plan, and attempt. It is a development artifact, not an approved item bank. No row has completed the two-human-reviewer requirement, no cross-survey link is approved, and no missing or skip code has been canonically recoded.
## Candidate coverage
| Survey | Ideation | Plan | Attempt | Current evidence boundary |
|---|---:|---:|---:|---|
| GSHS | 2 | 2 | 2 | Workbook labels/options only; component questionnaires still require direct review. |
| YRBS | 17 | 17 | 17 | 51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review. |
| NSDUH | 4 | 4 | 0 | Official API metadata supports youth ideation/plan only; no youth attempt field appears in the local full dictionary. |
Counts are source-record counts, not distinct item families. YRBS counts include annual versions; NSDUH counts include 20212024 raw youth records only.
## Material conflicts and exclusions
- GSHS exposes both `_b_` and unprefixed variants. They differ in naming and, for attempt, response form (binary versus frequency). They remain separate until questionnaire/component provenance is verified.
- GSHS self-harm variables are not included as suicide-attempt items.
- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.
- YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.
- NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.
- NSDUH youth attempt questions `YSUI03``YSUI05` are mentioned in notes, but no corresponding youth attempt variable appears in the local complete dictionary. Attempt coverage is therefore recorded as a source gap, not inferred from ideation wording.
## Gate 1 status
Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct official questionnaire/codebook review, exact population and skip-logic verification, canonical response/missing rules, component-specific GSHS provenance, and two actual human reviewers for primary outcomes.
## Reproduction
Run `research/stage1/build_primary_outcome_registry.py` in the locked research environment. The script reads repository-local dictionaries and Parquet schemas and rewrites this report plus `primary_outcome_registry.json`.