feat(stage1): register primary outcome candidates
This commit is contained in:
@@ -2,6 +2,35 @@
|
|||||||
|
|
||||||
Command failures and integration errors.
|
Command failures and integration errors.
|
||||||
|
|
||||||
|
---
|
||||||
|
## [ERR-20260920-030] gshs-workbook-column-name
|
||||||
|
|
||||||
|
**Logged**: 2026-09-20
|
||||||
|
**Priority**: low
|
||||||
|
**Status**: resolved
|
||||||
|
**Area**: docs
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
An exploratory GSHS workbook filter referenced `column` instead of the actual `column_name` field.
|
||||||
|
|
||||||
|
### Error
|
||||||
|
`KeyError: 'column'`
|
||||||
|
|
||||||
|
### Context
|
||||||
|
The workbook schema was printed in the same command and showed the correct field name. The failed command was read-only and changed no data.
|
||||||
|
|
||||||
|
### Suggested Fix
|
||||||
|
Inspect the sheet columns first and filter on `column_name`.
|
||||||
|
|
||||||
|
### Metadata
|
||||||
|
- Reproducible: yes
|
||||||
|
- Related Files: `Dataset/GSHS-全球学生健康调查数据/GSHS/02_documentation/GSHS_数据字典.xlsx`
|
||||||
|
- Pattern-Key: runtime.python-exception
|
||||||
|
|
||||||
|
### Resolution
|
||||||
|
- **Resolved**: 2026-09-20
|
||||||
|
- **Notes**: Re-ran the read-only extraction using `column_name` and obtained the intended candidate rows.
|
||||||
|
|
||||||
---
|
---
|
||||||
## [ERR-20260920-028] python-assignment-expression-comprehension
|
## [ERR-20260920-028] python-assignment-expression-comprehension
|
||||||
|
|
||||||
|
|||||||
+26
-11
@@ -1,9 +1,9 @@
|
|||||||
# 项目进度档案
|
# 项目进度档案
|
||||||
|
|
||||||
版本:1.7
|
版本:1.8
|
||||||
建立日期:2026-09-20
|
建立日期:2026-09-20
|
||||||
当前总状态:`in_progress`
|
当前总状态:`in_progress`
|
||||||
当前阶段:阶段 1 `in_progress`;输入结构与数据契约首轮审计完成
|
当前阶段:阶段 1 `in_progress`;主要结局来源登记骨架已建立,直接来源复核与双人审核尚未完成
|
||||||
|
|
||||||
## 1. 管理规则
|
## 1. 管理规则
|
||||||
|
|
||||||
@@ -28,7 +28,7 @@
|
|||||||
| 阶段 | 名称 | 状态 | 当前判定 |
|
| 阶段 | 名称 | 状态 | 当前判定 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 0 | 冻结研究问题、范围与版本 | `passed` | 四项交付物齐备,Gate 0 已通过;研究工具链补强与锁定完成 |
|
| 0 | 冻结研究问题、范围与版本 | `passed` | 四项交付物齐备,Gate 0 已通过;研究工具链补强与锁定完成 |
|
||||||
| 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约与三套候选入口结构审计完成;题目来源登记和人工复核尚未完成 |
|
| 1 | 有来源证据的题库和回答层 | `in_progress` | 数据契约、候选入口结构审计和主要结局来源登记骨架已完成;直接问卷核验、编码规则与人工双重审核尚未完成 |
|
||||||
| 2 | 可行性、可识别性和连接审计 | `not_started` | 等待 Gate 1 |
|
| 2 | 可行性、可识别性和连接审计 | `not_started` | 等待 Gate 1 |
|
||||||
| 3 | 冻结验证协议与已知真值模拟 | `not_started` | 等待 Gate 2 |
|
| 3 | 冻结验证协议与已知真值模拟 | `not_started` | 等待 Gate 2 |
|
||||||
| 4 | 直接协调和传统基线 | `not_started` | 等待 Gate 3 |
|
| 4 | 直接协调和传统基线 | `not_started` | 等待 Gate 3 |
|
||||||
@@ -67,7 +67,7 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
|
|||||||
- GSHS 现有说明记录主表 673,499 行、637 列、165 项调查、191 个非重叠组件;这些数字目前属于既有质量报告陈述,尚未由本轮执行重算。
|
- GSHS 现有说明记录主表 673,499 行、637 列、165 项调查、191 个非重叠组件;这些数字目前属于既有质量报告陈述,尚未由本轮执行重算。
|
||||||
- YRBS 原始库覆盖 1991–2023;现有整理包说明 1991–2019 已生成分析表,2021/2023 仍处于导入布局验证隔离状态。
|
- YRBS 原始库覆盖 1991–2023;现有整理包说明 1991–2019 已生成分析表,2021/2023 仍处于导入布局验证隔离状态。
|
||||||
- 系统 Python 3.13.7、Pandoc 3.11、31.82 GiB RAM 和 NVIDIA GeForce RTX 5060 Ti 16 GB 已核验;正式项目环境使用 CPython 3.12.14、PyMC 6.3.2、R 4.6.1、mirt/lavaan/survey 栈及 CUDA Toolkit 13.4.2,三套运行验收均通过。
|
- 系统 Python 3.13.7、Pandoc 3.11、31.82 GiB RAM 和 NVIDIA GeForce RTX 5060 Ti 16 GB 已核验;正式项目环境使用 CPython 3.12.14、PyMC 6.3.2、R 4.6.1、mirt/lavaan/survey 栈及 CUDA Toolkit 13.4.2,三套运行验收均通过。
|
||||||
- 当前根目录不是 Git 仓库;版本化与审计需使用明确的文件哈希和状态档案,或后续经用户授权建立版本控制。
|
- 当前根目录已建立 Git 仓库,`main` 跟踪 `origin/main`;远端为 `Jinovo/language-conditioned-psychometric-harmonization-model`,每项推进按强制协议独立提交并推送。
|
||||||
|
|
||||||
## 6. 文献初筛结论(尚待矩阵固化)
|
## 6. 文献初筛结论(尚待矩阵固化)
|
||||||
|
|
||||||
@@ -81,12 +81,13 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
|
|||||||
|
|
||||||
| 优先级 | 问题 | 影响 | 下一处理 |
|
| 优先级 | 问题 | 影响 | 下一处理 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| P0 | 官方题目文本、回答标签、适用总体和跳题规则尚未逐项核验 | 阶段 1 和确认性分析不能启动 | 在阶段 1 建立来源可追踪题库并保留 provisional 状态 |
|
| P0 | 官方题目文本、回答标签、适用总体和跳题规则尚未逐项核验 | Gate 1 与确认性分析不能通过 | 来源登记骨架已建立;下一步直接复核官方问卷/代码本,并保持 provisional 直到双人审核完成 |
|
||||||
| P0 | 题目共现、锚定图与可识别性尚未重算 | 不能确定 IRT/层级模型是否成立 | 完成阶段 1 后进入阶段 2 审计 |
|
| P0 | 题目共现、锚定图与可识别性尚未重算 | 不能确定 IRT/层级模型是否成立 | 完成阶段 1 后进入阶段 2 审计 |
|
||||||
| P1 | 数据来源许可及可分发范围尚未逐来源完成最终复核 | 影响复现包与衍生数据交付 | 已在清单中显式标为 review required;阶段 1 按实际纳入来源逐项确认,不默认允许再分发 |
|
| P1 | 数据来源许可及可分发范围尚未逐来源完成最终复核 | 影响复现包与衍生数据交付 | 已在清单中显式标为 review required;阶段 1 按实际纳入来源逐项确认,不默认允许再分发 |
|
||||||
| P1 | R、PyMC/ArviZ、心理测量包与 CUDA Toolkit 曾缺失 | 已解决:项目 Python/R 环境和 CUDA 端到端验收通过 | 后续仅按锁定文件维护;任何升级必须重跑三套烟雾测试 |
|
| P1 | R、PyMC/ArviZ、心理测量包与 CUDA Toolkit 曾缺失 | 已解决:项目 Python/R 环境和 CUDA 端到端验收通过 | 后续仅按锁定文件维护;任何升级必须重跑三套烟雾测试 |
|
||||||
| P1 | 文献初筛尚未形成可审计矩阵 | 研究定位和“贡献”表述仍不稳定 | 建立 literature matrix,记录 DOI、来源、证据层级和项目关系 |
|
| P1 | 文献初筛尚未形成可审计矩阵 | 研究定位和“贡献”表述仍不稳定 | 建立 literature matrix,记录 DOI、来源、证据层级和项目关系 |
|
||||||
| P2 | 根目录没有 Git 版本控制 | 变更追踪依赖手工哈希和档案 | 暂不擅自初始化;先维持文件哈希和行动日志 |
|
| P0 | NSDUH 本地完整字典未暴露青少年自杀尝试变量,虽 2022–2024 模块说明提到 `YSUI03`–`YSUI05` | NSDUH 不能被假定覆盖三项主要结局;跨工具任务可能只支持意念/计划 | 直接核验年度青少年问卷、代码本与公开使用文件抑制/派生说明 |
|
||||||
|
| P1 | GSHS `_b_` 与无前缀题目版本、YRBS 年度题目文本仍缺直接官方文档复核 | 不能批准题目家族或跨调查锚定 | 按组件/年度建立直接来源证据和冲突清单 |
|
||||||
|
|
||||||
## 8. 行动日志
|
## 8. 行动日志
|
||||||
|
|
||||||
@@ -673,9 +674,23 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
|
|||||||
- 新证据:GSHS 三个设计候选字段各缺 20 行;YRBS 1997 有 1 行设计字段缺失,2017/2019 的 PSU/stratum 全缺且权重仅部分存在;NSDUH 四个权重列在全部年度均非空且为正,需要官方指南解释适用语义。
|
- 新证据:GSHS 三个设计候选字段各缺 20 行;YRBS 1997 有 1 行设计字段缺失,2017/2019 的 PSU/stratum 全缺且权重仅部分存在;NSDUH 四个权重列在全部年度均非空且为正,需要官方指南解释适用语义。
|
||||||
- 质量边界:所有审核状态默认 `provisional`;本次未批准题目等价、构念、缺失重编码、锚题或复杂抽样方差方法;Gate 1 未通过。
|
- 质量边界:所有审核状态默认 `provisional`;本次未批准题目等价、构念、缺失重编码、锚题或复杂抽样方差方法;Gate 1 未通过。
|
||||||
- 工程修复:内联聚合的 Python 语法失败和 pandas 内存耗尽分别记录为已解决的 `ERR-20260920-028`、`ERR-20260920-029`;最终脚本使用标准 CSV 流式扫描。
|
- 工程修复:内联聚合的 Python 语法失败和 pandas 内存耗尽分别记录为已解决的 `ERR-20260920-028`、`ERR-20260920-029`;最终脚本使用标准 CSV 流式扫描。
|
||||||
- 推送状态:提交后立即推送。
|
- 推送状态:`pushed`;提交 `5c31417` 已推送至 `origin/main`。
|
||||||
- 下一步:建立主要结局来源登记骨架;优先核验 GSHS/NSDUH 的意念、计划、尝试题与适用总体,并追查 YRBS 2017/2019 设计字段缺口。
|
- 下一步:建立主要结局来源登记骨架;优先核验 GSHS/NSDUH 的意念、计划、尝试题与适用总体,并追查 YRBS 2017/2019 设计字段缺口。
|
||||||
|
|
||||||
|
### A-20260920-054:建立主要结局来源登记骨架
|
||||||
|
|
||||||
|
- 时间:2026-09-20
|
||||||
|
- 阶段:1(有来源证据的题库和回答层)
|
||||||
|
- 动作:完整读取工作流与进度档案;从 GSHS 数据字典、YRBS selected 字典、NSDUH complete 字典及 full/selected Parquet schema 中提取自杀意念、计划、尝试候选;严格分开原始青少年题、派生变量、成人模块、COVID 条件题与复合指标。
|
||||||
|
- 输入:GSHS `GSHS_数据字典.xlsx` 的 `Master_Columns`;YRBS `yrbs_selected_variable_dictionary.csv`;NSDUH `nsduh_complete_variable_dictionary.csv` 及 2021–2024 full/selected Parquet schema。
|
||||||
|
- 输出:`research/stage1/build_primary_outcome_registry.py`、`primary_outcome_registry.json`、`primary_outcome_registry.md`;更新 `.learnings/ERRORS.md` 的已解决 `ERR-20260920-030`。
|
||||||
|
- 检查:脚本通过 `py_compile`;JSON 可解析;65 条原始候选全部为 `provisional`、审核者为空、连接状态为 `not_evaluated`;分布为 GSHS 6 条、YRBS 51 条、NSDUH 8 条;另隔离 8 条 NSDUH 青少年派生记录和 40 条成人/COVID/派生或复合排除记录;语义断言和 `git diff --check` 通过。
|
||||||
|
- 新证据:GSHS 自杀尝试同时存在二分类与次数型版本,不能自动合并;YRBS 每个主要构念有 17 个年度候选,2021/2023 仍只保留元数据且数据隔离;NSDUH 本地完整字典只暴露青少年意念与计划,未找到青少年尝试变量,`YSUI03`–`YSUI05` 仅在模块迁移说明中出现。
|
||||||
|
- 质量边界:未直接核验年度 PDF/代码本页码;没有把 AI 检查登记为人类审核;没有批准任何构念等价、缺失重编码、题目家族或锚定关系。Gate 1 仍为 `in_progress`。
|
||||||
|
- 工程修复:一次只读工作簿筛选错误使用 `column` 而非 `column_name`,无数据改动;修正后成功并记录为已解决的 `ERR-20260920-030`。
|
||||||
|
- 推送状态:`push_pending`;完成本行动验证后提交并立即推送。
|
||||||
|
- 下一步:直接核验 NSDUH 年度青少年问卷/代码本中的 `YSUI03`–`YSUI05` 与公开数据字段关系,同时建立 GSHS 组件级题目版本来源映射。
|
||||||
|
|
||||||
## 9. 剩余工作量估算(2026-09-20 基线)
|
## 9. 剩余工作量估算(2026-09-20 基线)
|
||||||
|
|
||||||
| 阶段 | 主要工作 | 预计人周 | 主要波动来源 |
|
| 阶段 | 主要工作 | 预计人周 | 主要波动来源 |
|
||||||
@@ -696,8 +711,8 @@ Gate 0 当前判定:`passed`。四项交付物齐备;目标总体政策、
|
|||||||
|
|
||||||
下一次行动开始前重新读取 `Document/research-workflow.md` 与本文件。随后:
|
下一次行动开始前重新读取 `Document/research-workflow.md` 与本文件。随后:
|
||||||
|
|
||||||
环境补强已经完成。下一行动进入阶段 1:
|
阶段 1 下一行动:
|
||||||
|
|
||||||
1. 冻结阶段 1 数据实体 schema 与审核状态枚举;
|
1. 直接核验 NSDUH 2021–2024 青少年问卷/代码本中的 `YSUI03`–`YSUI05`,解释其与公开使用字段、跳题和可能抑制规则的关系;
|
||||||
2. 只读核验 GSHS、YRBS、NSDUH 候选入口的实际结构、行列数和设计字段;
|
2. 建立 GSHS 组件级题目版本与官方问卷来源映射,保持 `_b_` 与无前缀版本分离;
|
||||||
3. 建立主要结局的来源登记骨架,所有未完成实际双重核查的条目标为 `provisional`。
|
3. 随后形成回答编码/缺失规则草案,并把需要两名实际审核者处理的记录导出为审核队列。
|
||||||
|
|||||||
@@ -0,0 +1,280 @@
|
|||||||
|
"""Build a provisional, source-linked registry for the three primary outcomes.
|
||||||
|
|
||||||
|
This script deliberately does not approve cross-survey equivalence, recode
|
||||||
|
responses, or substitute AI review for the two required human reviewers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pyarrow.parquet as pq
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
STAGE_DIR = ROOT / "research" / "stage1"
|
||||||
|
GSHS_BOOK = ROOT / "Dataset" / "GSHS-全球学生健康调查数据" / "GSHS" / "02_documentation" / "GSHS_数据字典.xlsx"
|
||||||
|
YRBS_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "YRBS"
|
||||||
|
NSDUH_DIR = ROOT / "Dataset" / "可直接分析数据包_NSDUH_YRBS" / "NSDUH"
|
||||||
|
OUT_JSON = STAGE_DIR / "primary_outcome_registry.json"
|
||||||
|
OUT_MD = STAGE_DIR / "primary_outcome_registry.md"
|
||||||
|
|
||||||
|
PRIMARY_CONSTRUCTS = {"suicide_ideation", "suicide_plan", "suicide_attempt"}
|
||||||
|
GSHS_CONSTRUCT = {
|
||||||
|
"raw_mh_considersui": "suicide_ideation",
|
||||||
|
"raw_mh_b_considersui": "suicide_ideation",
|
||||||
|
"raw_mh_plansui": "suicide_plan",
|
||||||
|
"raw_mh_b_plansui": "suicide_plan",
|
||||||
|
"raw_mh_attemptsui": "suicide_attempt",
|
||||||
|
"raw_mh_b_attemptsui": "suicide_attempt",
|
||||||
|
}
|
||||||
|
NSDUH_YOUTH_RAW = {"YUSUITHK": "suicide_ideation", "YUSUIPLN": "suicide_plan"}
|
||||||
|
NSDUH_YOUTH_DERIVED = {"YUSUITHKYR": "suicide_ideation", "YUSUIPLNYR": "suicide_plan"}
|
||||||
|
|
||||||
|
|
||||||
|
def clean_text(value: object) -> str:
|
||||||
|
return html.unescape(str(value or "")).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def parse_json_object(value: object) -> object:
|
||||||
|
text = clean_text(value)
|
||||||
|
if not text:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return json.loads(text)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return {"unparsed_source_text": text}
|
||||||
|
|
||||||
|
|
||||||
|
def parquet_columns(path: Path) -> set[str]:
|
||||||
|
return set(pq.ParquetFile(path).schema_arrow.names)
|
||||||
|
|
||||||
|
|
||||||
|
def build_gshs() -> list[dict]:
|
||||||
|
frame = pd.read_excel(GSHS_BOOK, sheet_name="Master_Columns", dtype=str, keep_default_na=False)
|
||||||
|
frame = frame[frame["column_name"].isin(GSHS_CONSTRUCT)].copy()
|
||||||
|
records = []
|
||||||
|
for row in frame.sort_values("ordinal").to_dict("records"):
|
||||||
|
records.append(
|
||||||
|
{
|
||||||
|
"survey": "GSHS",
|
||||||
|
"year": None,
|
||||||
|
"source_variable": row["column_name"],
|
||||||
|
"construct": GSHS_CONSTRUCT[row["column_name"]],
|
||||||
|
"record_type": "raw_candidate",
|
||||||
|
"population": "school-attending students; exact component ages pending source review",
|
||||||
|
"time_window": "past 12 months (from source label; questionnaire verification pending)",
|
||||||
|
"question_text": None,
|
||||||
|
"source_label": clean_text(row["label_variants"]),
|
||||||
|
"answer_options": parse_json_object(row["value_labels_example"]),
|
||||||
|
"source_file": str(GSHS_BOOK.relative_to(ROOT)).replace("\\", "/"),
|
||||||
|
"source_url": None,
|
||||||
|
"page_or_section": "Master_Columns",
|
||||||
|
"present_in_local_data": True,
|
||||||
|
"source_verification_status": "pending_questionnaire_review",
|
||||||
|
"review_status": "provisional",
|
||||||
|
"reviewer_1": None,
|
||||||
|
"reviewer_2": None,
|
||||||
|
"link_status": "not_evaluated",
|
||||||
|
"notes": "The `_b_` and unprefixed forms remain separate item versions; labels alone do not prove equivalence.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def build_yrbs() -> list[dict]:
|
||||||
|
source = YRBS_DIR / "yrbs_selected_variable_dictionary.csv"
|
||||||
|
frame = pd.read_csv(source, dtype=str, keep_default_na=False)
|
||||||
|
frame = frame[frame["construct"].isin(PRIMARY_CONSTRUCTS)].copy()
|
||||||
|
records = []
|
||||||
|
for row in frame.sort_values(["year", "variable"]).to_dict("records"):
|
||||||
|
year = int(row["year"])
|
||||||
|
records.append(
|
||||||
|
{
|
||||||
|
"survey": "YRBS",
|
||||||
|
"year": year,
|
||||||
|
"source_variable": row["variable"],
|
||||||
|
"construct": row["construct"],
|
||||||
|
"record_type": "raw_candidate",
|
||||||
|
"population": "national high-school students; exact eligibility pending annual source review",
|
||||||
|
"time_window": clean_text(row["time_window"]),
|
||||||
|
"question_text": clean_text(row["original_question"]),
|
||||||
|
"answer_options": parse_json_object(row["answer_options_json"]),
|
||||||
|
"source_file": str(source.relative_to(ROOT)).replace("\\", "/"),
|
||||||
|
"source_url": clean_text(row["official_url"]),
|
||||||
|
"page_or_section": row["questionnaire_version"],
|
||||||
|
"present_in_local_data": year <= 2019,
|
||||||
|
"data_status": "available_selected_table" if year <= 2019 else "quarantined_pending_import_validation",
|
||||||
|
"source_verification_status": "pending_direct_pdf_review",
|
||||||
|
"review_status": "provisional",
|
||||||
|
"reviewer_1": None,
|
||||||
|
"reviewer_2": None,
|
||||||
|
"link_status": "not_evaluated",
|
||||||
|
"notes": "Repository dictionary candidate only. Stale source-machine absolute paths are intentionally not copied.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def build_nsduh() -> tuple[list[dict], list[dict], list[dict]]:
|
||||||
|
source = NSDUH_DIR / "nsduh_complete_variable_dictionary.csv"
|
||||||
|
frame = pd.read_csv(source, dtype=str, keep_default_na=False)
|
||||||
|
full_cols = parquet_columns(NSDUH_DIR / "nsduh_2021_2024_full.parquet")
|
||||||
|
selected_cols = parquet_columns(NSDUH_DIR / "nsduh_2021_2024_selected.parquet")
|
||||||
|
raw_records: list[dict] = []
|
||||||
|
derived_records: list[dict] = []
|
||||||
|
excluded_records: list[dict] = []
|
||||||
|
|
||||||
|
def common(row: dict, construct: str, record_type: str) -> dict:
|
||||||
|
variable = row["variable"]
|
||||||
|
return {
|
||||||
|
"survey": "NSDUH",
|
||||||
|
"year": int(row["year"]),
|
||||||
|
"source_variable": variable,
|
||||||
|
"construct": construct,
|
||||||
|
"record_type": record_type,
|
||||||
|
"population": "youth module; exact age/eligibility and skip logic pending codebook review",
|
||||||
|
"time_window": "past 12 months",
|
||||||
|
"question_text": clean_text(row["original_question_or_definition"]) or None,
|
||||||
|
"source_label": clean_text(row["official_title"]),
|
||||||
|
"source_file": str(source.relative_to(ROOT)).replace("\\", "/"),
|
||||||
|
"source_url": clean_text(row["official_source"]),
|
||||||
|
"page_or_section": variable,
|
||||||
|
"present_in_full_table": variable in full_cols,
|
||||||
|
"present_in_selected_table": variable in selected_cols,
|
||||||
|
"source_verification_status": "official_api_metadata_reviewed_codebook_pending",
|
||||||
|
"review_status": "provisional",
|
||||||
|
"reviewer_1": None,
|
||||||
|
"reviewer_2": None,
|
||||||
|
"link_status": "not_evaluated",
|
||||||
|
}
|
||||||
|
|
||||||
|
for row in frame[frame["variable"].isin(NSDUH_YOUTH_RAW)].to_dict("records"):
|
||||||
|
item = common(row, NSDUH_YOUTH_RAW[row["variable"]], "raw_candidate")
|
||||||
|
item["notes"] = "Keep 'not sure' and refusal/nonresponse distinct until skip and missing rules are verified."
|
||||||
|
raw_records.append(item)
|
||||||
|
|
||||||
|
for row in frame[frame["variable"].isin(NSDUH_YOUTH_DERIVED)].to_dict("records"):
|
||||||
|
item = common(row, NSDUH_YOUTH_DERIVED[row["variable"]], "derived_youth_indicator")
|
||||||
|
item["question_text"] = None
|
||||||
|
item["notes"] = "Derived/recode variable; do not count as an independent administered item."
|
||||||
|
derived_records.append(item)
|
||||||
|
|
||||||
|
exclusion_mask = frame["variable"].str.match(r"^(SUICTHNK|SUIPLANYR|IRSUITRYYR|IRCOSUITRYYR|YUCOSUI|COSUI|ADSUIT)")
|
||||||
|
for row in frame[exclusion_mask].to_dict("records"):
|
||||||
|
variable = row["variable"]
|
||||||
|
reason = "adult_module"
|
||||||
|
if "COSUI" in variable:
|
||||||
|
reason = "covid_conditioned"
|
||||||
|
elif variable.startswith(("IR", "AD")):
|
||||||
|
reason = "adult_derived_or_composite"
|
||||||
|
excluded_records.append(
|
||||||
|
{
|
||||||
|
"survey": "NSDUH",
|
||||||
|
"year": int(row["year"]),
|
||||||
|
"source_variable": variable,
|
||||||
|
"source_label": clean_text(row["official_title"]),
|
||||||
|
"exclusion_reason": reason,
|
||||||
|
"status": "excluded_from_primary_youth_item_candidates",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return raw_records, derived_records, excluded_records
|
||||||
|
|
||||||
|
|
||||||
|
def render_markdown(registry: dict) -> str:
|
||||||
|
records = registry["candidate_records"]
|
||||||
|
counts = Counter((r["survey"], r["construct"]) for r in records)
|
||||||
|
lines = [
|
||||||
|
"# Primary outcome source registry",
|
||||||
|
"",
|
||||||
|
"Version: 1.0 | Status: provisional | Generated: 2026-09-20",
|
||||||
|
"",
|
||||||
|
"## Scope and decision boundary",
|
||||||
|
"",
|
||||||
|
"This registry captures source-linked candidates for suicide ideation, plan, and attempt. It is a development artifact, not an approved item bank. No row has completed the two-human-reviewer requirement, no cross-survey link is approved, and no missing or skip code has been canonically recoded.",
|
||||||
|
"",
|
||||||
|
"## Candidate coverage",
|
||||||
|
"",
|
||||||
|
"| Survey | Ideation | Plan | Attempt | Current evidence boundary |",
|
||||||
|
"|---|---:|---:|---:|---|",
|
||||||
|
]
|
||||||
|
for survey in ["GSHS", "YRBS", "NSDUH"]:
|
||||||
|
boundary = {
|
||||||
|
"GSHS": "Workbook labels/options only; component questionnaires still require direct review.",
|
||||||
|
"YRBS": "51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review.",
|
||||||
|
"NSDUH": "Official API metadata supports youth ideation/plan only; no youth attempt field appears in the local full dictionary.",
|
||||||
|
}[survey]
|
||||||
|
lines.append(
|
||||||
|
f"| {survey} | {counts[(survey, 'suicide_ideation')]} | {counts[(survey, 'suicide_plan')]} | {counts[(survey, 'suicide_attempt')]} | {boundary} |"
|
||||||
|
)
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"Counts are source-record counts, not distinct item families. YRBS counts include annual versions; NSDUH counts include 2021–2024 raw youth records only.",
|
||||||
|
"",
|
||||||
|
"## Material conflicts and exclusions",
|
||||||
|
"",
|
||||||
|
"- GSHS exposes both `_b_` and unprefixed variants. They differ in naming and, for attempt, response form (binary versus frequency). They remain separate until questionnaire/component provenance is verified.",
|
||||||
|
"- GSHS self-harm variables are not included as suicide-attempt items.",
|
||||||
|
"- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.",
|
||||||
|
"- YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.",
|
||||||
|
"- NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.",
|
||||||
|
"- NSDUH youth attempt questions `YSUI03`–`YSUI05` are mentioned in notes, but no corresponding youth attempt variable appears in the local complete dictionary. Attempt coverage is therefore recorded as a source gap, not inferred from ideation wording.",
|
||||||
|
"",
|
||||||
|
"## Gate 1 status",
|
||||||
|
"",
|
||||||
|
"Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct official questionnaire/codebook review, exact population and skip-logic verification, canonical response/missing rules, component-specific GSHS provenance, and two actual human reviewers for primary outcomes.",
|
||||||
|
"",
|
||||||
|
"## Reproduction",
|
||||||
|
"",
|
||||||
|
"Run `research/stage1/build_primary_outcome_registry.py` in the locked research environment. The script reads repository-local dictionaries and Parquet schemas and rewrites this report plus `primary_outcome_registry.json`.",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
gshs = build_gshs()
|
||||||
|
yrbs = build_yrbs()
|
||||||
|
nsduh_raw, nsduh_derived, nsduh_excluded = build_nsduh()
|
||||||
|
registry = {
|
||||||
|
"schema_version": "1.0",
|
||||||
|
"generated_on": "2026-09-20",
|
||||||
|
"status": "provisional",
|
||||||
|
"allowed_uses": ["pipeline_development", "source_review_queue"],
|
||||||
|
"prohibited_uses": ["confirmatory_analysis", "approved_anchor_links", "claim_of_human_double_review"],
|
||||||
|
"candidate_records": gshs + yrbs + nsduh_raw,
|
||||||
|
"derived_records_not_independent_items": nsduh_derived,
|
||||||
|
"excluded_nsduh_records": nsduh_excluded,
|
||||||
|
"source_gaps": [
|
||||||
|
{
|
||||||
|
"survey": "NSDUH",
|
||||||
|
"construct": "suicide_attempt",
|
||||||
|
"status": "not_found_in_local_complete_dictionary",
|
||||||
|
"evidence": "YSUI03–YSUI05 are named in 2022–2024 module-move notes, but no youth attempt variable is exposed in the local 2021–2024 dictionary.",
|
||||||
|
"next_check": "Review annual youth questionnaires/codebooks and public-use suppression/derivation notes.",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"review_policy": {
|
||||||
|
"human_reviewers_required": 2,
|
||||||
|
"ai_review_is_human_review": False,
|
||||||
|
"default_review_status": "provisional",
|
||||||
|
"cross_survey_equivalence_default": "not_evaluated",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
OUT_JSON.write_text(json.dumps(registry, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
OUT_MD.write_text(render_markdown(registry), encoding="utf-8")
|
||||||
|
print(json.dumps({
|
||||||
|
"candidate_records": len(registry["candidate_records"]),
|
||||||
|
"derived_records": len(nsduh_derived),
|
||||||
|
"excluded_nsduh_records": len(nsduh_excluded),
|
||||||
|
"json": str(OUT_JSON.relative_to(ROOT)),
|
||||||
|
"report": str(OUT_MD.relative_to(ROOT)),
|
||||||
|
}, ensure_ascii=False))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,34 @@
|
|||||||
|
# Primary outcome source registry
|
||||||
|
|
||||||
|
Version: 1.0 | Status: provisional | Generated: 2026-09-20
|
||||||
|
|
||||||
|
## Scope and decision boundary
|
||||||
|
|
||||||
|
This registry captures source-linked candidates for suicide ideation, plan, and attempt. It is a development artifact, not an approved item bank. No row has completed the two-human-reviewer requirement, no cross-survey link is approved, and no missing or skip code has been canonically recoded.
|
||||||
|
|
||||||
|
## Candidate coverage
|
||||||
|
|
||||||
|
| Survey | Ideation | Plan | Attempt | Current evidence boundary |
|
||||||
|
|---|---:|---:|---:|---|
|
||||||
|
| GSHS | 2 | 2 | 2 | Workbook labels/options only; component questionnaires still require direct review. |
|
||||||
|
| YRBS | 17 | 17 | 17 | 51 annual candidates; 2021/2023 data remain quarantined; annual PDFs require direct review. |
|
||||||
|
| NSDUH | 4 | 4 | 0 | Official API metadata supports youth ideation/plan only; no youth attempt field appears in the local full dictionary. |
|
||||||
|
|
||||||
|
Counts are source-record counts, not distinct item families. YRBS counts include annual versions; NSDUH counts include 2021–2024 raw youth records only.
|
||||||
|
|
||||||
|
## Material conflicts and exclusions
|
||||||
|
|
||||||
|
- GSHS exposes both `_b_` and unprefixed variants. They differ in naming and, for attempt, response form (binary versus frequency). They remain separate until questionnaire/component provenance is verified.
|
||||||
|
- GSHS self-harm variables are not included as suicide-attempt items.
|
||||||
|
- YRBS metadata contains stale absolute paths from another machine and some automatically extracted neighboring-text contamination. Only exact primary-construct rows were retained, and their direct PDF review is still pending.
|
||||||
|
- YRBS 2021 and 2023 item metadata are retained for future migration checks, but their respondent data are quarantined pending layout/import validation.
|
||||||
|
- NSDUH adult, COVID-conditioned, imputed, recoded, and composite variables are not treated as primary raw youth items.
|
||||||
|
- NSDUH youth attempt questions `YSUI03`–`YSUI05` are mentioned in notes, but no corresponding youth attempt variable appears in the local complete dictionary. Attempt coverage is therefore recorded as a source gap, not inferred from ideation wording.
|
||||||
|
|
||||||
|
## Gate 1 status
|
||||||
|
|
||||||
|
Gate 1 remains **in progress**. Before any candidate can enter confirmatory analysis, the project still needs direct official questionnaire/codebook review, exact population and skip-logic verification, canonical response/missing rules, component-specific GSHS provenance, and two actual human reviewers for primary outcomes.
|
||||||
|
|
||||||
|
## Reproduction
|
||||||
|
|
||||||
|
Run `research/stage1/build_primary_outcome_registry.py` in the locked research environment. The script reads repository-local dictionaries and Parquet schemas and rewrites this report plus `primary_outcome_registry.json`.
|
||||||
Reference in New Issue
Block a user