diff --git a/AGENTS.md b/AGENTS.md index 1468911..7bb4a50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,6 +35,7 @@ If an automation Skill becomes a dependency of a CCPE runtime, agent, or committ - Prefer small, focused changes. - Keep Skill directories self-contained. - Use lowercase kebab-case for Skill names and file names. +- Run Python code and tests through the shared `skills-vault` conda environment by default, for example `conda run -n skills-vault python ...`. Use another environment only when a concrete dependency conflict requires it. - Preserve behavior during first migration; refactor only after baseline behavior is captured. - Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical. - Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository. diff --git a/docs/install-sync.md b/docs/install-sync.md index 53dab9e..51f9cd3 100644 --- a/docs/install-sync.md +++ b/docs/install-sync.md @@ -16,16 +16,20 @@ C:\Users\wangq\.agents\skills\ ## Current State -This repository does not yet include an automated sync script. +This repository includes install scripts for copying reviewed Skill source into `.agents/skills`. -For early migrations, install manually or with a narrowly scoped copy command after reviewing the Skill directory. +PowerShell: -Future repository-level scripts may support: +```powershell +powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations +``` -- listing Skills -- validating Skill structure -- installing selected Skills -- syncing selected Skills to `.agents/skills` -- checking registry consistency +Use `-Force` to back up and replace an existing installed copy: + +```powershell +powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force +``` + +`-All` installs only registry rows with an installable status and `default` install policy. Optional Skills must be named explicitly. Do not sync CCPE content into `.agents/skills` through this repository. diff --git a/docs/installation.md b/docs/installation.md index d0d4112..a0d3624 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -47,12 +47,22 @@ conda activate C:\Users\wangq\.conda\envs\skills-vault Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies. +When running Python from automation, prefer `conda run` so the intended environment is explicit even if the shell has not been activated: + +```powershell +conda run -n skills-vault python -m unittest discover -s skills\fix-title\tests -v +conda run -n skills-vault python .\skills\fix-title\scripts\fix_markdown_titles.py --help +``` + +Do not rely on bare `python` unless you have first verified it resolves to `C:\Users\wangq\.conda\envs\skills-vault\python.exe`. + ## Install One Skill PowerShell: ```powershell powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title +powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations ``` Git Bash / Linux / macOS: @@ -92,6 +102,7 @@ Use `-Force` or `--force` to back up the current installed directory and copy th ```powershell powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force +powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force ``` ```bash diff --git a/docs/superpowers/plans/2026-06-16-repair-markdown-citations.md b/docs/superpowers/plans/2026-06-16-repair-markdown-citations.md new file mode 100644 index 0000000..ae0d060 --- /dev/null +++ b/docs/superpowers/plans/2026-06-16-repair-markdown-citations.md @@ -0,0 +1,54 @@ +# Repair Markdown Citations Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a reusable Skill that repairs ChatGPT/Deep Research Markdown citation tokens into Markdown footnotes using metadata keyed by turn IDs. + +**Architecture:** Add a self-contained Skill under `skills/repair-markdown-citations`. The core behavior lives in a deterministic Python CLI; `SKILL.md` tells agents when to use it and how to handle exact metadata mapping versus fallback cleanup. + +**Tech Stack:** Python standard library, `unittest`, Markdown text processing. + +--- + +### Task 1: Tests First + +**Files:** +- Create: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py` + +- [ ] Write tests for token parsing, exact turn-ID replacement, deduplicated footnotes, fallback cleanup without metadata, and fenced code block safety. +- [ ] Run `python -m unittest discover -s skills/repair-markdown-citations/tests -v` and confirm the tests fail because the module does not exist yet. + +### Task 2: CLI Implementation + +**Files:** +- Create: `skills/repair-markdown-citations/scripts/repair_markdown_citations.py` + +- [ ] Implement parsing for `citeturn34view0turn20search2` tokens. +- [ ] Build citation records from metadata keyed by refs such as `turn34view0`. +- [ ] Replace citation tokens outside fenced code blocks with first-use footnote markers. +- [ ] Append a fresh `## 参考资料` section with deduplicated footnote definitions. +- [ ] If metadata is unavailable or incomplete, remove citation tokens and append a warning note instead of fabricating footnotes. +- [ ] Support `--metadata`, `--output`, `--in-place`, and `--dry-run`. + +### Task 3: Skill Package + +**Files:** +- Create: `skills/repair-markdown-citations/SKILL.md` +- Create: `skills/repair-markdown-citations/README.md` +- Create: `skills/repair-markdown-citations/fixtures/README.md` + +- [ ] Document the three capability levels. +- [ ] Make exact turn-ID mapping the required primary strategy. +- [ ] Document fallback behavior and safety rules. +- [ ] Keep the directory self-contained and lowercase kebab-case. + +### Task 4: Verification + +**Files:** +- Test: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py` +- Inspect: `skills/repair-markdown-citations/SKILL.md` + +- [ ] Run the unit test command. +- [ ] Run a lightweight frontmatter/naming check. +- [ ] Run `git diff --check`. +- [ ] Inspect `git status --short` and summarize changed files. diff --git a/registry/skills-index.md b/registry/skills-index.md index 3a0fca7..1fe6d29 100644 --- a/registry/skills-index.md +++ b/registry/skills-index.md @@ -14,6 +14,7 @@ Existing CCPE content is not migrated here. | Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration | | --- | --- | --- | --- | --- | --- | --- | | `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none | +| `repair-markdown-citations` | Repair ChatGPT/Deep Research Markdown citation tokens into standard footnotes and a deduplicated reference section using exact turn-ID metadata mapping. | `C:\Users\wangq\Documents\Codex\skills-vault\skills\repair-markdown-citations` | `installed` | `default` | `C:\Users\wangq\.agents\skills\repair-markdown-citations` | none | | `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none | ## Status Values diff --git a/skills/repair-markdown-citations/README.md b/skills/repair-markdown-citations/README.md new file mode 100644 index 0000000..f56c8dd --- /dev/null +++ b/skills/repair-markdown-citations/README.md @@ -0,0 +1,11 @@ +# Repair Markdown Citations + +Source Skill for repairing ChatGPT/Deep Research citation tokens in Markdown exports. + +The canonical entry point is `SKILL.md`; the deterministic transformer is in `scripts/repair_markdown_citations.py`. + +Run Python code through the shared repository environment: + +```powershell +conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run +``` diff --git a/skills/repair-markdown-citations/SKILL.md b/skills/repair-markdown-citations/SKILL.md new file mode 100644 index 0000000..3bab368 --- /dev/null +++ b/skills/repair-markdown-citations/SKILL.md @@ -0,0 +1,79 @@ +--- +name: repair-markdown-citations +description: Use when a Markdown export from ChatGPT Deep Research or another LLM contains broken citation tokens such as `citeturn34view0turn20search2` that need to be converted into standard Markdown footnotes and a deduplicated `## 参考资料` section. Use for repairing turn-ID citations, cleaning citation mojibake, preserving source links, and safely degrading when metadata is missing. +--- + +# Repair Markdown Citations + +## Purpose + +Repair ChatGPT/Deep Research Markdown citation tokens into readable Markdown. Use the bundled script for deterministic edits instead of rewriting the document by prompt. + +The primary rule is exact ID mapping: treat `turn34view0`, `turn20search2`, and similar IDs as citation primary keys. Do not assume the citation token order matches `safe_urls`, a footer link list, or source appearance order. + +## Capability Levels + +Level 1: no metadata cleanup. + +- Remove malformed `cite...` tokens outside fenced code blocks. +- Preserve existing links from an old `## 参考资料` section as ordinary source links. +- Add a warning that逐句脚注映射 could not be recovered. + +Level 2: `content_references` metadata. + +- Build a citation map from metadata refs keyed by turn ID. +- Replace citation tokens in正文 with Markdown footnote markers such as `[^1][^2]`. +- Generate a deduplicated `## 参考资料` section in first-citation order. + +Level 3: complete Deep Research export. + +- Prefer exact `ref_id` / `turn_index` / `ref_type` / `ref_index` fields wherever available. +- Use richer fields such as title, attribution, source, and site name when present. +- Treat source grouping, dates, snippets, and risk checks as future enrichment, not required first-pass behavior. + +## Workflow + +1. Confirm the input is a Markdown file exported from ChatGPT/Deep Research or converted from a report containing citation tokens. +2. Prefer an explicit metadata JSON file. If the user has one, pass it with `--metadata`. +3. If no metadata file is available, let the script look for an embedded HTML comment block named `deep-research-metadata`. +4. Choose output mode: + - Use `--output` for a non-destructive repaired copy. + - Use `--in-place` only when the user wants the original file rewritten. + - Use `--dry-run` before editing important files. +5. Inspect the resulting `## 参考资料` section and any `未解析引用 ID` warnings. + +## Commands + +```powershell +conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --output "C:\path\report.fixed.md" +conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --in-place +conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run +``` + +## Embedded Metadata + +The script can read a Markdown HTML comment block: + +```md + +``` + +Use embedded metadata only when an external JSON file is unavailable or when the report intentionally carries its own citation map. + +## Safety Rules + +- Do not map citations by `safe_urls` order or by footer link order. +- Do not fabricate footnotes when a turn ID has no URL in the citation map. +- Do not rewrite citation-like text inside fenced code blocks. +- Do not delete user-authored prose or non-reference sections. +- If metadata is missing or incomplete, preserve recoverable source links and clearly mark the output as degraded. + +## Validation + +Run the Skill tests after modifying behavior: + +```powershell +conda run -n skills-vault python -m unittest discover -s skills/repair-markdown-citations/tests -v +``` diff --git a/skills/repair-markdown-citations/agents/openai.yaml b/skills/repair-markdown-citations/agents/openai.yaml new file mode 100644 index 0000000..200bb25 --- /dev/null +++ b/skills/repair-markdown-citations/agents/openai.yaml @@ -0,0 +1,3 @@ +display_name: Repair Markdown Citations +short_description: Convert broken ChatGPT citation tokens into Markdown references. +default_prompt: Repair the broken citation tokens in this Markdown file and produce a clean reference section. diff --git a/skills/repair-markdown-citations/fixtures/README.md b/skills/repair-markdown-citations/fixtures/README.md new file mode 100644 index 0000000..aef8606 --- /dev/null +++ b/skills/repair-markdown-citations/fixtures/README.md @@ -0,0 +1,3 @@ +# Fixtures + +Place small anonymized Markdown and metadata examples here when adding regression coverage. diff --git a/skills/repair-markdown-citations/scripts/repair_markdown_citations.py b/skills/repair-markdown-citations/scripts/repair_markdown_citations.py new file mode 100644 index 0000000..360d123 --- /dev/null +++ b/skills/repair-markdown-citations/scripts/repair_markdown_citations.py @@ -0,0 +1,304 @@ +#!/usr/bin/env python3 +"""Repair ChatGPT citation tokens in Markdown exports.""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import sys +from dataclasses import dataclass +from typing import Any + + +CITATION_RE = re.compile(r"cite(?:turn\d+(?:view|search)\d+)+") +REF_ID_RE = re.compile(r"turn\d+(?:view|search)\d+") +FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})") +LINK_RE = re.compile(r"\[[^\]\n]+\]\([^) \n]+(?: [^)]+)?\)") +REFERENCE_HEADING_RE = re.compile(r"^## +参考资料\s*$", re.MULTILINE) +HTML_METADATA_RE = re.compile( + r"", + re.DOTALL, +) + + +@dataclass(frozen=True) +class CitationRecord: + ref_id: str + url: str + title: str = "" + attribution: str = "" + + def footnote_text(self) -> str: + label = self.title or self.url + if self.attribution and self.title: + return f"{self.attribution}：{self.title}。{self.url}" + return f"{label}。{self.url}" + + +def extract_ref_ids(token: str) -> list[str]: + return REF_ID_RE.findall(token) + + +def ref_id_from_ref(ref: dict[str, Any]) -> str | None: + turn_index = ref.get("turn_index") + ref_type = ref.get("ref_type") + ref_index = ref.get("ref_index") + if turn_index is None or ref_type not in {"view", "search"} or ref_index is None: + return None + return f"turn{turn_index}{ref_type}{ref_index}" + + +def _first_string(*values: Any) -> str: + for value in values: + if isinstance(value, str) and value.strip(): + return value.strip() + return "" + + +def record_from_ref(ref: dict[str, Any]) -> CitationRecord | None: + ref_id = _first_string(ref.get("ref_id"), ref.get("id")) or ref_id_from_ref(ref) + url = _first_string(ref.get("url"), ref.get("safe_url"), ref.get("source_url")) + title = _first_string(ref.get("title"), ref.get("name"), ref.get("text")) + attribution = _first_string(ref.get("attribution"), ref.get("source"), ref.get("site_name")) + if not ref_id or not url: + return None + return CitationRecord(ref_id=ref_id, url=url, title=title, attribution=attribution) + + +def _walk_dicts(value: Any) -> list[dict[str, Any]]: + found: list[dict[str, Any]] = [] + if isinstance(value, dict): + found.append(value) + for child in value.values(): + found.extend(_walk_dicts(child)) + elif isinstance(value, list): + for child in value: + found.extend(_walk_dicts(child)) + return found + + +def build_citation_map(metadata: dict[str, Any] | None) -> dict[str, CitationRecord]: + if not metadata: + return {} + + citation_map: dict[str, CitationRecord] = {} + for node in _walk_dicts(metadata): + record = record_from_ref(node) + if record and record.ref_id not in citation_map: + citation_map[record.ref_id] = record + + return citation_map + + +def split_existing_references(markdown: str) -> tuple[str, str]: + match = REFERENCE_HEADING_RE.search(markdown) + if not match: + return markdown.rstrip(), "" + return markdown[: match.start()].rstrip(), markdown[match.start() :].strip() + + +def collect_markdown_links(markdown: str) -> list[str]: + seen: set[str] = set() + links: list[str] = [] + for link in LINK_RE.findall(markdown): + if link not in seen: + seen.add(link) + links.append(link) + return links + + +def replace_citations( + markdown: str, citation_map: dict[str, CitationRecord] +) -> tuple[str, list[CitationRecord], list[str]]: + footnote_numbers: dict[str, int] = {} + ordered_records: list[CitationRecord] = [] + missing: list[str] = [] + + def marker_for_ref(ref_id: str) -> str: + if ref_id not in citation_map: + if ref_id not in missing: + missing.append(ref_id) + return "" + if ref_id not in footnote_numbers: + footnote_numbers[ref_id] = len(ordered_records) + 1 + ordered_records.append(citation_map[ref_id]) + return f"[^{footnote_numbers[ref_id]}]" + + output: list[str] = [] + in_fence = False + fence_marker = "" + + for line in markdown.splitlines(keepends=True): + body = line[:-1] if line.endswith("\n") else line + newline = "\n" if line.endswith("\n") else "" + if body.endswith("\r"): + body = body[:-1] + newline = "\r" + newline + + fence_match = FENCE_RE.match(body) + if fence_match: + marker = fence_match.group(2) + marker_char = marker[0] + if not in_fence: + in_fence = True + fence_marker = marker_char * len(marker) + elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker): + in_fence = False + fence_marker = "" + output.append(line) + continue + + if in_fence: + output.append(line) + continue + + def replace_token(match: re.Match[str]) -> str: + return "".join(marker_for_ref(ref_id) for ref_id in extract_ref_ids(match.group(0))) + + output.append(CITATION_RE.sub(replace_token, body) + newline) + + return "".join(output), ordered_records, missing + + +def strip_citation_tokens(markdown: str) -> str: + lines: list[str] = [] + in_fence = False + fence_marker = "" + + for line in markdown.splitlines(keepends=True): + body = line[:-1] if line.endswith("\n") else line + newline = "\n" if line.endswith("\n") else "" + if body.endswith("\r"): + body = body[:-1] + newline = "\r" + newline + + fence_match = FENCE_RE.match(body) + if fence_match: + marker = fence_match.group(2) + marker_char = marker[0] + if not in_fence: + in_fence = True + fence_marker = marker_char * len(marker) + elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker): + in_fence = False + fence_marker = "" + lines.append(line) + continue + + if in_fence: + lines.append(line) + else: + lines.append(CITATION_RE.sub("", body) + newline) + + return "".join(lines) + + +def render_reference_section( + records: list[CitationRecord], existing_links: list[str], missing: list[str], degraded: bool +) -> str: + lines = ["## 参考资料"] + + if degraded: + lines.append("") + lines.append("> 无法恢复逐句脚注映射：未找到可用元数据或部分 turn ID 缺少 URL。") + + if records: + lines.append("") + for index, record in enumerate(records, start=1): + lines.append(f"[^{index}]: {record.footnote_text()}") + + if existing_links: + lines.append("") + lines.append("### 原始来源链接") + for link in existing_links: + lines.append(f"- {link}") + + if missing: + lines.append("") + lines.append("### 未解析引用 ID") + for ref_id in missing: + lines.append(f"- `{ref_id}`") + + return "\n".join(lines) + + +def repair_markdown(markdown: str, metadata: dict[str, Any] | None = None) -> str: + body, existing_reference_section = split_existing_references(markdown) + existing_links = collect_markdown_links(existing_reference_section) + citation_map = build_citation_map(metadata) + + if citation_map: + replaced, records, missing = replace_citations(body, citation_map) + degraded = bool(missing) + cleaned = strip_citation_tokens(replaced) if missing else replaced + reference_section = render_reference_section(records, existing_links, missing, degraded) + return f"{cleaned.rstrip()}\n\n{reference_section}\n" + + cleaned = strip_citation_tokens(body) + reference_section = render_reference_section([], existing_links, [], degraded=True) + return f"{cleaned.rstrip()}\n\n{reference_section}\n" + + +def load_json(path: pathlib.Path) -> dict[str, Any]: + with path.open("r", encoding="utf-8") as handle: + value = json.load(handle) + if not isinstance(value, dict): + raise ValueError("metadata JSON must contain an object at the top level") + return value + + +def extract_embedded_metadata(markdown: str) -> dict[str, Any] | None: + match = HTML_METADATA_RE.search(markdown) + if not match: + return None + value = json.loads(match.group("json")) + if not isinstance(value, dict): + raise ValueError("embedded metadata JSON must contain an object at the top level") + return value + + +def parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Repair ChatGPT/Deep Research citation tokens in Markdown." + ) + parser.add_argument("path", type=pathlib.Path, help="Markdown file to repair") + parser.add_argument("--metadata", type=pathlib.Path, help="Deep Research metadata JSON") + parser.add_argument("-o", "--output", type=pathlib.Path, help="Write repaired Markdown here") + parser.add_argument("--in-place", action="store_true", help="Rewrite the input Markdown file") + parser.add_argument("--dry-run", action="store_true", help="Print repaired Markdown to stdout") + return parser.parse_args(argv) + + +def main(argv: list[str]) -> int: + args = parse_args(argv) + if sum(bool(value) for value in (args.output, args.in_place, args.dry_run)) != 1: + print("error: choose exactly one of --output, --in-place, or --dry-run", file=sys.stderr) + return 2 + + try: + markdown = args.path.read_text(encoding="utf-8") + metadata = load_json(args.metadata) if args.metadata else extract_embedded_metadata(markdown) + fixed = repair_markdown(markdown, metadata=metadata) + except (OSError, ValueError, json.JSONDecodeError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + + if args.dry_run: + sys.stdout.write(fixed) + return 0 + + destination = args.path if args.in_place else args.output + assert destination is not None + try: + destination.write_text(fixed, encoding="utf-8") + except OSError as exc: + print(f"error: cannot write {destination}: {exc}", file=sys.stderr) + return 1 + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/skills/repair-markdown-citations/tests/test_repair_markdown_citations.py b/skills/repair-markdown-citations/tests/test_repair_markdown_citations.py new file mode 100644 index 0000000..2244a5c --- /dev/null +++ b/skills/repair-markdown-citations/tests/test_repair_markdown_citations.py @@ -0,0 +1,185 @@ +from __future__ import annotations + +import importlib.util +import pathlib +import sys +import textwrap +import unittest + + +SCRIPT_PATH = ( + pathlib.Path(__file__).resolve().parents[1] + / "scripts" + / "repair_markdown_citations.py" +) + +spec = importlib.util.spec_from_file_location("repair_markdown_citations", SCRIPT_PATH) +repair = importlib.util.module_from_spec(spec) +assert spec is not None +assert spec.loader is not None +sys.modules["repair_markdown_citations"] = repair +spec.loader.exec_module(repair) + + +class RepairMarkdownCitationsTest(unittest.TestCase): + def test_extracts_turn_ids_from_chatgpt_citation_token(self) -> None: + token = "citeturn34view0turn20search2" + + self.assertEqual( + repair.extract_ref_ids(token), + ["turn34view0", "turn20search2"], + ) + + def test_replaces_tokens_with_first_use_footnotes_from_metadata(self) -> None: + markdown = textwrap.dedent( + """\ + 武汉早期疫情通报强调未发现明显人传人。citeturn34view0 + + 郑州暴雨和唐山事件报道也体现了不同报道框架。citeturn20search2turn34view6 + """ + ) + metadata = { + "content_references": [ + { + "items": [ + { + "refs": [ + { + "turn_index": 34, + "ref_type": "view", + "ref_index": 0, + "url": "https://wjw.hubei.gov.cn/example.shtml", + "title": "湖北省卫健委通报", + "attribution": "湖北省卫健委", + }, + { + "turn_index": 20, + "ref_type": "search", + "ref_index": 2, + "url": "https://www.xinhuanet.com/zhengzhou.htm", + "title": "新华社郑州暴雨报道", + }, + { + "turn_index": 34, + "ref_type": "view", + "ref_index": 6, + "url": "https://www.xinhuanet.com/tangshan.htm", + "title": "新华社唐山打人事件报道", + }, + ] + } + ] + } + ] + } + + fixed = repair.repair_markdown(markdown, metadata=metadata) + + self.assertIn("武汉早期疫情通报强调未发现明显人传人。[^1]", fixed) + self.assertIn("报道框架。[^2][^3]", fixed) + self.assertIn("## 参考资料", fixed) + self.assertIn( + "[^1]: 湖北省卫健委：湖北省卫健委通报。https://wjw.hubei.gov.cn/example.shtml", + fixed, + ) + self.assertIn( + "[^2]: 新华社郑州暴雨报道。https://www.xinhuanet.com/zhengzhou.htm", + fixed, + ) + self.assertIn( + "[^3]: 新华社唐山打人事件报道。https://www.xinhuanet.com/tangshan.htm", + fixed, + ) + + def test_deduplicates_repeated_references(self) -> None: + markdown = "Aciteturn1view0 Bciteturn1view0" + metadata = { + "content_references": [ + { + "items": [ + { + "refs": [ + { + "turn_index": 1, + "ref_type": "view", + "ref_index": 0, + "url": "https://example.com/a", + "title": "Example A", + } + ] + } + ] + } + ] + } + + fixed = repair.repair_markdown(markdown, metadata=metadata) + + self.assertIn("A[^1] B[^1]", fixed) + self.assertEqual(fixed.count("[^1]:"), 1) + + def test_without_metadata_removes_tokens_and_adds_warning(self) -> None: + markdown = "正文citeturn1view0\n\n[来源](https://example.com)" + + fixed = repair.repair_markdown(markdown, metadata=None) + + self.assertNotIn("cite", fixed) + self.assertIn("## 参考资料", fixed) + self.assertIn("[来源](https://example.com)", fixed) + self.assertIn("无法恢复逐句脚注映射", fixed) + + def test_loads_embedded_metadata_from_html_comment(self) -> None: + markdown = textwrap.dedent( + """\ + 正文citeturn1view0 + + + """ + ) + + metadata = repair.extract_embedded_metadata(markdown) + fixed = repair.repair_markdown(markdown, metadata=metadata) + + self.assertIn("正文[^1]", fixed) + self.assertIn("[^1]: Example A。https://example.com/a", fixed) + + def test_does_not_rewrite_tokens_inside_fenced_code_blocks(self) -> None: + markdown = textwrap.dedent( + """\ + 正文citeturn1view0 + + ```md + 示例citeturn1view0 + ``` + """ + ) + metadata = { + "content_references": [ + { + "items": [ + { + "refs": [ + { + "turn_index": 1, + "ref_type": "view", + "ref_index": 0, + "url": "https://example.com/a", + "title": "Example A", + } + ] + } + ] + } + ] + } + + fixed = repair.repair_markdown(markdown, metadata=metadata) + + self.assertIn("正文[^1]", fixed) + self.assertIn("示例citeturn1view0", fixed) + + +if __name__ == "__main__": + unittest.main()