feat: add markdown citation repair skill

This commit is contained in:
wantsong 2026-06-16 09:35:38 +08:00
parent 910926838c
commit 86d860044b
11 changed files with 664 additions and 8 deletions

View File

@ -35,6 +35,7 @@ If an automation Skill becomes a dependency of a CCPE runtime, agent, or committ
- Prefer small, focused changes. - Prefer small, focused changes.
- Keep Skill directories self-contained. - Keep Skill directories self-contained.
- Use lowercase kebab-case for Skill names and file names. - Use lowercase kebab-case for Skill names and file names.
- Run Python code and tests through the shared `skills-vault` conda environment by default, for example `conda run -n skills-vault python ...`. Use another environment only when a concrete dependency conflict requires it.
- Preserve behavior during first migration; refactor only after baseline behavior is captured. - Preserve behavior during first migration; refactor only after baseline behavior is captured.
- Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical. - Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical.
- Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository. - Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository.

View File

@ -16,16 +16,20 @@ C:\Users\wangq\.agents\skills\<skill-name>
## Current State ## Current State
This repository does not yet include an automated sync script. This repository includes install scripts for copying reviewed Skill source into `.agents/skills`.
For early migrations, install manually or with a narrowly scoped copy command after reviewing the Skill directory. PowerShell:
Future repository-level scripts may support: ```powershell
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
```
- listing Skills Use `-Force` to back up and replace an existing installed copy:
- validating Skill structure
- installing selected Skills ```powershell
- syncing selected Skills to `.agents/skills` powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
- checking registry consistency ```
`-All` installs only registry rows with an installable status and `default` install policy. Optional Skills must be named explicitly.
Do not sync CCPE content into `.agents/skills` through this repository. Do not sync CCPE content into `.agents/skills` through this repository.

View File

@ -47,12 +47,22 @@ conda activate C:\Users\wangq\.conda\envs\skills-vault
Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies. Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies.
When running Python from automation, prefer `conda run` so the intended environment is explicit even if the shell has not been activated:
```powershell
conda run -n skills-vault python -m unittest discover -s skills\fix-title\tests -v
conda run -n skills-vault python .\skills\fix-title\scripts\fix_markdown_titles.py --help
```
Do not rely on bare `python` unless you have first verified it resolves to `C:\Users\wangq\.conda\envs\skills-vault\python.exe`.
## Install One Skill ## Install One Skill
PowerShell: PowerShell:
```powershell ```powershell
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
``` ```
Git Bash / Linux / macOS: Git Bash / Linux / macOS:
@ -92,6 +102,7 @@ Use `-Force` or `--force` to back up the current installed directory and copy th
```powershell ```powershell
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
``` ```
```bash ```bash

View File

@ -0,0 +1,54 @@
# Repair Markdown Citations Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a reusable Skill that repairs ChatGPT/Deep Research Markdown citation tokens into Markdown footnotes using metadata keyed by turn IDs.
**Architecture:** Add a self-contained Skill under `skills/repair-markdown-citations`. The core behavior lives in a deterministic Python CLI; `SKILL.md` tells agents when to use it and how to handle exact metadata mapping versus fallback cleanup.
**Tech Stack:** Python standard library, `unittest`, Markdown text processing.
---
### Task 1: Tests First
**Files:**
- Create: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
- [ ] Write tests for token parsing, exact turn-ID replacement, deduplicated footnotes, fallback cleanup without metadata, and fenced code block safety.
- [ ] Run `python -m unittest discover -s skills/repair-markdown-citations/tests -v` and confirm the tests fail because the module does not exist yet.
### Task 2: CLI Implementation
**Files:**
- Create: `skills/repair-markdown-citations/scripts/repair_markdown_citations.py`
- [ ] Implement parsing for `citeturn34view0turn20search2` tokens.
- [ ] Build citation records from metadata keyed by refs such as `turn34view0`.
- [ ] Replace citation tokens outside fenced code blocks with first-use footnote markers.
- [ ] Append a fresh `## 参考资料` section with deduplicated footnote definitions.
- [ ] If metadata is unavailable or incomplete, remove citation tokens and append a warning note instead of fabricating footnotes.
- [ ] Support `--metadata`, `--output`, `--in-place`, and `--dry-run`.
### Task 3: Skill Package
**Files:**
- Create: `skills/repair-markdown-citations/SKILL.md`
- Create: `skills/repair-markdown-citations/README.md`
- Create: `skills/repair-markdown-citations/fixtures/README.md`
- [ ] Document the three capability levels.
- [ ] Make exact turn-ID mapping the required primary strategy.
- [ ] Document fallback behavior and safety rules.
- [ ] Keep the directory self-contained and lowercase kebab-case.
### Task 4: Verification
**Files:**
- Test: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
- Inspect: `skills/repair-markdown-citations/SKILL.md`
- [ ] Run the unit test command.
- [ ] Run a lightweight frontmatter/naming check.
- [ ] Run `git diff --check`.
- [ ] Inspect `git status --short` and summarize changed files.

View File

@ -14,6 +14,7 @@ Existing CCPE content is not migrated here.
| Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration | | Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration |
| --- | --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- | --- |
| `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none | | `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none |
| `repair-markdown-citations` | Repair ChatGPT/Deep Research Markdown citation tokens into standard footnotes and a deduplicated reference section using exact turn-ID metadata mapping. | `C:\Users\wangq\Documents\Codex\skills-vault\skills\repair-markdown-citations` | `installed` | `default` | `C:\Users\wangq\.agents\skills\repair-markdown-citations` | none |
| `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none | | `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none |
## Status Values ## Status Values

View File

@ -0,0 +1,11 @@
# Repair Markdown Citations
Source Skill for repairing ChatGPT/Deep Research citation tokens in Markdown exports.
The canonical entry point is `SKILL.md`; the deterministic transformer is in `scripts/repair_markdown_citations.py`.
Run Python code through the shared repository environment:
```powershell
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
```

View File

@ -0,0 +1,79 @@
---
name: repair-markdown-citations
description: Use when a Markdown export from ChatGPT Deep Research or another LLM contains broken citation tokens such as `citeturn34view0turn20search2` that need to be converted into standard Markdown footnotes and a deduplicated `## 参考资料` section. Use for repairing turn-ID citations, cleaning citation mojibake, preserving source links, and safely degrading when metadata is missing.
---
# Repair Markdown Citations
## Purpose
Repair ChatGPT/Deep Research Markdown citation tokens into readable Markdown. Use the bundled script for deterministic edits instead of rewriting the document by prompt.
The primary rule is exact ID mapping: treat `turn34view0`, `turn20search2`, and similar IDs as citation primary keys. Do not assume the citation token order matches `safe_urls`, a footer link list, or source appearance order.
## Capability Levels
Level 1: no metadata cleanup.
- Remove malformed `cite...` tokens outside fenced code blocks.
- Preserve existing links from an old `## 参考资料` section as ordinary source links.
- Add a warning that逐句脚注映射 could not be recovered.
Level 2: `content_references` metadata.
- Build a citation map from metadata refs keyed by turn ID.
- Replace citation tokens in正文 with Markdown footnote markers such as `[^1][^2]`.
- Generate a deduplicated `## 参考资料` section in first-citation order.
Level 3: complete Deep Research export.
- Prefer exact `ref_id` / `turn_index` / `ref_type` / `ref_index` fields wherever available.
- Use richer fields such as title, attribution, source, and site name when present.
- Treat source grouping, dates, snippets, and risk checks as future enrichment, not required first-pass behavior.
## Workflow
1. Confirm the input is a Markdown file exported from ChatGPT/Deep Research or converted from a report containing citation tokens.
2. Prefer an explicit metadata JSON file. If the user has one, pass it with `--metadata`.
3. If no metadata file is available, let the script look for an embedded HTML comment block named `deep-research-metadata`.
4. Choose output mode:
- Use `--output` for a non-destructive repaired copy.
- Use `--in-place` only when the user wants the original file rewritten.
- Use `--dry-run` before editing important files.
5. Inspect the resulting `## 参考资料` section and any `未解析引用 ID` warnings.
## Commands
```powershell
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --output "C:\path\report.fixed.md"
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --in-place
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
```
## Embedded Metadata
The script can read a Markdown HTML comment block:
```md
<!-- deep-research-metadata
{"content_references":[...]}
-->
```
Use embedded metadata only when an external JSON file is unavailable or when the report intentionally carries its own citation map.
## Safety Rules
- Do not map citations by `safe_urls` order or by footer link order.
- Do not fabricate footnotes when a turn ID has no URL in the citation map.
- Do not rewrite citation-like text inside fenced code blocks.
- Do not delete user-authored prose or non-reference sections.
- If metadata is missing or incomplete, preserve recoverable source links and clearly mark the output as degraded.
## Validation
Run the Skill tests after modifying behavior:
```powershell
conda run -n skills-vault python -m unittest discover -s skills/repair-markdown-citations/tests -v
```

View File

@ -0,0 +1,3 @@
display_name: Repair Markdown Citations
short_description: Convert broken ChatGPT citation tokens into Markdown references.
default_prompt: Repair the broken citation tokens in this Markdown file and produce a clean reference section.

View File

@ -0,0 +1,3 @@
# Fixtures
Place small anonymized Markdown and metadata examples here when adding regression coverage.

View File

@ -0,0 +1,304 @@
#!/usr/bin/env python3
"""Repair ChatGPT citation tokens in Markdown exports."""
from __future__ import annotations
import argparse
import json
import pathlib
import re
import sys
from dataclasses import dataclass
from typing import Any
CITATION_RE = re.compile(r"cite(?:turn\d+(?:view|search)\d+)+")
REF_ID_RE = re.compile(r"turn\d+(?:view|search)\d+")
FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})")
LINK_RE = re.compile(r"\[[^\]\n]+\]\([^) \n]+(?: [^)]+)?\)")
REFERENCE_HEADING_RE = re.compile(r"^## +参考资料\s*$", re.MULTILINE)
HTML_METADATA_RE = re.compile(
r"<!--\s*deep-research-metadata\s*(?P<json>\{.*?\})\s*-->",
re.DOTALL,
)
@dataclass(frozen=True)
class CitationRecord:
ref_id: str
url: str
title: str = ""
attribution: str = ""
def footnote_text(self) -> str:
label = self.title or self.url
if self.attribution and self.title:
return f"{self.attribution}{self.title}{self.url}"
return f"{label}{self.url}"
def extract_ref_ids(token: str) -> list[str]:
return REF_ID_RE.findall(token)
def ref_id_from_ref(ref: dict[str, Any]) -> str | None:
turn_index = ref.get("turn_index")
ref_type = ref.get("ref_type")
ref_index = ref.get("ref_index")
if turn_index is None or ref_type not in {"view", "search"} or ref_index is None:
return None
return f"turn{turn_index}{ref_type}{ref_index}"
def _first_string(*values: Any) -> str:
for value in values:
if isinstance(value, str) and value.strip():
return value.strip()
return ""
def record_from_ref(ref: dict[str, Any]) -> CitationRecord | None:
ref_id = _first_string(ref.get("ref_id"), ref.get("id")) or ref_id_from_ref(ref)
url = _first_string(ref.get("url"), ref.get("safe_url"), ref.get("source_url"))
title = _first_string(ref.get("title"), ref.get("name"), ref.get("text"))
attribution = _first_string(ref.get("attribution"), ref.get("source"), ref.get("site_name"))
if not ref_id or not url:
return None
return CitationRecord(ref_id=ref_id, url=url, title=title, attribution=attribution)
def _walk_dicts(value: Any) -> list[dict[str, Any]]:
found: list[dict[str, Any]] = []
if isinstance(value, dict):
found.append(value)
for child in value.values():
found.extend(_walk_dicts(child))
elif isinstance(value, list):
for child in value:
found.extend(_walk_dicts(child))
return found
def build_citation_map(metadata: dict[str, Any] | None) -> dict[str, CitationRecord]:
if not metadata:
return {}
citation_map: dict[str, CitationRecord] = {}
for node in _walk_dicts(metadata):
record = record_from_ref(node)
if record and record.ref_id not in citation_map:
citation_map[record.ref_id] = record
return citation_map
def split_existing_references(markdown: str) -> tuple[str, str]:
match = REFERENCE_HEADING_RE.search(markdown)
if not match:
return markdown.rstrip(), ""
return markdown[: match.start()].rstrip(), markdown[match.start() :].strip()
def collect_markdown_links(markdown: str) -> list[str]:
seen: set[str] = set()
links: list[str] = []
for link in LINK_RE.findall(markdown):
if link not in seen:
seen.add(link)
links.append(link)
return links
def replace_citations(
markdown: str, citation_map: dict[str, CitationRecord]
) -> tuple[str, list[CitationRecord], list[str]]:
footnote_numbers: dict[str, int] = {}
ordered_records: list[CitationRecord] = []
missing: list[str] = []
def marker_for_ref(ref_id: str) -> str:
if ref_id not in citation_map:
if ref_id not in missing:
missing.append(ref_id)
return ""
if ref_id not in footnote_numbers:
footnote_numbers[ref_id] = len(ordered_records) + 1
ordered_records.append(citation_map[ref_id])
return f"[^{footnote_numbers[ref_id]}]"
output: list[str] = []
in_fence = False
fence_marker = ""
for line in markdown.splitlines(keepends=True):
body = line[:-1] if line.endswith("\n") else line
newline = "\n" if line.endswith("\n") else ""
if body.endswith("\r"):
body = body[:-1]
newline = "\r" + newline
fence_match = FENCE_RE.match(body)
if fence_match:
marker = fence_match.group(2)
marker_char = marker[0]
if not in_fence:
in_fence = True
fence_marker = marker_char * len(marker)
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
in_fence = False
fence_marker = ""
output.append(line)
continue
if in_fence:
output.append(line)
continue
def replace_token(match: re.Match[str]) -> str:
return "".join(marker_for_ref(ref_id) for ref_id in extract_ref_ids(match.group(0)))
output.append(CITATION_RE.sub(replace_token, body) + newline)
return "".join(output), ordered_records, missing
def strip_citation_tokens(markdown: str) -> str:
lines: list[str] = []
in_fence = False
fence_marker = ""
for line in markdown.splitlines(keepends=True):
body = line[:-1] if line.endswith("\n") else line
newline = "\n" if line.endswith("\n") else ""
if body.endswith("\r"):
body = body[:-1]
newline = "\r" + newline
fence_match = FENCE_RE.match(body)
if fence_match:
marker = fence_match.group(2)
marker_char = marker[0]
if not in_fence:
in_fence = True
fence_marker = marker_char * len(marker)
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
in_fence = False
fence_marker = ""
lines.append(line)
continue
if in_fence:
lines.append(line)
else:
lines.append(CITATION_RE.sub("", body) + newline)
return "".join(lines)
def render_reference_section(
records: list[CitationRecord], existing_links: list[str], missing: list[str], degraded: bool
) -> str:
lines = ["## 参考资料"]
if degraded:
lines.append("")
lines.append("> 无法恢复逐句脚注映射:未找到可用元数据或部分 turn ID 缺少 URL。")
if records:
lines.append("")
for index, record in enumerate(records, start=1):
lines.append(f"[^{index}]: {record.footnote_text()}")
if existing_links:
lines.append("")
lines.append("### 原始来源链接")
for link in existing_links:
lines.append(f"- {link}")
if missing:
lines.append("")
lines.append("### 未解析引用 ID")
for ref_id in missing:
lines.append(f"- `{ref_id}`")
return "\n".join(lines)
def repair_markdown(markdown: str, metadata: dict[str, Any] | None = None) -> str:
body, existing_reference_section = split_existing_references(markdown)
existing_links = collect_markdown_links(existing_reference_section)
citation_map = build_citation_map(metadata)
if citation_map:
replaced, records, missing = replace_citations(body, citation_map)
degraded = bool(missing)
cleaned = strip_citation_tokens(replaced) if missing else replaced
reference_section = render_reference_section(records, existing_links, missing, degraded)
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
cleaned = strip_citation_tokens(body)
reference_section = render_reference_section([], existing_links, [], degraded=True)
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
def load_json(path: pathlib.Path) -> dict[str, Any]:
with path.open("r", encoding="utf-8") as handle:
value = json.load(handle)
if not isinstance(value, dict):
raise ValueError("metadata JSON must contain an object at the top level")
return value
def extract_embedded_metadata(markdown: str) -> dict[str, Any] | None:
match = HTML_METADATA_RE.search(markdown)
if not match:
return None
value = json.loads(match.group("json"))
if not isinstance(value, dict):
raise ValueError("embedded metadata JSON must contain an object at the top level")
return value
def parse_args(argv: list[str]) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Repair ChatGPT/Deep Research citation tokens in Markdown."
)
parser.add_argument("path", type=pathlib.Path, help="Markdown file to repair")
parser.add_argument("--metadata", type=pathlib.Path, help="Deep Research metadata JSON")
parser.add_argument("-o", "--output", type=pathlib.Path, help="Write repaired Markdown here")
parser.add_argument("--in-place", action="store_true", help="Rewrite the input Markdown file")
parser.add_argument("--dry-run", action="store_true", help="Print repaired Markdown to stdout")
return parser.parse_args(argv)
def main(argv: list[str]) -> int:
args = parse_args(argv)
if sum(bool(value) for value in (args.output, args.in_place, args.dry_run)) != 1:
print("error: choose exactly one of --output, --in-place, or --dry-run", file=sys.stderr)
return 2
try:
markdown = args.path.read_text(encoding="utf-8")
metadata = load_json(args.metadata) if args.metadata else extract_embedded_metadata(markdown)
fixed = repair_markdown(markdown, metadata=metadata)
except (OSError, ValueError, json.JSONDecodeError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if args.dry_run:
sys.stdout.write(fixed)
return 0
destination = args.path if args.in_place else args.output
assert destination is not None
try:
destination.write_text(fixed, encoding="utf-8")
except OSError as exc:
print(f"error: cannot write {destination}: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))

View File

@ -0,0 +1,185 @@
from __future__ import annotations
import importlib.util
import pathlib
import sys
import textwrap
import unittest
SCRIPT_PATH = (
pathlib.Path(__file__).resolve().parents[1]
/ "scripts"
/ "repair_markdown_citations.py"
)
spec = importlib.util.spec_from_file_location("repair_markdown_citations", SCRIPT_PATH)
repair = importlib.util.module_from_spec(spec)
assert spec is not None
assert spec.loader is not None
sys.modules["repair_markdown_citations"] = repair
spec.loader.exec_module(repair)
class RepairMarkdownCitationsTest(unittest.TestCase):
def test_extracts_turn_ids_from_chatgpt_citation_token(self) -> None:
token = "citeturn34view0turn20search2"
self.assertEqual(
repair.extract_ref_ids(token),
["turn34view0", "turn20search2"],
)
def test_replaces_tokens_with_first_use_footnotes_from_metadata(self) -> None:
markdown = textwrap.dedent(
"""\
武汉早期疫情通报强调未发现明显人传人citeturn34view0
郑州暴雨和唐山事件报道也体现了不同报道框架citeturn20search2turn34view6
"""
)
metadata = {
"content_references": [
{
"items": [
{
"refs": [
{
"turn_index": 34,
"ref_type": "view",
"ref_index": 0,
"url": "https://wjw.hubei.gov.cn/example.shtml",
"title": "湖北省卫健委通报",
"attribution": "湖北省卫健委",
},
{
"turn_index": 20,
"ref_type": "search",
"ref_index": 2,
"url": "https://www.xinhuanet.com/zhengzhou.htm",
"title": "新华社郑州暴雨报道",
},
{
"turn_index": 34,
"ref_type": "view",
"ref_index": 6,
"url": "https://www.xinhuanet.com/tangshan.htm",
"title": "新华社唐山打人事件报道",
},
]
}
]
}
]
}
fixed = repair.repair_markdown(markdown, metadata=metadata)
self.assertIn("武汉早期疫情通报强调未发现明显人传人。[^1]", fixed)
self.assertIn("报道框架。[^2][^3]", fixed)
self.assertIn("## 参考资料", fixed)
self.assertIn(
"[^1]: 湖北省卫健委湖北省卫健委通报。https://wjw.hubei.gov.cn/example.shtml",
fixed,
)
self.assertIn(
"[^2]: 新华社郑州暴雨报道。https://www.xinhuanet.com/zhengzhou.htm",
fixed,
)
self.assertIn(
"[^3]: 新华社唐山打人事件报道。https://www.xinhuanet.com/tangshan.htm",
fixed,
)
def test_deduplicates_repeated_references(self) -> None:
markdown = "Aciteturn1view0 Bciteturn1view0"
metadata = {
"content_references": [
{
"items": [
{
"refs": [
{
"turn_index": 1,
"ref_type": "view",
"ref_index": 0,
"url": "https://example.com/a",
"title": "Example A",
}
]
}
]
}
]
}
fixed = repair.repair_markdown(markdown, metadata=metadata)
self.assertIn("A[^1] B[^1]", fixed)
self.assertEqual(fixed.count("[^1]:"), 1)
def test_without_metadata_removes_tokens_and_adds_warning(self) -> None:
markdown = "正文citeturn1view0\n\n[来源](https://example.com)"
fixed = repair.repair_markdown(markdown, metadata=None)
self.assertNotIn("cite", fixed)
self.assertIn("## 参考资料", fixed)
self.assertIn("[来源](https://example.com)", fixed)
self.assertIn("无法恢复逐句脚注映射", fixed)
def test_loads_embedded_metadata_from_html_comment(self) -> None:
markdown = textwrap.dedent(
"""\
正文citeturn1view0
<!-- deep-research-metadata
{"content_references":[{"items":[{"refs":[{"turn_index":1,"ref_type":"view","ref_index":0,"url":"https://example.com/a","title":"Example A"}]}]}]}
-->
"""
)
metadata = repair.extract_embedded_metadata(markdown)
fixed = repair.repair_markdown(markdown, metadata=metadata)
self.assertIn("正文[^1]", fixed)
self.assertIn("[^1]: Example A。https://example.com/a", fixed)
def test_does_not_rewrite_tokens_inside_fenced_code_blocks(self) -> None:
markdown = textwrap.dedent(
"""\
正文citeturn1view0
```md
示例citeturn1view0
```
"""
)
metadata = {
"content_references": [
{
"items": [
{
"refs": [
{
"turn_index": 1,
"ref_type": "view",
"ref_index": 0,
"url": "https://example.com/a",
"title": "Example A",
}
]
}
]
}
]
}
fixed = repair.repair_markdown(markdown, metadata=metadata)
self.assertIn("正文[^1]", fixed)
self.assertIn("示例citeturn1view0", fixed)
if __name__ == "__main__":
unittest.main()