feat: add markdown citation repair skill
This commit is contained in:
parent
910926838c
commit
86d860044b
|
|
@ -35,6 +35,7 @@ If an automation Skill becomes a dependency of a CCPE runtime, agent, or committ
|
|||
- Prefer small, focused changes.
|
||||
- Keep Skill directories self-contained.
|
||||
- Use lowercase kebab-case for Skill names and file names.
|
||||
- Run Python code and tests through the shared `skills-vault` conda environment by default, for example `conda run -n skills-vault python ...`. Use another environment only when a concrete dependency conflict requires it.
|
||||
- Preserve behavior during first migration; refactor only after baseline behavior is captured.
|
||||
- Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical.
|
||||
- Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository.
|
||||
|
|
|
|||
|
|
@ -16,16 +16,20 @@ C:\Users\wangq\.agents\skills\<skill-name>
|
|||
|
||||
## Current State
|
||||
|
||||
This repository does not yet include an automated sync script.
|
||||
This repository includes install scripts for copying reviewed Skill source into `.agents/skills`.
|
||||
|
||||
For early migrations, install manually or with a narrowly scoped copy command after reviewing the Skill directory.
|
||||
PowerShell:
|
||||
|
||||
Future repository-level scripts may support:
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
|
||||
```
|
||||
|
||||
- listing Skills
|
||||
- validating Skill structure
|
||||
- installing selected Skills
|
||||
- syncing selected Skills to `.agents/skills`
|
||||
- checking registry consistency
|
||||
Use `-Force` to back up and replace an existing installed copy:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
|
||||
```
|
||||
|
||||
`-All` installs only registry rows with an installable status and `default` install policy. Optional Skills must be named explicitly.
|
||||
|
||||
Do not sync CCPE content into `.agents/skills` through this repository.
|
||||
|
|
|
|||
|
|
@ -47,12 +47,22 @@ conda activate C:\Users\wangq\.conda\envs\skills-vault
|
|||
|
||||
Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies.
|
||||
|
||||
When running Python from automation, prefer `conda run` so the intended environment is explicit even if the shell has not been activated:
|
||||
|
||||
```powershell
|
||||
conda run -n skills-vault python -m unittest discover -s skills\fix-title\tests -v
|
||||
conda run -n skills-vault python .\skills\fix-title\scripts\fix_markdown_titles.py --help
|
||||
```
|
||||
|
||||
Do not rely on bare `python` unless you have first verified it resolves to `C:\Users\wangq\.conda\envs\skills-vault\python.exe`.
|
||||
|
||||
## Install One Skill
|
||||
|
||||
PowerShell:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
|
||||
```
|
||||
|
||||
Git Bash / Linux / macOS:
|
||||
|
|
@ -92,6 +102,7 @@ Use `-Force` or `--force` to back up the current installed directory and copy th
|
|||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force
|
||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
|
||||
```
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -0,0 +1,54 @@
|
|||
# Repair Markdown Citations Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build a reusable Skill that repairs ChatGPT/Deep Research Markdown citation tokens into Markdown footnotes using metadata keyed by turn IDs.
|
||||
|
||||
**Architecture:** Add a self-contained Skill under `skills/repair-markdown-citations`. The core behavior lives in a deterministic Python CLI; `SKILL.md` tells agents when to use it and how to handle exact metadata mapping versus fallback cleanup.
|
||||
|
||||
**Tech Stack:** Python standard library, `unittest`, Markdown text processing.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Tests First
|
||||
|
||||
**Files:**
|
||||
- Create: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
|
||||
|
||||
- [ ] Write tests for token parsing, exact turn-ID replacement, deduplicated footnotes, fallback cleanup without metadata, and fenced code block safety.
|
||||
- [ ] Run `python -m unittest discover -s skills/repair-markdown-citations/tests -v` and confirm the tests fail because the module does not exist yet.
|
||||
|
||||
### Task 2: CLI Implementation
|
||||
|
||||
**Files:**
|
||||
- Create: `skills/repair-markdown-citations/scripts/repair_markdown_citations.py`
|
||||
|
||||
- [ ] Implement parsing for `citeturn34view0turn20search2` tokens.
|
||||
- [ ] Build citation records from metadata keyed by refs such as `turn34view0`.
|
||||
- [ ] Replace citation tokens outside fenced code blocks with first-use footnote markers.
|
||||
- [ ] Append a fresh `## 参考资料` section with deduplicated footnote definitions.
|
||||
- [ ] If metadata is unavailable or incomplete, remove citation tokens and append a warning note instead of fabricating footnotes.
|
||||
- [ ] Support `--metadata`, `--output`, `--in-place`, and `--dry-run`.
|
||||
|
||||
### Task 3: Skill Package
|
||||
|
||||
**Files:**
|
||||
- Create: `skills/repair-markdown-citations/SKILL.md`
|
||||
- Create: `skills/repair-markdown-citations/README.md`
|
||||
- Create: `skills/repair-markdown-citations/fixtures/README.md`
|
||||
|
||||
- [ ] Document the three capability levels.
|
||||
- [ ] Make exact turn-ID mapping the required primary strategy.
|
||||
- [ ] Document fallback behavior and safety rules.
|
||||
- [ ] Keep the directory self-contained and lowercase kebab-case.
|
||||
|
||||
### Task 4: Verification
|
||||
|
||||
**Files:**
|
||||
- Test: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
|
||||
- Inspect: `skills/repair-markdown-citations/SKILL.md`
|
||||
|
||||
- [ ] Run the unit test command.
|
||||
- [ ] Run a lightweight frontmatter/naming check.
|
||||
- [ ] Run `git diff --check`.
|
||||
- [ ] Inspect `git status --short` and summarize changed files.
|
||||
|
|
@ -14,6 +14,7 @@ Existing CCPE content is not migrated here.
|
|||
| Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none |
|
||||
| `repair-markdown-citations` | Repair ChatGPT/Deep Research Markdown citation tokens into standard footnotes and a deduplicated reference section using exact turn-ID metadata mapping. | `C:\Users\wangq\Documents\Codex\skills-vault\skills\repair-markdown-citations` | `installed` | `default` | `C:\Users\wangq\.agents\skills\repair-markdown-citations` | none |
|
||||
| `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none |
|
||||
|
||||
## Status Values
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
# Repair Markdown Citations
|
||||
|
||||
Source Skill for repairing ChatGPT/Deep Research citation tokens in Markdown exports.
|
||||
|
||||
The canonical entry point is `SKILL.md`; the deterministic transformer is in `scripts/repair_markdown_citations.py`.
|
||||
|
||||
Run Python code through the shared repository environment:
|
||||
|
||||
```powershell
|
||||
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
|
||||
```
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
---
|
||||
name: repair-markdown-citations
|
||||
description: Use when a Markdown export from ChatGPT Deep Research or another LLM contains broken citation tokens such as `citeturn34view0turn20search2` that need to be converted into standard Markdown footnotes and a deduplicated `## 参考资料` section. Use for repairing turn-ID citations, cleaning citation mojibake, preserving source links, and safely degrading when metadata is missing.
|
||||
---
|
||||
|
||||
# Repair Markdown Citations
|
||||
|
||||
## Purpose
|
||||
|
||||
Repair ChatGPT/Deep Research Markdown citation tokens into readable Markdown. Use the bundled script for deterministic edits instead of rewriting the document by prompt.
|
||||
|
||||
The primary rule is exact ID mapping: treat `turn34view0`, `turn20search2`, and similar IDs as citation primary keys. Do not assume the citation token order matches `safe_urls`, a footer link list, or source appearance order.
|
||||
|
||||
## Capability Levels
|
||||
|
||||
Level 1: no metadata cleanup.
|
||||
|
||||
- Remove malformed `cite...` tokens outside fenced code blocks.
|
||||
- Preserve existing links from an old `## 参考资料` section as ordinary source links.
|
||||
- Add a warning that逐句脚注映射 could not be recovered.
|
||||
|
||||
Level 2: `content_references` metadata.
|
||||
|
||||
- Build a citation map from metadata refs keyed by turn ID.
|
||||
- Replace citation tokens in正文 with Markdown footnote markers such as `[^1][^2]`.
|
||||
- Generate a deduplicated `## 参考资料` section in first-citation order.
|
||||
|
||||
Level 3: complete Deep Research export.
|
||||
|
||||
- Prefer exact `ref_id` / `turn_index` / `ref_type` / `ref_index` fields wherever available.
|
||||
- Use richer fields such as title, attribution, source, and site name when present.
|
||||
- Treat source grouping, dates, snippets, and risk checks as future enrichment, not required first-pass behavior.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Confirm the input is a Markdown file exported from ChatGPT/Deep Research or converted from a report containing citation tokens.
|
||||
2. Prefer an explicit metadata JSON file. If the user has one, pass it with `--metadata`.
|
||||
3. If no metadata file is available, let the script look for an embedded HTML comment block named `deep-research-metadata`.
|
||||
4. Choose output mode:
|
||||
- Use `--output` for a non-destructive repaired copy.
|
||||
- Use `--in-place` only when the user wants the original file rewritten.
|
||||
- Use `--dry-run` before editing important files.
|
||||
5. Inspect the resulting `## 参考资料` section and any `未解析引用 ID` warnings.
|
||||
|
||||
## Commands
|
||||
|
||||
```powershell
|
||||
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --output "C:\path\report.fixed.md"
|
||||
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --in-place
|
||||
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
|
||||
```
|
||||
|
||||
## Embedded Metadata
|
||||
|
||||
The script can read a Markdown HTML comment block:
|
||||
|
||||
```md
|
||||
<!-- deep-research-metadata
|
||||
{"content_references":[...]}
|
||||
-->
|
||||
```
|
||||
|
||||
Use embedded metadata only when an external JSON file is unavailable or when the report intentionally carries its own citation map.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Do not map citations by `safe_urls` order or by footer link order.
|
||||
- Do not fabricate footnotes when a turn ID has no URL in the citation map.
|
||||
- Do not rewrite citation-like text inside fenced code blocks.
|
||||
- Do not delete user-authored prose or non-reference sections.
|
||||
- If metadata is missing or incomplete, preserve recoverable source links and clearly mark the output as degraded.
|
||||
|
||||
## Validation
|
||||
|
||||
Run the Skill tests after modifying behavior:
|
||||
|
||||
```powershell
|
||||
conda run -n skills-vault python -m unittest discover -s skills/repair-markdown-citations/tests -v
|
||||
```
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
display_name: Repair Markdown Citations
|
||||
short_description: Convert broken ChatGPT citation tokens into Markdown references.
|
||||
default_prompt: Repair the broken citation tokens in this Markdown file and produce a clean reference section.
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
# Fixtures
|
||||
|
||||
Place small anonymized Markdown and metadata examples here when adding regression coverage.
|
||||
|
|
@ -0,0 +1,304 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Repair ChatGPT citation tokens in Markdown exports."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
CITATION_RE = re.compile(r"cite(?:turn\d+(?:view|search)\d+)+")
|
||||
REF_ID_RE = re.compile(r"turn\d+(?:view|search)\d+")
|
||||
FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})")
|
||||
LINK_RE = re.compile(r"\[[^\]\n]+\]\([^) \n]+(?: [^)]+)?\)")
|
||||
REFERENCE_HEADING_RE = re.compile(r"^## +参考资料\s*$", re.MULTILINE)
|
||||
HTML_METADATA_RE = re.compile(
|
||||
r"<!--\s*deep-research-metadata\s*(?P<json>\{.*?\})\s*-->",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CitationRecord:
|
||||
ref_id: str
|
||||
url: str
|
||||
title: str = ""
|
||||
attribution: str = ""
|
||||
|
||||
def footnote_text(self) -> str:
|
||||
label = self.title or self.url
|
||||
if self.attribution and self.title:
|
||||
return f"{self.attribution}：{self.title}。{self.url}"
|
||||
return f"{label}。{self.url}"
|
||||
|
||||
|
||||
def extract_ref_ids(token: str) -> list[str]:
|
||||
return REF_ID_RE.findall(token)
|
||||
|
||||
|
||||
def ref_id_from_ref(ref: dict[str, Any]) -> str | None:
|
||||
turn_index = ref.get("turn_index")
|
||||
ref_type = ref.get("ref_type")
|
||||
ref_index = ref.get("ref_index")
|
||||
if turn_index is None or ref_type not in {"view", "search"} or ref_index is None:
|
||||
return None
|
||||
return f"turn{turn_index}{ref_type}{ref_index}"
|
||||
|
||||
|
||||
def _first_string(*values: Any) -> str:
|
||||
for value in values:
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def record_from_ref(ref: dict[str, Any]) -> CitationRecord | None:
|
||||
ref_id = _first_string(ref.get("ref_id"), ref.get("id")) or ref_id_from_ref(ref)
|
||||
url = _first_string(ref.get("url"), ref.get("safe_url"), ref.get("source_url"))
|
||||
title = _first_string(ref.get("title"), ref.get("name"), ref.get("text"))
|
||||
attribution = _first_string(ref.get("attribution"), ref.get("source"), ref.get("site_name"))
|
||||
if not ref_id or not url:
|
||||
return None
|
||||
return CitationRecord(ref_id=ref_id, url=url, title=title, attribution=attribution)
|
||||
|
||||
|
||||
def _walk_dicts(value: Any) -> list[dict[str, Any]]:
|
||||
found: list[dict[str, Any]] = []
|
||||
if isinstance(value, dict):
|
||||
found.append(value)
|
||||
for child in value.values():
|
||||
found.extend(_walk_dicts(child))
|
||||
elif isinstance(value, list):
|
||||
for child in value:
|
||||
found.extend(_walk_dicts(child))
|
||||
return found
|
||||
|
||||
|
||||
def build_citation_map(metadata: dict[str, Any] | None) -> dict[str, CitationRecord]:
|
||||
if not metadata:
|
||||
return {}
|
||||
|
||||
citation_map: dict[str, CitationRecord] = {}
|
||||
for node in _walk_dicts(metadata):
|
||||
record = record_from_ref(node)
|
||||
if record and record.ref_id not in citation_map:
|
||||
citation_map[record.ref_id] = record
|
||||
|
||||
return citation_map
|
||||
|
||||
|
||||
def split_existing_references(markdown: str) -> tuple[str, str]:
|
||||
match = REFERENCE_HEADING_RE.search(markdown)
|
||||
if not match:
|
||||
return markdown.rstrip(), ""
|
||||
return markdown[: match.start()].rstrip(), markdown[match.start() :].strip()
|
||||
|
||||
|
||||
def collect_markdown_links(markdown: str) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
links: list[str] = []
|
||||
for link in LINK_RE.findall(markdown):
|
||||
if link not in seen:
|
||||
seen.add(link)
|
||||
links.append(link)
|
||||
return links
|
||||
|
||||
|
||||
def replace_citations(
|
||||
markdown: str, citation_map: dict[str, CitationRecord]
|
||||
) -> tuple[str, list[CitationRecord], list[str]]:
|
||||
footnote_numbers: dict[str, int] = {}
|
||||
ordered_records: list[CitationRecord] = []
|
||||
missing: list[str] = []
|
||||
|
||||
def marker_for_ref(ref_id: str) -> str:
|
||||
if ref_id not in citation_map:
|
||||
if ref_id not in missing:
|
||||
missing.append(ref_id)
|
||||
return ""
|
||||
if ref_id not in footnote_numbers:
|
||||
footnote_numbers[ref_id] = len(ordered_records) + 1
|
||||
ordered_records.append(citation_map[ref_id])
|
||||
return f"[^{footnote_numbers[ref_id]}]"
|
||||
|
||||
output: list[str] = []
|
||||
in_fence = False
|
||||
fence_marker = ""
|
||||
|
||||
for line in markdown.splitlines(keepends=True):
|
||||
body = line[:-1] if line.endswith("\n") else line
|
||||
newline = "\n" if line.endswith("\n") else ""
|
||||
if body.endswith("\r"):
|
||||
body = body[:-1]
|
||||
newline = "\r" + newline
|
||||
|
||||
fence_match = FENCE_RE.match(body)
|
||||
if fence_match:
|
||||
marker = fence_match.group(2)
|
||||
marker_char = marker[0]
|
||||
if not in_fence:
|
||||
in_fence = True
|
||||
fence_marker = marker_char * len(marker)
|
||||
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
|
||||
in_fence = False
|
||||
fence_marker = ""
|
||||
output.append(line)
|
||||
continue
|
||||
|
||||
if in_fence:
|
||||
output.append(line)
|
||||
continue
|
||||
|
||||
def replace_token(match: re.Match[str]) -> str:
|
||||
return "".join(marker_for_ref(ref_id) for ref_id in extract_ref_ids(match.group(0)))
|
||||
|
||||
output.append(CITATION_RE.sub(replace_token, body) + newline)
|
||||
|
||||
return "".join(output), ordered_records, missing
|
||||
|
||||
|
||||
def strip_citation_tokens(markdown: str) -> str:
|
||||
lines: list[str] = []
|
||||
in_fence = False
|
||||
fence_marker = ""
|
||||
|
||||
for line in markdown.splitlines(keepends=True):
|
||||
body = line[:-1] if line.endswith("\n") else line
|
||||
newline = "\n" if line.endswith("\n") else ""
|
||||
if body.endswith("\r"):
|
||||
body = body[:-1]
|
||||
newline = "\r" + newline
|
||||
|
||||
fence_match = FENCE_RE.match(body)
|
||||
if fence_match:
|
||||
marker = fence_match.group(2)
|
||||
marker_char = marker[0]
|
||||
if not in_fence:
|
||||
in_fence = True
|
||||
fence_marker = marker_char * len(marker)
|
||||
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
|
||||
in_fence = False
|
||||
fence_marker = ""
|
||||
lines.append(line)
|
||||
continue
|
||||
|
||||
if in_fence:
|
||||
lines.append(line)
|
||||
else:
|
||||
lines.append(CITATION_RE.sub("", body) + newline)
|
||||
|
||||
return "".join(lines)
|
||||
|
||||
|
||||
def render_reference_section(
|
||||
records: list[CitationRecord], existing_links: list[str], missing: list[str], degraded: bool
|
||||
) -> str:
|
||||
lines = ["## 参考资料"]
|
||||
|
||||
if degraded:
|
||||
lines.append("")
|
||||
lines.append("> 无法恢复逐句脚注映射：未找到可用元数据或部分 turn ID 缺少 URL。")
|
||||
|
||||
if records:
|
||||
lines.append("")
|
||||
for index, record in enumerate(records, start=1):
|
||||
lines.append(f"[^{index}]: {record.footnote_text()}")
|
||||
|
||||
if existing_links:
|
||||
lines.append("")
|
||||
lines.append("### 原始来源链接")
|
||||
for link in existing_links:
|
||||
lines.append(f"- {link}")
|
||||
|
||||
if missing:
|
||||
lines.append("")
|
||||
lines.append("### 未解析引用 ID")
|
||||
for ref_id in missing:
|
||||
lines.append(f"- `{ref_id}`")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def repair_markdown(markdown: str, metadata: dict[str, Any] | None = None) -> str:
|
||||
body, existing_reference_section = split_existing_references(markdown)
|
||||
existing_links = collect_markdown_links(existing_reference_section)
|
||||
citation_map = build_citation_map(metadata)
|
||||
|
||||
if citation_map:
|
||||
replaced, records, missing = replace_citations(body, citation_map)
|
||||
degraded = bool(missing)
|
||||
cleaned = strip_citation_tokens(replaced) if missing else replaced
|
||||
reference_section = render_reference_section(records, existing_links, missing, degraded)
|
||||
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
|
||||
|
||||
cleaned = strip_citation_tokens(body)
|
||||
reference_section = render_reference_section([], existing_links, [], degraded=True)
|
||||
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
|
||||
|
||||
|
||||
def load_json(path: pathlib.Path) -> dict[str, Any]:
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
value = json.load(handle)
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("metadata JSON must contain an object at the top level")
|
||||
return value
|
||||
|
||||
|
||||
def extract_embedded_metadata(markdown: str) -> dict[str, Any] | None:
|
||||
match = HTML_METADATA_RE.search(markdown)
|
||||
if not match:
|
||||
return None
|
||||
value = json.loads(match.group("json"))
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("embedded metadata JSON must contain an object at the top level")
|
||||
return value
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Repair ChatGPT/Deep Research citation tokens in Markdown."
|
||||
)
|
||||
parser.add_argument("path", type=pathlib.Path, help="Markdown file to repair")
|
||||
parser.add_argument("--metadata", type=pathlib.Path, help="Deep Research metadata JSON")
|
||||
parser.add_argument("-o", "--output", type=pathlib.Path, help="Write repaired Markdown here")
|
||||
parser.add_argument("--in-place", action="store_true", help="Rewrite the input Markdown file")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Print repaired Markdown to stdout")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
args = parse_args(argv)
|
||||
if sum(bool(value) for value in (args.output, args.in_place, args.dry_run)) != 1:
|
||||
print("error: choose exactly one of --output, --in-place, or --dry-run", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
try:
|
||||
markdown = args.path.read_text(encoding="utf-8")
|
||||
metadata = load_json(args.metadata) if args.metadata else extract_embedded_metadata(markdown)
|
||||
fixed = repair_markdown(markdown, metadata=metadata)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.dry_run:
|
||||
sys.stdout.write(fixed)
|
||||
return 0
|
||||
|
||||
destination = args.path if args.in_place else args.output
|
||||
assert destination is not None
|
||||
try:
|
||||
destination.write_text(fixed, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"error: cannot write {destination}: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,185 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import sys
|
||||
import textwrap
|
||||
import unittest
|
||||
|
||||
|
||||
SCRIPT_PATH = (
|
||||
pathlib.Path(__file__).resolve().parents[1]
|
||||
/ "scripts"
|
||||
/ "repair_markdown_citations.py"
|
||||
)
|
||||
|
||||
spec = importlib.util.spec_from_file_location("repair_markdown_citations", SCRIPT_PATH)
|
||||
repair = importlib.util.module_from_spec(spec)
|
||||
assert spec is not None
|
||||
assert spec.loader is not None
|
||||
sys.modules["repair_markdown_citations"] = repair
|
||||
spec.loader.exec_module(repair)
|
||||
|
||||
|
||||
class RepairMarkdownCitationsTest(unittest.TestCase):
|
||||
def test_extracts_turn_ids_from_chatgpt_citation_token(self) -> None:
|
||||
token = "citeturn34view0turn20search2"
|
||||
|
||||
self.assertEqual(
|
||||
repair.extract_ref_ids(token),
|
||||
["turn34view0", "turn20search2"],
|
||||
)
|
||||
|
||||
def test_replaces_tokens_with_first_use_footnotes_from_metadata(self) -> None:
|
||||
markdown = textwrap.dedent(
|
||||
"""\
|
||||
武汉早期疫情通报强调未发现明显人传人。citeturn34view0
|
||||
|
||||
郑州暴雨和唐山事件报道也体现了不同报道框架。citeturn20search2turn34view6
|
||||
"""
|
||||
)
|
||||
metadata = {
|
||||
"content_references": [
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"refs": [
|
||||
{
|
||||
"turn_index": 34,
|
||||
"ref_type": "view",
|
||||
"ref_index": 0,
|
||||
"url": "https://wjw.hubei.gov.cn/example.shtml",
|
||||
"title": "湖北省卫健委通报",
|
||||
"attribution": "湖北省卫健委",
|
||||
},
|
||||
{
|
||||
"turn_index": 20,
|
||||
"ref_type": "search",
|
||||
"ref_index": 2,
|
||||
"url": "https://www.xinhuanet.com/zhengzhou.htm",
|
||||
"title": "新华社郑州暴雨报道",
|
||||
},
|
||||
{
|
||||
"turn_index": 34,
|
||||
"ref_type": "view",
|
||||
"ref_index": 6,
|
||||
"url": "https://www.xinhuanet.com/tangshan.htm",
|
||||
"title": "新华社唐山打人事件报道",
|
||||
},
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||
|
||||
self.assertIn("武汉早期疫情通报强调未发现明显人传人。[^1]", fixed)
|
||||
self.assertIn("报道框架。[^2][^3]", fixed)
|
||||
self.assertIn("## 参考资料", fixed)
|
||||
self.assertIn(
|
||||
"[^1]: 湖北省卫健委：湖北省卫健委通报。https://wjw.hubei.gov.cn/example.shtml",
|
||||
fixed,
|
||||
)
|
||||
self.assertIn(
|
||||
"[^2]: 新华社郑州暴雨报道。https://www.xinhuanet.com/zhengzhou.htm",
|
||||
fixed,
|
||||
)
|
||||
self.assertIn(
|
||||
"[^3]: 新华社唐山打人事件报道。https://www.xinhuanet.com/tangshan.htm",
|
||||
fixed,
|
||||
)
|
||||
|
||||
def test_deduplicates_repeated_references(self) -> None:
|
||||
markdown = "Aciteturn1view0 Bciteturn1view0"
|
||||
metadata = {
|
||||
"content_references": [
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"refs": [
|
||||
{
|
||||
"turn_index": 1,
|
||||
"ref_type": "view",
|
||||
"ref_index": 0,
|
||||
"url": "https://example.com/a",
|
||||
"title": "Example A",
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||
|
||||
self.assertIn("A[^1] B[^1]", fixed)
|
||||
self.assertEqual(fixed.count("[^1]:"), 1)
|
||||
|
||||
def test_without_metadata_removes_tokens_and_adds_warning(self) -> None:
|
||||
markdown = "正文citeturn1view0\n\n[来源](https://example.com)"
|
||||
|
||||
fixed = repair.repair_markdown(markdown, metadata=None)
|
||||
|
||||
self.assertNotIn("cite", fixed)
|
||||
self.assertIn("## 参考资料", fixed)
|
||||
self.assertIn("[来源](https://example.com)", fixed)
|
||||
self.assertIn("无法恢复逐句脚注映射", fixed)
|
||||
|
||||
def test_loads_embedded_metadata_from_html_comment(self) -> None:
|
||||
markdown = textwrap.dedent(
|
||||
"""\
|
||||
正文citeturn1view0
|
||||
|
||||
<!-- deep-research-metadata
|
||||
{"content_references":[{"items":[{"refs":[{"turn_index":1,"ref_type":"view","ref_index":0,"url":"https://example.com/a","title":"Example A"}]}]}]}
|
||||
-->
|
||||
"""
|
||||
)
|
||||
|
||||
metadata = repair.extract_embedded_metadata(markdown)
|
||||
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||
|
||||
self.assertIn("正文[^1]", fixed)
|
||||
self.assertIn("[^1]: Example A。https://example.com/a", fixed)
|
||||
|
||||
def test_does_not_rewrite_tokens_inside_fenced_code_blocks(self) -> None:
|
||||
markdown = textwrap.dedent(
|
||||
"""\
|
||||
正文citeturn1view0
|
||||
|
||||
```md
|
||||
示例citeturn1view0
|
||||
```
|
||||
"""
|
||||
)
|
||||
metadata = {
|
||||
"content_references": [
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"refs": [
|
||||
{
|
||||
"turn_index": 1,
|
||||
"ref_type": "view",
|
||||
"ref_index": 0,
|
||||
"url": "https://example.com/a",
|
||||
"title": "Example A",
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||
|
||||
self.assertIn("正文[^1]", fixed)
|
||||
self.assertIn("示例citeturn1view0", fixed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Reference in New Issue