feat: add markdown citation repair skill
This commit is contained in:
parent
910926838c
commit
86d860044b
|
|
@ -35,6 +35,7 @@ If an automation Skill becomes a dependency of a CCPE runtime, agent, or committ
|
||||||
- Prefer small, focused changes.
|
- Prefer small, focused changes.
|
||||||
- Keep Skill directories self-contained.
|
- Keep Skill directories self-contained.
|
||||||
- Use lowercase kebab-case for Skill names and file names.
|
- Use lowercase kebab-case for Skill names and file names.
|
||||||
|
- Run Python code and tests through the shared `skills-vault` conda environment by default, for example `conda run -n skills-vault python ...`. Use another environment only when a concrete dependency conflict requires it.
|
||||||
- Preserve behavior during first migration; refactor only after baseline behavior is captured.
|
- Preserve behavior during first migration; refactor only after baseline behavior is captured.
|
||||||
- Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical.
|
- Add tests for transformations, file rewrites, parsing behavior, and safety checks when practical.
|
||||||
- Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository.
|
- Do not write secrets, tokens, machine-specific credentials, or generated cache files into the repository.
|
||||||
|
|
|
||||||
|
|
@ -16,16 +16,20 @@ C:\Users\wangq\.agents\skills\<skill-name>
|
||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
This repository does not yet include an automated sync script.
|
This repository includes install scripts for copying reviewed Skill source into `.agents/skills`.
|
||||||
|
|
||||||
For early migrations, install manually or with a narrowly scoped copy command after reviewing the Skill directory.
|
PowerShell:
|
||||||
|
|
||||||
Future repository-level scripts may support:
|
```powershell
|
||||||
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
|
||||||
|
```
|
||||||
|
|
||||||
- listing Skills
|
Use `-Force` to back up and replace an existing installed copy:
|
||||||
- validating Skill structure
|
|
||||||
- installing selected Skills
|
```powershell
|
||||||
- syncing selected Skills to `.agents/skills`
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
|
||||||
- checking registry consistency
|
```
|
||||||
|
|
||||||
|
`-All` installs only registry rows with an installable status and `default` install policy. Optional Skills must be named explicitly.
|
||||||
|
|
||||||
Do not sync CCPE content into `.agents/skills` through this repository.
|
Do not sync CCPE content into `.agents/skills` through this repository.
|
||||||
|
|
|
||||||
|
|
@ -47,12 +47,22 @@ conda activate C:\Users\wangq\.conda\envs\skills-vault
|
||||||
|
|
||||||
Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies.
|
Most lightweight Skills should use this shared environment. Create a dedicated environment only when a Skill has conflicting or heavy dependencies.
|
||||||
|
|
||||||
|
When running Python from automation, prefer `conda run` so the intended environment is explicit even if the shell has not been activated:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
conda run -n skills-vault python -m unittest discover -s skills\fix-title\tests -v
|
||||||
|
conda run -n skills-vault python .\skills\fix-title\scripts\fix_markdown_titles.py --help
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not rely on bare `python` unless you have first verified it resolves to `C:\Users\wangq\.conda\envs\skills-vault\python.exe`.
|
||||||
|
|
||||||
## Install One Skill
|
## Install One Skill
|
||||||
|
|
||||||
PowerShell:
|
PowerShell:
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title
|
||||||
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations
|
||||||
```
|
```
|
||||||
|
|
||||||
Git Bash / Linux / macOS:
|
Git Bash / Linux / macOS:
|
||||||
|
|
@ -92,6 +102,7 @@ Use `-Force` or `--force` to back up the current installed directory and copy th
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill fix-title -Force
|
||||||
|
powershell -ExecutionPolicy Bypass -File .\scripts\install-skill.ps1 -Skill repair-markdown-citations -Force
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
# Repair Markdown Citations Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Build a reusable Skill that repairs ChatGPT/Deep Research Markdown citation tokens into Markdown footnotes using metadata keyed by turn IDs.
|
||||||
|
|
||||||
|
**Architecture:** Add a self-contained Skill under `skills/repair-markdown-citations`. The core behavior lives in a deterministic Python CLI; `SKILL.md` tells agents when to use it and how to handle exact metadata mapping versus fallback cleanup.
|
||||||
|
|
||||||
|
**Tech Stack:** Python standard library, `unittest`, Markdown text processing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Tests First
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
|
||||||
|
|
||||||
|
- [ ] Write tests for token parsing, exact turn-ID replacement, deduplicated footnotes, fallback cleanup without metadata, and fenced code block safety.
|
||||||
|
- [ ] Run `python -m unittest discover -s skills/repair-markdown-citations/tests -v` and confirm the tests fail because the module does not exist yet.
|
||||||
|
|
||||||
|
### Task 2: CLI Implementation
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `skills/repair-markdown-citations/scripts/repair_markdown_citations.py`
|
||||||
|
|
||||||
|
- [ ] Implement parsing for `citeturn34view0turn20search2` tokens.
|
||||||
|
- [ ] Build citation records from metadata keyed by refs such as `turn34view0`.
|
||||||
|
- [ ] Replace citation tokens outside fenced code blocks with first-use footnote markers.
|
||||||
|
- [ ] Append a fresh `## 参考资料` section with deduplicated footnote definitions.
|
||||||
|
- [ ] If metadata is unavailable or incomplete, remove citation tokens and append a warning note instead of fabricating footnotes.
|
||||||
|
- [ ] Support `--metadata`, `--output`, `--in-place`, and `--dry-run`.
|
||||||
|
|
||||||
|
### Task 3: Skill Package
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `skills/repair-markdown-citations/SKILL.md`
|
||||||
|
- Create: `skills/repair-markdown-citations/README.md`
|
||||||
|
- Create: `skills/repair-markdown-citations/fixtures/README.md`
|
||||||
|
|
||||||
|
- [ ] Document the three capability levels.
|
||||||
|
- [ ] Make exact turn-ID mapping the required primary strategy.
|
||||||
|
- [ ] Document fallback behavior and safety rules.
|
||||||
|
- [ ] Keep the directory self-contained and lowercase kebab-case.
|
||||||
|
|
||||||
|
### Task 4: Verification
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Test: `skills/repair-markdown-citations/tests/test_repair_markdown_citations.py`
|
||||||
|
- Inspect: `skills/repair-markdown-citations/SKILL.md`
|
||||||
|
|
||||||
|
- [ ] Run the unit test command.
|
||||||
|
- [ ] Run a lightweight frontmatter/naming check.
|
||||||
|
- [ ] Run `git diff --check`.
|
||||||
|
- [ ] Inspect `git status --short` and summarize changed files.
|
||||||
|
|
@ -14,6 +14,7 @@ Existing CCPE content is not migrated here.
|
||||||
| Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration |
|
| Skill | Purpose | Source | Status | Install Policy | Installed Path | CCPE Registration |
|
||||||
| --- | --- | --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- | --- | --- |
|
||||||
| `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none |
|
| `fix-title` | Shift Markdown ATX heading depth for pasted LLM replies before insertion under parent sections. | `C:\Users\wangq\Documents\Codex\knowledge-vault\skills\fix-title` | `installed` | `default` | `C:\Users\wangq\.agents\skills\fix-title` | none |
|
||||||
|
| `repair-markdown-citations` | Repair ChatGPT/Deep Research Markdown citation tokens into standard footnotes and a deduplicated reference section using exact turn-ID metadata mapping. | `C:\Users\wangq\Documents\Codex\skills-vault\skills\repair-markdown-citations` | `installed` | `default` | `C:\Users\wangq\.agents\skills\repair-markdown-citations` | none |
|
||||||
| `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none |
|
| `voice-generation` | Generate spoken audio from Markdown scripts using the local `voice-gen` CLI and MiniMax `mmx`, including custom voice clone registration and voice registry management. | `D:\AI\ClaudeCode\voice-ge` | `installed` | `conditional` | `C:\Users\wangq\.agents\skills\voice-generation` | none |
|
||||||
|
|
||||||
## Status Values
|
## Status Values
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
# Repair Markdown Citations
|
||||||
|
|
||||||
|
Source Skill for repairing ChatGPT/Deep Research citation tokens in Markdown exports.
|
||||||
|
|
||||||
|
The canonical entry point is `SKILL.md`; the deterministic transformer is in `scripts/repair_markdown_citations.py`.
|
||||||
|
|
||||||
|
Run Python code through the shared repository environment:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,79 @@
|
||||||
|
---
|
||||||
|
name: repair-markdown-citations
|
||||||
|
description: Use when a Markdown export from ChatGPT Deep Research or another LLM contains broken citation tokens such as `citeturn34view0turn20search2` that need to be converted into standard Markdown footnotes and a deduplicated `## 参考资料` section. Use for repairing turn-ID citations, cleaning citation mojibake, preserving source links, and safely degrading when metadata is missing.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Repair Markdown Citations
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Repair ChatGPT/Deep Research Markdown citation tokens into readable Markdown. Use the bundled script for deterministic edits instead of rewriting the document by prompt.
|
||||||
|
|
||||||
|
The primary rule is exact ID mapping: treat `turn34view0`, `turn20search2`, and similar IDs as citation primary keys. Do not assume the citation token order matches `safe_urls`, a footer link list, or source appearance order.
|
||||||
|
|
||||||
|
## Capability Levels
|
||||||
|
|
||||||
|
Level 1: no metadata cleanup.
|
||||||
|
|
||||||
|
- Remove malformed `cite...` tokens outside fenced code blocks.
|
||||||
|
- Preserve existing links from an old `## 参考资料` section as ordinary source links.
|
||||||
|
- Add a warning that逐句脚注映射 could not be recovered.
|
||||||
|
|
||||||
|
Level 2: `content_references` metadata.
|
||||||
|
|
||||||
|
- Build a citation map from metadata refs keyed by turn ID.
|
||||||
|
- Replace citation tokens in正文 with Markdown footnote markers such as `[^1][^2]`.
|
||||||
|
- Generate a deduplicated `## 参考资料` section in first-citation order.
|
||||||
|
|
||||||
|
Level 3: complete Deep Research export.
|
||||||
|
|
||||||
|
- Prefer exact `ref_id` / `turn_index` / `ref_type` / `ref_index` fields wherever available.
|
||||||
|
- Use richer fields such as title, attribution, source, and site name when present.
|
||||||
|
- Treat source grouping, dates, snippets, and risk checks as future enrichment, not required first-pass behavior.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
1. Confirm the input is a Markdown file exported from ChatGPT/Deep Research or converted from a report containing citation tokens.
|
||||||
|
2. Prefer an explicit metadata JSON file. If the user has one, pass it with `--metadata`.
|
||||||
|
3. If no metadata file is available, let the script look for an embedded HTML comment block named `deep-research-metadata`.
|
||||||
|
4. Choose output mode:
|
||||||
|
- Use `--output` for a non-destructive repaired copy.
|
||||||
|
- Use `--in-place` only when the user wants the original file rewritten.
|
||||||
|
- Use `--dry-run` before editing important files.
|
||||||
|
5. Inspect the resulting `## 参考资料` section and any `未解析引用 ID` warnings.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --output "C:\path\report.fixed.md"
|
||||||
|
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --metadata "C:\path\report.metadata.json" --in-place
|
||||||
|
conda run -n skills-vault python .\scripts\repair_markdown_citations.py "C:\path\report.md" --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
## Embedded Metadata
|
||||||
|
|
||||||
|
The script can read a Markdown HTML comment block:
|
||||||
|
|
||||||
|
```md
|
||||||
|
<!-- deep-research-metadata
|
||||||
|
{"content_references":[...]}
|
||||||
|
-->
|
||||||
|
```
|
||||||
|
|
||||||
|
Use embedded metadata only when an external JSON file is unavailable or when the report intentionally carries its own citation map.
|
||||||
|
|
||||||
|
## Safety Rules
|
||||||
|
|
||||||
|
- Do not map citations by `safe_urls` order or by footer link order.
|
||||||
|
- Do not fabricate footnotes when a turn ID has no URL in the citation map.
|
||||||
|
- Do not rewrite citation-like text inside fenced code blocks.
|
||||||
|
- Do not delete user-authored prose or non-reference sections.
|
||||||
|
- If metadata is missing or incomplete, preserve recoverable source links and clearly mark the output as degraded.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Run the Skill tests after modifying behavior:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
conda run -n skills-vault python -m unittest discover -s skills/repair-markdown-citations/tests -v
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
display_name: Repair Markdown Citations
|
||||||
|
short_description: Convert broken ChatGPT citation tokens into Markdown references.
|
||||||
|
default_prompt: Repair the broken citation tokens in this Markdown file and produce a clean reference section.
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
# Fixtures
|
||||||
|
|
||||||
|
Place small anonymized Markdown and metadata examples here when adding regression coverage.
|
||||||
|
|
@ -0,0 +1,304 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Repair ChatGPT citation tokens in Markdown exports."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
CITATION_RE = re.compile(r"cite(?:turn\d+(?:view|search)\d+)+")
|
||||||
|
REF_ID_RE = re.compile(r"turn\d+(?:view|search)\d+")
|
||||||
|
FENCE_RE = re.compile(r"^( {0,3})(`{3,}|~{3,})")
|
||||||
|
LINK_RE = re.compile(r"\[[^\]\n]+\]\([^) \n]+(?: [^)]+)?\)")
|
||||||
|
REFERENCE_HEADING_RE = re.compile(r"^## +参考资料\s*$", re.MULTILINE)
|
||||||
|
HTML_METADATA_RE = re.compile(
|
||||||
|
r"<!--\s*deep-research-metadata\s*(?P<json>\{.*?\})\s*-->",
|
||||||
|
re.DOTALL,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class CitationRecord:
|
||||||
|
ref_id: str
|
||||||
|
url: str
|
||||||
|
title: str = ""
|
||||||
|
attribution: str = ""
|
||||||
|
|
||||||
|
def footnote_text(self) -> str:
|
||||||
|
label = self.title or self.url
|
||||||
|
if self.attribution and self.title:
|
||||||
|
return f"{self.attribution}:{self.title}。{self.url}"
|
||||||
|
return f"{label}。{self.url}"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_ref_ids(token: str) -> list[str]:
|
||||||
|
return REF_ID_RE.findall(token)
|
||||||
|
|
||||||
|
|
||||||
|
def ref_id_from_ref(ref: dict[str, Any]) -> str | None:
|
||||||
|
turn_index = ref.get("turn_index")
|
||||||
|
ref_type = ref.get("ref_type")
|
||||||
|
ref_index = ref.get("ref_index")
|
||||||
|
if turn_index is None or ref_type not in {"view", "search"} or ref_index is None:
|
||||||
|
return None
|
||||||
|
return f"turn{turn_index}{ref_type}{ref_index}"
|
||||||
|
|
||||||
|
|
||||||
|
def _first_string(*values: Any) -> str:
|
||||||
|
for value in values:
|
||||||
|
if isinstance(value, str) and value.strip():
|
||||||
|
return value.strip()
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def record_from_ref(ref: dict[str, Any]) -> CitationRecord | None:
|
||||||
|
ref_id = _first_string(ref.get("ref_id"), ref.get("id")) or ref_id_from_ref(ref)
|
||||||
|
url = _first_string(ref.get("url"), ref.get("safe_url"), ref.get("source_url"))
|
||||||
|
title = _first_string(ref.get("title"), ref.get("name"), ref.get("text"))
|
||||||
|
attribution = _first_string(ref.get("attribution"), ref.get("source"), ref.get("site_name"))
|
||||||
|
if not ref_id or not url:
|
||||||
|
return None
|
||||||
|
return CitationRecord(ref_id=ref_id, url=url, title=title, attribution=attribution)
|
||||||
|
|
||||||
|
|
||||||
|
def _walk_dicts(value: Any) -> list[dict[str, Any]]:
|
||||||
|
found: list[dict[str, Any]] = []
|
||||||
|
if isinstance(value, dict):
|
||||||
|
found.append(value)
|
||||||
|
for child in value.values():
|
||||||
|
found.extend(_walk_dicts(child))
|
||||||
|
elif isinstance(value, list):
|
||||||
|
for child in value:
|
||||||
|
found.extend(_walk_dicts(child))
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def build_citation_map(metadata: dict[str, Any] | None) -> dict[str, CitationRecord]:
|
||||||
|
if not metadata:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
citation_map: dict[str, CitationRecord] = {}
|
||||||
|
for node in _walk_dicts(metadata):
|
||||||
|
record = record_from_ref(node)
|
||||||
|
if record and record.ref_id not in citation_map:
|
||||||
|
citation_map[record.ref_id] = record
|
||||||
|
|
||||||
|
return citation_map
|
||||||
|
|
||||||
|
|
||||||
|
def split_existing_references(markdown: str) -> tuple[str, str]:
|
||||||
|
match = REFERENCE_HEADING_RE.search(markdown)
|
||||||
|
if not match:
|
||||||
|
return markdown.rstrip(), ""
|
||||||
|
return markdown[: match.start()].rstrip(), markdown[match.start() :].strip()
|
||||||
|
|
||||||
|
|
||||||
|
def collect_markdown_links(markdown: str) -> list[str]:
|
||||||
|
seen: set[str] = set()
|
||||||
|
links: list[str] = []
|
||||||
|
for link in LINK_RE.findall(markdown):
|
||||||
|
if link not in seen:
|
||||||
|
seen.add(link)
|
||||||
|
links.append(link)
|
||||||
|
return links
|
||||||
|
|
||||||
|
|
||||||
|
def replace_citations(
|
||||||
|
markdown: str, citation_map: dict[str, CitationRecord]
|
||||||
|
) -> tuple[str, list[CitationRecord], list[str]]:
|
||||||
|
footnote_numbers: dict[str, int] = {}
|
||||||
|
ordered_records: list[CitationRecord] = []
|
||||||
|
missing: list[str] = []
|
||||||
|
|
||||||
|
def marker_for_ref(ref_id: str) -> str:
|
||||||
|
if ref_id not in citation_map:
|
||||||
|
if ref_id not in missing:
|
||||||
|
missing.append(ref_id)
|
||||||
|
return ""
|
||||||
|
if ref_id not in footnote_numbers:
|
||||||
|
footnote_numbers[ref_id] = len(ordered_records) + 1
|
||||||
|
ordered_records.append(citation_map[ref_id])
|
||||||
|
return f"[^{footnote_numbers[ref_id]}]"
|
||||||
|
|
||||||
|
output: list[str] = []
|
||||||
|
in_fence = False
|
||||||
|
fence_marker = ""
|
||||||
|
|
||||||
|
for line in markdown.splitlines(keepends=True):
|
||||||
|
body = line[:-1] if line.endswith("\n") else line
|
||||||
|
newline = "\n" if line.endswith("\n") else ""
|
||||||
|
if body.endswith("\r"):
|
||||||
|
body = body[:-1]
|
||||||
|
newline = "\r" + newline
|
||||||
|
|
||||||
|
fence_match = FENCE_RE.match(body)
|
||||||
|
if fence_match:
|
||||||
|
marker = fence_match.group(2)
|
||||||
|
marker_char = marker[0]
|
||||||
|
if not in_fence:
|
||||||
|
in_fence = True
|
||||||
|
fence_marker = marker_char * len(marker)
|
||||||
|
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
|
||||||
|
in_fence = False
|
||||||
|
fence_marker = ""
|
||||||
|
output.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if in_fence:
|
||||||
|
output.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
def replace_token(match: re.Match[str]) -> str:
|
||||||
|
return "".join(marker_for_ref(ref_id) for ref_id in extract_ref_ids(match.group(0)))
|
||||||
|
|
||||||
|
output.append(CITATION_RE.sub(replace_token, body) + newline)
|
||||||
|
|
||||||
|
return "".join(output), ordered_records, missing
|
||||||
|
|
||||||
|
|
||||||
|
def strip_citation_tokens(markdown: str) -> str:
|
||||||
|
lines: list[str] = []
|
||||||
|
in_fence = False
|
||||||
|
fence_marker = ""
|
||||||
|
|
||||||
|
for line in markdown.splitlines(keepends=True):
|
||||||
|
body = line[:-1] if line.endswith("\n") else line
|
||||||
|
newline = "\n" if line.endswith("\n") else ""
|
||||||
|
if body.endswith("\r"):
|
||||||
|
body = body[:-1]
|
||||||
|
newline = "\r" + newline
|
||||||
|
|
||||||
|
fence_match = FENCE_RE.match(body)
|
||||||
|
if fence_match:
|
||||||
|
marker = fence_match.group(2)
|
||||||
|
marker_char = marker[0]
|
||||||
|
if not in_fence:
|
||||||
|
in_fence = True
|
||||||
|
fence_marker = marker_char * len(marker)
|
||||||
|
elif marker_char == fence_marker[0] and len(marker) >= len(fence_marker):
|
||||||
|
in_fence = False
|
||||||
|
fence_marker = ""
|
||||||
|
lines.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if in_fence:
|
||||||
|
lines.append(line)
|
||||||
|
else:
|
||||||
|
lines.append(CITATION_RE.sub("", body) + newline)
|
||||||
|
|
||||||
|
return "".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def render_reference_section(
|
||||||
|
records: list[CitationRecord], existing_links: list[str], missing: list[str], degraded: bool
|
||||||
|
) -> str:
|
||||||
|
lines = ["## 参考资料"]
|
||||||
|
|
||||||
|
if degraded:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("> 无法恢复逐句脚注映射:未找到可用元数据或部分 turn ID 缺少 URL。")
|
||||||
|
|
||||||
|
if records:
|
||||||
|
lines.append("")
|
||||||
|
for index, record in enumerate(records, start=1):
|
||||||
|
lines.append(f"[^{index}]: {record.footnote_text()}")
|
||||||
|
|
||||||
|
if existing_links:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("### 原始来源链接")
|
||||||
|
for link in existing_links:
|
||||||
|
lines.append(f"- {link}")
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("### 未解析引用 ID")
|
||||||
|
for ref_id in missing:
|
||||||
|
lines.append(f"- `{ref_id}`")
|
||||||
|
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def repair_markdown(markdown: str, metadata: dict[str, Any] | None = None) -> str:
|
||||||
|
body, existing_reference_section = split_existing_references(markdown)
|
||||||
|
existing_links = collect_markdown_links(existing_reference_section)
|
||||||
|
citation_map = build_citation_map(metadata)
|
||||||
|
|
||||||
|
if citation_map:
|
||||||
|
replaced, records, missing = replace_citations(body, citation_map)
|
||||||
|
degraded = bool(missing)
|
||||||
|
cleaned = strip_citation_tokens(replaced) if missing else replaced
|
||||||
|
reference_section = render_reference_section(records, existing_links, missing, degraded)
|
||||||
|
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
|
||||||
|
|
||||||
|
cleaned = strip_citation_tokens(body)
|
||||||
|
reference_section = render_reference_section([], existing_links, [], degraded=True)
|
||||||
|
return f"{cleaned.rstrip()}\n\n{reference_section}\n"
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: pathlib.Path) -> dict[str, Any]:
|
||||||
|
with path.open("r", encoding="utf-8") as handle:
|
||||||
|
value = json.load(handle)
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError("metadata JSON must contain an object at the top level")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def extract_embedded_metadata(markdown: str) -> dict[str, Any] | None:
|
||||||
|
match = HTML_METADATA_RE.search(markdown)
|
||||||
|
if not match:
|
||||||
|
return None
|
||||||
|
value = json.loads(match.group("json"))
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError("embedded metadata JSON must contain an object at the top level")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Repair ChatGPT/Deep Research citation tokens in Markdown."
|
||||||
|
)
|
||||||
|
parser.add_argument("path", type=pathlib.Path, help="Markdown file to repair")
|
||||||
|
parser.add_argument("--metadata", type=pathlib.Path, help="Deep Research metadata JSON")
|
||||||
|
parser.add_argument("-o", "--output", type=pathlib.Path, help="Write repaired Markdown here")
|
||||||
|
parser.add_argument("--in-place", action="store_true", help="Rewrite the input Markdown file")
|
||||||
|
parser.add_argument("--dry-run", action="store_true", help="Print repaired Markdown to stdout")
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> int:
|
||||||
|
args = parse_args(argv)
|
||||||
|
if sum(bool(value) for value in (args.output, args.in_place, args.dry_run)) != 1:
|
||||||
|
print("error: choose exactly one of --output, --in-place, or --dry-run", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
try:
|
||||||
|
markdown = args.path.read_text(encoding="utf-8")
|
||||||
|
metadata = load_json(args.metadata) if args.metadata else extract_embedded_metadata(markdown)
|
||||||
|
fixed = repair_markdown(markdown, metadata=metadata)
|
||||||
|
except (OSError, ValueError, json.JSONDecodeError) as exc:
|
||||||
|
print(f"error: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
sys.stdout.write(fixed)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
destination = args.path if args.in_place else args.output
|
||||||
|
assert destination is not None
|
||||||
|
try:
|
||||||
|
destination.write_text(fixed, encoding="utf-8")
|
||||||
|
except OSError as exc:
|
||||||
|
print(f"error: cannot write {destination}: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(sys.argv[1:]))
|
||||||
|
|
@ -0,0 +1,185 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import pathlib
|
||||||
|
import sys
|
||||||
|
import textwrap
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
|
||||||
|
SCRIPT_PATH = (
|
||||||
|
pathlib.Path(__file__).resolve().parents[1]
|
||||||
|
/ "scripts"
|
||||||
|
/ "repair_markdown_citations.py"
|
||||||
|
)
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location("repair_markdown_citations", SCRIPT_PATH)
|
||||||
|
repair = importlib.util.module_from_spec(spec)
|
||||||
|
assert spec is not None
|
||||||
|
assert spec.loader is not None
|
||||||
|
sys.modules["repair_markdown_citations"] = repair
|
||||||
|
spec.loader.exec_module(repair)
|
||||||
|
|
||||||
|
|
||||||
|
class RepairMarkdownCitationsTest(unittest.TestCase):
|
||||||
|
def test_extracts_turn_ids_from_chatgpt_citation_token(self) -> None:
|
||||||
|
token = "citeturn34view0turn20search2"
|
||||||
|
|
||||||
|
self.assertEqual(
|
||||||
|
repair.extract_ref_ids(token),
|
||||||
|
["turn34view0", "turn20search2"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_replaces_tokens_with_first_use_footnotes_from_metadata(self) -> None:
|
||||||
|
markdown = textwrap.dedent(
|
||||||
|
"""\
|
||||||
|
武汉早期疫情通报强调未发现明显人传人。citeturn34view0
|
||||||
|
|
||||||
|
郑州暴雨和唐山事件报道也体现了不同报道框架。citeturn20search2turn34view6
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
metadata = {
|
||||||
|
"content_references": [
|
||||||
|
{
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"refs": [
|
||||||
|
{
|
||||||
|
"turn_index": 34,
|
||||||
|
"ref_type": "view",
|
||||||
|
"ref_index": 0,
|
||||||
|
"url": "https://wjw.hubei.gov.cn/example.shtml",
|
||||||
|
"title": "湖北省卫健委通报",
|
||||||
|
"attribution": "湖北省卫健委",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"turn_index": 20,
|
||||||
|
"ref_type": "search",
|
||||||
|
"ref_index": 2,
|
||||||
|
"url": "https://www.xinhuanet.com/zhengzhou.htm",
|
||||||
|
"title": "新华社郑州暴雨报道",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"turn_index": 34,
|
||||||
|
"ref_type": "view",
|
||||||
|
"ref_index": 6,
|
||||||
|
"url": "https://www.xinhuanet.com/tangshan.htm",
|
||||||
|
"title": "新华社唐山打人事件报道",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||||
|
|
||||||
|
self.assertIn("武汉早期疫情通报强调未发现明显人传人。[^1]", fixed)
|
||||||
|
self.assertIn("报道框架。[^2][^3]", fixed)
|
||||||
|
self.assertIn("## 参考资料", fixed)
|
||||||
|
self.assertIn(
|
||||||
|
"[^1]: 湖北省卫健委:湖北省卫健委通报。https://wjw.hubei.gov.cn/example.shtml",
|
||||||
|
fixed,
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"[^2]: 新华社郑州暴雨报道。https://www.xinhuanet.com/zhengzhou.htm",
|
||||||
|
fixed,
|
||||||
|
)
|
||||||
|
self.assertIn(
|
||||||
|
"[^3]: 新华社唐山打人事件报道。https://www.xinhuanet.com/tangshan.htm",
|
||||||
|
fixed,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_deduplicates_repeated_references(self) -> None:
|
||||||
|
markdown = "Aciteturn1view0 Bciteturn1view0"
|
||||||
|
metadata = {
|
||||||
|
"content_references": [
|
||||||
|
{
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"refs": [
|
||||||
|
{
|
||||||
|
"turn_index": 1,
|
||||||
|
"ref_type": "view",
|
||||||
|
"ref_index": 0,
|
||||||
|
"url": "https://example.com/a",
|
||||||
|
"title": "Example A",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||||
|
|
||||||
|
self.assertIn("A[^1] B[^1]", fixed)
|
||||||
|
self.assertEqual(fixed.count("[^1]:"), 1)
|
||||||
|
|
||||||
|
def test_without_metadata_removes_tokens_and_adds_warning(self) -> None:
|
||||||
|
markdown = "正文citeturn1view0\n\n[来源](https://example.com)"
|
||||||
|
|
||||||
|
fixed = repair.repair_markdown(markdown, metadata=None)
|
||||||
|
|
||||||
|
self.assertNotIn("cite", fixed)
|
||||||
|
self.assertIn("## 参考资料", fixed)
|
||||||
|
self.assertIn("[来源](https://example.com)", fixed)
|
||||||
|
self.assertIn("无法恢复逐句脚注映射", fixed)
|
||||||
|
|
||||||
|
def test_loads_embedded_metadata_from_html_comment(self) -> None:
|
||||||
|
markdown = textwrap.dedent(
|
||||||
|
"""\
|
||||||
|
正文citeturn1view0
|
||||||
|
|
||||||
|
<!-- deep-research-metadata
|
||||||
|
{"content_references":[{"items":[{"refs":[{"turn_index":1,"ref_type":"view","ref_index":0,"url":"https://example.com/a","title":"Example A"}]}]}]}
|
||||||
|
-->
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
|
||||||
|
metadata = repair.extract_embedded_metadata(markdown)
|
||||||
|
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||||
|
|
||||||
|
self.assertIn("正文[^1]", fixed)
|
||||||
|
self.assertIn("[^1]: Example A。https://example.com/a", fixed)
|
||||||
|
|
||||||
|
def test_does_not_rewrite_tokens_inside_fenced_code_blocks(self) -> None:
|
||||||
|
markdown = textwrap.dedent(
|
||||||
|
"""\
|
||||||
|
正文citeturn1view0
|
||||||
|
|
||||||
|
```md
|
||||||
|
示例citeturn1view0
|
||||||
|
```
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
metadata = {
|
||||||
|
"content_references": [
|
||||||
|
{
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"refs": [
|
||||||
|
{
|
||||||
|
"turn_index": 1,
|
||||||
|
"ref_type": "view",
|
||||||
|
"ref_index": 0,
|
||||||
|
"url": "https://example.com/a",
|
||||||
|
"title": "Example A",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fixed = repair.repair_markdown(markdown, metadata=metadata)
|
||||||
|
|
||||||
|
self.assertIn("正文[^1]", fixed)
|
||||||
|
self.assertIn("示例citeturn1view0", fixed)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Loading…
Reference in New Issue