Build a Domain FAQ

A pipeline that keeps pace with a live domain

A domain FAQ took three months to write. Every entry was drafted by hand, reviewed by a subject expert, and published. Six months later, the software it documented had shipped four updates. Forum posts referenced features the FAQ had never covered. The experts who wrote it had moved on. Someone asked whether the FAQ was still maintained. Nobody was sure.

The pipeline in this book inverts that dependency. Content is ingested on a schedule from the web and from structured sources like GitHub repositories. Claude reads that content and generates draft FAQ entries. Automated checks filter noise, flag duplicates, and verify that each draft has a source before anything reaches a human reviewer. The reviewer’s job is to approve, reject, or edit — drafting from scratch is no longer part of the workload. A 200-entry FAQ that took three months to build can be audited and refreshed in an afternoon.

Seven stages from ingestion to searchable database

• the-pipeline-architecture — Eight stages from content ingestion to a searchable, reviewed FAQ database. The data format flowing between stages — the fields on an ingested document, a generated draft entry, and a reviewed entry — defined once so every subsequent stage builds to the same interface.
• seeds-and-web-crawling — A web crawler using trafilatura for content extraction, which handles the structural noise of most public pages and returns clean text. Seed file management: how to define crawl scope, version the seed list, and add sources without breaking what already runs.
• multi-source-crawling — Extending ingestion to GitHub repositories, forum scrapers, release note parsers, and changelog extractors — the sources where authoritative current behavior is documented for technical domains. Each connects to the same data format the web crawler produces.
• drafting-with-claude — Structured output via the Anthropic API, batch processing for cost control, and the prompt design that produces entries a reviewer can evaluate quickly. Tradeoffs between prompt length, entries per source chunk, and output schema strictness. The result: a drafting stage that processes a directory of ingested content and writes a directory of candidate entries.
• validation — Schema validation, duplicate detection against the approved database, and rejection logging. What filtering removes — malformed output, near-duplicates, entries with no traceable source — and what it leaves for the reviewer. Entries arriving at review have cleared all mechanical checks.
• human-review — A CLI interface where the reviewer sees each candidate entry alongside its source content. Approve, reject, or edit — each action logged with the reason. The database schema for approved entries and the commit format that records provenance from source to pipeline run to publication.
• embeddings-and-re-crawl — Embedding-based search that surfaces entries matching a user query on meaning. Drift detection that flags entries whose source content has changed since review. A nightly batch orchestrator that ties all stages into a single scheduled run.

A preview

When a project ships updates monthly, when its forums generate hundreds of posts a week, and when the original authors have moved to other work, the hand-authored FAQ becomes an archaeological artifact. Users learn to distrust it. They check Discord instead. The documentation that was supposed to reduce support burden becomes a source of confusion when it contradicts the current behavior of the software.

Volume is the pipeline’s job. Accuracy belongs to the reviewer.

Anyone building a knowledge base that has to stay current

Teams and individuals maintaining technical documentation, product FAQs, or community knowledge bases for domains that update faster than any editorial team can follow. The same pipeline that runs across all seven chapters also runs KaspaFAQ.com, a live domain knowledge base for the Kaspa blockchain community — real seeds, real crawl output, entries reviewed and served at production volume. Swap the seeds file and the pipeline runs on any topic.

A longer excerpt is available to newsletter subscribers.

Buy Ebook on Amazon