Generating help articles from pull requests.
The diff tells you what changed. It does not tell you what the screen now says.
Every Monday a job reads the pull requests merged that week, works out which help articles they affect, and opens one pull request in the docs repository with edits, new articles and deletions. A person reviews it. Merging is what publishes.
What it replaced
A product manager did this once a month. Read customer requests and support tickets, check what shipped, then write the article. Around two hours each, and most of that was not writing. It was opening the app to confirm what the screen actually says.
A feature that shipped early in the month had no article until the end of it. Small changes, a renamed button or a moved setting, were never worth a monthly slot, so they were skipped and came back as support tickets instead.
Why this is harder than a changelog
A changelog entry is one sentence about what changed. The pull request title gets you most of the way there.
A help article has to say which tab to open, what the control is called, and what happens when you click it. None of that is in the diff.
One change we shipped removed a few translation strings and swapped a set of radio buttons for checkboxes. The diff shows strings deleted. It does not tell you the name of the tab those controls sit on, or that customers can now pick several options where before they could pick one. Meanwhile a published article was still walking readers through the old single-choice control.
So this needs two things a changelog generator does not: a way to find which articles a code change affects, and a written description of your own interface.
Step zero: get your articles out of the CMS
We publish to WordPress, and at first we edited WordPress directly. That had to change before anything else could work.
So we moved every published article into a file in a git repository, 480 of them, and wrote a script that watches those files and pushes changes to WordPress.
You cannot automate what you cannot diff.
A draft has to arrive as a proposed change that a person can read next to the current version, edit and reject. A CMS database gives you none of that. A text file in git gives you all of it for free.
One word of warning about the file format. The frontmatter is YAML and the file is named .md, but the body is whatever your CMS stores. For WordPress that is Gutenberg HTML, and the sync passes it through untouched. If you write real Markdown in the body and send it through this script, your readers get raw asterisks. Convert on the way in or not at all.
What the step needs:
- One file per article. The filename is the URL slug, the folder is the category.
- Frontmatter carrying the CMS post id, title, SEO fields and related links. The post id is what lets the script update an article instead of duplicating it.
- A sync script that runs when a change lands on the main branch.
- A refusal to publish an empty body, so a blank file can never wipe a live page.
sync.js · one article, create or update
async function syncFile(file) {
const { data: fm, content } = matter(fs.readFileSync(file, 'utf8'));
const body = content.trim();
const slug = fm.slug || path.basename(file, '.md');
if (!fm.title) throw new Error('missing title');
if (!body) throw new Error('empty body, refusing to blank a live article');
// Only send what the file specifies. A field the file omits must preserve
// whatever is live, never blank it.
const payload = { title: fm.title, content: body,
categories: [await categoryId(fm.category)] };
const meta = {};
if (fm.seoTitle) meta.seo_title = fm.seoTitle;
if (fm.seoDesc) meta.seo_description = fm.seoDesc;
if (Object.keys(meta).length) payload.meta = meta;
// No id yet? Adopt a live post with the same slug before creating a duplicate.
let id = fm.cmsPostId;
const adopted = !id && (id = await lookupPostIdBySlug(slug));
if (id) {
// context=edit is load-bearing. Without it the CMS returns the RENDERED
// body, not the source you sent, so every article looks changed every run
// and this early exit never fires.
const live = await api(`/posts/${id}?context=edit`);
// Compare raw against raw, ignoring whitespace the CMS renormalises on save.
const changed = diffRaw(payload, live);
// Write an adopted id back even when nothing else changed, or you re-query
// by slug forever and the file never gains an id.
if (adopted) writeBack(file, fm, content, id);
if (!changed.length) return { action: 'unchanged', id };
// Note what is NOT in the payload: no status, no slug. A published article
// stays published, and the live URL never moves. Both are SEO-critical.
await api(`/posts/${id}`, { method: 'POST', body: JSON.stringify(payload) });
return { action: 'updated', id };
}
// Creating. Send the slug explicitly, or the CMS derives one from the title
// and the live URL stops matching the filename.
payload.slug = slug;
payload.status = ALLOWED.has(fm.status) ? fm.status : 'publish';
const post = await api('/posts', { method: 'POST', body: JSON.stringify(payload) });
writeBack(file, fm, content, post.id); // commit this with a message CI ignores
return { action: 'created', id: post.id };
}
Deleting is the part people get wrong. Never delete a ranked page into a 404. Remove the file, trash the post, and ship a 301 to the category index in the same change. Recovering the id needs one trick, because the file is already gone from disk:
sync.js · the delete path
function idFromParentCommit(file) {
// Every step here can fail on a fresh branch or a force-push, and an
// uncaught throw would skip the guard below instead of reporting it.
try {
const sha = execSync(`git log -1 --format=%H --skip=1 -- "${file}"`,
{ encoding: 'utf8' }).trim();
if (!sha) return null;
return matter(execSync(`git show "${sha}:${file}"`,
{ encoding: 'utf8' })).data.cmsPostId || null;
} catch { return null; }
}
async function syncDeletion(file, redirectTo) {
const id = idFromParentCommit(file);
if (!id) throw new Error('cannot recover the post id, refusing to delete');
if (!redirectTo) throw new Error('published article needs a 301, refusing to delete');
appendRedirect(slugOf(file), redirectTo); // into your web server config
await api(`/posts/${id}?force=false`, { method: 'DELETE' }); // trash, not destroy
}
Two more guards, both learned the expensive way: reject internal links missing their category segment, and never send a related-articles list shorter than the live one.
To do the migration itself, hand your export to a model one article at a time:
Migration prompt · one article at a time
Here is the body of a published help article, with its title, category,
URL slug and CMS post id.
Convert it to a file with YAML frontmatter containing:
title, seoTitle, seoDesc, category, cmsPostId, related
Rules:
- Keep the body markup exactly as it is. Do not convert or rewrite it.
- Copy the existing SEO description. Do not write a new one.
- Leave image tags untouched.
- Output the file and nothing else.
This is a move, not a rewrite. Rewrite hundreds of articles while migrating them and you cannot tell which step broke something.
Do this much and a writer can already edit a text file, open a pull request, and have it live on merge. What it does not do is notice. Someone still has to spot that a feature changed and go looking for the articles it broke. Everything after this automates that manual step, so the reviewer starts from a draft instead of a blank page.
The weekly job
Five stages. The model is involved in exactly one of them.
Five stages, one pull request out
1. Collect Pull requests merged in the window, across every product repo.
Drop the clearly internal ones.
2. Match For each one, find the help articles it might affect.
Group pull requests that share articles.
3. Draft One model call. It reads every group and returns, per pull request,
an edit, a new article, a deletion, or a skip with a stated reason.
4. Check Validate everything, drop what fails, apply the rest.
Write the pull request description.
5. Review One pull request. A person reads it, fills in screenshots, merges.
Merging is the only action that publishes.
Fetching the pull requests is a single search per repo:
merged:>=2026-07-01 base:main is:merged repo:ORG/NAME
Mind the window. Ours looks back a fixed seven days on a weekly cron, which means a run that fails and is not re-run loses that week permanently. Track the last successful run instead, or at minimum re-run by hand with a wider window. Cap your per-repo page size generously and paginate, or a busy week truncates silently.
The job itself is one scheduled workflow with the model called once in the middle, between two plain scripts:
.github/workflows/weekly-doc-update.yml
name: Weekly doc update
on:
schedule:
- cron: '30 3 * * 1' # Off the top of the hour on purpose: scheduled
# runs get dropped when everyone picks :00.
workflow_dispatch:
inputs:
since_days: { type: string, default: '7' }
dry_run: { type: choice, default: 'true', options: ['true', 'false'] }
permissions:
contents: write
pull-requests: write
jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v5
with: { node-version: '20' }
- run: npm ci
# The article index lives on a release, not in git.
- env: { GH_TOKEN: '${{ github.token }}' }
run: |
mkdir -p tools/embeddings
gh release download embeddings-latest -p index.jsonl \
-O tools/embeddings/index.jsonl --clobber
# 1. Fetch, match, group, write the model's input files.
- name: Prepare
id: prepare
env:
GH_TOKEN: ${{ secrets.SOURCE_REPO_TOKEN }} # read-only, product repos
run: |
node tools/run-weekly.js --prepare-only \
--since-days "${{ inputs.since_days || 7 }}"
# 2. The only model call. Reads batches.json, writes result.json.
# dry_run is empty on a cron trigger, so the schedule runs for real.
- name: Draft the changes
if: steps.prepare.outputs.has_batches == 'true' && inputs.dry_run != 'true'
uses: <your-model-provider-action>
with:
# Pin the model. An action release should not silently change
# what writes your documentation.
args: --model <pinned-model> --allowed-tools "Read,Write"
prompt: |
Read prompts/system-prompt.md, then .run-output/batches.json.
Write .run-output/result.json in exactly this shape and nothing else.
# 3. Validate, apply, render the pull request body.
- name: Apply
id: apply
if: steps.prepare.outputs.has_batches == 'true' && inputs.dry_run != 'true'
run: node tools/run-weekly.js --apply-only
- name: Open the pull request
if: steps.apply.outputs.has_changes == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # note: not the read-only one
run: |
set -euo pipefail
# Re-attach auth: the model action clears what checkout set up.
git remote set-url origin \
"https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git config user.name "docs-bot"
git config user.email "docs-bot@users.noreply.github.com"
git checkout -b "docs/auto-update-$(date -u +%F)-r${GITHUB_RUN_ID}"
git add -A articles .htaccess # -A, or deletions are left unstaged
git commit -m "docs: auto-update from product pull requests"
git push -u origin HEAD
gh pr create --fill --base main --body-file .run-output/pr-body.md
Give the model an explicit output schema. Ours is one entry per input group:
result.json
{ "batch_results": [ {
"batch_id": 1,
"edits": [ { "slug": "...", "source_pr": "...",
"operations": [ { "find_text": "...",
"replace_with": "..." } ] } ],
"new_articles": [ { "slug": "...", "source_pr": "...", "body": "..." } ],
"deletions": [ { "slug": "...", "source_pr": "...",
"redirect_to": "/help/x/" } ],
"skipped": [ { "source_pr": "...", "reason": "..." } ]
} ] }
Finding the right articles
Three signals, combined into one candidate set per pull request.
- Exact rules. A few changes point straight at one article, with no guessing. We have one article per review platform we import from, and the code has one file listing those platforms. If that file changes and the diff mentions Shopify, the Shopify article is the one to update. Write a rule per pattern like this that you have. They are rare, and they are never wrong.
- A path map you write by hand. Source directories and the article groups they belong to. Anything under the forms directory belongs to the forms articles.
- Meaning search. Compare the pull request against every article at once.
All three run on every pull request. The order decides who wins when two of them return the same article, not whether each one executes. Put the deterministic ones first because they are exact, not because they save you the search.
The settings we run
results per search: 30
similarity threshold: 0.62
hard cap: 50
- 30 results, because one interface change can genuinely affect a dozen articles. Ten misses them. A hundred fills the prompt with noise and costs more every run.
- 0.62 similarity was tuned by hand. Lower and you pull in articles that merely share vocabulary, which the model then edits for no reason. Higher and real matches fall out. If you copy one number from this post, do not copy this one. Run your own past changes through and look at what comes back.
- A hard cap of 50 stops one pull request sending your whole help centre to the model. Know what it costs you: a broad signal legitimately matches hundreds. A change to a single translation file maps to every article in that category, which for us is over 400. The cap then keeps whichever fifty came first out of the directory listing. Log what it dropped. Truncating in silence looks exactly like full coverage.
- Stop early when nothing matches. If neither deterministic signal fires and the search returns nothing above the threshold, this change does not affect the documentation. That saves the model call, not the search. You have already paid for that.
Why we embed the articles
The first two signals only fire when someone has already written a rule for the thing that changed. The third has to cover everything else, which means comparing one pull request against every article you have.
You cannot do that with a model. A thousand articles is a thousand calls, every week.
An embedding turns each article into a list of numbers standing for its meaning. Comparing two of them is arithmetic: multiply, add, done. So the pull request gets embedded once, and scoring it against a thousand articles happens on the runner in milliseconds and costs nothing.
embed.js and query.js
// Build. Title, headings and plain-text body together; headings carry a lot
// of signal for very little length.
const text = [fm.title, headings.join(' • '), stripHtml(body)].join('\n\n');
const hash = sha256(text);
if (!forceAll && existing.get(slug)?.text_hash === hash) return; // cache key
const [vector] = await embed([text]); // batch ~50
existing.set(slug, { slug, category, title, text_hash: hash, vector });
// Search. Math.min guards the case where a stale index and a new query were
// embedded at different dimensions, or every score comes back NaN.
function cosine(a, b) {
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < Math.min(a.length, b.length); i++) {
dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb) || 1);
}
Where the index lives
One release on the docs repository, tagged embeddings-latest, one index.jsonl asset, overwritten in place. The local copy is gitignored.
Three things we deliberately did not do:
- Not committed to git. The file is rewritten whenever any article changes. At roughly 9 KB per article, a thousand articles is about 10 MB, and every refresh would add another full copy to history forever. Our docs repository is also what gets deployed to the web server, so a committed index would ship to production as dead weight. Release assets are not part of a
git pull, which keeps it out of both. - Not a vector database. Another service, key and bill for a few thousand rows that fit in memory. Reach for one when a flat file stops working, not before.
- Not rebuilt on every run. That pays the full cost and wall time weekly for a corpus that barely changes. Hash the text instead, and prune entries whose file is gone. A deleted article whose vector lingers keeps matching, and the model gets asked to edit a file that no longer exists. Guard the prune: if the folder walk returns zero files, skip it, so one bad path cannot wipe the index.
512 dimensions instead of the default 1536 makes the index a third of the size and the comparison a third of the work, with no difference we could measure.
Keeping it current is one step on whatever publishes your articles:
One step on the publish workflow
- name: Re-embed changed articles
# Runs even when a publish partially failed, and on delete-only pushes.
if: ${{ !cancelled() && (steps.changes.outputs.files != ''
|| steps.changes.outputs.deleted != '') }}
working-directory: tools/embeddings
run: |
# The release will not exist on the very first run.
if gh release view embeddings-latest >/dev/null 2>&1; then
gh release download embeddings-latest -p index.jsonl -O index.jsonl --clobber
node embed.js && gh release upload embeddings-latest index.jsonl --clobber
else
node embed.js --all
gh release create embeddings-latest index.jsonl -t 'Article embeddings'
fi
The embeddings come from the files in the repository, not from the CMS, so one article failing to publish should never stall the index.
Write your interface down
A model cannot see your product. Give it a diff and nothing else and it will describe controls that sound completely reasonable and do not exist: a font picker, a corner-radius slider, a toggle with a plausible name. Those are the things products like yours usually have.
The fix is boring and it works. Write the interface down, one document per area, and feed it to the model alongside the diff. It looks like this:
One entry from the interface reference
## Widget editor: Design tab
Reached from: Widgets list, pencil icon on a card. Opens on "Review Source".
Sections, in order:
Badge Layout Customization Layout (Grid | List | Carousel), Card spacing (px)
Content Customization Show reviewer photo, Show star rating, Show date
Badge Color Customization Background, Text, Accent (all hex inputs)
Business plan and above only: Custom CSS (textarea, below Badge Color).
NOT on this tab: font selection, corner radius. Both are common requests and
neither exists.
That last line matters as much as the rest. Naming what is absent is how you stop a model inventing it.
This is the largest piece of work in the project and it decides whether the output is usable. It pays for itself twice, because a human writer needs the same reference. Expect it to drift behind the product, so maintain it rather than finish it. And expect your source code to have its own stale copies: an old comment describing behaviour that changed two releases ago will be believed and repeated to your customers.
Guards worth building before you need them
You can enforce a rule in two places, and they are not equally strong.
Write it in the prompt and you are asking the model to remember. It usually will. Write it in your own code, as a check that runs after the model replies, and it happens every time whether the model cooperated or not.
So put anything that could damage a live page in the code. Leave the prompt the decisions that need judgement.
One filter runs before the model. Dropping dependency bumps and refactors before they cost you a call is worth doing, but be careful how you match them:
The filter, wrong and right
// WRONG. \b matches a space, so "Build the new delete modal" is classed as a
// chore and dropped before matching ever runs. Nothing logs it.
const internal = /^(chore|build|refactor|test|ci|deps|bump|revert|wip)\b/i;
// RIGHT. Require the conventional-commit separator, so only "build:" and
// "build(scope):" match, not the English word "Build".
const internal = /^(chore|build|refactor|test|ci|deps|bump|revert|wip)(\([^)]*\))?\s*:/i;
A false keep costs a little model time. A false drop loses documentation with no signal anywhere. Bias hard toward keeping.
The rest run after the model returns, before anything touches the working tree:
validate() · runs before anything is applied
function validate(batches, result) {
// 1. Shape. A structural mistake must not read as "the model returned
// nothing", or you get a wall of errors one layer below the real fault.
if (!Array.isArray(result.batch_results)) {
fail(`No batch_results[]. Top-level keys: ${Object.keys(result).join(', ')}`);
}
for (const br of result.batch_results) {
const input = batches.find(b => b.id === br.batch_id);
// 2. Accounting, PER BATCH. Judge this across the whole run and one
// productive batch masks a different batch that dropped its input.
const missing = input.prs.filter(pr => !collectReferencedIds(br).has(pr.id));
if (missing.length) {
producedChanges(br)
? warn(`Folded into a sibling edit, confirm in review: ${missing}`)
: fail(`Batch ${br.batch_id} produced nothing and never named: ${missing}`);
}
// 3. Anchoring. Presence is not enough: replace() takes the FIRST match,
// so a phrase appearing twice silently rewrites the wrong one. Count,
// do not just test for existence.
for (const edit of br.edits) {
const body = read(edit.slug);
for (const op of edit.operations) {
const hits = countOccurrences(body, op.find_text); // guard empty needle
if (hits === 0) drop(edit, 'find_text not verbatim');
else if (hits > 1) drop(edit, `find_text ambiguous, appears ${hits} times`);
}
}
// 4. A published article may not be deleted without a redirect.
for (const del of br.deletions) {
if (del.cmsPostId && !del.redirect_to) drop(del, 'deletion with no 301');
}
}
}
Check 2 warns in one case and fails in the other. Two pull requests are often two halves of one change, and the model is right to fold them into a single edit crediting one of them. If the batch produced real changes, an unnamed input is probably that. If it produced nothing at all, it was dropped. Fail only the second, or you will spend a month turning off your own alarm.
Note what is not here: unresolved screenshot placeholders. This stage is where they get inserted, not blocked. That gate belongs at publish time, so the draft can carry them into the pull request for the human to fill in. Block them any earlier and you kill every change that needs a screenshot before anyone sees it.
The judgement calls go in the prompt:
System prompt · the rules that must hold
Every pull request id in this batch MUST appear somewhere in your response:
in an edit, a new article, a deletion, or a skipped entry. If a change needs
no documentation, say so as an explicit skip. Never drop one silently.
For every skip, give a reason a reviewer can disagree with. "Internal
refactor, no user-facing change" is a claim about the product, not a
formality. Say what you checked to reach it.
Every edit must quote find_text character for character from the article
excerpt you were given, and that quote must appear exactly once in the
article. If you cannot quote it exactly, do not propose the edit.
You are given neighbouring articles from the same category. Before writing
a new article, check that none of them already covers this task. If one
does, edit that one instead and say which.
Watch the word excerpt there. We pass the model the first 1,200 characters of each candidate, which keeps the prompt affordable and means the model can only ever edit the top of a long article. Anything below the fold is unreachable until you raise that budget. Decide the number deliberately; it silently caps what the whole system can do.
Then a third layer, cheap and worth it: run your own linter on the bot's pull request and let it go red before a human can merge. And render every skip and its reason in the pull request body. A reviewer can only disagree with a decision they can see, and no check above can tell you whether a sentence is true.
Doing the setup with AI
Three prompts, in order.
1. Work out how much of this you need. Export last month's merged pull requests first.
Prompt 1 · scope the work
Here is a JSON export of every pull request merged into main last month,
with title, description, author and changed file paths.
Answer only these three things:
1. Which of these changed something a customer can see in the interface?
2. For each one, can you tell from the file paths and diff alone what the
screen now looks like, what the control is called, and what happens when
someone uses it? Answer yes or no per item, with one line of reasoning.
3. Group the customer-visible ones by product area and name each group.
Wherever the answer to 2 is no, state exactly what information is missing.
That list is what I have to write down by hand.
Question 2 is the whole project in miniature. If the answer is mostly no, the written reference is where your time goes.
2. Build the path map.
Prompt 2 · map code to articles
Here is the directory tree of my product repository, and a list of my help
article titles with their categories.
Propose a mapping from source path prefixes to article groups, as JSON:
{ "src/components/forms/": ["form-setup", "form-automation"], ... }
Rules:
- Only map a path when you are confident. Skip the rest. Do not guess.
- Prefer specific prefixes over broad ones.
- List any code area with no matching articles. Those are documentation gaps.
- List any article group with no matching code path. Those may be stale.
The two lists at the end are worth as much as the map.
3. Write the interface down, one area at a time.
Prompt 3 · build the reference
Read the code for the [area name] screens in this repository and write a
reference document for someone who has never seen the product.
For each screen and tab, give me:
- Its exact visible name, copied from the interface strings, not paraphrased
- Every control on it, with its exact label
- What happens when each control is used
- Anything only visible on certain plans or behind a flag
- A short list of controls people assume exist here and that do NOT
Rules:
- Copy labels character for character from the source. Do not tidy them up.
- Where you cannot confirm a label in the code, write UNCONFIRMED and name
the file you checked.
- No marketing language. This is a reference, not a description.
Read it against the running app. The UNCONFIRMED lines are your checklist.
What is still open
Screenshots are the bottleneck. The draft is written in minutes, then waits on a person to capture the images. Automating that is the next piece of work.
Features behind a flag are a judgement call. Something released to a few accounts should not get a public article yet, and a model can only infer that from hints in the diff.
The short version
A changelog generator needs to filter. A help article generator needs to know your product, and the pull request does not contain that.
Put your articles in git before you put a model anywhere near them. Then start with prompt one: export last month's merged pull requests and ask how many of them you could actually document from the diff alone. That number tells you how much of your interface you are going to have to write down, and that is the real project. The prompting is the small part.