Your documentation is excellent, your engineers maintain it properly, and roughly a third of it has never appeared in a search result. The reason is almost always the same: the docs site renders in the browser, and what renders in the browser is not necessarily what a crawler receives.
Publication and findability are separated by three distinct events. The address has to become known. A crawler has to fetch it. What it fetches has to be judged worth keeping. Nothing announces itself when one of the three fails, and in a technology company each of them fails for reasons that look like good engineering practice from the inside.
That is the awkward part. The causes here are not neglect. They are a documentation framework chosen for authoring convenience, a versioning scheme chosen for correctness, and a staging environment chosen for safety — each defensible, each with a side effect nobody costed.
What a browser sees and a crawler does not
Documentation frameworks commonly assemble the page after load: the shell arrives, a script fetches the content, the navigation tree builds itself from a manifest. For a reader this is fast and pleasant. For discovery it means the delivered markup may contain a heading, a spinner and nothing else.
The navigation matters as much as the content. Where the sidebar tree is generated client-side from a manifest file, there are no links in the delivered markup at all — which means every page below the entry point is reachable only through the sitemap, the weakest of the four discovery channels. A framework can be excellent and still leave a documentation set effectively unlinked.
How correct versioning becomes a crawl problem
Publishing documentation per release is the right thing to do. It is also the fastest way to turn four hundred genuine pages into several thousand addresses that differ by a version string and three sentences.
Where the address count actually comes from
On a sixty-page corporate site none of this applies. On a documentation platform all four usually do.
- One tree per release. Twelve retained versions of four hundred pages is four thousand eight hundred addresses, of which four hundred are current. Crawl activity distributes across all of them unless something says otherwise.
- A sandbox or staging environment left reachable. Frequently a complete second copy of the entire site on a subdomain, occasionally with its own sitemap. It competes with production for the same terms and sometimes wins, because nothing distinguishes them.
- Anchor-based deep links treated as pages. Where the framework routes fragments as separate addresses, a single long reference page becomes forty.
- Slow generation under load. Documentation platforms that render on request rather than at build time answer slowly when crawled in bulk, at which point the crawler reduces its own rate. That is politeness, and the effect is fewer pages fetched.
The second item causes more damage than the others combined and is the easiest to fix. A reachable staging copy is not a marginal issue: it is an exact duplicate competing with production, and where it happens to be older it can be the one retained. Blocking it takes one line of configuration and is done by whoever owns the deployment pipeline.
Submitting is not indexing
What it does buy is genuine and limited to one thing: the opening stage no longer sits on the critical path. Instead of waiting for a crawler to arrive through a link that may not exist in the markup, the address is known immediately. For a documentation set whose navigation is client-side, that is frequently the only lever available until the framework changes — which makes it worth operating properly rather than dismissing.
What content and marketing control
Submission, sitemap contents, cross-links written into body text, and pages summarising documents.
- No deployment required
- Produces evidence for the next step
What the framework controls
Server-side rendering of the sidebar, canonical handling across versions, blocking the staging copy.
- Raise with a log extract attached
- Small changes, large effect
The Indexing Hub and the numbers that govern a schedule
This part of the work is what the Indexing Hub inside the Semalt workspace is for. Addresses go in, they are forwarded, the response is logged, and anything stuck becomes visible. The ceilings involved do not move, so a release plan should be drawn up with them in view rather than against them.
| Capability | Use on a documentation platform | Where it stops |
|---|---|---|
| Single-address tracking | the current version of the pages that matter commercially | a thousand addresses per day per account |
| Bulk submission | a framework migration or a change of address scheme | ten thousand addresses per batch |
| Sitemap handover | generated sitemap sets, one per version tree | three levels of nesting, a thousand files per job |
| IndexNow relay | release notes and changed reference pages | a notification, never an undertaking |
The nesting limit deserves attention on documentation platforms specifically, because generated sitemap sets nest naturally: an index pointing at per-version indexes pointing at per-section files is already at the boundary. Anything below that is not fully resolved, and the symptom is a whole section that never appears while everything around it does.
One further remark about the daily allowance, because documentation platforms hit it in an unusual way. The constraint is rarely the total number of pages; it is the churn. A platform that regenerates every address on every build looks, from the outside, like several thousand changed pages a week even when three sentences moved. Submitting all of them consumes the allowance without conveying anything, and it obscures the twelve pages that genuinely changed. Restricting submission to genuinely modified addresses is a pipeline decision and turns a saturated allowance into a comfortable one. The counters in the workspace make the difference visible within a week.
Reading the per-address log
Handing an address over is the easy half. The record of what followed is where the value sits: when a crawler called, with what status, and on failure with what detail. Three running totals accompany it — sent, discovered, failed.
| What the log shows | What it means | Who fixes it |
|---|---|---|
| No visit after several days | Known, but nothing points at it | Content — cross-links in body text, no deploy needed |
| Visit with an error status | Redirect chain, timeout, or a directive blocking it | Engineering — as a ticket with the status attached |
| Visit, then not retained | Thin, or indistinguishable from a sister version | Content plus engineering — canonical handling across versions |
| Retained, then dropped | Months without changes, links or traffic | Content — annual revision and a link from current material |
The third column is the useful one in a technology company, because it converts a vague complaint into an assignment. Two of the four rows never reach engineering at all, and the two that do arrive as reproducible defects rather than as marketing asking for attention. That difference decides whether anything gets scheduled.
Making coverage part of shipping
Four attachment points in a pipeline that already exists, none of which require a new process.
- Sitemap regeneration on build. If the sitemap is generated by the same pipeline that publishes the docs, it is never stale. If it is maintained separately, it is stale within two releases and nobody notices for months.
- Submission triggered by the release, not by a person. New and changed addresses handed over as part of the deployment step. A human-triggered submission gets skipped during the releases that matter most, which are the busy ones.
- Staging blocked by environment, not by memory. The directive belongs in the environment configuration so that any new preview environment inherits it. Blocking staging by hand works until somebody spins up a second one.
- Coverage checked at the same cadence as uptime. Not daily and not annually. A monthly look at how many current-version pages are actually retained catches a framework change before a quarter of content disappears.
The value of attaching all four to the pipeline rather than to a person is that they survive staff changes, which in this segment happen faster than documentation platforms do. A process that depends on somebody remembering to submit after each release lasts precisely as long as that person's tenure — and then fails silently, which is the worst of the available failure modes.
Four situations specific to technology companies
Twelve versions, one page
Every retained release publishes the same reference page. Search engines pick one, and it is not reliably the current one.
- Canonical to the current version
- Keep the rest reachable, not indexed
The staging subdomain
A complete second copy, reachable, occasionally with its own sitemap and its own inbound links.
- One configuration line to block
- Check quarterly, not once
Docs on a separate domain
Chosen for operational independence; starts with none of the main domain's accumulated history.
- A subdomain inherits less than a folder
- Cross-linking mitigates, does not solve
The deprecated endpoint page
Removed when the endpoint was retired, along with every reference other sites had built to it.
- Redirect to the successor deliberately
- Deleting discards accumulated references
Each of the four looks like a coverage failure and is in fact something else, which is exactly why resubmitting achieves nothing in any of them. The question of whether the underlying structure can support this market at all sits within technical SEO; the prior question of which page carries which capability term is answered during keyword research, and the editorial split between English and German material in our content strategy.
The order that works
- Confirm delivery first. Content and navigation present in the delivered markup. Until that holds, everything downstream is wasted effort, and it is a five-minute check.
- Then obtain links. Two from pages that themselves receive traffic. On a documentation platform the release notes are usually the strongest available source and the least used.
- Then the sitemap. Confirm the address appears in the file that is actually served, which on generated platforms is not always the file in the repository.
- Then submit. Once, through the Hub, with the relay active. Not repeatedly.
- Read the log after two to seven days. Earlier tells you nothing; later loses the connection to whatever changed.
- Respond according to the finding. Cross-link, raise a ticket, or rewrite. Three tasks an order of magnitude apart in cost, chosen from evidence.
Where a whole section changes at once, the sitemap route applies: a file or an address is handed over, nesting resolves to a depth of three, a job accepts up to a thousand files, two jobs run and twenty wait. Splitting by section keeps each log interpretable, which is what matters on the day a product family goes missing. Coverage then sits beside clicks and positions in the Semalt dashboard, so a page that never entered the index is visible before further work is aimed at it.
Why this is not a separate workstream
Index coverage is where every other piece of work becomes visible or does not. A German regulatory explainer that took two months to write and is never retained performs exactly as well as an unwritten one. A citation earned for a page that was never indexed is effort with no return.
That is the reason submission tooling belongs beside campaign control and reporting rather than in a separate application. Within My SEO everything sits on one surface: which terms are being pursued, which placements exist, what the model suggests changing, and whether the pages are actually in the index. Alongside each placement stands the authority score of the referring domain and how much traffic it sees. The practical gain is unglamorous: you learn that a page never entered the index before you commission three citations aimed at it.
Check documentation coverage in the dashboard
Questions from documentation and platform teams
Our docs render client-side. Is that fatal?
Not fatal, and not something to work around indefinitely either. Two things are worth separating: whether the content appears in the delivered markup, and whether the navigation does. Content-only is workable, because pages can be submitted individually and cross-linked from body text. Neither present is a genuine blocker, and the fix — rendering on the server or at build time — is usually a configuration change in the framework rather than a rewrite.
How do we handle twelve retained documentation versions?
Keep them reachable and index one. Point the versioned copies at the current one canonically, submit only the current tree, and give the current version a stable address that does not change with each release. Historic versions remain available to anyone who needs them and stop competing with the version almost everybody is looking for.
Should the documentation live on a subdomain or in a folder?
A folder inherits considerably more from the main domain, which matters most when the main domain is the stronger asset. Subdomains are frequently chosen for deployment independence, which is a legitimate reason — but the cost should be understood rather than discovered later. Where a subdomain is already in place, heavy cross-linking in both directions reduces the gap without eliminating it.
Does submitting the same address twice help?
No. A submission is a notification and does not become more persuasive through repetition. If a week goes by without a crawler calling, the explanation is nearly always an absence of inbound links. Put two in place from pages that themselves see traffic, then wait seven days — that route works considerably better than resubmitting, and whichever way it turns out you learn something usable.
We deleted our deprecated endpoint pages. Was that wrong?
Usually, yes. Those pages typically hold references from integration guides, forum answers and other companies' documentation, all of which disappear along with them. Redirecting to the successor page preserves that accumulated value and gives anyone following an old link something useful. Where no successor exists, a page explaining the deprecation and pointing at the alternative is still better than a not-found response.
How do we get engineering to prioritise any of this?
By sending evidence instead of an assessment. A log extract listing affected addresses, the status returned and a timestamp reads as a defect and gets a ticket number. "Our documentation is not ranking well" reads as a request for attention and gets a polite acknowledgement. The two are the same finding described differently, and only one of them ends up in a sprint.