A crawl finishing successfully does not mean it found every page that matters. The crawler may have exhausted its queue, reached a configured limit, followed only public links, or skipped a section it could not access. Those outcomes require different interpretations.
Before using a crawl as an audit inventory, reconcile what was expected with what was discovered and captured. The goal is a coverage explanation that another person can inspect, including the gaps that remain open.
Define the expected inventory independently
Use more than one source to describe the website. Useful inputs include XML sitemaps, navigation links, a content management export, a release route list, and the product owner's list of important journeys. Record the date and origin of each input.
These sources will disagree for legitimate reasons. A sitemap may omit account pages. Navigation may not expose campaign landing pages. A CMS export may contain drafts or old redirects. Keep those distinctions instead of treating any one list as the unquestionable source of truth.
Google describes sitemaps as a way to help search engines discover content. A sitemap is not a guarantee that all listed URLs will be crawled or indexed, and its existence does not prove that it lists every relevant page. For an audit, it is one inventory input among several.
Keep discovery and capture separate
A discovered URL is an address the crawler encountered. A captured page is a successful result under particular conditions. A redirect, timeout, excluded path, or unsupported response can sit between those two events.
Use distinct states in your coverage ledger. “Not captured” is too broad to guide the next action. Prefer categories such as captured, redirected, blocked, failed, excluded by scope, or not attempted because the limit was reached.
Here is a fictional reconciliation for an expected set of sixty URLs. The numbers illustrate the bookkeeping, not an expected success rate.
| Outcome | Count | Follow-up |
|---|---|---|
| Captured at the expected address | 43 | Review evidence and page grouping |
| Redirected to another captured address | 7 | Confirm destinations and record the mapping |
| Failed during capture | 4 | Inspect errors and retry when appropriate |
| Excluded by the agreed boundary | 3 | Keep the exclusion and reason visible |
| Not attempted after the page limit | 3 | Extend the run or narrow the inventory deliberately |
These categories account for all sixty expected addresses. Separately list any additional URLs found during crawling. Avoid quietly adding them to the denominator halfway through a coverage comparison.
Investigate missing links and dynamic routes
Some pages appear only after a user changes filters, submits a form, opens a menu, or completes an earlier step. Other pages exist but are not linked from the starting section. A crawl that follows ordinary links may never reach them.
Inspect how the missing destination is exposed. Is there a real link? Is its address generated only after a selection? Does the interface update content without changing the URL? The answers determine whether the gap belongs in URL discovery, state coverage, or a separate journey test.
Google's JavaScript SEO guidance explains the distinction between fetching a page and processing its rendered content. Your audit crawler has its own rendering capabilities and limits; do not assume it behaves exactly like a search engine or an interactive human session.
For pages that load asynchronously, use the dynamic content monitoring guide to define a meaningful captured state. A screenshot of a loading skeleton should not count as successful evidence of the finished interface.
Check boundaries before increasing the crawl limit
Increasing the page limit can reveal useful pages, but it can also spend the entire run on duplicate filter combinations. First inspect the addresses already discovered. Repeating sort parameters, calendar navigation, and faceted search can create many URLs with limited additional audit value.
Document the normalization rules you actually apply. Removing a tracking parameter may be sensible when you have confirmed it does not change content. Removing a product variant or language parameter can erase an important difference. Keep the original requested address alongside any normalized reference.
Respect access restrictions and the agreed authorization. An inaccessible account area needs an appropriate authorized evaluation process, not an attempt to bypass the restriction. When the public crawl cannot cover it, label the limitation and assign the next step.
Turn unresolved gaps into owned actions
For each gap, record four things: the expected page or state, the evidence of the problem, the person who can resolve it, and the effect on the proposed audit. This keeps a technical error from disappearing inside a percentage.
A timeout might require a later retry. A removed page might need confirmation from the content owner. An unreachable confirmation screen might require a safe test transaction. These are different actions, and they should not all become “increase crawl depth.”
Repeat only the work needed to resolve the uncertainty. Preserve the original run and its errors so the updated inventory does not imply that the first attempt had complete coverage.
Publish the coverage explanation with the report
The report should state its starting points, discovery sources, boundary, limits, successful captures, unresolved failures, and material exclusions. Readers should be able to understand both what the evidence contains and what it cannot answer.
Once reconciled, the inventory becomes a stronger input to representative page sampling. You can see an example of a bounded inventory in the public scoping report. Its small, disclosed source set makes the boundary inspectable. Apply the same discipline to larger websites rather than substituting a large URL count for a coverage explanation.