What should I define before running a website crawler?

Start with the pages and customer journeys that matter to the business. Build a defined URL set from the XML sitemap, navigation, priority landing pages, contact paths and known URL variants. Record each requested URL, its final destination, status, response type and discovery source.

Does a successful HTTP response mean a page is healthy?

No. A successful HTTP response is an access observation. It does not establish that the page is indexed, accessible, functionally correct, useful to visitors or capable of delivering an enquiry.

Should metadata rules apply to every successful response?

Apply metadata checks only to applicable HTML responses. Machine-readable endpoints may need different controls, so report them separately rather than treating their lack of HTML metadata as a defect.

How should a small business verify crawl and metadata repairs?

Use the same intended URL set, page-type rules and exclusions after repairs. Keep the original and rerun records together so the team can identify what changed, what passed and which exceptions remain.

How to Audit Website Crawlability and Metadata for a Small Business

Patrick HallermannOctober 26, 20267 min read

How should a small business audit website crawlability and metadata?

List the pages and business journeys that matter, then build a defined URL set from the XML sitemap, navigation, priority landing pages and known variants. Crawl that set and retain each requested URL, final destination, status and response type. Check access, discovery and indexing controls separately, including redirects, robots controls, noindex instructions, canonical targets, sitemap inclusion and internal links. Apply agreed title, description and other metadata rules only to applicable HTML responses. Group recurring defects, prioritise them by their effect on important pages, assign repairs, and rerun the same tests. Passing one check does not prove indexing, accessibility, functional correctness or enquiry delivery.

A website audit is a structured examination of technical health, search visibility, content, user experience, performance, accessibility, security and conversion paths.: The Complete Website Audit Guide: How to Audit a Website from Top to Bottom · As of 2026-09-26
Before a page can rank in Google, it must be crawled so Google can discover and read it, then indexed for retrieval.: The Complete Technical SEO Guide for Small Businesses (2026) - eSEOspace · As of 2026-06-01
Technical SEO focuses on website infrastructure including crawlability, indexing, page speed, security and structured data.: The Complete Technical SEO Guide for Small Businesses (2026) - eSEOspace · As of 2026-06-01
A Hallermann Consulting release check found no missing or invalid required metadata in 170 HTML responses on 2026-10-03 after recorded repairs; machine-readable responses were excluded from that HTML denominator.: One-site release crawl and metadata verification · As of 2026-10-03
In one Hallermann Consulting post-repair crawl on 2026-10-03, all 177 requested URL variants returned successful HTTP responses: 170 HTML responses and 7 machine-readable responses.: One-site release crawl and metadata verification · As of 2026-10-03

Define what the audit must prove

A website audit is a structured examination of technical health, search visibility, content, user experience, performance, accessibility, security and conversion paths. A crawlability and metadata audit should narrow that broad examination to an agreed URL set, the business journeys those pages support and the observations needed to make repair decisions.

Sources: The Complete Website Audit Guide: How to Audit a Website from Top to Bottom.

Write a short audit brief that names priority pages, the expected action on each page, known URL variants and endpoints intentionally excluded from the HTML review. Treat this brief as the boundary of the exercise, not as a promise that every aspect of website quality will be tested.

Name the priority pages and customer journeys they support.
State the expected visitor action for each important page.
Record intentional exclusions before collecting findings.

Build and crawl the intended URL set

Before a page can rank in Google, it must be crawled so Google can discover and read it, then indexed for retrieval. A small-business audit should therefore use an intended URL set rather than relying only on the addresses a crawler happens to discover.

Sources: The Complete Technical SEO Guide for Small Businesses (2026) - eSEOspace.

Start with XML sitemap routes, navigation links, priority service or product pages, contact paths, campaign destinations and known legacy or redirected addresses. For every entry, retain the requested URL, final URL, response status, response type, discovery source and crawl date so the same evidence can be checked after repair.

Seed the set from the sitemap, navigation and priority landing pages.
Keep requested and final URLs in separate fields.
Record status and response type for every crawl entry.

Check access and indexing controls separately

Technical SEO focuses on website infrastructure including crawlability, indexing, page speed, security and structured data. The audit should label access, discovery and indexing observations separately so one apparently clean result does not conceal a different problem.

Sources: The Complete Technical SEO Guide for Small Businesses (2026) - eSEOspace.

For each important URL, inspect the destination and redirect, any robots restriction or noindex instruction, the canonical target, XML sitemap membership and whether internal links support discovery. Record conflicting signals for investigation rather than declaring a page indexed or visible from crawler output alone.

Check broken destinations and unexpected redirects.
Review robots, noindex, canonical and sitemap signals separately.
Do not treat one clean signal as proof of search visibility.

Audit metadata on the correct response set

A Hallermann Consulting release check found no missing or invalid required metadata in 170 HTML responses on 2026-10-03 after recorded repairs; machine-readable responses were excluded from that HTML denominator. A small business can use the same denominator discipline by applying metadata rules only to applicable HTML responses and recording excluded response types separately.

Sources: One-site release crawl and metadata verification.

Define the required fields and acceptance rules for each page type before scanning, then retain the URL, observed value, expected rule and exception category for every finding. Keep missing, invalid, duplicate and conflicting values separate because their likely causes and repairs may differ. A clean metadata result does not establish indexing, accessibility, functional correctness, rankings or enquiry delivery.

Define page-type requirements before scanning.
Apply the rules only to applicable HTML responses.
Separate missing, invalid, duplicate and conflicting values.

Turn findings into an owned repair queue

A useful website audit asks whether the site works effectively for its users and the business behind it, rather than merely whether it looks good. Crawl and metadata findings should therefore be prioritised by the importance of the affected page, the customer journey at risk and the breadth of the pattern.

Sources: The Complete Website Audit Guide: How to Audit a Website from Top to Bottom.

Create one repair ticket per recurring pattern and include representative URLs, the likely cause, affected priority pages, expected behaviour, proposed action, owner and exact rerun test. A widespread defect on a key contact or service path usually deserves attention before an isolated warning on a low-priority page, even if a scanner assigns the opposite generic severity.

Group repeated symptoms by likely cause.
Put business-critical pages ahead of generic scanner severity.
Assign an owner, expected outcome and verification check.

Rerun the audit and qualify the result

In one Hallermann Consulting post-repair crawl on 2026-10-03, all 177 requested URL variants returned successful HTTP responses: 170 HTML responses and 7 machine-readable responses. That result is a reporting example rather than a benchmark; each business should state its own requested set, response-type split, observation period and method.

Sources: One-site release crawl and metadata verification.

Retain the original and rerun records together, including the URL set, observed results, exclusions, acceptance rules, repair date and remaining exceptions. The Hallermann Consulting record covered one website and one release window, and its URL variants were not independent websites or reliability samples. Successful responses and valid required metadata did not prove indexing, accessibility, functional correctness, rankings or successful enquiry delivery.

Rerun the same defined URL set and metadata rules.
Report HTML and machine-readable responses separately.
State what the rerun does not prove.

Crawlability and metadata audit control table

This compact control table makes the audit reproducible by pairing each stage with evidence that can be retained and checked after repair.

Audit stage	Evidence to retain	Decision or output
Define scope	Priority pages, customer journeys, expected actions and exclusions	A bounded audit brief
Crawl intended URLs	Requested URL, final URL, status, response type, discovery source and date	A retained URL-level crawl record
Review controls	Redirect, robots, noindex, canonical, sitemap and internal-link observations	Labelled access, discovery and indexing findings
Validate metadata	Applicable HTML set, page-type rules and URL-level exceptions	A metadata exception list with a clear denominator
Assign repairs	Affected pages, likely cause, owner, expected outcome and rerun test	An owned repair queue
Rerun and report	Repeated URL set, results, exclusions and unresolved exceptions	A qualified post-repair verification record

Access, indexing controls, HTML metadata and broader website outcomes remain separate. Passing one stage does not prove the others.

Editorial position

A small business should audit crawlability and metadata as one controlled workflow: define the important pages and journeys, crawl a known URL set, separate HTML pages from machine-readable endpoints, examine access and indexing signals independently, validate page-type metadata rules, prioritise defects by business impact, and rerun the same tests after repairs. A successful response alone does not prove that a page is indexed, accessible, functionally correct or capable of generating enquiries.

What follow-up questions matter most?

What should I define before running a website crawler?: Start with the pages and customer journeys that matter to the business. Build a defined URL set from the XML sitemap, navigation, priority landing pages, contact paths and known URL variants. Record each requested URL, its final destination, status, response type and discovery source.
Does a successful HTTP response mean a page is healthy?: No. A successful HTTP response is an access observation. It does not establish that the page is indexed, accessible, functionally correct, useful to visitors or capable of delivering an enquiry.
Should metadata rules apply to every successful response?: Apply metadata checks only to applicable HTML responses. Machine-readable endpoints may need different controls, so report them separately rather than treating their lack of HTML metadata as a defect.
How should a small business verify crawl and metadata repairs?: Use the same intended URL set, page-type rules and exclusions after repairs. Keep the original and rerun records together so the team can identify what changed, what passed and which exceptions remain.

What steps does this workflow follow?

Run a small-business crawlability and metadata audit

Define the audit boundary: List the important pages, customer journeys, expected user actions and intentional exclusions before collecting findings.
Build the intended URL set: Combine XML sitemap routes, navigation links, priority landing pages, contact paths and known URL variants.
Retain crawl evidence: For each requested URL, record the final URL, response status, response type, discovery source and crawl date.
Separate access from indexing: Review broken destinations, redirects, robots controls, noindex instructions, canonical targets, sitemap inclusion and internal discovery as distinct checks.
Test metadata on HTML pages: Define page-type requirements and apply them only to applicable HTML responses, retaining every exception at URL level.
Prioritise and assign repairs: Group repeated symptoms by likely cause, then assign an owner, expected outcome and rerun test according to business impact.
Rerun and report: Repeat the defined checks after repairs and report the tested set, response-type split, exclusions, remaining exceptions and limits.