Technical SEO checklist (2026): crawling to Core Web Vitals

A robots.txt file that returns a 500 error during a deploy can pause Google’s crawling of your whole site for twelve hours, and a noindex left in a template can remove a section from search without any error appearing anywhere. Technical SEO is mostly the discipline of checking the things that fail silently. This technical SEO checklist groups those checks into twelve areas, and for every item gives the reason it matters, a way to test it and the tool to use. Tick the items as you go; the list renders as plain checkboxes you can copy into a ticket.

How to use this checklist

Work in the order given. The first four groups decide whether a page can appear at all; the rest decide how well it competes. For a one-page spot check, the free audit at the top of this site runs the crawling, indexing, status, security and snippet items automatically in about a minute, with an owner and effort tag on each finding. For a full site, pair a crawler export with Search Console and work through each group by template rather than by page.

Each item follows the same pattern: what to check, then why it matters, how to test it and which tool does it fastest.

1. Crawling (including AI bots)

• robots.txt returns 200 with plain text, or 404, never 5xx. Why: Google says a 4xx (except 429) is treated as no restrictions, but on a 5xx it stops crawling for 12 hours and then relies on a cached copy for up to 30 days (Google’s robots.txt spec). Test: request /robots.txt and check the status and content type. Tool: the robots.txt tester; our audit’s “robots.txt” check.
• No Disallow rule blocks pages you want indexed, and none blocks CSS or JavaScript. Why: blocked resources mean Google renders a broken page; blocked URLs may still be indexed without a snippet. Test: test your key URLs and asset paths against Googlebot. Tool: robots.txt tester; Search Console URL Inspection.
• The file is under 500 KiB. Why: Google ignores anything after the limit. Test: check the response size. Tool: any HTTP client.
• Each AI crawler is allowed or blocked on purpose. Why: OpenAI says blocking OAI-SearchBot removes a site from ChatGPT search answers while GPTBot only controls training; Anthropic and Perplexity split their bots the same way; Google’s AI features use ordinary Googlebot and Google-Extended is a training and grounding token, not a crawler. Test: test one URL against OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, bingbot, Applebot, CCBot. Tool: robots.txt tester; the AI crawlers guide for the policy snippets.
• Your firewall or CDN does not block AI crawlers by user agent unless you intend it. Why: a WAF rule can return 403 to a crawler that robots.txt allows, and nothing in Search Console will tell you. Test: fetch a page with each crawler’s user agent string from outside your network. Tool: our audit’s “Fetch as AI crawler” check; the AI visibility checker.
• Only the most specific robots.txt group applies, and you have written it that way. Why: Google says a crawler uses the single group with the most specific matching user agent and ignores the * group when a named group exists, so a User-agent: GPTBot block with no Allow lines does not inherit your global rules. Test: read the file with that rule in mind. Tool: robots.txt tester.
• Crawl budget only if you are large. Why: Google says its crawl budget guidance is for sites with over a million pages changing weekly or over 10,000 changing daily. Test: if pages are crawled the day they are published, skip this. Tool: Search Console Crawl stats; the crawl budget entry.

2. Indexing

• No unintended noindex in a robots meta tag or X-Robots-Tag header. Why: it is the most common way a template or a staging flag removes a section from search. Test: check the HTML head and the response headers on every template. Tool: the meta tag checker; our audit’s “Indexability” check; the noindex entry.
• Pages you want excluded carry noindex and are not also blocked in robots.txt. Why: Google says robots directives can only be read if crawlers are allowed to access the page (Google’s robots meta tag documentation). Test: for each excluded page, confirm it is crawlable and carries the tag. Tool: robots.txt tester plus meta tag checker.
• An XML sitemap exists, is valid, is referenced in robots.txt and submitted in Search Console. Why: it is how you tell Google which URLs you consider canonical. Test: fetch the sitemap, validate it, count URLs. Tool: the sitemap checker; the XML sitemap entry.
• The sitemap lists only canonical, indexable, 200-status URLs, with accurate lastmod. Why: Google says it uses lastmod when it is consistently accurate and ignores priority and changefreq; redirecting or noindexed entries waste the file. Test: sample 50 sitemap URLs for status and canonical. Tool: sitemap checker; a crawler in list mode.
• No sitemap exceeds 50,000 URLs or 50 MB uncompressed. Why: Google’s documented limit. Test: count. Tool: sitemap checker.
• Page indexing report reviewed for “Crawled, currently not indexed” and “Discovered, currently not indexed”. Why: these are Google’s quality and priority verdicts, and they rarely come from a technical error. Test: open the report, export examples, read the pages. Tool: Search Console; the indexability entry.
• No soft 404s. Why: an empty category or “no results” page that returns 200 is reported as a soft 404 and keeps being crawled. Test: filter the Page indexing report for the reason. Tool: Search Console; our audit flags a 200 with almost no text.

3. Rendering

• The main content, links and canonical are in the raw HTML. Why: Google renders JavaScript in a queue “once Google’s resources allow”, and Google itself notes that not all bots can run JavaScript; most AI crawlers fetch raw HTML only. Test: compare the view-source word count with the rendered page. Tool: our audit’s “Text in the HTML” check warns below 200 words; the JavaScript rendering entry explains the pipeline.
• Rendered HTML matches raw HTML for titles, canonicals and robots directives. Why: a JavaScript framework that rewrites the canonical or injects noindex after load creates two versions Google may read differently. Test: inspect the rendered DOM in URL Inspection. Tool: Search Console URL Inspection; the canonical checker.
• No content depends on a click, scroll or tab to appear in the DOM. Why: Google indexes what is in the rendered DOM; it does not interact. Test: check accordions and tabs in the rendered HTML. Tool: URL Inspection.
• Lazy-loaded images use native loading="lazy" or expose the src without scroll events. Why: images that only load on scroll are never seen by a crawler. Test: view rendered HTML for img src. Tool: URL Inspection.

4. Architecture and internal links

• Every page you care about is linked from at least one other page. Why: Google’s link guidance says exactly this; orphan pages are discovered late or not at all. Test: crawl export, filter inlinks = 0. Tool: the website audit; the orphan page entry.
• Important pages are within three clicks of the home page. Why: depth correlates with how often Google recrawls a URL and with how much link value reaches it. Test: crawl depth column. Tool: a crawler.
• Links are real a elements with an href, and anchor text is descriptive. Why: Google says it reliably follows only a href links, and that “click here” tells it nothing. Test: crawl report of anchor text; check JavaScript-driven navigation. Tool: our audit’s “Anchor text” and “Internal links” checks.
• One URL per page: one hostname, one protocol, one trailing-slash convention, lower case. Why: each variant that returns 200 is a duplicate competing with the original. Test: request the www and non-www, http and https, slash and no-slash forms. Tool: the redirect checker; our audit’s “www / non-www consistency” check.
• Faceted and parameter URLs are controlled. Why: filters can generate millions of crawlable URLs that dilute crawling and cannibalise categories. Test: search the crawl export for ?. Tool: a crawler; noindex on filtered pages, then robots.txt once de-indexed.
• No internal links point at redirects or 404s. Why: each adds a hop or a dead end. Test: crawl export, filter outlinks to non-200. Tool: website audit; our audit samples up to 25 internal links.

5. Status codes and redirects

• Every canonical URL returns 200 directly. Why: search engines index 200s; other codes are errors, redirects or soft errors. Test: crawl status column. Tool: website audit; our audit’s “HTTP status” check.
• Permanent moves use 301 or 308, not 302 or 307. Why: Google treats 301 and 308 as canonicalisation signals and 302, 303 and 307 as temporary, which tends to keep the old URL indexed. Test: trace the redirect. Tool: redirect checker; the 301 vs 302 redirect entry.
• No redirect chains or loops. Why: Google says Googlebot generally follows 10 hops, but each hop adds latency and leaks signals, and loops end crawling. Test: trace home page, top landing pages and legacy URLs. Tool: redirect checker; the redirect chains post covers causes and fixes.
• http redirects to https and the alternate hostname redirects to the chosen one in a single hop. Why: two hops (http www to https www to https apex) is the most common chain on the web. Test: request each variant. Tool: redirect checker; our audit’s “HTTP to HTTPS redirect” check.
• Removed pages return 404 or 410, not 200 or a redirect to the home page. Why: Google treats a 404 as a strong signal to stop crawling the URL; a home page redirect is a soft 404. Test: request a known removed URL. Tool: redirect checker.
• Custom 404 page returns a real 404 status. Why: a “not found” template served with 200 is indexed as a page. Test: request a nonsense path. Tool: any HTTP client.

6. International

• Every language version lists itself and all alternates with hreflang. Why: Google says unreciprocated annotations are ignored (Google’s hreflang documentation). Test: check one template per market for return links. Tool: the hreflang checker; the hreflang entry.
• Codes are ISO 639-1 language with optional ISO 3166-1 Alpha 2 region; no “uk”, “eu” or region-only values. Why: Google says these invalid codes are ignored. Test: list the distinct codes in use. Tool: hreflang checker.
• An x-default points to the language selector or the fallback page. Why: it catches users whose language matches none of your versions. Test: look for it in the set. Tool: hreflang checker.
• hreflang URLs are absolute, canonical and return 200. Why: annotations to redirected or non-canonical URLs are discarded. Test: crawl the hreflang targets. Tool: hreflang checker plus redirect checker.
• Only one implementation method is used. Why: Google says HTML, HTTP header and sitemap methods are equivalent and combining them adds no benefit, only conflict risk. Test: check for both link tags and sitemap entries. Tool: sitemap checker.

7. Performance (LCP, INP, CLS)

• LCP is 2.5 seconds or less at the 75th percentile of real users. Why: Google says Core Web Vitals are used by its ranking systems; the target is field data, not a lab run (web.dev on Core Web Vitals). Test: Search Console Core Web Vitals report, mobile first. Tool: the Core Web Vitals checker; the LCP entry.
• INP is 200 milliseconds or less at p75. Why: INP measures responsiveness across the whole visit; third-party scripts and heavy handlers are the usual causes. Test: same report. Tool: Core Web Vitals checker; the INP entry.
• CLS is 0.1 or less at p75. Why: layout shift comes from images without dimensions, late-loading fonts and injected banners. Test: same report. Tool: Core Web Vitals checker; the CLS entry.
• Time to first byte is under 800 milliseconds. Why: every other metric waits for it. Test: field TTFB in the checker, or a server timing header. Tool: our audit’s “Server response time” check.
• Text responses are compressed and HTML is under 300 KB. Why: uncompressed or bloated HTML delays parsing on every visit. Test: check the Content-Encoding header and response size. Tool: our audit’s “Text compression” and “HTML document size” checks.
• Images declare width and height, use modern formats and are sized to the slot. Why: dimensions prevent CLS; AVIF and WebP cut LCP. Test: audit the hero image on each template. Tool: our audit’s “Image dimensions” check; Lighthouse opportunities.
• No render-blocking CSS or JavaScript above the fold. Why: it delays LCP directly. Test: Lighthouse opportunities. Tool: Core Web Vitals checker.

8. Mobile

• A responsive viewport meta tag is present. Why: Google indexes the mobile version of your content; without a viewport the page renders as desktop on phones. Test: check the head. Tool: our audit’s “Mobile viewport” check.
• Mobile and desktop serve the same content, structured data and robots directives. Why: Google’s mobile-first guidance asks for parity on all three. Test: compare the mobile and desktop renders. Tool: URL Inspection with a mobile user agent.
• No intrusive interstitials on page load. Why: Google’s page experience guidance lists them as a self-assessment question. Test: load the page on a phone. Tool: manual.
• Tap targets and text are usable without zooming. Why: Lighthouse accessibility and best-practice audits flag both. Test: Lighthouse mobile run. Tool: Core Web Vitals checker.

9. Structured data

• JSON-LD is used, parses, and describes only visible content. Why: Google recommends JSON-LD and says not to add structured data about information the user cannot see (Google’s structured data introduction). Test: validate each template. Tool: the schema markup validator; our audit’s “Structured data parses” check; the JSON-LD entry.
• Organization or Person markup includes sameAs links to official profiles. Why: it ties the site to a known entity for search and AI systems. Test: look for the node and the property. Tool: our audit’s “Entity markup” check.
• Article pages carry author, datePublished and dateModified that match the visible page. Why: mismatches between markup and page are a guideline violation and a trust problem. Test: compare. Tool: schema markup validator.
• No markup for rich results that no longer exist. Why: Google says FAQ rich results stopped appearing in Search from May 7, 2026, and HowTo rich results ended in 2023. The markup is harmless but earns nothing. Test: list the types in use. Tool: the rich results entry lists what is still live.
• Product, Review and SoftwareApplication markup use real values. Why: invented ratings are a manual action risk. Test: read the markup against the page. Tool: schema markup validator; the structured data entry.

10. HTTPS and security headers

• Every page is served over HTTPS with a valid certificate that is not about to expire. Why: Google has used HTTPS as a lightweight ranking signal since 2014, and browsers block mixed content. Test: check the certificate chain and expiry. Tool: our audit’s “HTTPS” and “TLS certificate” checks.
• Strict-Transport-Security is set with a long max-age, and includeSubDomains only once every subdomain is on HTTPS. Why: HSTS makes the browser upgrade every future request; MDN notes that includeSubDomains blocks any subdomain that is HTTP-only (MDN on Strict-Transport-Security). Test: read the response headers. Tool: our audit’s “HSTS” check.
• No mixed content. Why: http resources on an https page are blocked or warned. Test: scan the HTML for http:// asset URLs. Tool: our audit’s “Mixed content” check.
• X-Content-Type-Options: nosniff, a Referrer-Policy and framing protection (X-Frame-Options or CSP frame-ancestors) are set. Why: none affect ranking, but Lighthouse best practices and most security reviews flag their absence. Test: read the headers. Tool: our audit’s security checks.
• A Content-Security-Policy exists, at least in report-only mode. Why: it limits where scripts can load from. Test: read the headers. Tool: our audit’s “Content-Security-Policy” check.
• Server and framework version numbers are not disclosed in headers. Why: it is a small hardening step with no SEO cost. Test: read Server and X-Powered-By. Tool: our audit’s “Version disclosure” check.

11. Snippet controls for AI features

• No nosnippet or max-snippet:0 on pages you want in AI Overviews and AI Mode. Why: Google says these directives also prevent content being used as a direct input for AI Overviews and AI Mode. Test: search templates for the directives. Tool: meta tag checker; our audit’s “Snippet eligibility” check; the nosnippet entry.
• max-snippet values are deliberate, not a legal default. Why: a low limit caps how much Google may quote. Test: list the values in use. Tool: meta tag checker.
• data-nosnippet wraps only the text you actually want withheld. Why: it is per-element and easy to leave on a whole article wrapper. Test: search the templates. Tool: view source.
• Bing noarchive and nocache are deliberate. Why: Bing says noarchive keeps content out of its generative answers and training entirely, while nocache limits it to URL, title and snippet; both leave the page in normal results. Test: check for the tags. Tool: meta tag checker.
• Pages are indexed and snippet-eligible, which is Google’s stated condition for AI features. Why: Google says no special files, markup or Markdown are needed; indexing and snippet eligibility are. Test: URL Inspection. Tool: Search Console; the AI visibility checker.

12. Monitoring

• Search Console and Bing Webmaster Tools are verified, with sitemaps submitted. Why: they are the only free sources of each engine’s view of you. Test: open both. Tool: the two consoles.
• Page indexing, Core Web Vitals and Performance reports are reviewed monthly. Why: most technical regressions show up there first. Test: calendar it. Tool: Search Console.
• Certificate expiry, uptime and robots.txt status are alerted. Why: each fails silently and costs crawl time or trust. Test: confirm an alert fires. Tool: your monitoring service.
• A single-page scan runs after each release on the home page and one key template. Why: it catches leaked noindex, broken canonicals, lost headers and blocked crawlers within minutes. Test: add it to the release checklist. Tool: the free audit.
• A full crawl is compared against the previous one after any migration or redesign. Why: status, title, canonical and inlink diffs expose what changed. Test: diff the exports. Tool: a crawler; the SEO audit guide covers the process end to end.
• Log files or CDN analytics are checked quarterly for crawler behaviour. Why: they show which bots fetch what, and whether a spoofed crawler is scraping you. Test: filter by user agent and verify against published IP ranges. Tool: the AI crawlers guide explains verification.

What to do next

Clear the first four groups first; they decide whether pages can appear at all. Run the free audit on one URL per template to get the crawling, indexing, status, performance, security and snippet items scored with owners, then take the remaining groups to a crawler export. If you are blocking or allowing AI bots, the robots.txt examples for AI bots post has tested snippets, and if you are weighing whether to hand the work out, what technical SEO audit services include explains what to expect from a proposal.

Frequently asked questions

What is a technical SEO audit?

A technical SEO audit checks the machine-facing layer of a site: whether crawlers can reach pages, whether pages are indexable, how they render, how fast they load, whether redirects and canonicals point the right way, and whether structured data and security headers are correct. This checklist is the audit; run it top to bottom.

Which items matter most for a new website?

For a new site, confirm the five things that block everything else: the site is not noindexed, robots.txt allows crawling, an XML sitemap exists and is submitted, every page returns 200 over HTTPS with one hostname, and the main content is in the raw HTML. Do those before anything else on the list.

Does my Lighthouse score affect rankings?

Not directly. Google says Core Web Vitals are used by its ranking systems, and those are measured from real users at the 75th percentile, not from a Lighthouse lab run. Use Lighthouse to diagnose, and the Search Console Core Web Vitals report to know where you stand.

Should I block AI crawlers in robots.txt?

It is a policy decision with a cost. Blocking OAI-SearchBot removes you from ChatGPT search answers; blocking PerplexityBot and Claude-SearchBot removes you from those engines; blocking GPTBot, ClaudeBot and Google-Extended only affects training and grounding. Decide per crawler, not with one wildcard.

How often should I run through this checklist?

Fully after any migration, redesign or platform change and at least once a year. Between those, monitor the Search Console Page indexing and Core Web Vitals reports monthly, and run a single-page scan after each release to catch leaked noindex tags, broken canonicals and blocked crawlers.

How many of these items apply to a small site?

Most of them, but the effort is small. A ten-page site can clear crawling, indexing, redirects, HTTPS, mobile and snippet controls in an afternoon. Crawl budget, hreflang and large-site monitoring only apply if you have the scale or the markets that need them.