Technical SEO Defects in Litigation

Crawling and indexing are separate stages that fail separately

Two words get used interchangeably in these matters and should not be. Crawling is the retrieval of a URL by an automated agent; indexing is the decision to store and rank what was retrieved. They are sequential, independent, and fail for different reasons and by different parties' conduct.

The Page Indexing report in Search Console is the primary first-party record of both, and it states its reasons in fixed language. Not-indexed reasons include server error, redirect error, blocked by robots.txt, marked noindex, soft 404, not found, blocked by access forbidden, crawled but currently not indexed, discovered but currently not indexed, alternate page with a proper canonical tag, duplicate without a user-selected canonical, and page with redirect. Two warning states matter as well: indexed though blocked by robots.txt, and page indexed without content.

The forensic value is in which reason predominates, because different reasons point at different defendants. “Discovered — currently not indexed” at scale is Google saying it wanted to crawl but expected to overload the site: a hosting and capacity finding. “Crawled — currently not indexed” at scale is a content quality finding, and points toward whoever produced the content. “Soft 404” at scale after a launch points at redirect targets that are not equivalents, which points at whoever built the map. Reading the report as one number is how that distinction gets lost.

robots.txt, noindex and canonical: the three-way confusion

This is the most common technical misunderstanding in these matters, and the thing an expert is most often asked to explain to a court. The three mechanisms do different work and are not substitutes.

• robots.txt Disallow — a rule in a text file at the top level of a host telling crawlers not to request certain URLs. Google is explicit that it is not a mechanism for keeping a page out of Google: a disallowed page can still be indexed if other sites link to it.
• noindex — either a <meta name="robots" content="noindex"> element or an X-Robots-Tag HTTP response header. It removes a page from the index, but only once the page has been crawled.
• rel="canonical" — a <link> element naming which of several similar URLs is the one to index. It is a signal, not a directive, and it is neither an access control nor a removal tool.

The consequence is that Disallow and noindex cancel one another. Google states that for noindex to be effective the page must not be blocked by robots.txt and must otherwise be accessible. A page both blocked and marked noindex remains indexed indefinitely, and removal is not instant either — Google warns that depending on a page's importance it may take months for Googlebot to revisit.

There is an observable fingerprint for the mistake, and it is worth asking for by name: the Page Indexing warning “Indexed, though blocked by robots.txt” is the signature of a party that used Disallow intending deindexation.

JavaScript rendering, and the one genuinely binary test

Google processes JavaScript applications in three phases — crawling, rendering, indexing. Pages returning 200 are queued for rendering, and Google's own description is that a page may stay on that queue for a few seconds but can take longer, after which a headless browser renders it and executes the JavaScript.

Three failure modes recur. Blocked resources kill rendering outright: Google will not render JavaScript from blocked files or on blocked pages, so a Disallow on a script directory removes content from the index even though the page itself is crawlable. Fragment-based routing cannot be resolved reliably, so applications relying on hash URLs rather than the History API lose pages. And single-page applications generate soft 404s, because a client-routed “not found” view still returns 200.

What makes rendering valuable evidentially is that the test is binary and repeatable. Compare the raw HTML response with the rendered document — view source against the crawled page in URL Inspection, or a rendering crawler against a non-rendering one — and the disputed content either appears or it does not. In my experience it is the easiest technical finding to demonstrate to a jury, because nothing sits between the exhibit and the conclusion. Most findings here are not like that.

Crawl budget, faceted navigation and site scale

Crawl budget is shorthand for two things Google defines separately: the crawl capacity limit, meaning the maximum connection time Google will spend without overloading the server, and crawl demand, meaning Google's appetite for the site given its size, update frequency, quality, popularity and staleness.

Google describes the thresholds at which this becomes a real concern as a rough estimate, and the numbers matter because most sites in litigation are nowhere near them: more than a million unique pages changing about weekly, more than ten thousand changing daily, or a significant share of URLs sitting in “Discovered — currently not indexed.” Server behavior feeds back in directly: Google states that if a site slows down, or responds with 5xx errors or rate-limiting signals such as 429, the limit goes down and Google crawls less.

Faceted navigation — filter and sort controls generating a URL per combination — is how a moderate site becomes an enormous one. Five filters with four values each produce hundreds of crawlable permutations per category, most returning nearly identical content. The measurable effects are crawl requests consumed on parameterized URLs, visible in server logs by user agent and query string, and growth in “Duplicate without user-selected canonical.” Describe that mechanism from the site's own data rather than asserting a general rule, because the general rule is where cross-examination starts.

Status codes, soft 404s, and why server logs are the ground truth

A handful of status codes carry most of the weight. 5xx errors and 429 reduce crawl rate. A soft 404 is a 200 response for content that does not exist — in Google's words, a page returning a user-friendly “not found” message without a 404 status — and it wastes crawl and can cause redirect targets to be dropped. 404 and 410 both remove content, 410 being the explicit permanent signal. A 503 with a Retry-After header is the correct response to planned maintenance; a maintenance page served as 200 is a soft 404 at site scale. And 401 or 403 on pages meant to be public — staging authentication, a firewall rule, bot management blocking Googlebot — appears in Page Indexing under its own reasons.

Server logs settle all of this, and one methodological step decides whether a log analysis survives. Filter by the Googlebot user agent, verify by reverse DNS against Google's published crawler ranges, then tabulate status codes by URL directory over time. Skip the verification and the dataset includes spoofed user agents; once that is established on cross, the whole tabulation is compromised.

Core Web Vitals and their actual weight

Core Web Vitals are three field metrics of user experience. Google's stated good thresholds, published in its Search documentation, are Largest Contentful Paint within the first 2.5 seconds, Interaction to Next Paint under 200 milliseconds, and Cumulative Layout Shift below 0.1.

The part that decides disputes is what Google says about weight, and it is unhelpful to plaintiffs. Google states that there is no single signal, that its core ranking systems look at a variety of signals aligned with overall page experience, that Core Web Vitals are used by its ranking systems but that there is more to page experience than those scores, and — the sentence that gets read aloud on cross — that Google Search always seeks to show the most relevant content even if the page experience is sub-par.

My view is that this is the single most overstated theory in technical-SEO testimony. A claim that a site lost substantial rankings because of Core Web Vitals is very hard to sustain against Google's own documentation, and an expert advancing it should expect to spend a deposition defending a position the platform contradicts. Page experience operates mainly as a differentiator among comparably relevant results, so a defendant arguing that poor scores caused nothing has Google's guidance on its side. Overstating this costs credibility on the findings that were solid, which is a bad trade.

Turning a technical audit into an exhibit

A technical audit is a summary of voluminous machine-generated data, and that framing has three consequences a litigator should plan for from the outset.

Reproducibility. Record the crawler and version, the user-agent string, the rendering mode, the crawl date and duration, the seed URLs, the robots.txt handling setting, the concurrency and crawl-delay settings, and whether the crawl was authenticated. A crawl that cannot be re-run identically is not a method; it is a screenshot with more rows.

Underlying-data preservation. Keep the raw crawl database and log files, not only the summary tables. The summary is what goes to the jury; the raw data is what makes it admissible and what an opposing expert asks for first.

Separation of observation from opinion. “URL X returned HTTP 302 to URL Y on this date” is an observation, and close to unassailable. “This caused the traffic loss” is an opinion requiring a causal method. Keeping them in separate sections is not stylistic: it lets the observations survive a challenge that succeeds against the causal opinion.

The gap between a defect and a damages figure

Everything above establishes that a defect existed, when, and often who introduced it. None of it establishes that the defect caused a particular revenue loss, and this is where technical opinions most often fail.

The problems are ordinary and they compound. Multiple defects usually coexist, so attributing the loss to one requires isolating it from the others. Core updates run frequently enough to overlap almost any period of interest, and their dated record is public, so an opinion that does not address the overlap is answerable on its face. Demand shifts, competitor activity, paid media changes and the party's own content decisions all move the same curve.

The defensible posture is narrower than clients want. State the defect and its date range as observation. Quantify exposure at the level the data supports — affected URLs, impressions, query set — rather than at the level of company revenue. Where the defect was confined to a section, use the unaffected sections of the same site as an internal control, the strongest comparison available because it holds brand, seasonality and market constant. Then say plainly what the analysis cannot separate. Google's documentation on the Page Indexing report, JavaScript rendering and page experience is the reference point for all three.

Frequently Asked Questions

Published 2026-08-11