What Makes a Search Methodology Reliable

What the protocol is for

The reliability inquiry for a technical expert cannot attach to the field. There is no peer-reviewed literature on a proprietary ranking system, no published error rate, no licensure, and no controlled experiment available on a live engine. So it attaches to the analyst, and the question becomes whether the analysis has inputs, operations, and outputs somebody else can obtain by repeating the steps.

That is the point of writing a protocol down: it supplies the testability the field itself cannot. It is also what decides depositions. When every number traces to a named export and every exclusion is stated as a rule, the standard questions are answered from the page rather than from memory.

What follows is how I work. It is not the only defensible way to work, and it is deliberately unexciting. The parts that matter are the parts most reports leave out.

Name every source, and say where it came from

“Industry data” is not a source. Neither is “Google Analytics.” My standard is that a second analyst with the same access could locate the identical file from my description alone.

• Search Console. Google Search Console is the engine's own reporting interface for a site whose ownership has been verified. State the property and whether it is a domain or URL-prefix property, since they cover different sets of addresses; then the report, dimensions, filters, date range, and export date. Performance data covers a rolling sixteen-month window, so say whether the pull preceded its closing over the period at issue.
• Analytics. The platform and version, the property and data stream, the filters active on it, the date range, and the attribution and channel definitions in force — because redefining a channel produces a decline in a report with no change in reality.
• Server and CDN logs. Which hosts, date range, and format, and how requests attributed to a crawler were verified as genuine — user agent strings are trivially forged, and an unverified log analysis counts requests from anything claiming to be Googlebot.
• Crawls. The tool and version, the date and time of the run, and the configuration, which is part of the source rather than a footnote.
• Third-party exports. The vendor, the product, the plan or database, the export date, and what the vendor publishes about how the figure is produced.

To each I add who pulled it, from which account, and when — because the first thing an opposing expert asks about a spreadsheet is where it came from, and the second is whether anyone edited it.

Separate what was measured from what was modeled

This is the most useful distinction in the discipline, and most reports do not draw it at all.

Measured data is a record of events. Search Console reports the engine's own count of impressions and clicks for a verified property; analytics records sessions the tag observed; server logs record requests the server answered. Each has limitations, and each is still a record.

Modeled data is an estimate produced by a vendor from panel data, inference, and assumption. A rank tracker issues a query on a schedule from a chosen location and device and records where a URL appeared; it measures a synthetic query rather than what a user saw. A search volume figure is modeled and bucketed. A traffic estimate is an estimate multiplied by an estimate: an assumed click-through curve applied to an estimated position for an estimated volume. A backlink count describes one vendor's crawl index, not the web.

I use both, and I label every figure as one or the other in the report. The rule I apply is that a modeled number may support a description and may never carry a conclusion a measurement in the party's own possession would have tested. Where an estimate is the only data available — a competitor's site, or a period before the client began measuring — I state what the vendor publishes about how it is produced and which direction the error is likely to run. An estimate presented honestly is evidence; an estimate presented as a measurement is a cross-examination waiting to happen.

State the window, the comparison window, and why that one

Every date range is a choice, and an unexplained one looks convenient.

I state the analysis window, the comparison window, and the reason for the comparison. If the comparison is the equivalent period a year earlier, I say why year-over-year suits this site's demand pattern; if it is the immediately preceding period, I say what makes the two comparable. I align days of the week where the metric is traffic, because a 28-day window compared against a 31-day one produces a decline that means nothing.

I state the time zone the data is reported in, which sounds pedantic until two platforms' exhibits disagree by a day and the timeline argument turns on it, and the date of every export, because these platforms restate recent data.

And where the sixteen-month Search Console window has already closed over the period at issue, that is a fact about the evidence, not a gap to be papered over with a modeled substitute.

Write exclusions as rules, not as adjustments

Every analysis excludes something: bot traffic filtered, internal addresses removed, brand queries separated, staging subdomains dropped, an outlier day set aside. None of it is improper, and all of it is fatal when it surfaces for the first time in a deposition.

The distinction is between stating a rule and stating a result. “Brand queries were excluded” is a result. “Queries containing the strings listed in Appendix C, being the company name and its misspellings, were classified as brand and analyzed separately; the list is produced” is a rule. The first invites the question of what counted as brand and whether the answer was chosen to help. The second answers it and makes the choice checkable — a disagreement about method, which is a much better place to be than a disagreement about candor.

Where an exclusion materially moves the result, I report the figure both ways.

Make the run repeatable, and preserve the material that lets someone repeat it

Reproducibility is the reliability consideration a search analysis satisfies most directly, and it counts only if the other side can attempt it in fact.

For a crawl I record the tool and version, the start URL or supplied list, the user agent presented, whether JavaScript rendering was enabled, the depth and limits, the rate, whether robots.txt directives were respected, which subdomains and parameters were in or out, and the date and time of the run. Those settings change the results: a crawl with rendering disabled will not see content injected by script, and reporting it missing would be an artifact of my configuration rather than a finding about the site.

For platform data I record the exact filter expressions and export parameters rather than a description of them, and I keep and produce every query written against an export — formula, script, or database query — because the query is the method and the number is only its output.

Raw exports are preserved unaltered, with file names, sizes, and dates intact, and derived working files live separately so the path from source to chart is traceable in one direction. I keep a plain run log — date, time, what was run, against what, with what settings — written as the work happens rather than reconstructed afterward. Where a capture needs to be verifiable as unaltered, a hash value computed at capture and recorded in the log does work later for reliability and authentication both.

The test before serving: could a competent analyst, given only my description and the produced files, reach my numbers?

Control for confounds explicitly, and name the ones you could not control for

This is where opinions in this field fail, and it is not a drafting problem.

Search traffic moves for reasons unrelated to any defendant. The candidates are known and short, and I address each by name:

• Broad core updates — engine-wide changes to the ranking systems, announced and dated by the operator, rolled out over days or weeks — though not every change is announced, which is itself a limitation to state. I line the decline up against the published dates and check whether the shape of the change matches a rollout or a discrete event.
• Seasonality — tested against the site's own history over several years rather than asserted.
• Competitor and channel activity — new entrants or relaunches in the same query space, and cuts to advertising that reduce branded demand and therefore organic traffic.
• Measurement changes — an analytics tag removed or misconfigured in the very deployment under examination, a consent banner change, a channel redefinition. Part of a measured decline is sometimes an artifact, and finding that out from the other side is worse than finding it myself.

The work that tests them is segmentation: the affected URLs against untouched sections of the same site, organic against the site's other channels over the same window, the site against a peer set that saw the update but not the conduct. Check whether the loss is concentrated where the conduct was, because a sitewide decline concentrated nowhere is rarely explained by a defect in one template.

Then the sentence most reports omit: which confounds I could not exclude, and what that does to the opinion. If a core update landed inside the window and I cannot separate its effect from the conduct's, I say so and state the narrower opinion the data does support. That opinion survives. The expansive one draws the motion, and after the December 2023 amendment to Rule 702 the answer that an untested alternative goes to weight rather than admissibility is much weaker than it was.

State the limits of the opinion inside the opinion

I write a scope paragraph into every report, before the conclusions. It says four things: which pages or query sets the analysis covers, which period, what the opinion asserts, and what it does not.

In form: the analysis covers the product-detail templates on one subdomain between two stated dates and establishes how many were excluded from the index while a directive was present in the deployed templates; it does not address the category pages and does not quantify revenue.

Counsel occasionally reads that as weakness. It is the opposite. A stated boundary removes the easiest cross-examination in the discipline — walking a witness one step at a time past the edge of what the data reached until he either concedes or overstates. It also divides the labor cleanly: the technical findings stand on their own data, the economic consequences belong to a damages expert who takes them as inputs, and a ruling striking one does not automatically reach the other. An expert who can state where his opinion stops has one; an expert who cannot has an impression.

What the protocol costs and what it buys

It costs time at the front of the engagement, when the client wants an answer and the protocol wants a run log. It costs some conclusions: I have abandoned an approach because validation showed the error rate would not support the opinion I had been asked about, and I have narrowed opinions because a confound could not be separated out. Learning that before a report is served is better than learning it in a deposition, and it is the same work either way.

What it buys is testability — the one reliability consideration this field can actually satisfy — standards controlling the operation in the form of documented configurations, and an error rate wherever the analysis includes a sampling or classification step. It also answers the motion before the motion is filed, because every question the motion would ask has already been answered on the page.

Frequently Asked Questions

Published 2026-08-11