The Scraping Separation Trap: The UK Just Unbundled Google Visibility

For two decades, Google visibility felt like one dial. A publisher allowed crawling, earned a place in the index, and accepted that the same content could feed whatever search surface Google built next. A page could not stay fully discoverable in ordinary search while independently refusing a new generative answer surface.

The UK Competition and Markets Authority changed that architecture. Google received Strategic Market Status in October 2025 under the Digital Markets, Competition and Consumers Act. On January 28, 2026, the CMA published four proposed conduct requirements for Google Search. On June 3, it introduced the first requirement, giving publishers more control and bargaining power over the use of their content.

Read the CMA and gov.uk announcement carefully. The measure includes a tool for publishers to opt out of Google's generative AI search features without losing normal search rankings. That is not a ban on AI answers, a licensing mandate, or a global rule. It is an unbundling of uses in the UK.

The reporting on the new opt-out makes the practical change easier to see. Publishers can remain in ordinary search while making a separate decision about generative AI search. That choice may create leverage for future licensing or simply protect a click model. It also creates a new operating responsibility: someone must decide which use is valuable enough to allow.

That last sentence is the operational handoff. Editorial teams can decide what they want to publish, but product, search, legal, and revenue teams now share a setting that can change how the work is used. The setting needs an owner and a reason. Otherwise a technical default will silently become a business strategy.

The first switch is classic indexing. Googlebot crawls the page and the page can enter ordinary Google Search. Turn this off and the content loses the broadest discovery surface. It is the old, blunt control and still the one most publishers understand.

The second switch is AI-feature inclusion. It governs whether the content can be summarized or cited inside Google's generative search features. The new UK control makes this distinct from ordinary ranking for publishers covered by the conduct requirement. A publisher can therefore stay eligible for a search result while declining to supply the answer layer.

The third switch is AI-training access. Google-Extended is the relevant crawler token, not “Googlebot-Extended.” It addresses whether content may be used to improve certain generative AI models. Google's crawler documentation should be the reference point for implementation details. Training control is not the same as indexing control and does not automatically remove a page from an answer generated from existing indexed material.

The three switches answer different business questions: should humans find the page, should a machine use it in an answer, and should a future model learn from it? The CMA's importance is that at least two of those questions can now be separated for UK publishers. The strategic mistake is to keep describing visibility as one green number.

The implementation language should stay equally exact. A training exclusion is not an instruction to remove a page from search. An AI-feature opt-out is not a guarantee that another assistant will not cite the page. Classic indexing is not the same as a click, and a citation is not the same as qualified demand. Each event needs its own field in the measurement record.

The phrase “publishers won” is too simple. The CMA created choice; it did not make any choice costless. Allow everything maximizes reach and citation eligibility, but it also gives the answer engine more value and may replace the visit that paid for the original page.

Blocking AI features while keeping classic indexing defends a click-based model in theory. The page can still rank, but the AI answer can use a competitor instead. The category remains visible while the publisher disappears from the explanation that shapes the user's decision. This is protection of a visit, not protection of every impression.

Blocking training while allowing AI features is different. It keeps the page eligible for current retrieval or citation while declining to donate the archive to future model improvement. For a small site the leverage may be symbolic. For a large archive, it can establish a licensing posture and a record of what the publisher considers valuable.

Blocking both AI switches is the strongest protection posture and the largest bet. A subscription publisher with a powerful direct relationship may accept the loss of answer-layer reach. A service brand that depends on being named in a buyer's shortlist may not. There is no universal setting because the value of a click, a citation, and a training signal differs by business model.

The strategic trap is collective. If an entire category opts out, answer quality may fall and a licensing market may become viable. If one publisher opts out while competitors remain available, the engine can keep answering and simply remove that publisher's voice. A settings page can expose a prisoner's dilemma without solving it.

There is a second trade around leverage. A publisher that allows every use may receive more near-term visibility but has less evidence that its archive is scarce. A publisher that blocks every use may preserve negotiating power but lose the chance to shape the answer layer while it is still forming. The useful choice is rarely ideological. It is a question of which surface produces revenue, trust, or bargaining power for this business.

That choice should be recorded like a product decision. Name the owner, the market, the control being changed, the expected benefit, the risk accepted, and the measurement window. Set a review date. Without that record, the organization will forget why a switch was flipped and treat the default as policy forever.

Click-monetized publishers have the clearest case for testing an AI-feature opt-out. If the product is the visit, a complete answer is a substitute. Keep indexing, measure defended sessions and revenue, and make the experiment reversible. A traffic fear without a measured baseline is not a strategy.

Brands selling products or services face a different calculation. The click may be useful, but the shortlist or recommendation can be the more important moment. The default may be to remain in AI features, block training access where appropriate, and invest in accurate product and business context. Visibility is not automatically a visit, but absence is still a loss of consideration.

Premium or IP-heavy publishers can treat training access as inventory. They may allow a controlled answer relationship while preserving leverage over how the archive improves future systems. The business should log the setting, the market, the archive covered, and any licensing or platform communication that follows.

Regulated categories need a fourth lens: paraphrase risk. Sometimes the problem is not lost traffic but an answer that compresses a nuanced claim into something inaccurate or noncompliant. An AI-feature opt-out can be a claims-control measure where the cost of a wrong summary is greater than the value of the citation.

Make the decision per market. The control is UK-specific, and ordinary indexing, AI features, and training rules may behave differently in the US, EU, or APAC. A single global robots file cannot express a market-specific business decision. Keep a policy matrix instead.

A simple matrix can use the business model on one axis and the value of the answer surface on the other. A publisher whose primary product is the page itself starts by defending the visit. A brand whose primary product is consideration starts by defending accurate inclusion. A premium archive starts by defending training leverage. The matrix does not make the answer permanent. It makes the assumption visible enough to challenge.

Audit the setting after every meaningful product or legal change. A new AI feature can change the value of inclusion. A new licensing conversation can change the value of training. A ranking update can change the cost of refusing a feature. The policy should follow the business model and the evidence, not the vocabulary of the last press release.

The deeper change is jurisdictional. The UK has separated uses of content through a conduct requirement. The EU has adjacent leverage under the Digital Markets Act and transparency rules. The United States is developing a more market-led mix of crawl preferences, licensing, and platform policy. The same page may have different visibility economics depending on where the user is.

Google now has a choice of its own: regionalize the tooling or globalize it. Regionalization preserves local compliance but creates operational complexity. Globalization is simpler and could turn a UK remedy into the default expectation everywhere. The outcome matters less to a publisher that already records its settings and tests its results by market.

Measurement must split along the same lines. Track classic rankings, AI-feature inclusion and citations, AI-referred sessions, training access, and the market in which each event occurred. Keep the exact prompt, answer, cited sources, and timestamp when testing an answer surface. That is the record needed to know whether a setting defended a visit, lost a citation, or changed nothing.

Attribution will be imperfect, so make the test narrow. Hold the content, market, and question set steady where possible. Compare the period before and after a switch, watch whether competitors become the cited default, and note changes in branded searches and direct sessions. A policy that cannot produce a clear read after a defined window is not yet a policy. It is a belief attached to a robots file.

The work also becomes more cross-functional. Search engineering can implement and verify the controls. Editorial can decide which archives are valuable. Revenue can estimate the value of a visit versus an impression or citation. Legal can preserve the market scope and the organization's stated rationale. The decision is durable only when all four views are in the record.

A useful review asks five plain questions. Which market is covered? Which control changed? What did the publisher expect to protect? What evidence would show the trade was worth it? Who can reverse the setting? Those questions keep the regulatory event attached to an operating decision instead of turning it into another abstract debate about scraping.

Keep one more distinction in view: a publisher can be discoverable without being authoritative. The answer may mention the page and still paraphrase it badly, cite it without sending a visit, or use a competitor as the source for the important sentence. Inclusion, accuracy, citation, and qualified demand belong in separate fields. That is the only way to learn what a control actually changed.

Once the record exists, the setting becomes testable. If the defended visit never arrives, reverse the decision. If the citation is accurate and valuable, keep it. If the answer surface changes the economics, renegotiate from evidence. The point of the new separation is not to choose a side once. It is to give publishers a choice they can inspect.

Keep the implementation reversible and the evidence exportable. A future regulator, platform change, or licensing discussion may require the organization to show what it allowed and when. The exact prompt, answer, URL, timestamp, crawler setting, and internal decision owner are more useful than a retrospective claim that visibility “went down.”

The old dial is not coming back together. AI visibility is already a different measurement problem from classic SEO, and the UK rule makes the split operational rather than theoretical. Decide per switch, per market, and per business model. Default settings are still choices, just unexamined ones.