AI companies can build agents that browse, compare, click and buy. That does not mean websites have agreed to let them in.
The first wave of AI access disputes focused on model training. Publishers and creators wanted to know whether their content was being copied into training datasets without permission or compensation. The newer conflict is broader because agents are not just collecting information. They are trying to use the web.
That matters for ecommerce, travel, finance, media and any business that depends on a visitor reaching a page, seeing an offer, logging into an account or completing a transaction through the company’s own interface. A website built for people can look very different when the visitor is an automated assistant acting through a browser, API or cloud session.
Cloudflare has framed this as a split between three kinds of automated activity: Search, Training and Agent. Search crawling supports discoverability. Training crawling collects content for model development. Agent activity is user-directed automation, such as chat fetch bots and browser-use agents visiting pages on behalf of a person.
That taxonomy is becoming central to the fight. A business may want to appear in search results but refuse training. It may welcome a search crawler but block a shopping agent from loading ad-supported pages or checking out through a customer account. The old all-or-nothing bot model is breaking down.
Cloudflare is turning AI access into a policy layer
Cloudflare introduced separate Search, Training and Agent controls on July 1, 2026. On September 15, new domains began receiving default settings that block Training and Agent bots on pages displaying ads while allowing Search. Cloudflare also says mixed-purpose crawlers that combine Search and Training are treated differently under its newer settings because they can force website owners to choose between search visibility and AI data use.
Cloudflare’s September update also introduced Disallow AI Training, a setting intended to let sites refuse AI training while preserving search visibility where crawler operators support the separation. Cloudflare said mixed-use crawlers represented 36.6% of verified crawler traffic on its network, while fewer than 1% of Cloudflare sites block Search bots and 17% restrict AI training.
The company’s July reporting was even broader. Cloudflare said more than 50% of internet traffic was non-human, and that 52% of crawler requests it classified by purpose were for AI training as of June 2026, up from 22% in spring 2025. Those figures come from Cloudflare’s own network and Radar data, but they show why infrastructure providers are turning crawler access into a product category.
Cloudflare is also testing monetization. Its Pay Per Crawl feature remains in closed beta, according to Cloudflare’s documentation, and lets site owners set a price for AI crawler access. If a crawler does not present payment intent, the system can respond with HTTP 402 Payment Required. That is a major shift from the historical web bargain, where crawlers received access in exchange for visibility and referral traffic.
Ecommerce is where the access fight becomes visible
Tech Help Canada previously covered Amazon blocking Meta’s Muse AI shopping agent. Axios reported in September that Amazon said third-party applications that make purchases on behalf of customers should “operate openly and respect service provider decisions” about participation. That line captures the retailer position: a customer’s desire to use an agent does not automatically mean a platform has accepted that agent.
The Amazon-Perplexity court fight shows how unsettled the issue remains. In August 2026, the Ninth Circuit vacated a preliminary injunction Amazon had won against Perplexity over Comet’s AI Assistant and sent the case back for further proceedings. The appeals court said Amazon was unlikely to show, on the preliminary record, that Perplexity itself accessed Amazon’s computers under the federal Computer Fraud and Abuse Act because the user was accessing Amazon with the Assistant as a tool.
That ruling did not settle the broader permission issue. The opinion also said Amazon had told Perplexity that its AI products were not permitted to access the Amazon Store, and that a user-agent string identifying the AI agent would have allowed Amazon to block it. In other words, the legal question was narrower than the operational fight. Retailers still want agents to identify themselves. Agent companies still want their tools to work across the web.
For ecommerce companies, the concern is not limited to server load. An agent can affect who controls product discovery, who sees ads, how prices are compared, whether affiliate and marketplace economics are bypassed, how credentials are handled, and who owns the customer relationship. If the agent becomes the shopper’s interface, the store risks becoming a supplier behind someone else’s assistant.
Robots.txt cannot carry the whole burden
The Robots Exclusion Protocol still matters, but it was not built to solve every AI-agent problem. RFC 9309 says the protocol is not a substitute for valid content security measures. It lets sites publish crawl preferences, but it does not authenticate the crawler, prove intent, enforce payment or decide whether a human-directed agent counts as a crawler.
OpenAI’s crawler documentation shows why the distinction has become harder. It separates OAI-SearchBot for search, GPTBot for training and ChatGPT-User for certain user-triggered actions in ChatGPT and Custom GPTs. OpenAI says ChatGPT-User is not used for automatic web crawling and that, because these actions are initiated by a user, robots.txt rules may not apply.
That is exactly the difficult category. A site may view the visit as automation. The AI company may view it as a person using a tool. The user may view it as a browser assistant. A single page request can sit inside three competing interpretations.
For background on the older protocol, Tech Help Canada’s robots.txt guide explains how crawler directives work.
New protocols are trying to create a front door for agents
The industry is responding with proposed standards and commercial gateways rather than one universal rule.
Sierra and Meta announced the Personal Agent Protocol on October 6, 2026, with partners including Genesys, Instinct, Rocket, Shopify, Stripe and Walmart. Sierra described it as an open standard for how personal agents interact with businesses. A later Sierra draft, known as Poppy, described a standard file at /.well-known/poppy.json, agent sessions, OAuth sign-in, scoped tokens and ways for companies to route agents through websites, APIs or company-run agents.
That approach gives each party something it wants. The person can grant permission. The agent can connect through a predictable route. The company can know when it is dealing with an agent, what the agent is allowed to do and which channels are available.
The IETF AI Preferences Working Group is also active, with drafts aimed at expressing preferences about how content is collected and processed for AI model development, deployment and use. These efforts are still developing, and none has settled every question around ecommerce, login sessions, payments, ads, training, summaries or agent identity.
What this changes for businesses
For businesses, the practical question is no longer whether AI bots should be blocked. The decision is becoming more specific.
Search visibility: sites may need to keep search crawlers open while refusing model training or limiting AI summaries.
Training use: content owners are gaining more ways to express or enforce whether content can be used for AI training.
Live agent access: retailers, booking platforms and account-based services must decide whether user-directed agents can browse, log in, transact or trigger support workflows.
Monetization: publishers and data-rich sites may push toward licensing, pay-per-crawl, pay-per-use or direct protocol partnerships.
Security and liability: agents handling credentials, screenshots, account data or purchases introduce risks that ordinary crawlers did not create.
This also complicates AI visibility strategy. A site that blocks too much may disappear from AI-assisted discovery. A site that allows everything may lose control over how its content, inventory or customer experience is used. The stronger approach is likely to be selective: allow beneficial discovery, restrict unwanted training, define rules for agents and monitor what actually reaches the server.
The web is becoming negotiable
The web is not closing to automation wholesale. Permission may come through crawler settings, signed bot identity, OAuth, APIs, merchant partnerships, paid access systems, court rulings or new standards. It may also be denied by a WAF rule, a terms-of-service notice, a CAPTCHA, a 403 response or a retailer deciding that an outside shopping agent threatens its business model.
AI adoption will not be determined only by model quality. It will also depend on whether agents can get permission to act inside the websites people rely on.
Get new small business insights by email
Practical ideas and useful articles to help you make better business decisions.
Tech Help Canada Staff researches, writes, and reviews practical content for business owners and professionals. Our coverage spans business, marketing, SEO, technology, and the tools and systems people use to grow and operate online. We focus on clear, useful information backed by research, hands-on experience, and editorial review. Learn more about our team and editorial standards. Need help with something? Contact Us
