TheMarketshop AI Blog

Published August 22, 2026

The Realities of Using AI for Research

When you hand a research task to an AI, it helps to know what that AI can actually reach. The web an AI sees is not the same web you see, and the gap between the two is getting wider.

A lot of websites block AI

A growing number of sites block AI crawlers, particularly the ones that collect training data for future models.

News organizations have been the most aggressive about this, and the pattern is lopsided in a way that should give you pause. In a study titled "Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web", Nicolas Steinacker-Olsztyn of Saarland University, Devashish Gosain of IIT Bombay, and Ha Dao of the Max Planck Institute for Informatics examined more than 4,000 sites and found that 60% of reputable news outlets disallowed at least one AI crawler in their robots.txt file, compared to just 9.1% of misinformation sites. The average reputable news site blocked 15.5 different AI user agents. The average misinformation site blocked fewer than one. Over 80% of the misinformation sites in the study did not block a single AI crawler. The paper was presented at the ACM Web Conference in 2026.

The gap is also widening. The same researchers tracked archived snapshots and found blocking by reputable sites climbed from 23% in September 2023 to nearly 60% by May 2025. A separate NewsGuard analysis found that 67% of the news sites it rates as top quality restrict AI access to their journalism.

That creates a problem: the easiest websites for an AI to access are not necessarily the best websites.

At the same time, AI-generated content keeps piling up

Estimates vary a lot depending on what you count and how you measure it.

The SEO firm Graphite sampled 55,400 English-language articles from Common Crawl and ran them through three separate AI detectors. Their finding: articles that are primarily AI-generated made up about half of new published articles in early 2026, having risen sharply after ChatGPT launched and then plateaued near 50% since early 2025. Ahrefs took a different cut, analyzing 900,000 newly created pages, and found that 74.2% contained at least some AI-generated content, though only 2.5% looked purely machine-written with no human editing.

NOTE:  these are industry studies rather than peer-reviewed research, AI detectors have real false positive rates, and the human/AI line has gone blurry now that people routinely use AI to outline, draft, and edit work that they then rewrite. What the studies agree on is the direction: a large and growing share of new web content has a machine somewhere in its production chain.

That raises the possibility of a feedback loop. AI-generated information gets published, future AI systems encounter it, and they produce more content shaped partly by earlier AI output.

Researchers have shown this can go badly. In a 2024 paper in Nature, Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal (working across Oxford, Cambridge, Imperial College London, and the University of Toronto) trained successive generations of models on the previous generation's output and watched quality degrade into nonsense. They named the phenomenon model collapse.

But the conditions in that experiment were deliberately extreme. Follow-up work has found that keeping a portion of real human data in the mix, or accumulating data instead of replacing it, sharply reduces the effect. In May 2026, Fariba Jangjoo (NTNU), Matteo Marsili (ICTP Trieste), and Yasser Roudi (King's College London) published a paper in Physical Review Letters showing that in a class of simpler statistical models, adding even a single data point from outside the closed loop was enough to prevent collapse. A preprint version is on arXiv if you want to read it without a subscription.

So model collapse is a real and documented failure mode. However, there is not good evidence that today's major AI systems are experiencing it at any meaningful scale.

AI search is different from AI training

Here is a distinction that clears up a lot of confusion: when you ask ChatGPT to research something, it usually isn't crawling websites one by one.

The process is closer to this:

Question → Web search → Relevant results → Selected pages → AI answer

An AI can use search providers to discover pages even when its own training crawler was never allowed to index the site.

A website can also block a training crawler while still allowing a search crawler, and the big AI companies deliberately separate the two. Per OpenAI's own crawler documentation, it runs GPTBot for training data, OAI-SearchBot to surface pages in ChatGPT's search features, and ChatGPT-User for live fetches that a person triggers directly. Each one can be allowed or blocked independently in robots.txt, and plenty of publishers now block the training bot while keeping the search bot open.

There's a wrinkle worth knowing about. In December 2025, OpenAI revised that documentation and removed the robots.txt compliance language for ChatGPT-User, on the reasoning that a fetch you personally asked for isn't automated crawling. So the three bots don't all play by the same rules.

The practical takeaway: blocking AI does not necessarily mean disappearing from AI search results. It may, however, keep the AI from reading and analyzing the full page, leaving it working from a title, a URL, and a search snippet.

NOTE: robots.txt is a sign, not a lock. It's a text file asking bots to stay out, with nothing enforcing it. Any AI company could ignore it entirely and there'd be no technical barrier. What keeps them mostly honest is lawsuits, licensing deals they don't want to blow up, and the fact that server logs make cheating checkable.

AI agents create another complication

Things get more interesting with agents.

Some agents run a browser on the AI company's servers. Those are relatively easy for a website to identify, since they arrive from known data center IP ranges and often announce themselves.

Other agents drive your existing browser. In that case the website might see:

  • Your normal IP address
  • Your Chrome browser
  • Your cookies
  • Your existing login
  • Your normal computer

From the website's perspective, that can look a lot like you personally browsing.

This is actively being worked on. Cloudflare proposed a standard called Web Bot Auth that lets an agent cryptographically sign each request to prove which operator it belongs to, and an IETF working group was chartered around it in early 2026. Major AI companies, CDNs, and browser infrastructure providers have adopted it. But adoption is still uneven, and a signed request tells the site which agent is calling, not which human is behind it.

Absent something like that, websites fall back on detecting automation. They look at browser automation signals, browsing speed, mouse behavior, scrolling patterns, how quickly pages open, and whether the activity looks unusually repetitive.

If an agent opens 100 pages in three minutes and clicks everything instantly, it's obvious that software is involved. If an agent moves slowly through your normal Chrome browser using your existing login and your home internet connection, telling that apart from a human gets much harder.

So is this actually a problem?

I think there are two things worth watching.

First, the access asymmetry. High-quality human-created information is increasingly sitting behind paywalls, licensing agreements, or crawler restrictions. Cheap AI-generated content is trivial to produce and usually free to access. That points toward an internet where the easiest information for AI systems to reach is not the best information available.

Second, "bot traffic" is losing its meaning. Websites used to block crawlers coming from known servers. Increasingly, the thing on the other end is software operating a real person's browser, on that person's behalf, from that person's machine. The distinction is shifting away from human versus AI and toward a human driving the browser directly versus software driving it for them. That makes both AI web access and bot detection considerably messier.

What this means when you use AI for research

Keep the access gap in mind. An AI can give you a genuinely useful overview and help you find sources fast, but don't assume it reached everything worth reading. Some of the strongest sources are behind paywalls, in subscription databases, or on sites that block AI outright, and the AI has no way to tell you what it never saw.

For anything that matters, still do some searching yourself and compare what you find against what the AI handed you. If you're a student doing academic work, that goes double: your library's databases hold a large body of peer-reviewed material that no consumer AI tool is crawling.

The AI is a good research assistant. It is not a complete one.

Disclosure: AI was used to assist with editing and refining this article. The ideas and final content are my own.

Sources