Except for Google, users on Reddit have limited access to the majority of major search engines. But how is this feasible, and why? Reddit has blocked access to other search engines. Reddit has blocked the Bing search engine due to a new update to the robot.txt file, which Microsoft acknowledged on July 1.
Reddit CEO Steve Huffman does not feel that all web information, particularly their own, is free for AI models and search engines to scan and use. Source: Search Engine Land.
Why Reddit is blocking search engines and AI models
Reddit, a popular online community, has decided to block search engines like Microsoft and AI models from accessing its content. This means those search engines can’t look at Reddit posts unless they make a special deal with Reddit. This decision was announced by Reddit’s CEO, Steve Huffman.
Why is Reddit Doing This?
Blocking these search engines hasn’t been easy for Reddit. Steve Huffman explained to a tech news site, The Verge:
- Control Over Data: Without these deals, Reddit can’t control or even know how its data is used. This means they can’t ensure their data is used properly.
- Big Companies: Companies like Microsoft, Anthropic, and Perplexity seem to think they can use any content on the internet for free. Reddit wants to make sure their data isn’t just taken without permission.
- Changing Times: The way search engines work is changing. Now, they summarize and use data for training AI, which mixes up the value exchange. Previously, search engines would send traffic back to sites they crawled, but now it’s not so clear.
What Does Microsoft Think?
Microsoft’s AI CEO, Mustafa Suleyman, mentioned that web content has been seen as “freeware,” meaning anyone can copy and use it. He said:
- Fair Use: Since the 1990s, the understanding has been that online content is fair game for anyone to use, like freeware.
Google is Not Blocked
Interestingly, Reddit hasn’t blocked Google. That’s because Google made a deal with Reddit, paying them $60 million a year for the content. This deal was announced in February.
Microsoft’s Response
When Reddit started blocking search engines, a Microsoft spokesperson said:
- Respecting Rules: Microsoft respects the robots.txt standard, which tells search engines what they can and can’t access. They stopped crawling Reddit after the new rules were set on July 1, which blocked all crawling.
Why This Matters
Reddit is in a strong position because of its deal with Google. It also gets a lot of attention and traffic from being prominent in Google Search results. But other content creators might still need the visibility and traffic from AI search and answer engines. They might need to use special strategies called generative engine optimization (GEO) to get noticed.
Fun Statistics
- $60 million: The amount Google pays Reddit annually for access to its content.
- 100% Block: Reddit fully blocked Microsoft and other search engines unless they make a deal.
- 1990s: Since the 1990s, online content has been considered fair use by many, like freeware.
- July 1: The date when Reddit’s new robots.txt rules started blocking all crawling.
What Can We Learn?
This situation shows how important it is for websites to control their own data. It also highlights how search engines and AI models are changing the way they use online content. As users, it’s interesting to see how these big companies interact and what rules they follow.
