Reddit is blocking the Bing search engine because of a new upgrade to the robot.txt file, which Microsoft acknowledged on July 1. The upgrade upholds the platform’s policy prohibiting site scraping. Moreover, Reddit has prohibited AI chatbots and outside search engines from accessing its content for free as part of the revised policy. This shift has had significant ramifications for users of search engines other than Google.
“We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” said a Microsoft spokesperson as quoted by Search Engine Land.
As per Search Engine Land, Reddit upgraded the robots.txt file on the first of July and began prohibiting several search engines and AI bots from crawling material on the network. According to the study, the revised robots.txt file monitored IP to display multiple versions of the robots.txt file to search engines and humans.
Furthermore, Google reportedly paid Reddit $60 million for access to its data, allowing the company to continue providing updated Reddit results. In contrast, competing search engines, including Bing, DuckDuckGo, Mojeek, and Qwant, are no longer able to display recent Reddit material in their search results. While previous Reddit content remains visible in these search engines, they are unable to crawl the site for new posts and comments due to new restrictions.
Microsoft confirmed that Bing’s ability to crawl Reddit has been disabled. This illustrates the wider impact of Reddit’s decision on search engines besides Google.
Reddit’s policy change is part of a broader crackdown on web scraping, which involves using bots to gather data from websites. The firm has implemented these limits to prevent unauthorised usage of its data for AI model training.
Meanwhile, Reddit’s chief legal officer, Ben Lee, noted that the new policy is designed to send a message to companies who do not have agreements with Reddit that they should not access its data. This strategy compels search engines to agree to sponsored partnerships to continue indexing recent Reddit content.
In addition, Reddit has stated that the modifications are not related to its recent collaboration with Google.
Comments
0 comment