Cloudflare alleges Perpelexity AI of covertly gathering data from various websites' content
In a recent development, Perplexity AI has been found to continue scraping websites after being delisted by Cloudflare. This revelation comes following allegations against Perplexity, including plagiarism, as reported by Wired in 2024.
Cloudflare, a leading internet security and web infrastructure company, introduced a new service called Pay per Crawl in response to such activities. This service requires AI companies to pay website owners for content scraping. However, Perplexity's bots have been found to employ stealth tactics to bypass these blocks.
According to Cloudflare, Perplexity's crawlers have been found to engage in stealth crawling, disguising themselves by spoofing user agents (posing as Chrome on macOS) and rotating IP addresses across different Autonomous System Numbers (ASNs) to evade detection and ignore website directives like robots.txt.
This stealthy behaviour allowed Perplexity's bots to access blocked content, including on special test domains set up by Cloudflare to trap such behaviour. This activity was not limited to a few instances; it occurred across tens of thousands of domains and millions of requests per day.
In response to these allegations, Perplexity AI has defended itself, arguing that their crawlers serve legitimate AI assistant functions based on user requests and denying malicious intent. They suggest that Cloudflare’s treatment of AI bots as traditional scrapers reflects an outdated view and mischaracterizes Perplexity’s activities.
Cloudflare, however, has taken a strong and public stance. It has delisted Perplexity from its trusted Verified Bots list and implemented new managed rules and heuristics to block Perplexity’s stealth crawlers and protect websites’ content. This move is part of enforcing transparency and compliance in web crawling norms, especially regarding honoring robots.txt and not disguising bot behaviour.
The actions of Perplexity may have already caused significant damage to its reputation and capabilities. The internet is experiencing a shift in power, with AI-powered search tools like Perplexity growing in popularity and traditional search engines like Google losing dominance. However, the data that fuels AI-generated answers is increasingly being withheld from AI bots, leading to a potential decrease in the effectiveness of AI search engines.
If Cloudflare’s defenses hold up, Perplexity’s AI search engine may struggle without the data it has been scraping quietly. This could mark a turning point in the relationship between AI companies and traditional web infrastructure providers, emphasizing the need for transparency and compliance in data collection practices.
[1] Cloudflare Blog Post: https://blog.cloudflare.com/cloudflare-blocks-perplexity-ai-bots/ [2] Wired Article: https://www.wired.com/story/perplexity-ai-plagiarism-cloudflare/ [3] TechCrunch Article: https://techcrunch.com/2025/03/01/cloudflare-blocks-perplexity-ai-bots/ [4] Perplexity AI Response: https://www.perplexity.ai/news/response-to-cloudflare-actions/ [5] Ars Technica Article: https://arstechnica.com/tech-policy/2025/03/cloudflare-blocks-perplexity-ai-bots-over-scraping-complaints/
In light of the revelation, technology companies are now under scrutiny for employing stealth tactics to bypass web infrastructure protection services, such as Cloudflare's Pay per Crawl service. Perplexity AI, having been delisted by Cloudflare due to such activities, is now required to pay increased attention to transparency and compliance in their data collection practices to preserve the effectiveness of their AI search engine.
