AI bot traffic grew by 18% overall from May 2024 to May 2025, with some individual crawlers showing explosive growth rates exceeding 300%. The math is brutal. GPTBot's share grew from 4.7% in July 2024 to 11.7% in July 2025, while ClaudeBot increased from 6% to nearly 10%, and Meta's crawler jumped from 0.9% to 7.5%. These aren't just numbers on a dashboard. They represent a fundamental shift in how content flows through the internet.
The Scale is Staggering
In July 2024, training accounted for 72% of AI crawling. By July 2025, it had risen to 79%. This isn't passive indexing like traditional search crawlers. Fetcher bots can hit websites with over 39,000 requests per minute, with AI crawlers returning every 6 hours rather than crawling once and moving on. At Fusion AI, we've watched clients' server logs transform from recognizable patterns to something resembling a distributed attack. One Dubai-based e-commerce client saw AI crawler requests jump 400% in six months.
The extraction-to-referral ratio tells the real story. For every visitor Anthropic refers back to a website, its crawlers have already visited tens of thousands of pages, highlighting the imbalance between how much content AI systems consume and how little traffic they return. Some platforms show ratios as extreme as 70,900:1, with Anthropic (Claude) showing 50,000:1 to 70,900:1 crawl-to-refer ratios.
The Economic Threat
AI Overviews now appear for 13.14% of all queries, more than doubling from 6.49% in January 2025. When AI Overviews are present, click-through rates plummet to just 8%, compared to 15% for traditional search results. Google search traffic to publishers declined globally by a third in the year to November, with referrals from Google Discover down 21% year on year.
Major publishers are feeling the squeeze. CNN suffered traffic declining between 27% and 38% year-over-year, while educational platform Chegg reported a 49% decline in non-subscriber traffic between January 2024 and January 2025. The traditional web economy - create content, attract traffic, monetize eyeballs - is breaking down as AI systems extract value without proportional compensation.
The Bot Blocking Response
Publishers are fighting back. 79% of top news sites block AI training bots via robots.txt, with 71% also blocking AI retrieval bots, and PerplexityBot blocked by 67%. About 5.6 million websites have added OpenAI's GPTBot to their disallow list, up from 3.3 million at the start of July 2025 - an increase of almost 70 percent.
But robots.txt is a gentleman's agreement, not law. Across all AI bots, 13.26% of requests ignored robots.txt directives in Q2 2025, up from 3.3% in Q4 2024. Robots.txt only serves as "a leakage point," with bots able to scrape content indirectly or bypass robots.txt entirely. Perplexity is alleged to ignore robots.txt altogether.
What GCC Businesses Need to Do
The approach depends on your business model. Content publishers monetizing through advertising face an existential crisis. E-commerce and service businesses have different calculations. At Fusion AI, we're advising Dubai and GCC clients to implement layered defenses: robots.txt for compliant crawlers, rate limiting for aggressive ones, and server-level blocks for bad actors.
For businesses that want discovery, the strategy shifts from blocking to optimization for AI citation. 92.36% of AI Overview citations come from domains ranking in the top 10, with brands like Bank of America achieving 32.2% visibility across AI platforms in banking queries. This isn't traditional SEO. It's about becoming the authoritative source that AI systems cite.
The Infrastructure Reality
One site owner reported Meta/Facebook crawlers hitting over 300,000 times in a single day, effectively becoming a DOS attack, with crawlers potentially going around in interminable loops. This isn't theoretical infrastructure stress. It's measurable server costs and potential downtime.
From Fusion AI's perspective working with enterprises across the UAE, the question isn't whether AI crawlers will impact your business. They already are. The question is whether you'll adapt your content strategy, infrastructure, and business model to the new reality or watch your traffic evaporate as AI systems harvest your content without compensation.
The Path Forward
The data shows this isn't a temporary disruption. Publishers who blocked AI crawlers via robots.txt saw total traffic reduced by 23% and human traffic by 14% for large publishers. The old web is dying. The new one rewards direct relationships, premium experiences, and content that can't be summarized in a paragraph.
Every CTO in DIFC should audit their crawler traffic this quarter. Every content strategist needs an AI visibility plan. Every business model built on search traffic needs a backup plan. The crawlers aren't slowing down. The question is whether you'll be ready when they arrive at your door.