The Agentic Web Index: AI Bot Traffic Statistics | Known Agents

Overview Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking. Bot vs. Human Traffic 35% ↓ 2% Compared to the previous 90 days The amount of visits from bots vs. humans Agentrification 28% ↑ 11% Compared to the previous 90 days The percentage of bot traffic that's AI-related AI Chat Referral Volume 0.1% ↓ 9% Compared to the previous 90 days The percentage of human website visits that come from AI chat See Referral Trends ↓ Robots.txt Effectiveness 98.5% How often bots follow robots.txt rules See Per Agent ↓ Traffic by Agent Type AI Agent AI AgentUses an actual web browser to autonomously complete complex tasks on behalf of a human user AI Assistant AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer AI Coding Agent AI Coding AgentFetches documentation and other resources to help build software AI Data Provider AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service AI Data Scraper AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs AI Search Crawler AI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results Archiver ArchiverCaptures and stores historical website snapshots for long-term digital preservation Automated Agent Automated AgentAutomates browser interactions programmatically without direct human supervision Developer Helper Developer HelperAssists with testing, debugging, and ensuring website functionality Fetcher FetcherRetrieves web page metadata to power app features like link previews or feeds Intelligence Gatherer Intelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targeting Scraper ScraperExtracts large amounts of web data, often without explicit website permission Search Engine Crawler Search Engine CrawlerSystematically scans and indexes web pages to include in search results Security Scanner Security ScannerScans websites for security vulnerabilities, threats, and configuration weaknesses SEO Crawler SEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunities Uncategorized UncategorizedNot yet assigned a type Undocumented AI Agent Undocumented AI AgentCrawls websites without disclosing its purpose, collecting data for an unknown AI use case Hover over each agent type for more information about what they do Top Agent Types Agent types with the most activity Top Visiting Agents bingbotSRCH Search Engine CrawlerSystematically scans and indexes web pages to include in search results 8.2% GooglebotSRCH Search Engine CrawlerSystematically scans and indexes web pages to include in search results 7.9% AhrefsBotSEO SEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunities 6.2% Known AgentDEV Developer HelperAssists with testing, debugging, and ensuring website functionality 5.5% ChatGPT-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 3.3% ClaudeBotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 3.2% PetalBotSRCH AI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results 3.2% SemrushBotSEO SEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunities 3.0% facebookexternalhitFTCH FetcherRetrieves web page metadata to power app features like link previews or feeds 2.6% meta-externalagentSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 2.3% AmazonbotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 2.3% MJ12botSEO SEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunities 2.2% Amzn-SearchBotSRCH AI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results 2.2% DotBotSEO SEO CrawlerAnalyzes website structure and content to identify SEO improvement opportunities 2.1% ApplebotSRCH AI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results 2.1% Agents with the most activity Top Operators Operators with the most activity AI Scraping Activity These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Scrapers and AI Data Providers. Traffic by Agent Type AI Data Provider AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service AI Data Scraper AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs AI scraping activity by agent type over time Top Visited Website Categories Law and Government 6.0% Online Communities 5.9% Home and Garden 5.5% Computers and Electronics 5.3% Shopping 5.2% Jobs and Education 5.2% Pets and Animals 5.1% Business and Industrial 5.0% Autos and Vehicles 4.8% Games 4.8% Hobbies and Leisure 4.6% Internet and Telecom 4.5% Beauty and Fitness 4.1% Website categories with most activity Top Agents ClaudeBotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 27.0% meta-externalagentSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 19.5% AmazonbotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 19.2% GPTBotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 9.4% BytespiderSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 7.3% ShapBotPVDR AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service 5.6% GoogleOtherSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 3.9% CCBotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 2.3% TimpibotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 1.8% YouBotPVDR AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service 1.4% VelenPublicWebCrawlerSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 0.8% DiffbotPVDR AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service 0.5% DeepSeekBotSCRP AI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs 0.4% TerraCottaPVDR AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service 0.3% AIWebIndexPVDR AI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service 0.2% Agents doing the most AI scraping Top Operators Operators doing the most AI scraping AI Fetching Activity These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Coding Agents and AI Assistants. Traffic by Agent Type AI Assistant AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer AI Coding Agent AI Coding AgentFetches documentation and other resources to help build software AI fetching activity by agent type over time Top Visited Website Categories Reference 4.0% Science 3.7% Finance 3.4% Law and Government 2.6% Computers and Electronics 2.3% Beauty and Fitness 2.1% Travel and Transportation 1.9% Business and Industrial 1.8% Internet and Telecom 1.7% Health 1.6% Autos and Vehicles 1.3% Sports 1.3% Real Estate 1.3% Website categories with most activity Top Agents ChatGPT-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 86.4% DuckAssistBotASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 3.3% Perplexity-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 2.7% Claude-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 2.5% MistralAI-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 2.1% Claude-CodeCODE AI Coding AgentFetches documentation and other resources to help build software 1.1% Shap-UserASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 1.0% Google-NotebookLMASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 0.4% CursorCODE AI Coding AgentFetches documentation and other resources to help build software 0.4% Gemini-Deep-ResearchASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 0.0% meta-externalfetcherASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 0.0% GoogleAgent-URLContextASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 0.0% CodeCODE AI Coding AgentFetches documentation and other resources to help build software 0.0% kagi-fetcherASST AI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer 0.0% QualifiedBotASST