ai.robots.txt/table-of-bot-metrics.md

Name	Operator	Respects robots.txt	Data use	Visit regularity	Description
AddSearchBot	Unclear at this time.	Unclear at this time.	AI Search Crawlers	Unclear at this time.	AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results. More info can be found at https://darkvisitors.com/agents/agents/addsearchbot
AI2Bot	Ai2	Yes	Content is used to train open language models.	No information provided.	Explores 'certain domains' to find web content.
Ai2Bot-Dolma	Ai2	Yes	Content is used to train open language models.	No information provided.	Explores 'certain domains' to find web content.
aiHitBot	aiHit	Yes	A massive, artificial intelligence/machine learning, automated system.	No information provided.	Scrapes data for AI systems.
Amazonbot	Amazon	Yes	Service improvement and enabling answers for Alexa users.	No information provided.	Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.
Andibot	Andi	Unclear at this time	Search engine using generative AI, AI Search Assistant	No information provided.	Scrapes website and provides AI summary.
anthropic-ai	Anthropic	Unclear at this time.	Scrapes data to train Anthropic's AI products.	No information provided.	Scrapes data to train LLMs and AI products offered by Anthropic.
Applebot	Unclear at this time.	Unclear at this time.	AI Search Crawlers	Unclear at this time.	Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain references to the website. More info can be found at https://darkvisitors.com/agents/agents/applebot
Applebot-Extended	Apple	Yes	Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.	Unclear at this time.	Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
Awario	Awario	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Awario is an AI data scraper operated by Awario. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/awario
bedrockbot	Amazon	Yes	Data scraping for custom AI applications.	Unclear at this time.	Connects to and crawls URLs that have been selected for use in a user's AWS bedrock application.
bigsur.ai	Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for businesses	Unclear at this time.	AI Assistants	Unclear at this time.	bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for businesses. More info can be found at https://darkvisitors.com/agents/agents/bigsur-ai
Brightbot 1.0	https://brightdata.com/brightbot	Unclear at this time.	LLM/AI training.	At least one per minute.	Scrapes data to train LLMs and AI products focused on website customer support, uses residential IPs and legit-looking user-agents to disguise itself.
Bytespider	ByteDance	No	LLM training.	Unclear at this time.	Downloads data to train LLMS, including ChatGPT competitors.
CCBot	Common Crawl Foundation	Yes	Provides open crawl dataset, used for many purposes, including Machine Learning/AI.	Monthly at present.	Web archive going back to 2008. Cited in thousands of research papers per year.
ChatGPT Agent	OpenAI	Yes	AI Agents	Unclear at this time.	ChatGPT Agent is an AI agent created by OpenAI that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user. More info can be found at https://darkvisitors.com/agents/agents/chatgpt-agent
ChatGPT-User	OpenAI	Yes	Takes action based on user prompts.	Only when prompted by a user.	Used by plugins in ChatGPT to answer queries based on user input.
Claude-SearchBot	Anthropic	Yes	Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.	No information provided.	Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.
Claude-User	Anthropic	Yes	Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.	No information provided.	Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.
Claude-Web	Anthropic	Unclear at this time.	Undocumented AI Agents	Unclear at this time.	Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can provide more detail, please contact us. More info can be found at https://darkvisitors.com/agents/agents/claude-web
ClaudeBot	Anthropic	Yes	Scrapes data to train Anthropic's AI products.	No information provided.	Scrapes data to train LLMs and AI products offered by Anthropic.
CloudVertexBot	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	CloudVertexBot is a Google-operated crawler available to site owners to request targeted crawls of their own sites for AI training purposes on the Vertex AI platform. More info can be found at https://darkvisitors.com/agents/agents/cloudvertexbot
cohere-ai	Cohere	Unclear at this time.	Retrieves data to provide responses to user-initiated prompts.	Takes action based on user prompts.	Retrieves data based on user prompts.
cohere-training-data-crawler	Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products	Unclear at this time.	AI Data Scrapers	Unclear at this time.	cohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products. More info can be found at https://darkvisitors.com/agents/agents/cohere-training-data-crawler
Cotoyogi	ROIS	Yes	AI LLM Scraper.	No information provided.	Scrapes data for AI training in Japanese language.
Crawlspace	Crawlspace	Yes	Scrapes data	Unclear at this time.	Provides crawling services for any purpose, probably including AI model training.
Datenbank Crawler	Datenbank	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Datenbank Crawler is an AI data scraper operated by Datenbank. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/datenbank-crawler
DeepSeekBot	DeepSeek	Unclear at this time.	Training language models and improving AI products	Unclear at this time.	DeepSeekBot is a web crawler used by DeepSeek to train its language models and improve its AI products.
Devin	Devin AI	Unclear at this time.	AI Assistants	Unclear at this time.	Devin is an AI assistant operated by Devin AI. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/devin
Diffbot	Diffbot	At the discretion of Diffbot users.	Aggregates structured web data for monitoring and AI model training.	Unclear at this time.	Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.
DuckAssistBot	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	DuckAssistBot is used by DuckDuckGo's DuckAssist feature to fetch content and generate realtime AI answers to user searches. More info can be found at https://darkvisitors.com/agents/agents/duckassistbot
Echobot Bot	Echobox	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Echobot Bot is an AI data scraper operated by Echobox. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/echobot-bot
EchoboxBot	Echobox	Unclear at this time.	Data collection to support AI-powered products.	Unclear at this time.	Supports company's AI-powered social and email management products.
FacebookBot	Meta/Facebook	Yes	Training language models	Up to 1 page per second	Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically.
facebookexternalhit	Meta/Facebook	No	Ostensibly only for sharing, but likely used as an AI crawler as well	Unclear at this time.	Note that excluding FacebookExternalHit will block incorporating OpenGraph data when sharing in social media, including rich links in Apple's Messages app. According to Meta, its purpose is "to crawl the content of an app or website that was shared on one of Meta’s family of apps…". However, see discussions here and here for evidence to the contrary.
Factset_spyderbot	Factset	Unclear at this time.	AI model training.	No information provided.	Scrapes data for AI training.
FirecrawlAgent	Firecrawl	Yes	AI scraper and LLM training	No information provided.	Scrapes data for AI systems and LLM training.
FriendlyCrawler	Unknown	Yes	We are using the data from the crawler to build datasets for machine learning experiments.	Unclear at this time.	Unclear who the operator is; but data is used for training/machine learning.
Gemini-Deep-Research	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	Gemini-Deep-Research is the agent responsible for collecting and scanning resources used in Google Gemini's Deep Research feature, which acts as a personal research assistant. More info can be found at https://darkvisitors.com/agents/agents/gemini-deep-research
Google-CloudVertexBot	Google	Yes	Build and manage AI models for businesses employing Vertex AI	No information.	Google-CloudVertexBot crawls sites on the site owners' request when building Vertex AI Agents.
Google-Extended	Google	Yes	LLM training.	No information.	Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.
Google-Firebase	Google	Unclear at this time.	Used as part of AI apps developed by users of Google's Firebase AI products.	Unclear at this time.	Supports Google's Firebase AI products.
GoogleAgent-Mariner	Google	Unclear at this time.	AI Agents	Unclear at this time.	GoogleAgent-Mariner is an AI agent created by Google that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user. More info can be found at https://darkvisitors.com/agents/agents/googleagent-mariner
GoogleOther	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GoogleOther-Image	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GoogleOther-Video	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GPTBot	OpenAI	Yes	Scrapes data to train OpenAI's products.	No information.	Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.
iaskspider/2.0	iAsk	No	Crawls sites to provide answers to user queries.	Unclear at this time.	Used to provide answers to user queries.
ICC-Crawler	NICT	Yes	Scrapes data to train and support AI technologies.	No information.	Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.
ImagesiftBot	ImageSift	Yes	ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products	No information.	Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.
img2dataset	img2dataset	Unclear at this time.	Scrapes images for use in LLMs.	At the discretion of img2dataset users.	Downloads large sets of images into datasets for LLM training or other purposes.
ISSCyberRiskCrawler	ISS-Corporate	No	Scrapes data to train machine learning models.	No information.	Used to train machine learning based models to quantify cyber risk.
Kangaroo Bot	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Kangaroo Bot is used by the company Kangaroo LLM to download data to train AI models tailored to Australian language and culture. More info can be found at https://darkvisitors.com/agents/agents/kangaroo-bot
LinerBot	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	LinerBot is the web crawler used by Liner AI assistant to gather information from academic sources and websites to provide accurate answers with line-by-line source citations for research and scholarly work. More info can be found at https://darkvisitors.com/agents/agents/linerbot
meta-externalagent	Meta	Yes	Used to train models and improve products.	No information.	"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly."
Meta-ExternalAgent	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Meta-ExternalAgent is a web crawler used by Meta to download training data for its AI models and improve its products by indexing content directly. More info can be found at https://darkvisitors.com/agents/agents/meta-externalagent
meta-externalfetcher	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher
Meta-ExternalFetcher	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher
meta-webindexer	Meta	Unclear at this time.	AI Assistants	Unhinged, more than 1 per second.	As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."
MistralAI-User	Mistral	Unclear at this time.	AI Assistants	Unclear at this time.	MistralAI-User is an AI assistant operated by Mistral. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/mistralai-user
MistralAI-User/1.0	Mistral AI	Yes	Takes action based on user prompts.	Only when prompted by a user.	MistralAI-User is for user actions in LeChat. When users ask LeChat a question, it may visit a web page to help answer and include a link to the source in its response.
MyCentralAIScraperBot	Unclear at this time.	Unclear at this time.	AI data scraper	Unclear at this time.	Operator and data use is unclear at this time.
netEstate Imprint Crawler	netEstate	Unclear at this time.	AI Data Scrapers	Unclear at this time.	netEstate Imprint Crawler is an AI data scraper operated by netEstate. If you think this is incorrect or can provide additional detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/netestate-imprint-crawler
NovaAct	Unclear at this time.	Unclear at this time.	AI Agents	Unclear at this time.	Nova Act is an AI agent created by Amazon that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user. More info can be found at https://darkvisitors.com/agents/agents/novaact
OAI-SearchBot	OpenAI	Yes	Search result generation.	No information.	Crawls sites to surface as results in SearchGPT.
omgili	Webz.io	Yes	Data is sold.	No information.	Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.
omgilibot	Webz.io	Yes	Data is sold.	No information.	Legacy user agent initially used for Omgili search engine. Unknown if still used, omgili agent still used by Webz.io.
OpenAI	OpenAI	Yes	Unclear at this time.	Unclear at this time.	The purpose of this bot is unclear at this time but it is a member of OpenAI's suite of crawlers.
Operator	Unclear at this time.	Unclear at this time.	AI Agents	Unclear at this time.	Operator is an AI agent created by OpenAI that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user. More info can be found at https://darkvisitors.com/agents/agents/operator
PanguBot	the Chinese company Huawei	Unclear at this time.	AI Data Scrapers	Unclear at this time.	PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu. More info can be found at https://darkvisitors.com/agents/agents/pangubot
Panscient	Panscient	Yes	Data collection and analysis using machine learning and AI.	The Panscient web crawler will request a page at most once every second from the same domain name or the same IP address.	Compiles data on businesses and business professionals that is structured using AI and machine learning.
panscient.com	Panscient	Yes	Data collection and analysis using machine learning and AI.	The Panscient web crawler will request a page at most once every second from the same domain name or the same IP address.	Compiles data on businesses and business professionals that is structured using AI and machine learning.
Perplexity-User	Perplexity	No	Used to answer queries at the request of users.	Only when prompted by a user.	Visit web pages to help provide an accurate answer and include links to the page in Perplexity response.
PerplexityBot	Perplexity	Yes	Search result generation.	No information.	Crawls sites to surface as results in Perplexity.
PetalBot	Huawei	Yes	Used to provide recommendations in Hauwei assistant and AI search services.	No explicit frequency provided.	Operated by Huawei to provide search and AI assistant services.
PhindBot	phind	Unclear at this time.	AI-enhanced search engine.	No explicit frequency provided.	Company offers an AI agent that uses AI and generate extra web query on the fly
Poseidon Research Crawler	Poseidon Research	Unclear at this time.	AI research crawler	No explicit frequency provided.	Lab focused on scaling the interpretability research necessary to make better AI systems possible.
QualifiedBot	Qualified	Unclear at this time.	Company offers AI agents and other related products; usage can be assumed to support said products.	No explicit frequency provided.	Operated by Qualified as part of their suite of AI product offerings.
QuillBot	Quillbot	Unclear at this time.	Company offers AI detection, writing tools and other services.	No explicit frequency provided.	Operated by QuillBot as part of their suite of AI product offerings.
quillbot.com	Quillbot	Unclear at this time.	Company offers AI detection, writing tools and other services.	No explicit frequency provided.	Operated by QuillBot as part of their suite of AI product offerings.
SBIntuitionsBot	SB Intuitions	Yes	Uses data gathered in AI development and information analysis.	No information.	AI development and information analysis
Scrapy	Zyte	Unclear at this time.	Scrapes data for a variety of uses including training AI.	No information.	"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets."
SemrushBot-OCOB	Semrush	Yes	Crawls your site for ContentShake AI tool.	Roughly once every 10 seconds.	Data collected is used for the ContentShake AI tool reports.
SemrushBot-SWA	Semrush	Yes	Checks URLs on your site for SEO Writing Assistant.	Roughly once every 10 seconds.	Data collected is used for the SEO Writing Assistant tool to check if URL is accessible.
ShapBot	Parallel	Yes	Collects data for Parallel's web APIs.	Unclear at this time.	ShapBot helps discover and index websites for Parallel's web APIs.
Sidetrade indexer bot	Sidetrade	Unclear at this time.	Extracts data for a variety of uses including training AI.	No information.	AI product training.
TerraCotta	Ceramic AI	Yes	AI Agents	Unclear at this time.	Downloads data to train LLMs.
Thinkbot	Thinkbot	No	Insights on AI integration and automation.	Unclear at this time.	Collects data for analysis on AI usage and automation.
TikTokSpider	ByteDance	Unclear at this time.	LLM training.	Unclear at this time.	Downloads data to train LLMS, as per Bytespider.
Timpibot	Timpi	Unclear at this time.	Scrapes data for use in training LLMs.	No information.	Makes data available for training AI models.
VelenPublicWebCrawler	Velen Crawler	Yes	Scrapes data for business data sets and machine learning models.	No information.	"Our goal with this crawler is to build business datasets and machine learning models to better understand the web."
WARDBot	WEBSPARK	Unclear at this time.	AI Data Scrapers	Unclear at this time.	WARDBot is an AI data scraper operated by WEBSPARK. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us. More info can be found at https://darkvisitors.com/agents/agents/wardbot
Webzio-Extended	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Webzio-Extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models. More info can be found at https://darkvisitors.com/agents/agents/webzio-extended
wpbot	QuantumCloud	Unclear at this time; opt out provided via Google Form	Live chat support and lead generation.	Unclear at this time.	wpbot is a used to support the functionality of the AI Chatbot for WordPress plugin. It supports the use of customer models, data collection and customer support.
YaK	Meltwater	Unclear at this time.	According to the Meltwater Consumer Intelligence page 'By applying AI, data science, and market research expertise to a live feed of global data sources, we transform unstructured data into actionable insights allowing better decision-making'.	Unclear at this time.	Retrieves data used for Meltwater's AI enabled consumer intelligence suite
YandexAdditional	Yandex	Yes	Scrapes/analyzes data for the YandexGPT LLM.	No information.	Retrieves data used for YandexGPT quick answers features.
YandexAdditionalBot	Yandex	Yes	Scrapes/analyzes data for the YandexGPT LLM.	No information.	Retrieves data used for YandexGPT quick answers features.
YouBot	You	Yes	Scrapes data for search engine and LLMs.	No information.	Retrieves data used for You.com web search engine and LLMs.