dark-visitors
56d03d46fb
Update from Dark Visitors
2025-09-26 00:54:30 +00:00
ai.robots.txt
38d60b928c
Merge pull request #179 from ai-robots-txt/deepseekbot
...
chore(robots.json): add DeepSeekBot
2025-09-25 10:17:38 +00:00
Glyn Normington
e2266bbc1d
Merge pull request #179 from ai-robots-txt/deepseekbot
...
chore(robots.json): add DeepSeekBot
2025-09-25 11:17:23 +01:00
Cory Dransfeldt
bf347bdf91
chore(robots.json): add DeepSeekBot
2025-09-24 09:42:01 -07:00
nisbet-hubbard
c6e7d69dd5
Update README.md
2025-09-24 09:39:21 -07:00
ai.robots.txt
8906f6b447
Update from Dark Visitors
2025-09-12 00:53:06 +00:00
dark-visitors
2fd93029ca
Update from Dark Visitors
2025-09-11 00:55:06 +00:00
ai.robots.txt
b6338ddc73
Merge pull request #176 from emersion/TerraCotta
...
Add TerraCotta
2025-09-10 14:11:16 +00:00
Cory Dransfeldt
7ffbf33baf
Merge pull request #176 from emersion/TerraCotta
...
Add TerraCotta
2025-09-10 07:11:04 -07:00
Simon Ser
50870ba911
Add TerraCotta
...
Example server log:
X.X.X.X - - [10/Sep/2025:00:06:17 +0000] "GET /archives/dri-devel/2023-November/430875.html HTTP/1.1" 200 7388 "-" "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta "
2025-09-10 12:04:08 +02:00
dark-visitors
dd391bf960
Update from Dark Visitors
2025-09-10 00:53:56 +00:00
ai.robots.txt
229d1b4dbc
Merge pull request #174 from karolyi/master
...
Update Brightbot operator and details; add meta-webindexer entry
2025-09-09 02:45:32 +00:00
Glyn Normington
4d506ca322
Merge pull request #174 from karolyi/master
...
Update Brightbot operator and details; add meta-webindexer entry
2025-09-09 03:45:23 +01:00
László Károlyi
ec508ab434
Update Brightbot operator and details; add meta-webindexer entry
...
- Update Brightbot operator to https://brightdata.com/brightbot .
- Change Brightbot frequency to "At least one per minute."
- Expand Brightbot description with disguise tactics link.
- Add new entry for meta-webindexer under Meta operator.
- Set meta-webindexer respect to "Unclear at this time."
- Define meta-webindexer function as "AI Assistants."
- Set meta-webindexer frequency to "Unhinged, more than 1 per second."
- Include meta-webindexer description on improving Meta AI search.
2025-09-08 21:42:06 +02:00
dark-visitors
0ed29412c9
Update from Dark Visitors
2025-08-28 00:55:47 +00:00
ai.robots.txt
2ad1c3e831
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
...
chore: add Google-Firebase and ShapBot
2025-08-27 10:13:17 +00:00
Glyn Normington
1677278c5a
Merge pull request #168 from ai-robots-txt/google-firebase-shapbot
...
chore: add Google-Firebase and ShapBot
2025-08-27 11:13:04 +01:00
Cory Dransfeldt
1a8edfa84a
chore: add Google-Firebase and ShapBot
2025-08-26 19:39:42 -07:00
dark-visitors
cf073d49f2
Update from Dark Visitors
2025-08-15 01:01:42 +00:00
ai.robots.txt
6d3f3e1712
Merge pull request #167 from ai-robots-txt/OpenAI-bot
...
chore: adds OpenAI but user agent
2025-08-14 01:45:05 +00:00
Glyn Normington
784b8440a5
Merge pull request #167 from ai-robots-txt/OpenAI-bot
...
chore: adds OpenAI but user agent
2025-08-14 02:44:53 +01:00
Cory Dransfeldt
0e687a5b58
chore: adds OpenAI but user agent
2025-08-13 09:01:55 -07:00
ai.robots.txt
ff9fc26404
Update from Dark Visitors
2025-08-01 01:12:53 +00:00
dark-visitors
146a229662
Update from Dark Visitors
2025-07-31 01:05:21 +00:00
ai.robots.txt
64f9d6ce9c
Update from Dark Visitors
2025-07-30 01:05:21 +00:00
dark-visitors
085dd1071e
Update from Dark Visitors
2025-07-29 01:11:26 +00:00
ai.robots.txt
9565c11d4c
Merge pull request #166 from ai-robots-txt/summaly-bot
...
fix: remove summaly
2025-07-28 19:19:15 +00:00
Glyn Normington
8869442615
Merge pull request #166 from ai-robots-txt/summaly-bot
...
fix: remove summaly
2025-07-28 20:19:06 +01:00
ai.robots.txt
27420d6fed
Merge pull request #164 from GlitzSmarter/main
...
Adding YaK
2025-07-28 18:29:57 +00:00
Glyn Normington
44e58a5ece
Merge pull request #164 from GlitzSmarter/main
...
Adding YaK
2025-07-28 19:29:48 +01:00
Glyn Normington
12c5368e04
Fix syntax
2025-07-28 19:29:19 +01:00
Glitz Smarter
c15065544a
Rewriting function in YaK
...
Rewrote the function to make it clear the quote is according to the page on the Clearwater website.
2025-07-28 12:54:25 -04:00
Cory Dransfeldt
9171625db6
fix: remove summaly
2025-07-28 08:56:21 -07:00
Gregory
8b188d0612
Added YaK user-agent to list of robots
...
Added Meltwater's AI to list of bots that use AI and scrape websites.
2025-07-26 23:50:11 -04:00
ai.robots.txt
47769c8429
Update from Dark Visitors
2025-07-17 01:04:20 +00:00
dark-visitors
9843c31303
Update from Dark Visitors
2025-07-16 01:03:43 +00:00
ai.robots.txt
567e94a6cf
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
...
Simplify htaccess rewrite rule
2025-07-08 16:00:16 +00:00
Glyn Normington
30c4c037d9
Merge pull request #160 from sdavids/users/sdavids/feat/htaccess
...
Simplify htaccess rewrite rule
2025-07-08 16:59:52 +01:00
Sebastian Davids
c7407e721d
Simplify htaccess rewrite rule
...
https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_f
fix #159
Signed-off-by: Sebastian Davids <sdavids@gmx.de >
2025-07-08 12:33:24 +02:00
dark-visitors
f6f504012f
Update from Dark Visitors
2025-07-08 01:01:25 +00:00
ai.robots.txt
63f1f56307
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
...
chore(robots.json): add SummalyBot + Thinkbot
2025-07-07 16:55:31 +00:00
Glyn Normington
91a5b3c995
Merge pull request #158 from ai-robots-txt/thinkbot-summaly
...
chore(robots.json): add SummalyBot + Thinkbot
2025-07-07 17:55:22 +01:00
Cory Dransfeldt
c5dd4e98b6
chore(robots.json): add SummalyBot + Thinkbot
2025-07-07 09:49:26 -07:00
dark-visitors
70ff0c9fbd
Update from Dark Visitors
2025-07-03 01:00:49 +00:00
ai.robots.txt
20a242d390
Merge pull request #156 from nisbet-hubbard/patch-9
...
Update funtion and description for SemrushBot
2025-07-02 15:58:19 +00:00
Cory Dransfeldt
91ad302d68
Merge pull request #156 from nisbet-hubbard/patch-9
...
Update funtion and description for SemrushBot
2025-07-02 08:58:08 -07:00
dark-visitors
79bc22523f
Update from Dark Visitors
2025-07-02 01:01:05 +00:00
ai.robots.txt
2bfcee794d
Update from Dark Visitors
2025-06-29 01:07:47 +00:00
dark-visitors
7eb2099d3f
Update from Dark Visitors
2025-06-28 00:58:57 +00:00
nisbet-hubbard
accc48f327
Update funtion and description for SemrushBot
2025-06-21 08:48:58 +08:00
