Commit Graph

Select branches

Hide Pull Requests

MyCentralAIScraperBot

OpenAI-bot

bing-ai-opt-out-instructions

crawler-policy-update

google-cloudvertexbot

main

meta-externalagent

missing-dependency

mistral-bot

qualified-bot

semrush-bots

thinkbot-summaly

workflow-debug

workflow-fix

wpbot

#1

#10

#100

#100

#102

#105

#107

#109

#109

#110

#110

#112

#113

#114

#114

#115

#116

#118

#118

#119

#119

#121

#121

#122

#122

#123

#123

#125

#126

#126

#128

#128

#129

#129

#130

#130

#131

#131

#132

#132

#133

#134

#134

#136

#138

#139

#139

#14

#140

#141

#142

#143

#145

#148

#15

#150

#150

#151

#151

#152

#152

#153

#153

#154

#154

#156

#156

#158

#158

#16

#160

#164

#164

#166

#167

#167

#168

#17

#173

#173

#174

#174

#176

#176

#177

#179

#18

#2

#21

#22

#24

#25

#27

#28

#29

#31

#32

#33

#35

#36

#37

#38

#39

#41

#42

#43

#45

#46

#47

#49

#50

#51

#54

#55

#56

#57

#58

#6

#61

#62

#63

#64

#65

#66

#67

#68

#71

#73

#79

#8

#80

#83

#84

#86

#91

#91

#92

#92

#94

#94

#95

#95

#97

#98

#98

1.40

v1

v1.1

v1.10

v1.11

v1.12

v1.13

v1.14

v1.15

v1.16

v1.17

v1.18

v1.19

v1.2

v1.20

v1.21

v1.22

v1.23

v1.24

v1.25

v1.26

v1.27

v1.28

v1.29

v1.3

v1.30

v1.31

v1.32

v1.33

v1.34

v1.35

v1.36

v1.37

v1.38

v1.39

v1.4

v1.40

v1.41

v1.5

v1.6

v1.7

v1.8

v1.9

Mono Color

• 56d03d46fb Update from Dark Visitors main v1.41 dark-visitors 2025-09-26 00:54:30 +00:00
• 38d60b928c Merge pull request #179 from ai-robots-txt/deepseekbot ai.robots.txt 2025-09-25 10:17:38 +00:00
• e2266bbc1d Merge pull request #179 from ai-robots-txt/deepseekbot Glyn Normington 2025-09-25 11:17:23 +01:00
• bf347bdf91 chore(robots.json): add DeepSeekBot Cory Dransfeldt 2025-09-24 09:42:01 -07:00
• c6e7d69dd5 Update README.md nisbet-hubbard 2025-09-14 19:41:36 +08:00
• 8906f6b447 Update from Dark Visitors ai.robots.txt 2025-09-12 00:53:06 +00:00
• 2fd93029ca Update from Dark Visitors dark-visitors 2025-09-11 00:55:06 +00:00
• b6338ddc73 Merge pull request #176 from emersion/TerraCotta ai.robots.txt 2025-09-10 14:11:16 +00:00
• 7ffbf33baf Merge pull request #176 from emersion/TerraCotta Cory Dransfeldt 2025-09-10 07:11:04 -07:00
• 50870ba911 Add TerraCotta Simon Ser 2025-09-10 12:04:08 +02:00
• dd391bf960 Update from Dark Visitors dark-visitors 2025-09-10 00:53:56 +00:00
• 229d1b4dbc Merge pull request #174 from karolyi/master ai.robots.txt 2025-09-09 02:45:32 +00:00
• 4d506ca322 Merge pull request #174 from karolyi/master Glyn Normington 2025-09-09 03:45:23 +01:00
• ec508ab434 Update Brightbot operator and details; add meta-webindexer entry László Károlyi 2025-09-08 21:42:06 +02:00
• 0ed29412c9 Update from Dark Visitors v1.40 dark-visitors 2025-08-28 00:55:47 +00:00
• 2ad1c3e831 Merge pull request #168 from ai-robots-txt/google-firebase-shapbot 1.40 ai.robots.txt 2025-08-27 10:13:17 +00:00
• 1677278c5a Merge pull request #168 from ai-robots-txt/google-firebase-shapbot Glyn Normington 2025-08-27 11:13:04 +01:00
• 1a8edfa84a chore: add Google-Firebase and ShapBot Cory Dransfeldt 2025-08-26 19:39:42 -07:00
• cf073d49f2 Update from Dark Visitors dark-visitors 2025-08-15 01:01:42 +00:00
• 6d3f3e1712 Merge pull request #167 from ai-robots-txt/OpenAI-bot v1.39 ai.robots.txt 2025-08-14 01:45:05 +00:00
• 784b8440a5 Merge pull request #167 from ai-robots-txt/OpenAI-bot Glyn Normington 2025-08-14 02:44:53 +01:00
• 0e687a5b58 chore: adds OpenAI but user agent OpenAI-bot Cory Dransfeldt 2025-08-13 09:01:55 -07:00
• ff9fc26404 Update from Dark Visitors ai.robots.txt 2025-08-01 01:12:53 +00:00
• 146a229662 Update from Dark Visitors dark-visitors 2025-07-31 01:05:21 +00:00
• 64f9d6ce9c Update from Dark Visitors ai.robots.txt 2025-07-30 01:05:21 +00:00
• 085dd1071e Update from Dark Visitors dark-visitors 2025-07-29 01:11:26 +00:00
• 9565c11d4c Merge pull request #166 from ai-robots-txt/summaly-bot ai.robots.txt 2025-07-28 19:19:15 +00:00
• 8869442615 Merge pull request #166 from ai-robots-txt/summaly-bot Glyn Normington 2025-07-28 20:19:06 +01:00
• 27420d6fed Merge pull request #164 from GlitzSmarter/main ai.robots.txt 2025-07-28 18:29:57 +00:00
• 44e58a5ece Merge pull request #164 from GlitzSmarter/main Glyn Normington 2025-07-28 19:29:48 +01:00
• 12c5368e04 Fix syntax Glyn Normington 2025-07-28 19:29:19 +01:00
• c15065544a Rewriting function in YaK Glitz Smarter 2025-07-28 12:54:25 -04:00
• 9171625db6 fix: remove summaly Cory Dransfeldt 2025-07-28 08:56:21 -07:00
• 8b188d0612 Added YaK user-agent to list of robots Gregory 2025-07-26 23:50:11 -04:00
• 47769c8429 Update from Dark Visitors v1.38 ai.robots.txt 2025-07-17 01:04:20 +00:00
• 9843c31303 Update from Dark Visitors dark-visitors 2025-07-16 01:03:43 +00:00
• 567e94a6cf Merge pull request #160 from sdavids/users/sdavids/feat/htaccess ai.robots.txt 2025-07-08 16:00:16 +00:00
• 30c4c037d9 Merge pull request #160 from sdavids/users/sdavids/feat/htaccess Glyn Normington 2025-07-08 16:59:52 +01:00
• c7407e721d Simplify htaccess rewrite rule Sebastian Davids 2025-07-08 12:31:53 +02:00
• f6f504012f Update from Dark Visitors dark-visitors 2025-07-08 01:01:25 +00:00
• 63f1f56307 Merge pull request #158 from ai-robots-txt/thinkbot-summaly ai.robots.txt 2025-07-07 16:55:31 +00:00
• 91a5b3c995 Merge pull request #158 from ai-robots-txt/thinkbot-summaly Glyn Normington 2025-07-07 17:55:22 +01:00
• c5dd4e98b6 chore(robots.json): add SummalyBot + Thinkbot Cory Dransfeldt 2025-07-05 10:32:54 -07:00
• 92d721f0c6 chore(robots.json): add SummalyBot + Thinkbot thinkbot-summaly Cory Dransfeldt 2025-07-05 10:32:54 -07:00
• 70ff0c9fbd Update from Dark Visitors dark-visitors 2025-07-03 01:00:49 +00:00
• 20a242d390 Merge pull request #156 from nisbet-hubbard/patch-9 ai.robots.txt 2025-07-02 15:58:19 +00:00
• 91ad302d68 Merge pull request #156 from nisbet-hubbard/patch-9 Cory Dransfeldt 2025-07-02 08:58:08 -07:00
• 79bc22523f Update from Dark Visitors dark-visitors 2025-07-02 01:01:05 +00:00
• 2bfcee794d Update from Dark Visitors ai.robots.txt 2025-06-29 01:07:47 +00:00
• 7eb2099d3f Update from Dark Visitors dark-visitors 2025-06-28 00:58:57 +00:00
• accc48f327 Update funtion and description for SemrushBot nisbet-hubbard 2025-06-21 08:48:58 +08:00
• 4ed17b8e4a Update from Dark Visitors dark-visitors 2025-06-17 01:00:21 +00:00
• 5326c202b5 Merge pull request #154 from paulrudy/main ai.robots.txt 2025-06-16 15:12:42 +00:00
• a31ae1e6d0 Merge pull request #154 from paulrudy/main Cory Dransfeldt 2025-06-16 08:12:31 -07:00
• 7535893aec re-add facebookexternalhit paulrudy 2025-06-15 16:39:17 -07:00
• eb05f2f527 Merge pull request #153 from sergiospagnuolo/Poseidon v1.37 ai.robots.txt 2025-06-14 14:04:03 +00:00
• 26a46c409d Merge pull request #153 from sergiospagnuolo/Poseidon Cory Dransfeldt 2025-06-14 07:03:52 -07:00
• 2b68568ac2 Update from Dark Visitors dark-visitors 2025-06-14 00:58:11 +00:00
• b05f2fee00 Update robots.json with new crawler Sérgio Spagnuolo 2025-06-13 17:15:13 -03:00
• e53d81c66d Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ai.robots.txt 2025-06-13 09:28:41 +00:00
• 20e327e74e Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot Glyn Normington 2025-06-13 10:28:32 +01:00
• 8f17718e76 Fix typo Glyn Normington 2025-06-13 10:28:12 +01:00
• d760f9216f chore(robots.json): adds MyCentralAIScraperBot MyCentralAIScraperBot Cory Dransfeldt 2025-06-12 13:08:29 -07:00
• 842e2256e8 Merge pull request #150 from ai-robots-txt/semrush-bots ai.robots.txt 2025-06-12 07:12:00 +00:00
• 229ea20426 Merge pull request #150 from ai-robots-txt/semrush-bots Glyn Normington 2025-06-12 08:11:51 +01:00
• 14d68f05ba chore(robots.json): adds additional SemrushBot user agents semrush-bots Cory Dransfeldt 2025-06-11 13:50:53 -07:00
• cf598b6b71 Update from Dark Visitors v1.36 dark-visitors 2025-06-10 01:00:37 +00:00
• 3759a6bf14 chore(robots.json): adds EchoboxBot (#148) ai.robots.txt 2025-06-09 15:44:36 +00:00
• 7867c3e26c chore(robots.json): adds EchoboxBot (#148) Cory Dransfeldt 2025-06-09 08:44:25 -07:00
• e21f6ae1b6 Update from Dark Visitors v1.35 dark-visitors 2025-06-06 00:59:25 +00:00
• ac7ed17e71 Merge pull request #145 from ai-robots-txt/aws-bedrockbot ai.robots.txt 2025-06-05 16:51:17 +00:00
• 81747e6772 Merge pull request #145 from ai-robots-txt/aws-bedrockbot Glyn Normington 2025-06-05 17:51:03 +01:00
• 528d77bf07 chore(robots.json): adds bedrockbot Cory Dransfeldt 2025-06-05 09:14:23 -07:00
• 77393df5aa Update from Dark Visitors dark-visitors 2025-06-05 00:59:28 +00:00
• 75ea75a95b Merge pull request #143 from ai-robots-txt/panscient v1.34 ai.robots.txt 2025-06-04 18:04:06 +00:00
• 2fca1ddcf1 Merge pull request #143 from ai-robots-txt/panscient Glyn Normington 2025-06-04 19:03:53 +01:00
• 9c28c63a0c Merge pull request #142 from ai-robots-txt/quillbot ai.robots.txt 2025-06-04 17:54:57 +00:00
• 395c013eea Merge pull request #142 from ai-robots-txt/quillbot Cory Dransfeldt 2025-06-04 10:54:46 -07:00
• 4568d69b0e chore(robots.json): adds Panscient Cory Dransfeldt 2025-06-04 10:54:14 -07:00
• 03831a7eb5 chore(robots.json): adds Quillbot Cory Dransfeldt 2025-06-04 10:46:58 -07:00
• 2b5a59a303 Update from Dark Visitors dark-visitors 2025-06-04 01:00:07 +00:00
• 3efabc603d Merge pull request #141 from Ivan-Chupin/patch-1 v1.33 ai.robots.txt 2025-06-03 23:28:48 +00:00
• b35f9a31d7 Merge pull request #141 from Ivan-Chupin/patch-1 Cory Dransfeldt 2025-06-03 16:28:36 -07:00
• 8f75f4a2f5 Add SBIntuitionsBot Ivan Chupin 2025-06-04 03:48:42 +05:00
• 080946c360 Merge pull request #140 from ai-robots-txt/yandex-bots v1.32 ai.robots.txt 2025-06-03 19:51:25 +00:00
• 7eec033cad Merge pull request #140 from ai-robots-txt/yandex-bots Glyn Normington 2025-06-03 20:51:14 +01:00
• 3187fd8a32 chore(robots.json): adds YandexAdditional crawlers Cory Dransfeldt 2025-06-03 12:24:16 -07:00
• d239e7e5ad Merge pull request #139 from ai-robots-txt/workflow-fix ai.robots.txt 2025-06-03 01:52:35 +00:00
• 9dbf34010a Merge pull request #139 from ai-robots-txt/workflow-fix Glyn Normington 2025-06-03 02:52:23 +01:00
• 87016d1504 Update from Dark Visitors dark-visitors 2025-06-03 01:00:29 +00:00
• 899ce01c55 chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard workflow-fix Cory Dransfeldt 2025-06-02 14:47:31 -07:00
• fa3ee881ac chore(ai_robots_update.yml): possible workflow fix workflow-debug ai.robots.txt 2025-06-02 21:49:47 +00:00
• 6135251465 chore(ai_robots_update.yml): possible workflow fix Cory Dransfeldt 2025-06-02 14:49:27 -07:00
• b2ff663255 chore(ai_robots_update.yml): possible workflow fix Cory Dransfeldt 2025-06-02 14:47:31 -07:00
• 4af776f0a0 Merge pull request #136 from ai-robots-txt/imgproxy-revert Glyn Normington 2025-06-02 20:21:10 +01:00
• 1dd66b6969 Revert "chore(robots.json): adds imgproxy crawler" Cory Dransfeldt 2025-06-02 11:53:06 -07:00
• 814df6b9a0 Merge pull request #134 from not-not-the-imp/patch-1 Cory Dransfeldt 2025-05-31 16:03:16 -07:00
• 268922f8f2 Update robots.json Cory Dransfeldt 2025-05-31 16:02:05 -07:00
• 4259b25ccc Update robots.json Cory Dransfeldt 2025-05-31 16:01:09 -07:00
• d22b9ec51a Update robots.json Cory Dransfeldt 2025-05-31 16:00:13 -07:00