Default Branch

56d03d46fb · Update from Dark Visitors · Updated 2025-09-26 02:54:30 +02:00

Branches

0e687a5b58 · chore: adds OpenAI but user agent · Updated 2025-08-13 18:01:55 +02:00

21
0
Included

92d721f0c6 · chore(robots.json): add SummalyBot + Thinkbot · Updated 2025-07-05 19:35:31 +02:00

43
1

d760f9216f · chore(robots.json): adds MyCentralAIScraperBot · Updated 2025-06-12 22:08:29 +02:00

64
0
Included

14d68f05ba · chore(robots.json): adds additional SemrushBot user agents · Updated 2025-06-11 22:50:53 +02:00

64
0
Included

899ce01c55 · chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard · Updated 2025-06-02 23:56:09 +02:00

89
0
Included

fa3ee881ac · chore(ai_robots_update.yml): possible workflow fix · Updated 2025-06-02 23:49:47 +02:00

90
3

b1d9a60a38 · chore(robots.json): adds wpbot · Updated 2025-05-21 20:40:33 +02:00

107
0
Included

8a8001cbec · chore(README): updates the opening line of our README to clarify the types of agents we block · Updated 2025-05-20 22:55:25 +02:00

113
0
Included

08909d958a · fix(robots.json): remove duplicate entry for Meta-ExternalFetcher · Updated 2025-05-17 22:34:58 +02:00

120
2

ca918a963f · chore(robots.json): adds Google-CloudVertexBot · Updated 2025-05-16 06:16:49 +02:00

119
0
Included

5fba0b746d · chore(robots.json): adds MistralAI-User/1.0 crawler · Updated 2025-05-16 05:45:20 +02:00

120
0
Included

3475579089 · chore(robots.json): adds QualifiedBot crawler · Updated 2025-05-15 01:46:32 +02:00

128
1

36a52a88d8 · Bing AI opt-out instructions · Updated 2025-05-13 05:20:18 +02:00

132
0
Included

9e372d0696 · Ensure dependency installed · Updated 2025-01-05 02:45:33 +01:00    Epsilon_02

232
0
Included