(crawler) Reduce log spam from error handling in new sitemap fetcher

Reduce the use of 3rd party code in the crawler

ROADMAP.md

@@ -1,4 +1,4 @@
-# Roadmap 2024-2025
+# Roadmap 2025
 
 This is a roadmap with major features planned for Marginalia Search.
 
@@ -30,12 +30,6 @@ Retaining the ability to independently crawl the web is still strongly desirable
 The search engine has a bit of a problem showing spicy content mixed in with the results.  It would be desirable to have a way to filter this out.  It's likely something like a URL blacklist (e.g. [UT1](https://dsi.ut-capitole.fr/blacklists/index_en.php) )
 combined with naive bayesian filter would go a long way, or something more sophisticated...?
 
-## Web Design Overhaul
-
-The design is kinda clunky and hard to maintain, and needlessly outdated-looking.  
-
-In progress: PR [#127](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/127)  -- demo available at https://test.marginalia.nu/
-
 ## Additional Language Support
 
 It would be desirable if the search engine supported more languages than English.  This is partially about
@@ -62,8 +56,31 @@ filter for any API consumer.
 
 I've talked to the stract dev and he does not think it's a good idea to mimic their optics language, which is quite ad-hoc, but instead to work together to find some new common description language for this. 
 
+## Show favicons next to search results
+
+This is expected from search engines.  Basic proof of concept sketch of fetching this data has been done, but the feature is some way from being reality. 
+
+## Specialized crawler for github
+
+One of the search engine's biggest limitations right now is that it does not index github at all.   A specialized crawler that fetches at least the readme.md would go a long way toward providing search capabilities in this domain.
+
 # Completed
 
+## Web Design Overhaul (COMPLETED 2025-01)
+
+The design is kinda clunky and hard to maintain, and needlessly outdated-looking.  
+
+PR [#127](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/127)
+
+## Finalize RSS support (COMPLETED 2024-11)
+
+Marginalia has experimental RSS preview support for a few domains.  This works well and
+it should be extended to all domains.  It would also be interesting to offer search of the
+RSS data itself, or use the RSS set to feed a special live index that updates faster than the
+main dataset. 
+
+Completed with PR [#122](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/122) and PR [#125](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/125)
+
 ## Proper Position Index (COMPLETED 2024-09)
 
 The search engine uses a fixed width bit mask to indicate word positions.  It has the benefit
@@ -76,11 +93,3 @@ list, as is the civilized way of doing this.
 
 Completed with PR [#99](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/99)
 
-## Finalize RSS support (COMPLETED 2024-11)
-
-Marginalia has experimental RSS preview support for a few domains.  This works well and
-it should be extended to all domains.  It would also be interesting to offer search of the
-RSS data itself, or use the RSS set to feed a special live index that updates faster than the
-main dataset. 
-
-Completed with PR [#122](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/122) and PR [#125](https://github.com/MarginaliaSearch/MarginaliaSearch/pull/125)

build.gradle

@@ -47,7 +47,7 @@ ext {
     dockerImageBase='container-registry.oracle.com/graalvm/jdk:23'
     dockerImageTag='latest'
     dockerImageRegistry='marginalia'
-    jibVersion = '3.4.3'
+    jibVersion = '3.4.4'
 
 }
 

code/common/db/java/nu/marginalia/db/DbDomainQueries.java

@@ -20,7 +20,10 @@ public class DbDomainQueries {
     private final HikariDataSource dataSource;
 
     private static final Logger logger = LoggerFactory.getLogger(DbDomainQueries.class);
+
     private final Cache<EdgeDomain, Integer> domainIdCache = CacheBuilder.newBuilder().maximumSize(10_000).build();
+    private final Cache<Integer, EdgeDomain> domainNameCache = CacheBuilder.newBuilder().maximumSize(10_000).build();
+    private final Cache<String, List<DomainWithNode>> siblingsCache = CacheBuilder.newBuilder().maximumSize(10_000).build();
 
     @Inject
     public DbDomainQueries(HikariDataSource dataSource)
@@ -30,16 +33,21 @@ public class DbDomainQueries {
 
 
     public Integer getDomainId(EdgeDomain domain) throws NoSuchElementException {
-        try (var connection = dataSource.getConnection()) {
+        try {
-
             return domainIdCache.get(domain, () -> {
-                try (var stmt = connection.prepareStatement("SELECT ID FROM EC_DOMAIN WHERE DOMAIN_NAME=?")) {
+                try (var connection = dataSource.getConnection();
+                     var stmt = connection.prepareStatement("SELECT ID FROM EC_DOMAIN WHERE DOMAIN_NAME=?")) {
+
                     stmt.setString(1, domain.toString());
                     var rsp = stmt.executeQuery();
                     if (rsp.next()) {
                         return rsp.getInt(1);
                     }
                 }
+                catch (SQLException ex) {
+                    throw new RuntimeException(ex);
+                }
+
                 throw new NoSuchElementException();
             });
         }
@@ -49,9 +57,6 @@ public class DbDomainQueries {
         catch (ExecutionException ex) {
             throw new RuntimeException(ex.getCause());
         }
-        catch (SQLException ex) {
-            throw new RuntimeException(ex);
-        }
     }
 
     public OptionalInt tryGetDomainId(EdgeDomain domain) {
@@ -84,47 +89,55 @@ public class DbDomainQueries {
     }
 
     public Optional<EdgeDomain> getDomain(int id) {
-        try (var connection = dataSource.getConnection()) {
 
+        EdgeDomain existing = domainNameCache.getIfPresent(id);
+        if (existing != null) {
+            return Optional.of(existing);
+        }
+
+        try (var connection = dataSource.getConnection()) {
             try (var stmt = connection.prepareStatement("SELECT DOMAIN_NAME FROM EC_DOMAIN WHERE ID=?")) {
                 stmt.setInt(1, id);
                 var rsp = stmt.executeQuery();
                 if (rsp.next()) {
-                    return Optional.of(new EdgeDomain(rsp.getString(1)));
+                    var val = new EdgeDomain(rsp.getString(1));
+                    domainNameCache.put(id, val);
+                    return Optional.of(val);
                 }
                 return Optional.empty();
             }
         }
-        catch (UncheckedExecutionException ex) {
-            throw new RuntimeException(ex.getCause());
-        }
         catch (SQLException ex) {
             throw new RuntimeException(ex);
         }
     }
 
-    public List<DomainWithNode> otherSubdomains(EdgeDomain domain, int cnt) {
+    public List<DomainWithNode> otherSubdomains(EdgeDomain domain, int cnt) throws ExecutionException {
-        List<DomainWithNode> ret = new ArrayList<>();
+        String topDomain = domain.topDomain;
 
-        try (var conn = dataSource.getConnection();
+        return siblingsCache.get(topDomain, () -> {
-             var stmt = conn.prepareStatement("SELECT DOMAIN_NAME, NODE_AFFINITY FROM EC_DOMAIN WHERE DOMAIN_TOP = ? LIMIT ?")) {
+            List<DomainWithNode> ret = new ArrayList<>();
-            stmt.setString(1, domain.topDomain);
-            stmt.setInt(2, cnt);
 
-            var rs = stmt.executeQuery();
+            try (var conn = dataSource.getConnection();
-            while (rs.next()) {
+                 var stmt = conn.prepareStatement("SELECT DOMAIN_NAME, NODE_AFFINITY FROM EC_DOMAIN WHERE DOMAIN_TOP = ? LIMIT ?")) {
-                var sibling = new EdgeDomain(rs.getString(1));
+                stmt.setString(1, topDomain);
+                stmt.setInt(2, cnt);
 
-                if (sibling.equals(domain))
+                var rs = stmt.executeQuery();
-                    continue;
+                while (rs.next()) {
+                    var sibling = new EdgeDomain(rs.getString(1));
 
-                ret.add(new DomainWithNode(sibling, rs.getInt(2)));
+                    if (sibling.equals(domain))
+                        continue;
+
+                    ret.add(new DomainWithNode(sibling, rs.getInt(2)));
+                }
+            } catch (SQLException e) {
+                logger.error("Failed to get domain neighbors");
             }
-        } catch (SQLException e) {
+            return ret;
-            logger.error("Failed to get domain neighbors");
+        });
-        }
 
-        return ret;
     }
 
     public record DomainWithNode (EdgeDomain domain, int nodeAffinity) {

code/common/db/java/nu/marginalia/db/DbDomainStatsExportMultitool.java

@@ -1,118 +0,0 @@
-package nu.marginalia.db;
-
-import com.zaxxer.hikari.HikariDataSource;
-
-import java.sql.Connection;
-import java.sql.PreparedStatement;
-import java.sql.SQLException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.OptionalInt;
-
-/** Class used in exporting data.  This is intended to be used for a brief time
- * and then discarded, not kept around as a service.
- */
-public class DbDomainStatsExportMultitool implements AutoCloseable {
-    private final Connection connection;
-    private final int nodeId;
-    private final PreparedStatement knownUrlsQuery;
-    private final PreparedStatement visitedUrlsQuery;
-    private final PreparedStatement goodUrlsQuery;
-    private final PreparedStatement domainNameToId;
-
-    private final PreparedStatement allDomainsQuery;
-    private final PreparedStatement crawlQueueDomains;
-    private final PreparedStatement indexedDomainsQuery;
-
-    public DbDomainStatsExportMultitool(HikariDataSource dataSource, int nodeId) throws SQLException {
-        this.connection = dataSource.getConnection();
-        this.nodeId = nodeId;
-
-        knownUrlsQuery = connection.prepareStatement("""
-                SELECT KNOWN_URLS
-                FROM EC_DOMAIN INNER JOIN DOMAIN_METADATA
-                    ON EC_DOMAIN.ID=DOMAIN_METADATA.ID
-                WHERE DOMAIN_NAME=?
-                """);
-        visitedUrlsQuery = connection.prepareStatement("""
-                SELECT VISITED_URLS
-                FROM EC_DOMAIN INNER JOIN DOMAIN_METADATA
-                    ON EC_DOMAIN.ID=DOMAIN_METADATA.ID
-                WHERE DOMAIN_NAME=?
-                """);
-        goodUrlsQuery = connection.prepareStatement("""
-                SELECT GOOD_URLS
-                FROM EC_DOMAIN INNER JOIN DOMAIN_METADATA
-                    ON EC_DOMAIN.ID=DOMAIN_METADATA.ID
-                WHERE DOMAIN_NAME=?
-                """);
-        domainNameToId = connection.prepareStatement("""
-                SELECT ID
-                FROM EC_DOMAIN
-                WHERE DOMAIN_NAME=?
-                """);
-        allDomainsQuery = connection.prepareStatement("""
-                SELECT DOMAIN_NAME
-                FROM EC_DOMAIN
-                """);
-        crawlQueueDomains = connection.prepareStatement("""
-                SELECT DOMAIN_NAME
-                FROM CRAWL_QUEUE
-                """);
-        indexedDomainsQuery = connection.prepareStatement("""
-                SELECT DOMAIN_NAME
-                FROM EC_DOMAIN
-                WHERE INDEXED > 0
-                """);
-    }
-
-    public OptionalInt getVisitedUrls(String domainName) throws SQLException {
-        return executeNameToIntQuery(domainName, visitedUrlsQuery);
-    }
-
-    public OptionalInt getDomainId(String domainName) throws SQLException {
-        return executeNameToIntQuery(domainName, domainNameToId);
-    }
-
-    public List<String> getCrawlQueueDomains() throws SQLException {
-        return executeListQuery(crawlQueueDomains, 100);
-    }
-    public List<String> getAllIndexedDomains() throws SQLException {
-        return executeListQuery(indexedDomainsQuery, 100_000);
-    }
-
-    private OptionalInt executeNameToIntQuery(String domainName, PreparedStatement statement)
-            throws SQLException {
-        statement.setString(1, domainName);
-        var rs = statement.executeQuery();
-
-        if (rs.next()) {
-            return OptionalInt.of(rs.getInt(1));
-        }
-
-        return OptionalInt.empty();
-    }
-
-    private List<String> executeListQuery(PreparedStatement statement, int sizeHint) throws SQLException {
-        List<String> ret = new ArrayList<>(sizeHint);
-
-        var rs = statement.executeQuery();
-
-        while (rs.next()) {
-            ret.add(rs.getString(1));
-        }
-
-        return ret;
-    }
-
-    @Override
-    public void close() throws SQLException {
-        knownUrlsQuery.close();
-        goodUrlsQuery.close();
-        visitedUrlsQuery.close();
-        allDomainsQuery.close();
-        crawlQueueDomains.close();
-        domainNameToId.close();
-        connection.close();
-    }
-}

code/common/service/java/nu/marginalia/service/module/DatabaseModule.java

@@ -89,7 +89,7 @@ public class DatabaseModule extends AbstractModule {
             config.addDataSourceProperty("prepStmtCacheSize", "250");
             config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");
 
-            config.setMaximumPoolSize(5);
+            config.setMaximumPoolSize(Integer.getInteger("db.poolSize", 5));
             config.setMinimumIdle(2);
 
             config.setMaxLifetime(Duration.ofMinutes(9).toMillis());

code/functions/live-capture/java/nu/marginalia/livecapture/BrowserlessClient.java

@@ -15,7 +15,9 @@ import java.util.Map;
 
 /** Client for local browserless.io API */
 public class BrowserlessClient implements AutoCloseable {
+
     private static final Logger logger = LoggerFactory.getLogger(BrowserlessClient.class);
+    private static final String BROWSERLESS_TOKEN = System.getProperty("live-capture.browserless-token", "BROWSERLESS_TOKEN");
 
     private final HttpClient httpClient = HttpClient.newBuilder()
             .version(HttpClient.Version.HTTP_1_1)
@@ -36,7 +38,7 @@ public class BrowserlessClient implements AutoCloseable {
         );
 
         var request = HttpRequest.newBuilder()
-                .uri(browserlessURI.resolve("/content"))
+                .uri(browserlessURI.resolve("/content?token="+BROWSERLESS_TOKEN))
                 .method("POST", HttpRequest.BodyPublishers.ofString(
                         gson.toJson(requestData)
                 ))
@@ -63,7 +65,7 @@ public class BrowserlessClient implements AutoCloseable {
         );
 
         var request = HttpRequest.newBuilder()
-                .uri(browserlessURI.resolve("/screenshot"))
+                .uri(browserlessURI.resolve("/screenshot?token="+BROWSERLESS_TOKEN))
                 .method("POST", HttpRequest.BodyPublishers.ofString(
                         gson.toJson(requestData)
                 ))

code/functions/live-capture/java/nu/marginalia/rss/model/FeedItem.java

@@ -1,6 +1,6 @@
 package nu.marginalia.rss.model;
 
-import com.apptasticsoftware.rssreader.Item;
+import nu.marginalia.rss.svc.SimpleFeedParser;
 import org.apache.commons.lang3.StringUtils;
 import org.jetbrains.annotations.NotNull;
 import org.jsoup.Jsoup;
@@ -18,37 +18,33 @@ public record FeedItem(String title,
     public static final int MAX_DESC_LENGTH = 255;
     public static final DateTimeFormatter DATE_FORMAT = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
 
-    public static FeedItem fromItem(Item item, boolean keepFragment) {
+    public static FeedItem fromItem(SimpleFeedParser.ItemData item, boolean keepFragment) {
-        String title = item.getTitle().orElse("");
+        String title = item.title();
         String date = getItemDate(item);
         String description = getItemDescription(item);
         String url;
 
-        if (keepFragment || item.getLink().isEmpty()) {
+        if (keepFragment) {
-            url = item.getLink().orElse("");
+            url = item.url();
         }
         else {
             try {
-                String link = item.getLink().get();
+                String link = item.url();
                 var linkUri = new URI(link);
                 var cleanUri = new URI(linkUri.getScheme(), linkUri.getAuthority(), linkUri.getPath(), linkUri.getQuery(), null);
                 url = cleanUri.toString();
             }
             catch (Exception e) {
                 // fallback to original link if we can't clean it, this is not a very important step
-                url = item.getLink().get();
+                url = item.url();
             }
         }
 
         return new FeedItem(title, date, description, url);
     }
 
-    private static String getItemDescription(Item item) {
+    private static String getItemDescription(SimpleFeedParser.ItemData item) {
-        Optional<String> description = item.getDescription();
+        String rawDescription = item.description();
-        if (description.isEmpty())
-            return "";
-
-        String rawDescription = description.get();
         if (rawDescription.indexOf('<') >= 0) {
             rawDescription = Jsoup.parseBodyFragment(rawDescription).text();
         }
@@ -58,15 +54,18 @@ public record FeedItem(String title,
 
     // e.g. http://fabiensanglard.net/rss.xml does dates like this:  1 Apr 2021 00:00:00 +0000
     private static final DateTimeFormatter extraFormatter = DateTimeFormatter.ofPattern("d MMM yyyy HH:mm:ss Z");
-    private static String getItemDate(Item item) {
+    private static String getItemDate(SimpleFeedParser.ItemData item) {
         Optional<ZonedDateTime> zonedDateTime = Optional.empty();
         try {
             zonedDateTime = item.getPubDateZonedDateTime();
         }
         catch (Exception e) {
-            zonedDateTime = item.getPubDate()
+            try {
-                    .map(extraFormatter::parse)
+                zonedDateTime = Optional.of(ZonedDateTime.from(extraFormatter.parse(item.pubDate())));
-                    .map(ZonedDateTime::from);
+            }
+            catch (Exception e2) {
+                // ignore
+            }
         }
 
         return zonedDateTime.map(date -> date.format(DATE_FORMAT)).orElse("");

code/functions/live-capture/java/nu/marginalia/rss/svc/FeedFetcherService.java

@@ -1,7 +1,5 @@
 package nu.marginalia.rss.svc;
 
-import com.apptasticsoftware.rssreader.Item;
-import com.apptasticsoftware.rssreader.RssReader;
 import com.google.inject.Inject;
 import com.opencsv.CSVReader;
 import nu.marginalia.WmsaHome;
@@ -20,7 +18,6 @@ import nu.marginalia.storage.FileStorageService;
 import nu.marginalia.storage.model.FileStorage;
 import nu.marginalia.storage.model.FileStorageType;
 import nu.marginalia.util.SimpleBlockingThreadPool;
-import org.apache.commons.io.input.BOMInputStream;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -32,7 +29,6 @@ import java.net.URISyntaxException;
 import java.net.http.HttpClient;
 import java.net.http.HttpRequest;
 import java.net.http.HttpResponse;
-import java.nio.charset.StandardCharsets;
 import java.sql.SQLException;
 import java.time.*;
 import java.time.format.DateTimeFormatter;
@@ -48,8 +44,6 @@ public class FeedFetcherService {
     private static final int MAX_FEED_ITEMS = 10;
     private static final Logger logger = LoggerFactory.getLogger(FeedFetcherService.class);
 
-    private final RssReader rssReader = new RssReader();
-
     private final FeedDb feedDb;
     private final FileStorageService fileStorageService;
     private final NodeConfigurationService nodeConfigurationService;
@@ -72,17 +66,6 @@ public class FeedFetcherService {
         this.nodeConfigurationService = nodeConfigurationService;
         this.serviceHeartbeat = serviceHeartbeat;
         this.executorClient = executorClient;
-
-
-        // Add support for some alternate date tags for atom
-        rssReader.addItemExtension("issued", this::setDateFallback);
-        rssReader.addItemExtension("created", this::setDateFallback);
-    }
-
-    private void setDateFallback(Item item, String value) {
-        if (item.getPubDate().isEmpty()) {
-            item.setPubDate(value);
-        }
     }
 
     public enum UpdateMode {
@@ -371,12 +354,7 @@ public class FeedFetcherService {
 
     public FeedItems parseFeed(String feedData, FeedDefinition definition) {
         try {
-            feedData = sanitizeEntities(feedData);
+            List<SimpleFeedParser.ItemData> rawItems = SimpleFeedParser.parse(feedData);
-
-            List<Item> rawItems = rssReader.read(
-                    // Massage the data to maximize the possibility of the flaky XML parser consuming it
-                    new BOMInputStream(new ByteArrayInputStream(feedData.trim().getBytes(StandardCharsets.UTF_8)), false)
-            ).toList();
 
             boolean keepUriFragment = rawItems.size() < 2 || areFragmentsDisparate(rawItems);
 
@@ -399,33 +377,6 @@ public class FeedFetcherService {
         }
     }
 
-    private static final Map<String, String> HTML_ENTITIES = Map.of(
-            "&raquo;", "»",
-            "&laquo;", "«",
-            "&mdash;", "--",
-            "&ndash;", "-",
-            "&rsquo;", "'",
-            "&lsquo;", "'",
-            "&quot;", "\"",
-            "&nbsp;", ""
-    );
-
-    /** The XML parser will blow up if you insert HTML entities in the feed XML,
-     * which is unfortunately relatively common.  Replace them as far as is possible
-     * with their corresponding characters
-     */
-    static String sanitizeEntities(String feedData) {
-        String result = feedData;
-        for (Map.Entry<String, String> entry : HTML_ENTITIES.entrySet()) {
-            result = result.replace(entry.getKey(), entry.getValue());
-        }
-
-        // Handle lone ampersands not part of a recognized XML entity
-        result = result.replaceAll("&(?!(amp|lt|gt|apos|quot);)", "&amp;");
-
-        return result;
-    }
-
     /** Decide whether to keep URI fragments in the feed items.
      * <p></p>
      * We keep fragments if there are multiple different fragments in the items.
@@ -433,16 +384,16 @@ public class FeedFetcherService {
      * @param items The items to check
      * @return True if we should keep the fragments, false otherwise
      */
-    private boolean areFragmentsDisparate(List<Item> items) {
+    private boolean areFragmentsDisparate(List<SimpleFeedParser.ItemData> items) {
         Set<String> seenFragments = new HashSet<>();
 
         try {
             for (var item : items) {
-                if (item.getLink().isEmpty()) {
+                if (item.url().isBlank()) {
                     continue;
                 }
 
-                var link = item.getLink().get();
+                var link = item.url();
                 if (!link.contains("#")) {
                     continue;
                 }

code/functions/live-capture/java/nu/marginalia/rss/svc/FeedJournal.java

@@ -10,6 +10,7 @@ import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
+import java.util.function.BiConsumer;
 
 /** Utility for recording fetched feeds to a journal, useful in debugging feed parser issues.
  */
@@ -59,6 +60,17 @@ public interface FeedJournal extends AutoCloseable {
             urlWriter.put(url);
             contentsWriter.put(contents);
         }
+    }
+
+    static void replay(Path journalPath, BiConsumer<String, String> urlAndContent) throws IOException {
+        try (SlopTable table = new SlopTable(journalPath)) {
+            final StringColumn.Reader urlReader = urlColumn.open(table);
+            final StringColumn.Reader contentsReader = contentsColumn.open(table);
+
+            while (urlReader.hasRemaining()) {
+                urlAndContent.accept(urlReader.get(), contentsReader.get());
+            }
+        }
 
     }
 }

code/functions/live-capture/java/nu/marginalia/rss/svc/SimpleFeedParser.java

@@ -0,0 +1,94 @@
+package nu.marginalia.rss.svc;
+
+import com.apptasticsoftware.rssreader.DateTimeParser;
+import com.apptasticsoftware.rssreader.util.Default;
+import org.jsoup.Jsoup;
+import org.jsoup.parser.Parser;
+
+import java.time.ZonedDateTime;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class SimpleFeedParser {
+
+    private static final DateTimeParser dateTimeParser = Default.getDateTimeParser();
+
+    public record ItemData (
+            String title,
+            String description,
+            String url,
+            String pubDate
+    ) {
+        public boolean isWellFormed() {
+            return title != null && !title.isBlank() &&
+                    description != null && !description.isBlank() &&
+                    url != null && !url.isBlank() &&
+                    pubDate != null && !pubDate.isBlank();
+        }
+
+        public Optional<ZonedDateTime> getPubDateZonedDateTime() {
+            try {
+                return Optional.ofNullable(dateTimeParser.parse(pubDate()));
+            }
+            catch (Exception e) {
+                return Optional.empty();
+            }
+        }
+
+    }
+
+    public static List<ItemData> parse(String content) {
+        var doc = Jsoup.parse(content, Parser.xmlParser());
+        List<ItemData> ret = new ArrayList<>();
+
+        doc.select("item, entry").forEach(element -> {
+            String link = "";
+            String title = "";
+            String description = "";
+            String pubDate = "";
+
+            for (String attr : List.of("title", "dc:title")) {
+                if (!title.isBlank())
+                    break;
+                var tag = element.getElementsByTag(attr).first();
+                if (tag != null) {
+                    title = tag.text();
+                }
+            }
+
+            for (String attr : List.of("title", "summary", "content", "description", "dc:description")) {
+                if (!description.isBlank())
+                    break;
+                var tag = element.getElementsByTag(attr).first();
+                if (tag != null) {
+                    description = tag.text();
+                }
+            }
+
+            for (String attr : List.of("pubDate", "published", "updated", "issued", "created", "dc:date")) {
+                if (!pubDate.isBlank())
+                    break;
+                var tag = element.getElementsByTag(attr).first();
+                if (tag != null) {
+                    pubDate = tag.text();
+                }
+            }
+
+            for (String attr : List.of("link", "url")) {
+                if (!link.isBlank())
+                    break;
+                var tag = element.getElementsByTag(attr).first();
+                if (tag != null) {
+                    link = tag.text();
+                }
+            }
+
+            ret.add(new ItemData(title, description, link, pubDate));
+        });
+
+
+        return ret;
+    }
+
+}

code/functions/live-capture/test/nu/marginalia/livecapture/BrowserlessClientTest.java

@@ -2,16 +2,21 @@ package nu.marginalia.livecapture;
 
 import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
 import org.junit.jupiter.api.Test;
 import org.testcontainers.containers.GenericContainer;
 import org.testcontainers.junit.jupiter.Testcontainers;
 import org.testcontainers.utility.DockerImageName;
 
 import java.net.URI;
+import java.util.Map;
 
 @Testcontainers
+@Tag("slow")
 public class BrowserlessClientTest {
-    static GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("browserless/chrome")).withExposedPorts(3000);
+    static GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("browserless/chrome"))
+            .withEnv(Map.of("TOKEN", "BROWSERLESS_TOKEN"))
+            .withExposedPorts(3000);
 
     @BeforeAll
     public static void setup() {

code/functions/live-capture/test/nu/marginalia/rss/svc/TestXmlSanitization.java

@@ -1,50 +0,0 @@
-package nu.marginalia.rss.svc;
-
-import com.apptasticsoftware.rssreader.Item;
-import com.apptasticsoftware.rssreader.RssReader;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.List;
-import java.util.Optional;
-
-public class TestXmlSanitization {
-
-    @Test
-    public void testPreservedEntities() {
-        Assertions.assertEquals("&", FeedFetcherService.sanitizeEntities("&"));
-        Assertions.assertEquals("<", FeedFetcherService.sanitizeEntities("<"));
-        Assertions.assertEquals(">", FeedFetcherService.sanitizeEntities(">"));
-        Assertions.assertEquals("'", FeedFetcherService.sanitizeEntities("'"));
-    }
-
-    @Test
-    public void testNlnetTitleTag() {
-        // The NLnet atom feed puts HTML tags in the entry/title tags, which breaks the vanilla RssReader code
-
-        // Verify we're able to consume and strip out the HTML tags
-        RssReader r = new RssReader();
-
-        List<Item> items = r.read(ClassLoader.getSystemResourceAsStream("nlnet.atom")).toList();
-
-        Assertions.assertEquals(1, items.size());
-        for (var item : items) {
-            Assertions.assertEquals(Optional.of("50 Free and Open Source Projects Selected for NGI Zero grants"), item.getTitle());
-        }
-    }
-
-    @Test
-    public void testStrayAmpersand() {
-        Assertions.assertEquals("Bed &amp; Breakfast", FeedFetcherService.sanitizeEntities("Bed & Breakfast"));
-    }
-
-    @Test
-    public void testTranslatedHtmlEntity() {
-        Assertions.assertEquals("Foo -- Bar", FeedFetcherService.sanitizeEntities("Foo &mdash; Bar"));
-    }
-
-    @Test
-    public void testTranslatedHtmlEntityQuot() {
-        Assertions.assertEquals("\"Bob\"", FeedFetcherService.sanitizeEntities("&quot;Bob&quot;"));
-    }
-}

code/index/api/java/nu/marginalia/index/api/IndexClient.java

@@ -16,20 +16,19 @@ import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.Comparator;
-import java.util.Iterator;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
-
+import java.util.concurrent.atomic.AtomicInteger;
-import static java.lang.Math.clamp;
+import java.util.function.Consumer;
 
 @Singleton
 public class IndexClient {
     private static final Logger logger = LoggerFactory.getLogger(IndexClient.class);
     private final GrpcMultiNodeChannelPool<IndexApiGrpc.IndexApiBlockingStub> channelPool;
     private final DomainBlacklistImpl blacklist;
-    private static final ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();
+    private static final ExecutorService executor = Executors.newCachedThreadPool();
 
     @Inject
     public IndexClient(GrpcChannelPoolFactory channelPoolFactory, DomainBlacklistImpl blacklist) {
@@ -51,40 +50,37 @@ public class IndexClient {
 
     /** Execute a query on the index partitions and return the combined results. */
     public AggregateQueryResponse executeQueries(RpcIndexQuery indexRequest, Pagination pagination) {
-        List<CompletableFuture<Iterator<RpcDecoratedResultItem>>> futures =
-                channelPool.call(IndexApiGrpc.IndexApiBlockingStub::query)
-                        .async(executor)
-                        .runEach(indexRequest);
 
         final int requestedMaxResults = indexRequest.getQueryLimits().getResultsTotal();
-        final int resultsUpperBound = requestedMaxResults * channelPool.getNumNodes();
 
-        List<RpcDecoratedResultItem> results = new ArrayList<>(resultsUpperBound);
+        AtomicInteger totalNumResults = new AtomicInteger(0);
 
-        for (var future : futures) {
+        List<RpcDecoratedResultItem> results =
-            try {
+                channelPool.call(IndexApiGrpc.IndexApiBlockingStub::query)
-                future.get().forEachRemaining(results::add);
+                        .async(executor)
-            }
+                        .runEach(indexRequest)
-            catch (Exception e) {
+                        .stream()
-                logger.error("Downstream exception", e);
+                        .map(future -> future.thenApply(iterator -> {
-            }
+                            List<RpcDecoratedResultItem> ret = new ArrayList<>(requestedMaxResults);
-        }
+                            iterator.forEachRemaining(ret::add);
+                            totalNumResults.addAndGet(ret.size());
+                            return ret;
+                        }))
+                        .mapMulti((CompletableFuture<List<RpcDecoratedResultItem>> fut, Consumer<List<RpcDecoratedResultItem>> c) ->{
+                            try {
+                                c.accept(fut.join());
+                            } catch (Exception e) {
+                                logger.error("Error while fetching results", e);
+                            }
+                        })
+                        .flatMap(List::stream)
+                        .filter(item -> !isBlacklisted(item))
+                        .sorted(comparator)
+                        .skip(Math.max(0, (pagination.page - 1) * pagination.pageSize))
+                        .limit(pagination.pageSize)
+                        .toList();
 
-        // Sort the results by ranking score and remove blacklisted domains
+        return new AggregateQueryResponse(results, pagination.page(), totalNumResults.get());
-        results.sort(comparator);
-        results.removeIf(this::isBlacklisted);
-
-        int numReceivedResults = results.size();
-
-        // pagination is typically 1-indexed, so we need to adjust the start and end indices
-        int indexStart = (pagination.page - 1) * pagination.pageSize;
-        int indexEnd = (pagination.page) * pagination.pageSize;
-
-        results = results.subList(
-                clamp(indexStart, 0, Math.max(0, results.size() - 1)), // from is inclusive, so subtract 1 from size()
-                clamp(indexEnd, 0, results.size()));
-
-        return new AggregateQueryResponse(results, pagination.page(), numReceivedResults);
     }
 
     private boolean isBlacklisted(RpcDecoratedResultItem item) {

code/processes/converting-process/java/nu/marginalia/converting/sideload/warc/WarcSideloader.java

@@ -106,11 +106,7 @@ public class WarcSideloader implements SideloadSource, AutoCloseable {
                 return false;
 
             var url = new EdgeUrl(warcResponse.target());
-            if (!Objects.equals(url.getDomain(), domain)) {
+            return Objects.equals(url.getDomain(), domain);
-                return false;
-            }
-
-            return true;
         } catch (Exception e) {
             logger.warn("Failed to process response", e);
         }

code/processes/converting-process/test/nu/marginalia/converting/CrawlingThenConvertingIntegrationTest.java

@@ -8,6 +8,7 @@ import nu.marginalia.converting.model.ProcessedDomain;
 import nu.marginalia.converting.processor.DomainProcessor;
 import nu.marginalia.crawl.CrawlerMain;
 import nu.marginalia.crawl.DomainStateDb;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.HttpFetcher;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
@@ -200,23 +201,23 @@ public class CrawlingThenConvertingIntegrationTest {
 
     @Test
     public void crawlRobotsTxt() throws Exception {
-        var specs = new CrawlerMain.CrawlSpecRecord("search.marginalia.nu", 5,
+        var specs = new CrawlerMain.CrawlSpecRecord("marginalia-search.com", 5,
-                        List.of("https://search.marginalia.nu/search?q=hello+world")
+                        List.of("https://marginalia-search.com/search?q=hello+world")
         );
 
         CrawledDomain domain = crawl(specs);
         assertFalse(domain.doc.isEmpty());
         assertEquals("OK", domain.crawlerStatus);
-        assertEquals("search.marginalia.nu", domain.domain);
+        assertEquals("marginalia-search.com", domain.domain);
 
         Set<String> allUrls = domain.doc.stream().map(doc -> doc.url).collect(Collectors.toSet());
-        assertTrue(allUrls.contains("https://search.marginalia.nu/search"), "We expect a record for entities that are forbidden");
+        assertTrue(allUrls.contains("https://marginalia-search.com/search"), "We expect a record for entities that are forbidden");
 
         var output = process();
 
         assertNotNull(output);
         assertFalse(output.documents.isEmpty());
-        assertEquals(new EdgeDomain("search.marginalia.nu"), output.domain);
+        assertEquals(new EdgeDomain("marginalia-search.com"), output.domain);
         assertEquals(DomainIndexingState.ACTIVE, output.state);
 
         for (var doc : output.documents) {
@@ -246,7 +247,7 @@ public class CrawlingThenConvertingIntegrationTest {
     private CrawledDomain crawl(CrawlerMain.CrawlSpecRecord specs, Predicate<EdgeDomain> domainBlacklist) throws Exception {
         List<SerializableCrawlData> data = new ArrayList<>();
 
-        try (var recorder = new WarcRecorder(fileName);
+        try (var recorder = new WarcRecorder(fileName, new Cookies());
              var db = new DomainStateDb(dbTempFile))
         {
             new CrawlerRetreiver(httpFetcher, new DomainProber(domainBlacklist), specs, db, recorder).crawlDomain();

code/processes/crawling-process/build.gradle

@@ -55,7 +55,6 @@ dependencies {
     implementation libs.zstd
     implementation libs.jwarc
     implementation libs.crawlercommons
-    implementation libs.okhttp3
     implementation libs.jsoup
     implementation libs.opencsv
     implementation libs.fastutil

code/processes/crawling-process/java/nu/marginalia/crawl/CrawlerMain.java

@@ -33,8 +33,6 @@ import nu.marginalia.service.module.DatabaseModule;
 import nu.marginalia.storage.FileStorageService;
 import nu.marginalia.storage.model.FileStorageId;
 import nu.marginalia.util.SimpleBlockingThreadPool;
-import okhttp3.ConnectionPool;
-import okhttp3.Dispatcher;
 import org.jetbrains.annotations.NotNull;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -85,6 +83,7 @@ public class CrawlerMain extends ProcessMainClass {
 
     @Inject
     public CrawlerMain(UserAgent userAgent,
+                       HttpFetcherImpl httpFetcher,
                        ProcessHeartbeatImpl heartbeat,
                        MessageQueueFactory messageQueueFactory, DomainProber domainProber,
                        FileStorageService fileStorageService,
@@ -98,6 +97,7 @@ public class CrawlerMain extends ProcessMainClass {
         super(messageQueueFactory, processConfiguration, gson, CRAWLER_INBOX);
 
         this.userAgent = userAgent;
+        this.fetcher = httpFetcher;
         this.heartbeat = heartbeat;
         this.domainProber = domainProber;
         this.fileStorageService = fileStorageService;
@@ -111,10 +111,6 @@ public class CrawlerMain extends ProcessMainClass {
                 Integer.getInteger("crawler.poolSize", 256),
                 1);
 
-        fetcher = new HttpFetcherImpl(userAgent,
-                new Dispatcher(),
-                new ConnectionPool(5, 10, TimeUnit.SECONDS)
-        );
 
         // Wait for the blacklist to be loaded before starting the crawl
         blacklist.waitUntilLoaded();
@@ -132,6 +128,10 @@ public class CrawlerMain extends ProcessMainClass {
         System.setProperty("sun.net.client.defaultConnectTimeout", "30000");
         System.setProperty("sun.net.client.defaultReadTimeout", "30000");
 
+        // Set the maximum number of connections to keep alive in the connection pool
+        System.setProperty("jdk.httpclient.idleTimeout", "15"); // 15 seconds
+        System.setProperty("jdk.httpclient.connectionPoolSize", "256");
+
         // We don't want to use too much memory caching sessions for https
         System.setProperty("javax.net.ssl.sessionCacheSize", "2048");
 
@@ -364,10 +364,10 @@ public class CrawlerMain extends ProcessMainClass {
                 Files.deleteIfExists(tempFile);
             }
 
-            try (var warcRecorder = new WarcRecorder(newWarcFile); // write to a temp file for now
+            try (var warcRecorder = new WarcRecorder(newWarcFile, fetcher); // write to a temp file for now
                  var retriever = new CrawlerRetreiver(fetcher, domainProber, specification, domainStateDb, warcRecorder);
-                 CrawlDataReference reference = getReference();
+                 CrawlDataReference reference = getReference()
-                 )
+            )
             {
                 // Resume the crawl if it was aborted
                 if (Files.exists(tempFile)) {

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/ContentTags.java

@@ -1,6 +1,6 @@
 package nu.marginalia.crawl.fetcher;
 
-import okhttp3.Request;
+import java.net.http.HttpRequest;
 
 /** Encapsulates request modifiers; the ETag and Last-Modified tags for a resource */
 public record ContentTags(String etag, String lastMod) {
@@ -17,14 +17,14 @@ public record ContentTags(String etag, String lastMod) {
     }
 
     /** Paints the tags onto the request builder. */
-    public void paint(Request.Builder getBuilder) {
+    public void paint(HttpRequest.Builder getBuilder) {
 
         if (etag != null) {
-            getBuilder.addHeader("If-None-Match", etag);
+            getBuilder.header("If-None-Match", etag);
         }
 
         if (lastMod != null) {
-            getBuilder.addHeader("If-Modified-Since", lastMod);
+            getBuilder.header("If-Modified-Since", lastMod);
         }
     }
 }

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/Cookies.java

@@ -1,33 +1,14 @@
 package nu.marginalia.crawl.fetcher;
 
-import okhttp3.Cookie;
+import java.io.IOException;
-import okhttp3.CookieJar;
+import java.net.CookieHandler;
-import okhttp3.HttpUrl;
+import java.net.URI;
-
-import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 
-public class Cookies {
+public class Cookies extends CookieHandler {
-    final ThreadLocal<ConcurrentHashMap<String, List<Cookie>>> cookieJar = ThreadLocal.withInitial(ConcurrentHashMap::new);
+    final ThreadLocal<ConcurrentHashMap<String, List<String>>> cookieJar = ThreadLocal.withInitial(ConcurrentHashMap::new);
-
-    public CookieJar getJar() {
-        return new CookieJar() {
-
-            @Override
-            public void saveFromResponse(HttpUrl url, List<Cookie> cookies) {
-
-                if (!cookies.isEmpty()) {
-                    cookieJar.get().put(url.host(), cookies);
-                }
-            }
-
-            @Override
-            public List<Cookie> loadForRequest(HttpUrl url) {
-                return cookieJar.get().getOrDefault(url.host(), Collections.emptyList());
-            }
-        };
-    }
 
     public void clear() {
         cookieJar.get().clear();
@@ -38,6 +19,16 @@ public class Cookies {
     }
 
     public List<String> getCookies() {
-        return cookieJar.get().values().stream().flatMap(List::stream).map(Cookie::toString).toList();
+        return cookieJar.get().values().stream().flatMap(List::stream).toList();
+    }
+
+    @Override
+    public Map<String, List<String>> get(URI uri, Map<String, List<String>> requestHeaders) throws IOException {
+        return cookieJar.get();
+    }
+
+    @Override
+    public void put(URI uri, Map<String, List<String>> responseHeaders) throws IOException {
+        cookieJar.get().putAll(responseHeaders);
     }
 }

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/HttpFetcher.java

@@ -3,6 +3,7 @@ package nu.marginalia.crawl.fetcher;
 import com.google.inject.ImplementedBy;
 import crawlercommons.robots.SimpleRobotRules;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
+import nu.marginalia.crawl.retreival.CrawlDelayTimer;
 import nu.marginalia.model.EdgeDomain;
 import nu.marginalia.model.EdgeUrl;
 import nu.marginalia.model.body.HttpFetchResult;
@@ -11,10 +12,10 @@ import nu.marginalia.model.crawldata.CrawlerDomainStatus;
 import java.util.List;
 
 @ImplementedBy(HttpFetcherImpl.class)
-public interface HttpFetcher {
+public interface HttpFetcher extends AutoCloseable {
     void setAllowAllContentTypes(boolean allowAllContentTypes);
 
-    List<String> getCookies();
+    Cookies getCookies();
     void clearCookies();
 
     DomainProbeResult probeDomain(EdgeUrl url);
@@ -27,7 +28,9 @@ public interface HttpFetcher {
     HttpFetchResult fetchContent(EdgeUrl url,
                                  WarcRecorder recorder,
                                  ContentTags tags,
-                                 ProbeType probeType) throws HttpFetcherImpl.RateLimitException, Exception;
+                                 ProbeType probeType) throws Exception;
+
+    List<EdgeUrl> fetchSitemapUrls(String rootSitemapUrl, CrawlDelayTimer delayTimer);
 
     SimpleRobotRules fetchRobotRules(EdgeDomain domain, WarcRecorder recorder);
 

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/HttpFetcherImpl.java

@@ -1,35 +1,41 @@
 package nu.marginalia.crawl.fetcher;
 
 import com.google.inject.Inject;
+import com.google.inject.Singleton;
 import crawlercommons.robots.SimpleRobotRules;
 import crawlercommons.robots.SimpleRobotRulesParser;
 import nu.marginalia.UserAgent;
-import nu.marginalia.crawl.fetcher.socket.FastTerminatingSocketFactory;
-import nu.marginalia.crawl.fetcher.socket.IpInterceptingNetworkInterceptor;
 import nu.marginalia.crawl.fetcher.socket.NoSecuritySSL;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
+import nu.marginalia.crawl.retreival.CrawlDelayTimer;
 import nu.marginalia.model.EdgeDomain;
 import nu.marginalia.model.EdgeUrl;
 import nu.marginalia.model.body.ContentTypeLogic;
 import nu.marginalia.model.body.DocumentBodyExtractor;
 import nu.marginalia.model.body.HttpFetchResult;
 import nu.marginalia.model.crawldata.CrawlerDomainStatus;
-import okhttp3.ConnectionPool;
+import org.jsoup.Jsoup;
-import okhttp3.Dispatcher;
+import org.jsoup.nodes.Document;
-import okhttp3.OkHttpClient;
+import org.jsoup.parser.Parser;
-import okhttp3.Request;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
-import java.io.InterruptedIOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.net.URLDecoder;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.net.http.HttpTimeoutException;
+import java.nio.charset.StandardCharsets;
 import java.time.Duration;
-import java.util.List;
+import java.util.*;
-import java.util.Objects;
+import java.util.concurrent.Executors;
-import java.util.Optional;
+import java.util.zip.GZIPInputStream;
-import java.util.concurrent.TimeUnit;
 
 
+@Singleton
 public class HttpFetcherImpl implements HttpFetcher {
 
     private final Logger logger = LoggerFactory.getLogger(getClass());
@@ -40,39 +46,28 @@ public class HttpFetcherImpl implements HttpFetcher {
     private static final SimpleRobotRulesParser robotsParser = new SimpleRobotRulesParser();
     private static final ContentTypeLogic contentTypeLogic = new ContentTypeLogic();
 
+    private final Duration requestTimeout = Duration.ofSeconds(10);
+
     @Override
     public void setAllowAllContentTypes(boolean allowAllContentTypes) {
         contentTypeLogic.setAllowAllContentTypes(allowAllContentTypes);
     }
 
-    private final OkHttpClient client;
+    private final HttpClient client;
-
-    private static final FastTerminatingSocketFactory ftSocketFactory = new FastTerminatingSocketFactory();
-
-    private OkHttpClient createClient(Dispatcher dispatcher, ConnectionPool pool) {
-        var builder = new OkHttpClient.Builder();
-        if (dispatcher != null) {
-            builder.dispatcher(dispatcher);
-        }
-
-        return builder.sslSocketFactory(NoSecuritySSL.buildSocketFactory(), (X509TrustManager) NoSecuritySSL.trustAllCerts[0])
-            .socketFactory(ftSocketFactory)
-            .hostnameVerifier(NoSecuritySSL.buildHostnameVerifyer())
-            .addNetworkInterceptor(new IpInterceptingNetworkInterceptor())
-            .connectionPool(pool)
-            .cookieJar(cookies.getJar())
-            .followRedirects(true)
-            .followSslRedirects(true)
-            .connectTimeout(8, TimeUnit.SECONDS)
-            .readTimeout(10, TimeUnit.SECONDS)
-            .writeTimeout(10, TimeUnit.SECONDS)
-            .build();
 
+    private HttpClient createClient() {
+        return HttpClient.newBuilder()
+                .sslContext(NoSecuritySSL.buildSslContext())
+                .cookieHandler(cookies)
+                .followRedirects(HttpClient.Redirect.NORMAL)
+                .connectTimeout(Duration.ofSeconds(8))
+                .executor(Executors.newCachedThreadPool())
+                .build();
     }
 
     @Override
-    public List<String> getCookies() {
+    public Cookies getCookies() {
-        return cookies.getCookies();
+        return cookies;
     }
 
     @Override
@@ -81,26 +76,24 @@ public class HttpFetcherImpl implements HttpFetcher {
     }
 
     @Inject
-    public HttpFetcherImpl(UserAgent userAgent,
+    public HttpFetcherImpl(UserAgent userAgent)
-                           Dispatcher dispatcher,
-                           ConnectionPool connectionPool)
     {
-        this.client = createClient(dispatcher, connectionPool);
+        this.client = createClient();
         this.userAgentString = userAgent.uaString();
         this.userAgentIdentifier = userAgent.uaIdentifier();
     }
 
     public HttpFetcherImpl(String userAgent) {
-        this.client = createClient(null, new ConnectionPool());
+        this.client = createClient();
         this.userAgentString = userAgent;
         this.userAgentIdentifier = userAgent;
     }
 
     // Not necessary in prod, but useful in test
     public void close() {
-        client.dispatcher().executorService().shutdown();
+        client.close();
-        client.connectionPool().evictAll();
     }
+
     /**
      * Probe the domain to see if it is reachable, attempting to identify which schema to use,
      * and if there are any redirects.  This is done by one or more HEAD requests.
@@ -110,19 +103,26 @@ public class HttpFetcherImpl implements HttpFetcher {
      */
     @Override
     public DomainProbeResult probeDomain(EdgeUrl url) {
-        var head = new Request.Builder().head().addHeader("User-agent", userAgentString)
+        HttpRequest head;
-                .url(url.toString())
+        try {
-                .build();
+            head = HttpRequest.newBuilder()
+                    .HEAD()
+                    .uri(url.asURI())
+                    .header("User-agent", userAgentString)
+                    .timeout(requestTimeout)
+                    .build();
+        } catch (URISyntaxException e) {
+            return new DomainProbeResult.Error(CrawlerDomainStatus.ERROR, "Invalid URL");
+        }
 
-        var call = client.newCall(head);
+        try {
+            var rsp = client.send(head, HttpResponse.BodyHandlers.discarding());
+            EdgeUrl rspUri = new EdgeUrl(rsp.uri());
 
-        try (var rsp = call.execute()) {
+            if (!Objects.equals(rspUri.domain, url.domain)) {
-            EdgeUrl requestUrl = new EdgeUrl(rsp.request().url().toString());
+                return new DomainProbeResult.Redirect(rspUri.domain);
-
-            if (!Objects.equals(requestUrl.domain, url.domain)) {
-                return new DomainProbeResult.Redirect(requestUrl.domain);
             }
-            return new DomainProbeResult.Ok(requestUrl);
+            return new DomainProbeResult.Ok(rspUri);
         }
         catch (Exception ex) {
             return new DomainProbeResult.Error(CrawlerDomainStatus.ERROR, ex.getMessage());
@@ -140,21 +140,25 @@ public class HttpFetcherImpl implements HttpFetcher {
                                                    WarcRecorder warcRecorder,
                                                    ContentTags tags) throws RateLimitException {
         if (tags.isEmpty() && contentTypeLogic.isUrlLikeBinary(url)) {
-            var headBuilder = new Request.Builder().head()
-                    .addHeader("User-agent", userAgentString)
-                    .addHeader("Accept-Encoding", "gzip")
-                    .url(url.toString());
 
-            var head = headBuilder.build();
+            try {
-            var call = client.newCall(head);
+                var headBuilder = HttpRequest.newBuilder()
+                    .HEAD()
+                    .uri(url.asURI())
+                    .header("User-agent", userAgentString)
+                    .header("Accept-Encoding", "gzip")
+                    .timeout(requestTimeout)
+                    ;
 
-            try (var rsp = call.execute()) {
+                var rsp = client.send(headBuilder.build(), HttpResponse.BodyHandlers.discarding());
-                var contentTypeHeader = rsp.header("Content-type");
+                var headers = rsp.headers();
+
+                var contentTypeHeader = headers.firstValue("Content-Type").orElse(null);
 
                 if (contentTypeHeader != null && !contentTypeLogic.isAllowableContentType(contentTypeHeader)) {
-                    warcRecorder.flagAsFailedContentTypeProbe(url, contentTypeHeader, rsp.code());
+                    warcRecorder.flagAsFailedContentTypeProbe(url, contentTypeHeader, rsp.statusCode());
 
-                    return new ContentTypeProbeResult.BadContentType(contentTypeHeader, rsp.code());
+                    return new ContentTypeProbeResult.BadContentType(contentTypeHeader, rsp.statusCode());
                 }
 
                 // Update the URL to the final URL of the HEAD request, otherwise we might end up doing
@@ -168,27 +172,27 @@ public class HttpFetcherImpl implements HttpFetcher {
                 // too many eyebrows when looking at the logs on the target server.  Overall it's probably desirable
                 // that it looks like the traffic makes sense, as opposed to looking like a broken bot.
 
-                var redirectUrl = new EdgeUrl(rsp.request().url().toString());
+                var redirectUrl = new EdgeUrl(rsp.uri());
                 EdgeUrl ret;
 
                 if (Objects.equals(redirectUrl.domain, url.domain)) ret = redirectUrl;
                 else ret = url;
 
                 // Intercept rate limiting
-                if (rsp.code() == 429) {
+                if (rsp.statusCode() == 429) {
-                    throw new HttpFetcherImpl.RateLimitException(Objects.requireNonNullElse(rsp.header("Retry-After"), "1"));
+                    throw new HttpFetcherImpl.RateLimitException(headers.firstValue("Retry-After").orElse("1"));
                 }
 
                 return new ContentTypeProbeResult.Ok(ret);
             }
+            catch (HttpTimeoutException ex) {
+                warcRecorder.flagAsTimeout(url);
+                return new ContentTypeProbeResult.Timeout(ex);
+            }
             catch (RateLimitException ex) {
                 throw ex;
             }
-            catch (InterruptedIOException ex) {
+            catch (Exception ex) {
-                warcRecorder.flagAsTimeout(url);
-
-                return new ContentTypeProbeResult.Timeout(ex);
-            } catch (Exception ex) {
                 logger.error("Error during fetching {}[{}]", ex.getClass().getSimpleName(), ex.getMessage());
 
                 warcRecorder.flagAsError(url, ex);
@@ -210,13 +214,15 @@ public class HttpFetcherImpl implements HttpFetcher {
                                            ProbeType probeType)
         throws Exception
     {
-        var getBuilder = new Request.Builder().get();
+        var getBuilder = HttpRequest.newBuilder()
-
+                .GET()
-        getBuilder.url(url.toString())
+                .uri(url.asURI())
-                .addHeader("Accept-Encoding", "gzip")
+                .header("User-agent", userAgentString)
-                .addHeader("Accept-Language", "en,*;q=0.5")
+                .header("Accept-Encoding", "gzip")
-                .addHeader("Accept", "text/html, application/xhtml+xml, text/*;q=0.8")
+                .header("Accept-Language", "en,*;q=0.5")
-                .addHeader("User-agent", userAgentString);
+                .header("Accept", "text/html, application/xhtml+xml, text/*;q=0.8")
+                .timeout(requestTimeout)
+                ;
 
         contentTags.paint(getBuilder);
 
@@ -242,6 +248,126 @@ public class HttpFetcherImpl implements HttpFetcher {
         return new SitemapRetriever();
     }
 
+    @Override
+    public List<EdgeUrl> fetchSitemapUrls(String root, CrawlDelayTimer delayTimer) {
+        try {
+            List<EdgeUrl> ret = new ArrayList<>();
+
+            Set<String> seenUrls = new HashSet<>();
+            Set<String> seenSitemaps = new HashSet<>();
+
+            Deque<EdgeUrl> sitemapQueue = new LinkedList<>();
+
+            EdgeUrl rootSitemapUrl = new EdgeUrl(root);
+
+            sitemapQueue.add(rootSitemapUrl);
+
+            int fetchedSitemaps = 0;
+
+            while (!sitemapQueue.isEmpty() && ret.size() < 20_000 && ++fetchedSitemaps < 10) {
+                var head = sitemapQueue.removeFirst();
+
+                switch (fetchSitemap(head)) {
+                    case SitemapResult.SitemapUrls(List<String> urls) -> {
+
+                        for (var url : urls) {
+                            if (seenUrls.add(url)) {
+                                EdgeUrl.parse(url)
+                                        .filter(u -> u.domain.equals(rootSitemapUrl.domain))
+                                        .ifPresent(ret::add);
+                            }
+                        }
+
+                    }
+                    case SitemapResult.SitemapReferences(List<String> refs) -> {
+                        for (var ref : refs) {
+                            if (seenSitemaps.add(ref)) {
+                                EdgeUrl.parse(ref)
+                                        .filter(url -> url.domain.equals(rootSitemapUrl.domain))
+                                        .ifPresent(sitemapQueue::addFirst);
+                            }
+                        }
+                    }
+                    case SitemapResult.SitemapError() -> {}
+                }
+
+                delayTimer.waitFetchDelay();
+            }
+
+            return ret;
+        }
+        catch (Exception ex) {
+            logger.error("Error while fetching sitemaps via {}: {} ({})", root, ex.getClass().getSimpleName(), ex.getMessage());
+            return List.of();
+        }
+    }
+
+
+    private SitemapResult fetchSitemap(EdgeUrl sitemapUrl) throws URISyntaxException, IOException, InterruptedException {
+        HttpRequest getRequest = HttpRequest.newBuilder()
+                .GET()
+                .uri(sitemapUrl.asURI())
+                .header("Accept-Encoding", "gzip")
+                .header("Accept", "text/*, */*;q=0.9")
+                .header("User-agent", userAgentString)
+                .timeout(requestTimeout)
+                .build();
+
+        var response = client.send(getRequest, HttpResponse.BodyHandlers.ofInputStream());
+        if (response.statusCode() != 200) {
+            return new SitemapResult.SitemapError();
+        }
+
+        try (InputStream inputStream = response.body()) {
+
+            InputStream parserStream;
+            if (sitemapUrl.path.endsWith(".gz")) {
+                parserStream = new GZIPInputStream(inputStream);
+            }
+            else {
+                parserStream = inputStream;
+            }
+
+            Document parsedSitemap = Jsoup.parse(parserStream, "UTF-8", sitemapUrl.toString(), Parser.xmlParser());
+            if (parsedSitemap.childrenSize() == 0) {
+                return new SitemapResult.SitemapError();
+            }
+
+            String rootTagName = parsedSitemap.child(0).tagName();
+
+            return switch (rootTagName.toLowerCase()) {
+                case "sitemapindex" -> {
+                    List<String> references = new ArrayList<>();
+                    for (var locTag : parsedSitemap.getElementsByTag("loc")) {
+                        references.add(URLDecoder.decode(locTag.text().trim(), StandardCharsets.UTF_8));
+                    }
+                    yield new SitemapResult.SitemapReferences(Collections.unmodifiableList(references));
+                }
+                case "urlset" -> {
+                    List<String> urls = new ArrayList<>();
+                    for (var locTag : parsedSitemap.select("url > loc")) {
+                        urls.add(URLDecoder.decode(locTag.text().trim(), StandardCharsets.UTF_8));
+                    }
+                    yield new SitemapResult.SitemapUrls(Collections.unmodifiableList(urls));
+                }
+                case "rss", "atom" -> {
+                    List<String> urls = new ArrayList<>();
+                    for (var locTag : parsedSitemap.select("link, url")) {
+                        urls.add(locTag.text().trim());
+                    }
+                    yield new SitemapResult.SitemapUrls(Collections.unmodifiableList(urls));
+                }
+                default -> new SitemapResult.SitemapError();
+            };
+        }
+    }
+
+    private sealed interface SitemapResult {
+        record SitemapUrls(List<String> urls) implements SitemapResult {}
+        record SitemapReferences(List<String> sitemapRefs) implements SitemapResult {}
+        record SitemapError() implements SitemapResult {}
+    }
+
     @Override
     public SimpleRobotRules fetchRobotRules(EdgeDomain domain, WarcRecorder recorder) {
         var ret = fetchAndParseRobotsTxt(new EdgeUrl("https", domain, null, "/robots.txt", null), recorder);
@@ -257,14 +383,15 @@ public class HttpFetcherImpl implements HttpFetcher {
 
     private Optional<SimpleRobotRules> fetchAndParseRobotsTxt(EdgeUrl url, WarcRecorder recorder) {
         try {
-            var getBuilder = new Request.Builder().get();
+            var getRequest = HttpRequest.newBuilder()
+                    .GET()
+                    .uri(url.asURI())
+                    .header("Accept-Encoding", "gzip")
+                    .header("Accept", "text/*, */*;q=0.9")
+                    .header("User-agent", userAgentString)
+                    .timeout(requestTimeout);
 
-            getBuilder.url(url.toString())
+            HttpFetchResult result = recorder.fetch(client, getRequest.build());
-                    .addHeader("Accept-Encoding", "gzip")
-                    .addHeader("Accept", "text/*, */*;q=0.9")
-                    .addHeader("User-agent", userAgentString);
-
-            HttpFetchResult result = recorder.fetch(client, getBuilder.build());
 
             return DocumentBodyExtractor.asBytes(result).mapOpt((contentType, body) ->
                 robotsParser.parseContent(url.toString(),

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/socket/IpInterceptingNetworkInterceptor.java

@@ -1,31 +0,0 @@
-package nu.marginalia.crawl.fetcher.socket;
-
-import okhttp3.Interceptor;
-import okhttp3.Response;
-import org.jetbrains.annotations.NotNull;
-
-import java.io.IOException;
-
-
-/** An interceptor that intercepts network requests and adds the remote IP address as
- * a header in the response.  This is used to pass the remote IP address to the Warc
- * writer, as this information is not available in the response.
- */
-public class IpInterceptingNetworkInterceptor implements Interceptor  {
-    private static final String pseudoHeaderName = "X-Marginalia-Remote-IP";
-
-    @NotNull
-    @Override
-    public Response intercept(@NotNull Interceptor.Chain chain) throws IOException {
-        String IP = chain.connection().socket().getInetAddress().getHostAddress();
-
-        return chain.proceed(chain.request())
-                .newBuilder()
-                .addHeader(pseudoHeaderName, IP)
-                .build();
-    }
-
-    public static String getIpFromResponse(Response response) {
-        return response.header(pseudoHeaderName);
-    }
-}

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/socket/NoSecuritySSL.java

@@ -27,7 +27,7 @@ public class NoSecuritySSL {
             }
     };
 
-    public static SSLSocketFactory buildSocketFactory() {
+    public static SSLContext buildSslContext() {
         try {
             // Install the all-trusting trust manager
             final SSLContext sslContext = SSLContext.getInstance("TLS");
@@ -40,14 +40,11 @@ public class NoSecuritySSL {
             clientSessionContext.setSessionCacheSize(2048);
 
             // Create a ssl socket factory with our all-trusting manager
-            return sslContext.getSocketFactory();
+            return sslContext;
         }
         catch (Exception e) {
             throw new RuntimeException(e);
         }
     }
 
-    public static HostnameVerifier buildHostnameVerifyer() {
-        return (hn, session) -> true;
-    }
 }

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/warc/WarcInputBuffer.java

@@ -1,14 +1,14 @@
 package nu.marginalia.crawl.fetcher.warc;
 
-import okhttp3.Headers;
-import okhttp3.Response;
 import org.apache.commons.io.input.BOMInputStream;
 import org.netpreserve.jwarc.WarcTruncationReason;
 
 import java.io.*;
+import java.net.http.HttpHeaders;
+import java.net.http.HttpResponse;
 import java.nio.file.Files;
 import java.nio.file.Path;
-import java.util.Objects;
+import java.util.Map;
 import java.util.zip.GZIPInputStream;
 
 /** Input buffer for temporary storage of a HTTP response
@@ -17,8 +17,9 @@ import java.util.zip.GZIPInputStream;
  * */
 public abstract class WarcInputBuffer implements AutoCloseable {
     protected WarcTruncationReason truncationReason = WarcTruncationReason.NOT_TRUNCATED;
-    protected Headers headers;
+    protected HttpHeaders headers;
-    WarcInputBuffer(Headers headers) {
+
+    WarcInputBuffer(HttpHeaders headers) {
         this.headers = headers;
     }
 
@@ -30,7 +31,7 @@ public abstract class WarcInputBuffer implements AutoCloseable {
 
     public final WarcTruncationReason truncationReason() { return truncationReason; }
 
-    public final Headers headers() { return headers; }
+    public final HttpHeaders headers() { return headers; }
 
     /** Create a buffer for a response.
      *  If the response is small and not compressed, it will be stored in memory.
@@ -38,26 +39,27 @@ public abstract class WarcInputBuffer implements AutoCloseable {
      *  and suppressed from the headers.
      *  If an error occurs, a buffer will be created with no content and an error status.
      */
-    static WarcInputBuffer forResponse(Response rsp) {
+    static WarcInputBuffer forResponse(HttpResponse<InputStream> rsp) {
         if (rsp == null)
             return new ErrorBuffer();
 
-        try {
+        var headers = rsp.headers();
-            String contentLengthHeader = Objects.requireNonNullElse(rsp.header("Content-Length"), "-1");
+
-            int contentLength = Integer.parseInt(contentLengthHeader);
+        try (var is = rsp.body()) {
-            String contentEncoding = rsp.header("Content-Encoding");
+            int contentLength = (int) headers.firstValueAsLong("Content-Length").orElse(-1L);
+            String contentEncoding = headers.firstValue("Content-Encoding").orElse(null);
 
             if (contentEncoding == null && contentLength > 0 && contentLength < 8192) {
                 // If the content is small and not compressed, we can just read it into memory
-                return new MemoryBuffer(rsp, contentLength);
+                return new MemoryBuffer(headers, is, contentLength);
             }
             else {
                 // Otherwise, we unpack it into a file and read it from there
-                return new FileBuffer(rsp);
+                return new FileBuffer(headers, is);
             }
         }
         catch (Exception ex) {
-            return new ErrorBuffer(rsp);
+            return new ErrorBuffer();
         }
 
     }
@@ -99,12 +101,8 @@ public abstract class WarcInputBuffer implements AutoCloseable {
 /** Pseudo-buffer for when we have an error */
 class ErrorBuffer extends WarcInputBuffer {
     public ErrorBuffer() {
-        super(Headers.of());
+        super(HttpHeaders.of(Map.of(), (k,v)->false));
-        truncationReason = WarcTruncationReason.UNSPECIFIED;
-    }
 
-    public ErrorBuffer(Response rsp) {
-        super(rsp.headers());
         truncationReason = WarcTruncationReason.UNSPECIFIED;
     }
 
@@ -125,12 +123,12 @@ class ErrorBuffer extends WarcInputBuffer {
 /** Buffer for when we have the response in memory */
 class MemoryBuffer extends WarcInputBuffer {
     byte[] data;
-    public MemoryBuffer(Response response, int size) {
+    public MemoryBuffer(HttpHeaders headers, InputStream responseStream, int size) {
-        super(response.headers());
+        super(headers);
 
         var outputStream = new ByteArrayOutputStream(size);
 
-        copy(response.body().byteStream(), outputStream);
+        copy(responseStream, outputStream);
 
         data = outputStream.toByteArray();
     }
@@ -154,19 +152,15 @@ class MemoryBuffer extends WarcInputBuffer {
 class FileBuffer extends WarcInputBuffer {
     private final Path tempFile;
 
-    public FileBuffer(Response response) throws IOException {
+    public FileBuffer(HttpHeaders headers, InputStream responseStream) throws IOException {
-        super(suppressContentEncoding(response.headers()));
+        super(suppressContentEncoding(headers));
 
         this.tempFile = Files.createTempFile("rsp", ".html");
 
-        if (response.body() == null) {
-            truncationReason = WarcTruncationReason.DISCONNECT;
-            return;
-        }
 
-        if ("gzip".equals(response.header("Content-Encoding"))) {
+        if ("gzip".equalsIgnoreCase(headers.firstValue("Content-Encoding").orElse(""))) {
             try (var out = Files.newOutputStream(tempFile)) {
-                copy(new GZIPInputStream(response.body().byteStream()), out);
+                copy(new GZIPInputStream(responseStream), out);
             }
             catch (Exception ex) {
                 truncationReason = WarcTruncationReason.UNSPECIFIED;
@@ -174,7 +168,7 @@ class FileBuffer extends WarcInputBuffer {
         }
         else {
             try (var out = Files.newOutputStream(tempFile)) {
-                copy(response.body().byteStream(), out);
+                copy(responseStream, out);
             }
             catch (Exception ex) {
                 truncationReason = WarcTruncationReason.UNSPECIFIED;
@@ -182,22 +176,13 @@ class FileBuffer extends WarcInputBuffer {
         }
     }
 
-    private static Headers suppressContentEncoding(Headers headers) {
+    private static HttpHeaders suppressContentEncoding(HttpHeaders headers) {
-        var builder = new Headers.Builder();
+        return HttpHeaders.of(headers.map(), (k, v) -> {
-
-        headers.toMultimap().forEach((k, values) -> {
             if ("Content-Encoding".equalsIgnoreCase(k)) {
-                return;
+                return false;
-            }
-            if ("Transfer-Encoding".equalsIgnoreCase(k)) {
-                return;
-            }
-            for (var value : values) {
-                builder.add(k, value);
             }
+            return !"Transfer-Encoding".equalsIgnoreCase(k);
         });
-
-        return builder.build();
     }
 
 

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/warc/WarcProtocolReconstructor.java

@@ -1,11 +1,12 @@
 package nu.marginalia.crawl.fetcher.warc;
 
-import okhttp3.Protocol;
-import okhttp3.Response;
 import org.apache.commons.lang3.StringUtils;
 
 import java.net.URI;
 import java.net.URLEncoder;
+import java.net.http.HttpClient;
+import java.net.http.HttpHeaders;
+import java.net.http.HttpResponse;
 import java.nio.charset.StandardCharsets;
 import java.util.*;
 import java.util.stream.Collectors;
@@ -75,13 +76,13 @@ public class WarcProtocolReconstructor {
         return "HTTP/" + version + " " + statusCode + " " + statusMessage + "\r\n" + headerString + "\r\n\r\n";
     }
 
-    static String getResponseHeader(Response response, long size) {
+    static String getResponseHeader(HttpResponse<?> response, long size) {
-        String version = response.protocol() == Protocol.HTTP_1_1 ? "1.1" : "2.0";
+        String version = response.version() == HttpClient.Version.HTTP_1_1 ? "1.1" : "2.0";
 
-        String statusCode = String.valueOf(response.code());
+        String statusCode = String.valueOf(response.statusCode());
-        String statusMessage = STATUS_CODE_MAP.getOrDefault(response.code(), "Unknown");
+        String statusMessage = STATUS_CODE_MAP.getOrDefault(response.statusCode(), "Unknown");
 
-        String headerString = getHeadersAsString(response, size);
+        String headerString = getHeadersAsString(response.headers(), size);
 
         return "HTTP/" + version + " " + statusCode + " " + statusMessage + "\r\n" + headerString + "\r\n\r\n";
     }
@@ -148,10 +149,10 @@ public class WarcProtocolReconstructor {
         return joiner.toString();
     }
 
-    static private String getHeadersAsString(Response response, long responseSize) {
+    static private String getHeadersAsString(HttpHeaders headers, long responseSize) {
         StringJoiner joiner = new StringJoiner("\r\n");
 
-        response.headers().toMultimap().forEach((k, values) -> {
+        headers.map().forEach((k, values) -> {
             String headerCapitalized = capitalizeHeader(k);
 
             // Omit pseudoheaders injected by the crawler itself
@@ -179,8 +180,8 @@ public class WarcProtocolReconstructor {
         return joiner.toString();
     }
 
-    // okhttp gives us flattened headers, so we need to reconstruct Camel-Kebab-Case style
+    // okhttp gave us flattened headers, so we need to reconstruct Camel-Kebab-Case style
-    // for the WARC parser's sake...
+    // for the WARC parser's sake...  (do we still need this, mr chesterton?)
     static private String capitalizeHeader(String k) {
         return Arrays.stream(StringUtils.split(k, '-'))
                 .map(StringUtils::capitalize)

code/processes/crawling-process/java/nu/marginalia/crawl/fetcher/warc/WarcRecorder.java

@@ -1,13 +1,11 @@
 package nu.marginalia.crawl.fetcher.warc;
 
 import nu.marginalia.crawl.fetcher.ContentTags;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
-import nu.marginalia.crawl.fetcher.socket.IpInterceptingNetworkInterceptor;
 import nu.marginalia.model.EdgeDomain;
 import nu.marginalia.model.EdgeUrl;
 import nu.marginalia.model.body.HttpFetchResult;
-import okhttp3.OkHttpClient;
-import okhttp3.Request;
 import org.jetbrains.annotations.Nullable;
 import org.netpreserve.jwarc.*;
 import org.slf4j.Logger;
@@ -18,16 +16,18 @@ import java.io.InputStream;
 import java.net.InetAddress;
 import java.net.URI;
 import java.net.URISyntaxException;
+import java.net.http.HttpClient;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.NoSuchAlgorithmException;
+import java.time.Duration;
 import java.time.Instant;
 import java.util.*;
 
 /** Based on JWarc's fetch method, APL 2.0 license
  * <p></p>
- * This class wraps OkHttp's OkHttpClient and records the HTTP request and response in a WARC file,
+ * This class wraps HttpClient and records the HTTP request and response in a WARC file,
  * as best is possible given not all the data is available at the same time and needs to
  * be reconstructed.
  */
@@ -47,20 +47,22 @@ public class WarcRecorder implements AutoCloseable {
     // Affix a version string in case we need to change the format in the future
     // in some way
     private final String warcRecorderVersion = "1.0";
-
+    private final Cookies cookies;
-    // We need to know if the site uses cookies so this can be reported among the search results
-    // -- flip this to true if we see any cookies.  This information will also be painted on any
-    // revisited pages.  It's not 100% perfect and a bit order dependent, but it's good enough.
-    private final WarcXCookieInformationHeader cookieInformation = new WarcXCookieInformationHeader();
-
     /**
      * Create a new WarcRecorder that will write to the given file
      *
      * @param warcFile The file to write to
      */
-    public WarcRecorder(Path warcFile) throws IOException {
+    public WarcRecorder(Path warcFile, HttpFetcherImpl fetcher) throws IOException {
         this.warcFile = warcFile;
         this.writer = new WarcWriter(warcFile);
+        this.cookies = fetcher.getCookies();
+    }
+
+    public WarcRecorder(Path warcFile, Cookies cookies) throws IOException {
+        this.warcFile = warcFile;
+        this.writer = new WarcWriter(warcFile);
+        this.cookies = cookies;
     }
 
     /**
@@ -70,37 +72,41 @@ public class WarcRecorder implements AutoCloseable {
     public WarcRecorder() throws IOException {
         this.warcFile = Files.createTempFile("warc", ".warc.gz");
         this.writer = new WarcWriter(this.warcFile);
+        this.cookies = new Cookies();
 
         temporaryFile = true;
     }
 
-    public HttpFetchResult fetch(OkHttpClient client, Request request) throws NoSuchAlgorithmException,
+    public HttpFetchResult fetch(HttpClient client,
-            IOException,
+                                 java.net.http.HttpRequest request)
-            URISyntaxException,
+            throws NoSuchAlgorithmException, IOException, URISyntaxException, InterruptedException
-            InterruptedException
     {
-        URI requestUri = request.url().uri();
+        URI requestUri = request.uri();
 
         WarcDigestBuilder responseDigestBuilder = new WarcDigestBuilder();
         WarcDigestBuilder payloadDigestBuilder = new WarcDigestBuilder();
 
-        String ip;
         Instant date = Instant.now();
 
-        var call = client.newCall(request);
+        var response = client.send(request, java.net.http.HttpResponse.BodyHandlers.ofInputStream());
 
-        cookieInformation.update(client, request.url());
 
-        try (var response = call.execute();
+        Map<String, List<String>> extraHeaders = new HashMap<>();
-             WarcInputBuffer inputBuffer = WarcInputBuffer.forResponse(response))
+
+        // Not entirely sure why we need to do this, but keeping it due to Chesterton's Fence
+        extraHeaders.putAll(request.headers().map());
+
+        try (WarcInputBuffer inputBuffer = WarcInputBuffer.forResponse(response))
         {
+            if (cookies.hasCookies()) {
+                extraHeaders.put("X-Has-Cookies", List.of("1"));
+            }
+
             byte[] responseHeaders = WarcProtocolReconstructor.getResponseHeader(response, inputBuffer.size()).getBytes(StandardCharsets.UTF_8);
 
             ResponseDataBuffer responseDataBuffer = new ResponseDataBuffer(inputBuffer.size() + responseHeaders.length);
             InputStream inputStream = inputBuffer.read();
 
-            ip = IpInterceptingNetworkInterceptor.getIpFromResponse(response);
-
             responseDataBuffer.put(responseHeaders);
             responseDataBuffer.updateDigest(responseDigestBuilder, 0, responseHeaders.length);
 
@@ -123,17 +129,15 @@ public class WarcRecorder implements AutoCloseable {
 
             // It looks like this might be the same as requestUri, but it's not;
             // it's the URI after resolving redirects.
-            final URI responseUri = response.request().url().uri();
+            final URI responseUri = response.uri();
 
             WarcResponse.Builder responseBuilder = new WarcResponse.Builder(responseUri)
                     .blockDigest(responseDigestBuilder.build())
                     .date(date)
                     .body(MediaType.HTTP_RESPONSE, responseDataBuffer.copyBytes());
 
-            cookieInformation.paint(responseBuilder);
+            InetAddress inetAddress = InetAddress.getByName(responseUri.getHost());
-
+            responseBuilder.ipAddress(inetAddress);
-            if (ip != null) responseBuilder.ipAddress(InetAddress.getByName(ip));
-
             responseBuilder.payloadDigest(payloadDigestBuilder.build());
             responseBuilder.truncated(inputBuffer.truncationReason());
 
@@ -150,8 +154,8 @@ public class WarcRecorder implements AutoCloseable {
             byte[] httpRequestString = WarcProtocolReconstructor
                     .getHttpRequestString(
                             response.request().method(),
-                            response.request().headers().toMultimap(),
+                            response.request().headers().map(),
-                            request.headers().toMultimap(),
+                            extraHeaders,
                             requestUri)
                     .getBytes();
 
@@ -167,10 +171,29 @@ public class WarcRecorder implements AutoCloseable {
             warcRequest.http(); // force HTTP header to be parsed before body is consumed so that caller can use it
             writer.write(warcRequest);
 
+            if (Duration.between(date, Instant.now()).compareTo(Duration.ofSeconds(9)) > 0
+                    && inputBuffer.size() < 2048
+                    && !request.uri().getPath().endsWith("robots.txt")) // don't bail on robots.txt
+            {
+                // Fast detection and mitigation of crawler traps that respond with slow
+                // small responses, with a high branching factor
+
+                // Note we bail *after* writing the warc records, this will effectively only
+                // prevent link extraction from the document.
+
+                logger.warn("URL {} took too long to fetch ({}s) and was too small for the effort ({}b)",
+                        requestUri,
+                        Duration.between(date, Instant.now()).getSeconds(),
+                        inputBuffer.size()
+                );
+
+                return new HttpFetchResult.ResultException(new IOException("Likely crawler trap"));
+            }
+
             return new HttpFetchResult.ResultOk(responseUri,
-                    response.code(),
+                    response.statusCode(),
                     inputBuffer.headers(),
-                    ip,
+                    inetAddress.getHostAddress(),
                     responseDataBuffer.data,
                     dataStart,
                     responseDataBuffer.length() - dataStart);
@@ -246,7 +269,9 @@ public class WarcRecorder implements AutoCloseable {
                     .date(Instant.now())
                     .body(MediaType.HTTP_RESPONSE, responseDataBuffer.copyBytes());
 
-            cookieInformation.paint(builder);
+            if (cookies.hasCookies()) {
+                builder.addHeader("X-Has-Cookies", "1");
+            }
 
             var reference = builder.build();
 

code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java

@@ -12,7 +12,6 @@ import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
 import nu.marginalia.crawl.logic.LinkFilterSelector;
 import nu.marginalia.crawl.retreival.revisit.CrawlerRevisitor;
 import nu.marginalia.crawl.retreival.revisit.DocumentWithReference;
-import nu.marginalia.crawl.retreival.sitemap.SitemapFetcher;
 import nu.marginalia.ip_blocklist.UrlBlocklist;
 import nu.marginalia.link_parser.LinkParser;
 import nu.marginalia.model.EdgeDomain;
@@ -53,7 +52,6 @@ public class CrawlerRetreiver implements AutoCloseable {
     private final WarcRecorder warcRecorder;
     private final CrawlerRevisitor crawlerRevisitor;
 
-    private final SitemapFetcher sitemapFetcher;
     int errorCount = 0;
 
     public CrawlerRetreiver(HttpFetcher fetcher,
@@ -71,7 +69,6 @@ public class CrawlerRetreiver implements AutoCloseable {
 
         crawlFrontier = new DomainCrawlFrontier(new EdgeDomain(domain), specs.urls(), specs.crawlDepth());
         crawlerRevisitor = new CrawlerRevisitor(crawlFrontier, this, warcRecorder);
-        sitemapFetcher = new SitemapFetcher(crawlFrontier, fetcher.createSitemapRetriever());
 
         // We must always crawl the index page first, this is assumed when fingerprinting the server
         var fst = crawlFrontier.peek();
@@ -145,9 +142,11 @@ public class CrawlerRetreiver implements AutoCloseable {
         // Add external links to the crawl frontier
         crawlFrontier.addAllToQueue(domainLinks.getUrls(rootUrl.proto));
 
-        // Add links from the sitemap to the crawl frontier
-        sitemapFetcher.downloadSitemaps(robotsRules, rootUrl);
 
+        // Fetch sitemaps
+        for (var sitemap : robotsRules.getSitemaps()) {
+            crawlFrontier.addAllToQueue(fetcher.fetchSitemapUrls(sitemap, delayTimer));
+        }
 
         while (!crawlFrontier.isEmpty()
             && !crawlFrontier.isCrawlDepthReached()
@@ -271,10 +270,7 @@ public class CrawlerRetreiver implements AutoCloseable {
             }
 
             // Download the sitemap if available
-            if (feedLink.isPresent()) {
+            feedLink.ifPresent(s -> fetcher.fetchSitemapUrls(s, timer));
-                sitemapFetcher.downloadSitemaps(List.of(feedLink.get()));
-                timer.waitFetchDelay(0);
-            }
 
             // Grab the favicon if it exists
             fetchWithRetry(faviconUrl, timer, HttpFetcher.ProbeType.DISABLED, ContentTags.empty());

code/processes/crawling-process/java/nu/marginalia/crawl/retreival/sitemap/SitemapFetcher.java

@@ -1,72 +0,0 @@
-package nu.marginalia.crawl.retreival.sitemap;
-
-import crawlercommons.robots.SimpleRobotRules;
-import nu.marginalia.crawl.fetcher.SitemapRetriever;
-import nu.marginalia.crawl.retreival.DomainCrawlFrontier;
-import nu.marginalia.model.EdgeUrl;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.util.HashSet;
-import java.util.List;
-import java.util.Optional;
-import java.util.Set;
-
-public class SitemapFetcher {
-
-    private final DomainCrawlFrontier crawlFrontier;
-    private final SitemapRetriever sitemapRetriever;
-    private static final Logger logger = LoggerFactory.getLogger(SitemapFetcher.class);
-
-    public SitemapFetcher(DomainCrawlFrontier crawlFrontier, SitemapRetriever sitemapRetriever) {
-        this.crawlFrontier = crawlFrontier;
-        this.sitemapRetriever = sitemapRetriever;
-    }
-
-    public void downloadSitemaps(SimpleRobotRules robotsRules, EdgeUrl rootUrl) {
-        List<String> urls = robotsRules.getSitemaps();
-
-        if (urls.isEmpty()) {
-            urls = List.of(rootUrl.withPathAndParam("/sitemap.xml", null).toString());
-        }
-
-        downloadSitemaps(urls);
-    }
-
-    public void downloadSitemaps(List<String> urls) {
-
-        Set<String> checkedSitemaps = new HashSet<>();
-
-        for (var rawUrl : urls) {
-            Optional<EdgeUrl> parsedUrl = EdgeUrl.parse(rawUrl);
-            if (parsedUrl.isEmpty()) {
-                continue;
-            }
-
-            EdgeUrl url = parsedUrl.get();
-
-            // Let's not download sitemaps from other domains for now
-            if (!crawlFrontier.isSameDomain(url)) {
-                continue;
-            }
-
-            if (checkedSitemaps.contains(url.path))
-                continue;
-
-            var sitemap =  sitemapRetriever.fetchSitemap(url);
-            if (sitemap.isEmpty()) {
-                continue;
-            }
-
-            // ensure we don't try to download this sitemap again
-            // (don't move this up, as we may want to check the same
-            // path with different protocols until we find one that works)
-
-            checkedSitemaps.add(url.path);
-
-            crawlFrontier.addAllToQueue(sitemap);
-        }
-
-        logger.debug("Queue is now {}", crawlFrontier.queueSize());
-    }
-}

code/processes/crawling-process/model/build.gradle

@@ -36,7 +36,6 @@ dependencies {
     implementation libs.gson
     implementation libs.commons.io
     implementation libs.commons.lang3
-    implementation libs.okhttp3
     implementation libs.jsoup
     implementation libs.snakeyaml
     implementation libs.zstd

code/processes/crawling-process/model/java/nu/marginalia/model/body/HttpFetchResult.java

@@ -1,17 +1,17 @@
 package nu.marginalia.model.body;
 
 import nu.marginalia.contenttype.ContentType;
-import okhttp3.Headers;
+import org.jetbrains.annotations.Nullable;
 import org.jsoup.Jsoup;
 import org.jsoup.nodes.Document;
 import org.netpreserve.jwarc.MessageHeaders;
 import org.netpreserve.jwarc.WarcResponse;
 
 import java.io.ByteArrayInputStream;
-import java.io.IOException;
 import java.io.InputStream;
 import java.net.InetAddress;
 import java.net.URI;
+import java.net.http.HttpHeaders;
 import java.util.Optional;
 
 /* FIXME:  This interface has a very unfortunate name that is not very descriptive.
@@ -56,42 +56,26 @@ public sealed interface HttpFetchResult {
      */
     record ResultOk(URI uri,
                     int statusCode,
-                    Headers headers,
+                    HttpHeaders headers,
                     String ipAddress,
                     byte[] bytesRaw,
                     int bytesStart,
                     int bytesLength
     ) implements HttpFetchResult {
 
+        public ResultOk(URI uri, int status, MessageHeaders headers, String ipAddress, byte[] bytes, int bytesStart, int length) {
+            this(uri, status, HttpHeaders.of(headers.map(), (k,v) -> true), ipAddress, bytes, bytesStart, length);
+        }
+
         public boolean isOk() {
             return statusCode >= 200 && statusCode < 300;
         }
 
-        public ResultOk(URI uri,
-                        int statusCode,
-                        MessageHeaders headers,
-                        String ipAddress,
-                        byte[] bytesRaw,
-                        int bytesStart,
-                        int bytesLength) {
-            this(uri, statusCode, convertHeaders(headers), ipAddress, bytesRaw, bytesStart, bytesLength);
-        }
-
-        private static Headers convertHeaders(MessageHeaders headers) {
-            var ret = new Headers.Builder();
-            for (var header : headers.map().entrySet()) {
-                for (var value : header.getValue()) {
-                    ret.add(header.getKey(), value);
-                }
-            }
-            return ret.build();
-        }
-
         public InputStream getInputStream() {
             return new ByteArrayInputStream(bytesRaw, bytesStart, bytesLength);
         }
 
-        public Optional<Document> parseDocument() throws IOException {
+        public Optional<Document> parseDocument() {
             return DocumentBodyExtractor.asString(this).flatMapOpt((contentType, body) -> {
                 if (contentType.is("text/html")) {
                     return Optional.of(Jsoup.parse(body));
@@ -102,8 +86,9 @@ public sealed interface HttpFetchResult {
             });
         }
 
+        @Nullable
         public String header(String name) {
-            return headers.get(name);
+            return headers.firstValue(name).orElse(null);
         }
 
     }

code/processes/crawling-process/model/java/nu/marginalia/parquet/crawldata/CrawledDocumentParquetRecordFileWriter.java

@@ -165,27 +165,28 @@ public class CrawledDocumentParquetRecordFileWriter implements AutoCloseable {
             contentType = "";
         }
 
-        String headersStr = null;
         StringJoiner headersStrBuilder = new StringJoiner("\n");
-        for (var header : headers) {
+        for (var header : headers.map().entrySet()) {
-            headersStrBuilder.add(header.getFirst() + ": " + header.getSecond());
+            for (var value : header.getValue()) {
+                headersStrBuilder.add(header.getKey() + ": " + value);
+            }
         }
-        headersStr = headersStrBuilder.toString();
+        String headersStr = headersStrBuilder.toString();
 
 
         write(new CrawledDocumentParquetRecord(
                 domain,
                 response.target(),
                 fetchOk.ipAddress(),
-                WarcXCookieInformationHeader.hasCookies(response),
+                headers.firstValue("X-Has-Cookies").orElse("0").equals("1"),
                 fetchOk.statusCode(),
                 response.date(),
                 contentType,
                 bodyBytes,
                 headersStr,
-                headers.get("ETag"),
+                headers.firstValue("ETag").orElse(null),
-                headers.get("Last-Modified"))
+                headers.firstValue("Last-Modified").orElse(null)
-        );
+        ));
     }
 
 

code/processes/crawling-process/model/java/org/netpreserve/jwarc/WarcXCookieInformationHeader.java

@@ -1,35 +0,0 @@
-package org.netpreserve.jwarc;
-
-import okhttp3.HttpUrl;
-import okhttp3.OkHttpClient;
-
-/** Encapsulates out-of-band information about whether a website uses cookies,
- * using a non-standard WARC header "X-Has-Cookies".
- */
-public class WarcXCookieInformationHeader {
-    private boolean hasCookies = false;
-    private static final String headerName = "X-Has-Cookies";
-
-    public void update(OkHttpClient client, HttpUrl url) {
-        if (!hasCookies) {
-            hasCookies = !client.cookieJar().loadForRequest(url).isEmpty();
-        }
-    }
-
-    public boolean hasCookies() {
-        return hasCookies;
-    }
-
-    public void paint(WarcResponse.Builder builder) {
-        builder.addHeader(headerName, hasCookies ? "1" : "0");
-    }
-    public void paint(WarcXResponseReference.Builder builder) {
-        builder.addHeader(headerName, hasCookies ? "1" : "0");
-    }
-
-    public static boolean hasCookies(WarcRecord record) {
-        return record.headers().contains(headerName, "1");
-    }
-
-
-}

code/processes/crawling-process/test/nu/marginalia/crawl/retreival/CrawlerWarcResynchronizerTest.java

@@ -1,11 +1,9 @@
 package nu.marginalia.crawl.retreival;
 
-import nu.marginalia.crawl.fetcher.socket.IpInterceptingNetworkInterceptor;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
 import nu.marginalia.model.EdgeDomain;
 import nu.marginalia.model.EdgeUrl;
-import okhttp3.OkHttpClient;
-import okhttp3.Request;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -15,6 +13,8 @@ import org.netpreserve.jwarc.WarcResponse;
 
 import java.io.IOException;
 import java.net.URISyntaxException;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.NoSuchAlgorithmException;
@@ -27,11 +27,10 @@ import static org.junit.jupiter.api.Assertions.fail;
 class CrawlerWarcResynchronizerTest {
     Path fileName;
     Path outputFile;
-    OkHttpClient httpClient;
+    HttpClient httpClient;
     @BeforeEach
     public void setUp() throws Exception {
-        httpClient = new OkHttpClient.Builder()
+        httpClient = HttpClient.newBuilder()
-                .addNetworkInterceptor(new IpInterceptingNetworkInterceptor())
                 .build();
 
         fileName = Files.createTempFile("test", ".warc.gz");
@@ -46,7 +45,7 @@ class CrawlerWarcResynchronizerTest {
 
     @Test
     void run() throws IOException, URISyntaxException {
-        try (var oldRecorder = new WarcRecorder(fileName)) {
+        try (var oldRecorder = new WarcRecorder(fileName, new Cookies())) {
             fetchUrl(oldRecorder, "https://www.marginalia.nu/");
             fetchUrl(oldRecorder, "https://www.marginalia.nu/log/");
             fetchUrl(oldRecorder, "https://www.marginalia.nu/feed/");
@@ -56,7 +55,7 @@ class CrawlerWarcResynchronizerTest {
 
         var crawlFrontier = new DomainCrawlFrontier(new EdgeDomain("www.marginalia.nu"), List.of(), 100);
 
-        try (var newRecorder = new WarcRecorder(outputFile)) {
+        try (var newRecorder = new WarcRecorder(outputFile, new Cookies())) {
             new CrawlerWarcResynchronizer(crawlFrontier, newRecorder).run(fileName);
         }
 
@@ -79,10 +78,11 @@ class CrawlerWarcResynchronizerTest {
     }
 
     void fetchUrl(WarcRecorder recorder, String url) throws NoSuchAlgorithmException, IOException, URISyntaxException, InterruptedException {
-        var req = new Request.Builder().url(url)
+        var req = HttpRequest.newBuilder()
-                .addHeader("User-agent", "test.marginalia.nu")
+                .uri(new java.net.URI(url))
-                .addHeader("Accept-Encoding", "gzip")
+                .header("User-agent", "test.marginalia.nu")
-                .get().build();
+                .header("Accept-Encoding", "gzip")
+                .GET().build();
         recorder.fetch(httpClient, req);
     }
 }

code/processes/crawling-process/test/nu/marginalia/crawl/retreival/fetcher/ContentTypeProberTest.java

@@ -2,6 +2,7 @@ package nu.marginalia.crawl.retreival.fetcher;
 
 import com.sun.net.httpserver.HttpServer;
 import nu.marginalia.crawl.fetcher.ContentTags;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.HttpFetcher;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
@@ -79,7 +80,7 @@ class ContentTypeProberTest {
         htmlRedirEndpoint = EdgeUrl.parse("http://localhost:" + port + "/redir.gz").get();
 
         fetcher = new HttpFetcherImpl("test");
-        recorder = new WarcRecorder(warcFile);
+        recorder = new WarcRecorder(warcFile, new Cookies());
     }
 
     @AfterEach

code/processes/crawling-process/test/nu/marginalia/crawl/retreival/fetcher/WarcRecorderTest.java

@@ -2,13 +2,11 @@ package nu.marginalia.crawl.retreival.fetcher;
 
 import nu.marginalia.UserAgent;
 import nu.marginalia.crawl.fetcher.ContentTags;
-import nu.marginalia.crawl.fetcher.socket.IpInterceptingNetworkInterceptor;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
 import nu.marginalia.model.EdgeUrl;
 import nu.marginalia.parquet.crawldata.CrawledDocumentParquetRecordFileReader;
 import nu.marginalia.parquet.crawldata.CrawledDocumentParquetRecordFileWriter;
-import okhttp3.OkHttpClient;
-import okhttp3.Request;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -19,6 +17,8 @@ import org.netpreserve.jwarc.WarcXResponseReference;
 
 import java.io.IOException;
 import java.net.URISyntaxException;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.security.NoSuchAlgorithmException;
@@ -31,17 +31,16 @@ class WarcRecorderTest {
     Path fileNameWarc;
     Path fileNameParquet;
     WarcRecorder client;
-    OkHttpClient httpClient;
+
+    HttpClient httpClient;
     @BeforeEach
     public void setUp() throws Exception {
-        httpClient = new OkHttpClient.Builder()
+        httpClient = HttpClient.newBuilder().build();
-                .addNetworkInterceptor(new IpInterceptingNetworkInterceptor())
-                .build();
 
         fileNameWarc = Files.createTempFile("test", ".warc");
         fileNameParquet = Files.createTempFile("test", ".parquet");
 
-        client = new WarcRecorder(fileNameWarc);
+        client = new WarcRecorder(fileNameWarc, new Cookies());
     }
 
     @AfterEach
@@ -52,10 +51,13 @@ class WarcRecorderTest {
 
     @Test
     void fetch() throws NoSuchAlgorithmException, IOException, URISyntaxException, InterruptedException {
-        client.fetch(httpClient, new Request.Builder().url("https://www.marginalia.nu/")
+        client.fetch(httpClient,
-                .addHeader("User-agent", "test.marginalia.nu")
+                HttpRequest.newBuilder()
-                .addHeader("Accept-Encoding", "gzip")
+                        .uri(new java.net.URI("https://www.marginalia.nu/"))
-                .get().build());
+                        .header("User-agent", "test.marginalia.nu")
+                        .header("Accept-Encoding", "gzip")
+                        .GET().build()
+        );
 
         Map<String, String> sampleData = new HashMap<>();
         try (var warcReader = new WarcReader(fileNameWarc)) {
@@ -76,7 +78,7 @@ class WarcRecorderTest {
     @Test
     public void flagAsSkipped() throws IOException, URISyntaxException {
 
-        try (var recorder = new WarcRecorder(fileNameWarc)) {
+        try (var recorder = new WarcRecorder(fileNameWarc, new Cookies())) {
             recorder.writeReferenceCopy(new EdgeUrl("https://www.marginalia.nu/"),
                     "text/html",
                     200,
@@ -100,7 +102,7 @@ class WarcRecorderTest {
     @Test
     public void flagAsSkippedNullBody() throws IOException, URISyntaxException {
 
-        try (var recorder = new WarcRecorder(fileNameWarc)) {
+        try (var recorder = new WarcRecorder(fileNameWarc, new Cookies())) {
             recorder.writeReferenceCopy(new EdgeUrl("https://www.marginalia.nu/"),
                     "text/html",
                     200,
@@ -112,7 +114,7 @@ class WarcRecorderTest {
 
     @Test
     public void testSaveImport() throws URISyntaxException, IOException {
-        try (var recorder = new WarcRecorder(fileNameWarc)) {
+        try (var recorder = new WarcRecorder(fileNameWarc, new Cookies())) {
             recorder.writeReferenceCopy(new EdgeUrl("https://www.marginalia.nu/"),
                     "text/html",
                     200,
@@ -136,19 +138,23 @@ class WarcRecorderTest {
 
     @Test
     public void testConvertToParquet() throws NoSuchAlgorithmException, IOException, URISyntaxException, InterruptedException {
-        client.fetch(httpClient, new Request.Builder().url("https://www.marginalia.nu/")
+        client.fetch(httpClient, HttpRequest.newBuilder()
-                .addHeader("User-agent", "test.marginalia.nu")
+                .uri(new java.net.URI("https://www.marginalia.nu/"))
-                .addHeader("Accept-Encoding", "gzip")
+                .header("User-agent", "test.marginalia.nu")
-                .get().build());
+                .header("Accept-Encoding", "gzip")
-        client.fetch(httpClient, new Request.Builder().url("https://www.marginalia.nu/log/")
+                .GET().build());
-                .addHeader("User-agent", "test.marginalia.nu")
+
-                .addHeader("Accept-Encoding", "gzip")
+        client.fetch(httpClient, HttpRequest.newBuilder()
-                .get().build());
+                .uri(new java.net.URI("https://www.marginalia.nu/log/"))
-        client.fetch(httpClient, new Request.Builder().url("https://www.marginalia.nu/sanic.png")
+                .header("User-agent", "test.marginalia.nu")
-                .addHeader("User-agent", "test.marginalia.nu")
+                .header("Accept-Encoding", "gzip")
-                .addHeader("Accept-Encoding", "gzip")
+                .GET().build());
-                .get().build());
+
-        client.close();
+        client.fetch(httpClient, HttpRequest.newBuilder()
+                .uri(new java.net.URI("https://www.marginalia.nu/sanic.png"))
+                .header("User-agent", "test.marginalia.nu")
+                .header("Accept-Encoding", "gzip")
+                .GET().build());
 
         CrawledDocumentParquetRecordFileWriter.convertWarc(
                 "www.marginalia.nu",

code/processes/crawling-process/test/nu/marginalia/crawling/HttpFetcherTest.java

@@ -4,6 +4,7 @@ import nu.marginalia.crawl.fetcher.ContentTags;
 import nu.marginalia.crawl.fetcher.HttpFetcher;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
+import nu.marginalia.crawl.retreival.CrawlDelayTimer;
 import nu.marginalia.model.EdgeUrl;
 import nu.marginalia.model.body.ContentTypeLogic;
 import nu.marginalia.model.body.DocumentBodyExtractor;
@@ -37,6 +38,12 @@ class HttpFetcherTest {
         }
     }
 
+    @Test
+    void testSitemapMarginalia() {
+        var fetcher = new HttpFetcherImpl("nu.marginalia.edge-crawler");
+        fetcher.fetchSitemapUrls("https://www.marginalia.nu/sitemap.xml", new CrawlDelayTimer(1)).forEach(System.out::println);
+    }
+
     @Test
     void fetchText() throws Exception {
         var fetcher = new HttpFetcherImpl("nu.marginalia.edge-crawler");

code/processes/crawling-process/test/nu/marginalia/crawling/retreival/CrawlerMockFetcherTest.java

@@ -3,11 +3,9 @@ package nu.marginalia.crawling.retreival;
 import crawlercommons.robots.SimpleRobotRules;
 import nu.marginalia.crawl.CrawlerMain;
 import nu.marginalia.crawl.DomainStateDb;
-import nu.marginalia.crawl.fetcher.ContentTags;
+import nu.marginalia.crawl.fetcher.*;
-import nu.marginalia.crawl.fetcher.HttpFetcher;
-import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
-import nu.marginalia.crawl.fetcher.SitemapRetriever;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
+import nu.marginalia.crawl.retreival.CrawlDelayTimer;
 import nu.marginalia.crawl.retreival.CrawlerRetreiver;
 import nu.marginalia.crawl.retreival.DomainProber;
 import nu.marginalia.model.EdgeDomain;
@@ -17,7 +15,6 @@ import nu.marginalia.model.crawldata.CrawledDocument;
 import nu.marginalia.model.crawldata.CrawlerDocumentStatus;
 import nu.marginalia.model.crawldata.SerializableCrawlData;
 import nu.marginalia.test.CommonTestData;
-import okhttp3.Headers;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -27,6 +24,7 @@ import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
 import java.net.URISyntaxException;
+import java.net.http.HttpHeaders;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.sql.SQLException;
@@ -122,7 +120,7 @@ public class CrawlerMockFetcherTest {
         public void setAllowAllContentTypes(boolean allowAllContentTypes) {}
 
         @Override
-        public List<String> getCookies() { return List.of();}
+        public Cookies getCookies() { return new Cookies();}
 
         @Override
         public void clearCookies() {}
@@ -149,7 +147,7 @@ public class CrawlerMockFetcherTest {
                     return new HttpFetchResult.ResultOk(
                             url.asURI(),
                             200,
-                            new Headers.Builder().build(),
+                            HttpHeaders.of(Map.of(), (k,v)->true),
                             "127.0.0.1",
                             bodyBytes,
                             0,
@@ -164,6 +162,11 @@ public class CrawlerMockFetcherTest {
             return new HttpFetchResult.ResultNone();
         }
 
+        @Override
+        public List<EdgeUrl> fetchSitemapUrls(String rootSitemapUrl, CrawlDelayTimer delayTimer) {
+            return List.of();
+        }
+
         @Override
         public SimpleRobotRules fetchRobotRules(EdgeDomain domain, WarcRecorder recorder) {
             return new SimpleRobotRules();
@@ -174,5 +177,9 @@ public class CrawlerMockFetcherTest {
             return Mockito.mock(SitemapRetriever.class);
         }
 
+        @Override
+        public void close() {
+
+        }
     }
 }

code/processes/crawling-process/test/nu/marginalia/crawling/retreival/CrawlerRetreiverTest.java

@@ -5,6 +5,7 @@ import nu.marginalia.WmsaHome;
 import nu.marginalia.atags.model.DomainLinks;
 import nu.marginalia.crawl.CrawlerMain;
 import nu.marginalia.crawl.DomainStateDb;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.HttpFetcher;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
@@ -180,7 +181,7 @@ class CrawlerRetreiverTest {
                 new EdgeDomain("www.marginalia.nu"),
                 List.of(), 100);
         var resync = new CrawlerWarcResynchronizer(revisitCrawlFrontier,
-                new WarcRecorder(tempFileWarc2)
+                new WarcRecorder(tempFileWarc2, new Cookies())
         );
 
         // truncate the size of the file to simulate a crash
@@ -458,7 +459,7 @@ class CrawlerRetreiverTest {
                 List.of(), 100);
 
         var resync = new CrawlerWarcResynchronizer(revisitCrawlFrontier,
-                new WarcRecorder(tempFileWarc3)
+                new WarcRecorder(tempFileWarc3, new Cookies())
         );
 
         // truncate the size of the file to simulate a crash
@@ -509,7 +510,7 @@ class CrawlerRetreiverTest {
     }
 
     private void doCrawlWithReferenceStream(CrawlerMain.CrawlSpecRecord specs, SerializableCrawlDataStream stream) {
-        try (var recorder = new WarcRecorder(tempFileWarc2);
+        try (var recorder = new WarcRecorder(tempFileWarc2, new Cookies());
              var db = new DomainStateDb(tempFileDb)
         ) {
             new CrawlerRetreiver(httpFetcher, new DomainProber(d -> true), specs, db, recorder).crawlDomain(new DomainLinks(),
@@ -522,7 +523,7 @@ class CrawlerRetreiverTest {
 
     @NotNull
     private DomainCrawlFrontier doCrawl(Path tempFileWarc1, CrawlerMain.CrawlSpecRecord specs) {
-        try (var recorder = new WarcRecorder(tempFileWarc1);
+        try (var recorder = new WarcRecorder(tempFileWarc1, new Cookies());
              var db = new DomainStateDb(tempFileDb)
         ) {
             var crawler = new CrawlerRetreiver(httpFetcher, new DomainProber(d -> true), specs, db, recorder);

code/processes/live-crawling-process/build.gradle

@@ -56,7 +56,6 @@ dependencies {
     implementation libs.zstd
     implementation libs.jwarc
     implementation libs.crawlercommons
-    implementation libs.okhttp3
     implementation libs.jsoup
     implementation libs.opencsv
     implementation libs.fastutil

code/services-application/search-service/java/nu/marginalia/search/command/SearchParameters.java

@@ -84,18 +84,33 @@ public record SearchParameters(WebsiteUrl url,
     }
 
     public String renderUrl() {
-        String path = String.format("/search?query=%s&profile=%s&js=%s&adtech=%s&recent=%s&searchTitle=%s&newfilter=%s&page=%d",
-                URLEncoder.encode(query, StandardCharsets.UTF_8),
-                URLEncoder.encode(profile.filterId, StandardCharsets.UTF_8),
-                URLEncoder.encode(js.value, StandardCharsets.UTF_8),
-                URLEncoder.encode(adtech.value, StandardCharsets.UTF_8),
-                URLEncoder.encode(recent.value, StandardCharsets.UTF_8),
-                URLEncoder.encode(searchTitle.value, StandardCharsets.UTF_8),
-                Boolean.valueOf(newFilter).toString(),
-                page
-                );
 
-        return path;
+        StringBuilder pathBuilder = new StringBuilder("/search?");
+        pathBuilder.append("query=").append(URLEncoder.encode(query, StandardCharsets.UTF_8));
+
+        if (profile != SearchProfile.NO_FILTER) {
+            pathBuilder.append("&profile=").append(URLEncoder.encode(profile.filterId, StandardCharsets.UTF_8));
+        }
+        if (js != SearchJsParameter.DEFAULT) {
+            pathBuilder.append("&js=").append(URLEncoder.encode(js.value, StandardCharsets.UTF_8));
+        }
+        if (adtech != SearchAdtechParameter.DEFAULT) {
+            pathBuilder.append("&adtech=").append(URLEncoder.encode(adtech.value, StandardCharsets.UTF_8));
+        }
+        if (recent != SearchRecentParameter.DEFAULT) {
+            pathBuilder.append("&recent=").append(URLEncoder.encode(recent.value, StandardCharsets.UTF_8));
+        }
+        if (searchTitle != SearchTitleParameter.DEFAULT) {
+            pathBuilder.append("&searchTitle=").append(URLEncoder.encode(searchTitle.value, StandardCharsets.UTF_8));
+        }
+        if (page != 1) {
+            pathBuilder.append("&page=").append(page);
+        }
+        if (newFilter) {
+            pathBuilder.append("&newfilter=").append(Boolean.valueOf(newFilter).toString());
+        }
+
+        return pathBuilder.toString();
     }
 
     public RpcTemporalBias.Bias temporalBias() {

code/services-application/search-service/java/nu/marginalia/search/command/commands/SearchCommand.java

@@ -3,27 +3,22 @@ package nu.marginalia.search.command.commands;
 import com.google.inject.Inject;
 import io.jooby.MapModelAndView;
 import io.jooby.ModelAndView;
-import nu.marginalia.search.JteRenderer;
 import nu.marginalia.search.SearchOperator;
 import nu.marginalia.search.command.SearchCommandInterface;
 import nu.marginalia.search.command.SearchParameters;
 import nu.marginalia.search.model.DecoratedSearchResults;
 import nu.marginalia.search.model.NavbarModel;
 
-import java.io.IOException;
 import java.util.Map;
 import java.util.Optional;
 
 public class SearchCommand implements SearchCommandInterface {
     private final SearchOperator searchOperator;
-    private final JteRenderer jteRenderer;
 
 
     @Inject
-    public SearchCommand(SearchOperator searchOperator,
+    public SearchCommand(SearchOperator searchOperator){
-                         JteRenderer jteRenderer) throws IOException {
         this.searchOperator = searchOperator;
-        this.jteRenderer = jteRenderer;
     }
 
     @Override

code/services-application/search-service/java/nu/marginalia/search/svc/SearchSiteInfoService.java

@@ -28,6 +28,7 @@ import org.slf4j.LoggerFactory;
 import java.sql.SQLException;
 import java.util.*;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.function.Supplier;
@@ -67,8 +68,12 @@ public class SearchSiteInfoService {
         this.screenshotService = screenshotService;
         this.dataSource = dataSource;
         this.searchSiteSubscriptions = searchSiteSubscriptions;
+
+        Thread.ofPlatform().name("Recently Added Domains Model Updater").start(this::modelUpdater);
     }
 
+    private volatile SiteOverviewModel cachedOverviewModel = new SiteOverviewModel(List.of());
+
     @GET
     @Path("/site")
     public ModelAndView<?> handleOverview(@QueryParam String domain) {
@@ -77,23 +82,48 @@ public class SearchSiteInfoService {
             return new MapModelAndView("redirect.jte", Map.of("url", "/site/"+domain));
         }
 
-        List<SiteOverviewModel.DiscoveredDomain> domains = new ArrayList<>();
-
-        try (var conn = dataSource.getConnection();
-             var stmt = conn.prepareStatement("SELECT DOMAIN_NAME, DISCOVER_DATE FROM EC_DOMAIN WHERE NODE_AFFINITY = 0 ORDER BY ID DESC LIMIT 10")) {
-
-            var rs = stmt.executeQuery();
-            while (rs.next()) {
-                domains.add(new SiteOverviewModel.DiscoveredDomain(rs.getString("DOMAIN_NAME"), rs.getString("DISCOVER_DATE")));
-            }
-        }
-        catch (SQLException ex) {
-            throw new RuntimeException();
-        }
-
         return new MapModelAndView("siteinfo/start.jte",
                 Map.of("navbar", NavbarModel.SITEINFO,
-                        "model", new SiteOverviewModel(domains)));
+                        "model", cachedOverviewModel));
+    }
+
+    private void modelUpdater() {
+        while (!Thread.interrupted()) {
+            List<SiteOverviewModel.DiscoveredDomain> domains = new ArrayList<>();
+
+            // This query can be quite expensive, so we can't run it on demand
+            // for every request. Instead, we run it every 15 minutes and cache
+            // the result.
+
+            try (var conn = dataSource.getConnection();
+                 var stmt = conn.prepareStatement("""
+                    SELECT DOMAIN_NAME, DISCOVER_DATE
+                    FROM EC_DOMAIN
+                    WHERE NODE_AFFINITY = 0
+                    ORDER BY ID DESC
+                    LIMIT 10
+                    """))
+            {
+                var rs = stmt.executeQuery();
+                while (rs.next()) {
+                    domains.add(new SiteOverviewModel.DiscoveredDomain(
+                            rs.getString("DOMAIN_NAME"),
+                            rs.getString("DISCOVER_DATE"))
+                    );
+                }
+            } catch (SQLException ex) {
+                logger.warn("Failed to get recently added domains: {}", ex.getMessage());
+            }
+
+            cachedOverviewModel = new SiteOverviewModel(domains);
+
+            try {
+                TimeUnit.MINUTES.sleep(15);
+            } catch (InterruptedException e) {
+                Thread.currentThread().interrupt();
+                break;
+            }
+        }
     }
 
     public record SiteOverviewModel(List<DiscoveredDomain> domains) {
@@ -107,7 +137,7 @@ public class SearchSiteInfoService {
             @PathParam String domainName,
             @QueryParam String view,
             @QueryParam Integer page
-    ) throws SQLException {
+    ) throws SQLException, ExecutionException {
 
         if (null == domainName || domainName.isBlank()) {
             return null;
@@ -193,7 +223,7 @@ public class SearchSiteInfoService {
         );
     }
 
-    private SiteInfoWithContext listInfo(Context context, String domainName) {
+    private SiteInfoWithContext listInfo(Context context, String domainName) throws ExecutionException {
 
         var domain = new EdgeDomain(domainName);
         final int domainId = domainQueries.tryGetDomainId(domain).orElse(-1);

code/services-application/search-service/resources/jte/serp/part/searchform.jte

@@ -36,10 +36,11 @@
 
     </div>
 
+    @if (filters.showRecentOption.isSet()) <input type="hidden" name="js" value="${filters.removeJsOption.value()}"> @endif
+    @if (filters.reduceAdtechOption.isSet()) <input type="hidden" name="adtech" value="${filters.reduceAdtechOption.value()}"> @endif
+    @if (filters.searchTitleOption.isSet()) <input type="hidden" name="searchTitle" value="${filters.searchTitleOption.value()}"> @endif
+    @if (filters.showRecentOption.isSet()) <input type="hidden" name="recent" value="${filters.showRecentOption.value()}"> @endif
 
-    <input type="hidden" name="js" value="${filters.removeJsOption.value()}">
-    <input type="hidden" name="adtech" value="${filters.reduceAdtechOption.value()}">
-    <input type="hidden" name="searchTitle" value="${filters.searchTitleOption.value()}">
     <input type="hidden" name="profile" value="${profile}">
-    <input type="hidden" name="recent" value="${filters.showRecentOption.value()}">
+
 </form>

code/services-application/search-service/resources/jte/serp/start.jte

@@ -36,7 +36,7 @@
         <div class="text-slate-700 dark:text-white text-sm p-4">
             <div class="fas fa-gift mr-1 text-margeblue dark:text-slate-200"></div>
             This is the new design and home of Marginalia Search.
-            You can about what this entails <a href="https://about.marginalia-search.com/article/redesign/"  class="underline text-liteblue dark:text-blue-200">here</a>.
+            You can read about what this entails <a href="https://about.marginalia-search.com/article/redesign/"  class="underline text-liteblue dark:text-blue-200">here</a>.
         <p class="my-4"></p>
         The old version of Marginalia Search remains available at
         <a href="https://old-search.marginalia.nu/" class="underline text-liteblue dark:text-blue-200">https://old-search.marginalia.nu/</a>.

code/services-application/search-service/resources/jte/siteinfo/view/overview.jte

@@ -1,5 +1,4 @@
 @import nu.marginalia.db.DbDomainQueries
-@import nu.marginalia.model.EdgeDomain
 @import nu.marginalia.search.svc.SearchSiteInfoService
 @import nu.marginalia.search.svc.SearchSiteInfoService.*
 @import nu.marginalia.search.model.UrlDetails
@@ -81,35 +80,6 @@
 
         @endif
 
-
-        @if (!siteInfo.siblingDomains().isEmpty())
-            <div class="mx-3 flex place-items-baseline space-x-2 p-2 bg-gray-100 dark:bg-gray-600 rounded">
-                <i class="fas fa-globe"></i>
-                <span>Related Subdomains</span>
-            </div>
-
-            <table class="min-w-full divide-y divide-gray-200 dark:divide-gray-600 mx-4">
-                <thead>
-                <tr class="bg-gray-50 dark:bg-gray-700">
-                    <th scope="col" class="px-2 py-2 text-left text-xs font-medium text-gray-500 dark:text-gray-100 uppercase tracking-wider">Domain Name</th>
-                </tr>
-                </thead>
-                <tbody class="bg-white dark:bg-gray-800 divide-y divide-gray-200 dark:divide-gray-600 text-xs">
-                @for (DbDomainQueries.DomainWithNode sibling : siteInfo.siblingDomains())
-                    <tr>
-                        <td class="px-3 py-6 md:py-3 whitespace-nowrap">
-                            <a class="text-liteblue dark:text-blue-200" href="/site/${sibling.domain().toString()}">${sibling.domain().toString()}</a>
-
-                            @if (!sibling.isIndexed())
-                                <i class="ml-1 fa-regular fa-question-circle text-gray-400 dark:text-gray-600 text-xs" title="Not indexed"></i>
-                            @endif
-                        </td>
-                    </tr>
-                @endfor
-                </tbody>
-            </table>
-        @endif
-
         @if (siteInfo.domainInformation().isUnknownDomain())
             <div class="mx-3 flex place-items-baseline space-x-2 p-2 bg-gray-100 dark:bg-gray-600 rounded">
                 <i class="fa-regular fa-circle-question"></i>
@@ -178,6 +148,36 @@
             </form>
         @endif
 
+
+        @if (!siteInfo.siblingDomains().isEmpty())
+            <div class="mx-3 flex place-items-baseline space-x-2 p-2 bg-gray-100 dark:bg-gray-600 rounded">
+                <i class="fas fa-globe"></i>
+                <span>Related Subdomains</span>
+            </div>
+
+            <table class="min-w-full divide-y divide-gray-200 dark:divide-gray-600 mx-4">
+                <thead>
+                <tr class="bg-gray-50 dark:bg-gray-700">
+                    <th scope="col" class="px-2 py-2 text-left text-xs font-medium text-gray-500 dark:text-gray-100 uppercase tracking-wider">Domain Name</th>
+                </tr>
+                </thead>
+                <tbody class="bg-white dark:bg-gray-800 divide-y divide-gray-200 dark:divide-gray-600 text-xs">
+                @for (DbDomainQueries.DomainWithNode sibling : siteInfo.siblingDomains())
+                    <tr>
+                        <td class="px-3 py-6 md:py-3 whitespace-nowrap">
+                            <a class="text-liteblue dark:text-blue-200" href="/site/${sibling.domain().toString()}">${sibling.domain().toString()}</a>
+
+                            @if (!sibling.isIndexed())
+                                <i class="ml-1 fa-regular fa-question-circle text-gray-400 dark:text-gray-600 text-xs" title="Not indexed"></i>
+                            @endif
+                        </td>
+                    </tr>
+                @endfor
+                </tbody>
+            </table>
+        @endif
+
+
         @if (siteInfo.isKnown())
             <div class="mx-3 flex place-items-baseline space-x-2 p-2 bg-gray-100 dark:bg-gray-600 rounded">
                 <i class="fas fa-chart-simple"></i>

code/tools/integration-test/test/nu/marginalia/IntegrationTest.java

@@ -10,6 +10,7 @@ import nu.marginalia.api.searchquery.model.results.PrototypeRankingParameters;
 import nu.marginalia.converting.processor.DomainProcessor;
 import nu.marginalia.converting.writer.ConverterBatchWriter;
 import nu.marginalia.crawl.fetcher.ContentTags;
+import nu.marginalia.crawl.fetcher.Cookies;
 import nu.marginalia.crawl.fetcher.HttpFetcherImpl;
 import nu.marginalia.crawl.fetcher.warc.WarcRecorder;
 import nu.marginalia.functions.searchquery.QueryFactory;
@@ -120,7 +121,7 @@ public class IntegrationTest {
     public void run() throws Exception {
 
         /** CREATE WARC */
-        try (WarcRecorder warcRecorder = new WarcRecorder(warcData)) {
+        try (WarcRecorder warcRecorder = new WarcRecorder(warcData, new Cookies())) {
             warcRecorder.writeWarcinfoHeader("127.0.0.1", new EdgeDomain("www.example.com"),
                     new HttpFetcherImpl.DomainProbeResult.Ok(new EdgeUrl("https://www.example.com/")));
 

code/tools/screenshot-capture-tool/java/nu/marginalia/screenshot/ScreenshotCaptureToolMain.java

@@ -28,7 +28,7 @@ import java.util.concurrent.TimeUnit;
 public class ScreenshotCaptureToolMain {
 
     private static final Logger logger = LoggerFactory.getLogger(ScreenshotCaptureToolMain.class);
-
+    private static final String BROWSERLESS_TOKEN = System.getenv("live-capture.browserless-token");
     public static void main(String[] args) {
         DatabaseModule databaseModule = new DatabaseModule(false);
         var ds = databaseModule.provideConnection();
@@ -107,7 +107,7 @@ public class ScreenshotCaptureToolMain {
             );
 
             var request = HttpRequest.newBuilder()
-                    .uri(new URI("http://browserless:3000/screenshot"))
+                    .uri(new URI("http://browserless:3000/screenshot?token=" + BROWSERLESS_TOKEN))
                     .method("POST", HttpRequest.BodyPublishers.ofString(
                             gson.toJson(requestData)
                     ))

run/install/barebones-1/docker-compose.yml.template

@@ -72,11 +72,11 @@ services:
     image: "mariadb:lts"
     container_name: "mariadb"
     env_file: "${INSTALL_DIR}/env/mariadb.env"
-    command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
+    command: ['mariadbd', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
     ports:
       - "127.0.0.1:3306:3306/tcp"
     healthcheck:
-      test: mysqladmin ping -h 127.0.0.1 -u ${uval} --password=${pval}
+      test: mariadb-admin ping -h 127.0.0.1 -u ${uval} --password=${pval}
       start_period: 5s
       interval: 5s
       timeout: 5s

run/install/barebones-2/docker-compose.yml.template

@@ -103,11 +103,11 @@ services:
     image: "mariadb:lts"
     container_name: "mariadb"
     env_file: "${INSTALL_DIR}/env/mariadb.env"
-    command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
+    command: ['mariadbd', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
     ports:
       - "127.0.0.1:3306:3306/tcp"
     healthcheck:
-      test: mysqladmin ping -h 127.0.0.1 -u ${uval} --password=${pval}
+      test: mariadb-admin ping -h 127.0.0.1 -u ${uval} --password=${pval}
       start_period: 5s
       interval: 5s
       timeout: 5s

run/install/marginalia-prod-like/docker-compose.yml.template

@@ -129,11 +129,11 @@ services:
     image: "mariadb:lts"
     container_name: "mariadb"
     env_file: "${INSTALL_DIR}/env/mariadb.env"
-    command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
+    command: ['mariadbd', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
     ports:
       - "127.0.0.1:3306:3306/tcp"
     healthcheck:
-      test: mysqladmin ping -h 127.0.0.1 -u ${uval} --password=${pval}
+      test: mariadb-admin ping -h 127.0.0.1 -u ${uval} --password=${pval}
       start_period: 5s
       interval: 5s
       timeout: 5s

run/install/no-docker/docker-compose.yml.template

@@ -3,11 +3,11 @@ services:
     image: "mariadb:lts"
     container_name: "mariadb"
     env_file: "${INSTALL_DIR}/env/mariadb.env"
-    command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
+    command: ['mariadbd', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
     ports:
       - "127.0.0.1:3306:3306/tcp"
     healthcheck:
-      test: mysqladmin ping -h 127.0.0.1 -u ${uval} --password=${pval}
+      test: mariadb-admin ping -h 127.0.0.1 -u ${uval} --password=${pval}
       start_period: 5s
       interval: 5s
       timeout: 5s