mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-10-05 21:22:39 +02:00
62b696b1c3
The primary motivation for this is that in production, the large number of partitioned services has lead to an intermittent exhaustion of available database connections, as each service has a connection pool. The decision to have a separate executor service dates back from when the index service was very slow to start, and the executor didn't always spin off its memory-hungry tasks into separate processes, which meant the executor would sometimes OOM and crash, and it was undesirable to bring the index down with it.
The execution subsystem is responsible for the execution of long running tasks on each index node. It lives in the executor-service module.
It accomplishes this using the message queue and actor library, which permits program state to survive crashes and reboots.
The subsystem exposes four APIs:
- Execution API - for starting and stopping tasks, also contains miscellaneous commands
- Crawl API - for managing the crawl workflow
- Sideload API - for sideloading data
- Export API - for exporting data