Viktor Lofgren
8503030f18
(search) Fix rare exception in scribe.rip substitution
deploy-0311
deploy-0310
2025-07-27 19:38:52 +02:00
Viktor Lofgren
744f7d3ef7
(search) Fix rare exception in scribe.rip substitution
deploy-0309
2025-07-27 19:34:03 +02:00
Viktor Lofgren
215e12afe9
(index) Shrink query buffer size
deploy-0308
2025-07-27 17:33:46 +02:00
Viktor Lofgren
2716bce918
(index) Adjust timeout logic for evaluation
deploy-0307
2025-07-27 17:28:34 +02:00
Viktor Lofgren
caf2e6fbb7
(index) Adjust timeout logic for evaluation
deploy-0306
2025-07-27 17:27:07 +02:00
Viktor Lofgren
233f0acfb1
(index) Further reduce query buffer size
deploy-0305
2025-07-27 17:13:08 +02:00
Viktor Lofgren
e3a4ff02e9
(index) Abandon ongoing evaluation tasks if time is up
deploy-0304
2025-07-27 17:04:01 +02:00
Viktor Lofgren
c786283ae1
(index) Reduce quer buffer size
deploy-0303
deploy-0302
2025-07-27 16:57:55 +02:00
Viktor Lofgren
a3f65ac0e0
(deploy) Trigger index deployment
deploy-0301
2025-07-27 16:50:23 +02:00
Viktor
aba1a32af0
Merge pull request #217 from MarginaliaSearch/uncompressed-spans-file
...
Index optimizations
2025-07-27 16:49:27 +02:00
Viktor Lofgren
c9c442345b
(perf) Change execution test to use processing rate instead of count
2025-07-27 16:39:51 +02:00
Viktor Lofgren
2e126ba30e
(perf) Change execution test to use processing rate instead of count
2025-07-27 16:37:20 +02:00
Viktor Lofgren
2087985f49
(index) Implement work stealing in IndexQueryExecution as a better approach to backpressure
2025-07-27 16:29:57 +02:00
Viktor Lofgren
2b13ebd18b
(index) Tweak evaluation backlog handling
2025-07-27 16:08:16 +02:00
Viktor Lofgren
6d92c125fe
(perf) Fix perf test
2025-07-27 15:50:28 +02:00
Viktor Lofgren
f638cfa39a
(index) Avoid possibility of negative timeout
2025-07-27 15:39:12 +02:00
Viktor Lofgren
89447c12af
(index) Avoid possibility of negative timeout
2025-07-27 15:24:47 +02:00
Viktor Lofgren
c71fc46f04
(perf) Update perf test with execution scenario
2025-07-27 15:22:07 +02:00
Viktor Lofgren
f96874d828
(sequence) Implement a largestValue abort condition for minDistance()
...
This is something like 3500% faster in certain common scenarios
2025-07-27 15:05:50 +02:00
Viktor Lofgren
583a84d5a0
(index) Clean up of the index query execution logic
2025-07-27 15:05:50 +02:00
Viktor Lofgren
f65b946448
(index) Clean up code
2025-07-27 15:05:50 +02:00
Viktor Lofgren
3682815855
(index) Optimize sequence intersection for the n=1 case
2025-07-26 19:14:32 +02:00
Viktor Lofgren
3a94357660
(index) Perf test tool (WIP!)
2025-07-26 11:49:33 +02:00
Viktor Lofgren
673b0d3de1
(index) Perf test tool (WIP!)
2025-07-26 11:49:31 +02:00
Viktor Lofgren
ea942bc664
(spans) Add signature to the footer of the spans file, including a version byte so we can detect whether ot use the old or new decoding logic
2025-07-25 12:07:18 +02:00
Viktor Lofgren
7ed5083c54
(index) Don't split results into chunks
2025-07-25 11:45:07 +02:00
Viktor Lofgren
08bb2c097b
(refac) Clean up the data model used in the index service
2025-07-25 10:54:07 +02:00
Viktor Lofgren
495fb325be
(sequence) Correct sequence intersection bug introduced in optimizations
2025-07-25 10:48:33 +02:00
Viktor Lofgren
05c25bbaec
(chore) Clean up
2025-07-24 23:43:27 +02:00
Viktor Lofgren
2a028b84f3
(chore) Clean up
2025-07-24 20:12:56 +02:00
Viktor Lofgren
a091a23623
(ranking) Remove unnecessary metadata retrievals
2025-07-24 20:08:09 +02:00
Viktor Lofgren
e8897acb45
(ranking) Remove unnecessary metadata retrievals
2025-07-24 20:05:39 +02:00
Viktor Lofgren
b89ffcf2be
(index) Evaluate hash based idx mapping in ForwardIndexReader
2025-07-24 19:47:27 +02:00
Viktor Lofgren
dbcc9055b0
(index) Evaluate using MinMaxPriorityQueue as guts of ResultPriorityQueue
2025-07-24 19:31:51 +02:00
Viktor Lofgren
d9740557f4
(sequence) Optimize intersection logic with a fast abort condition
2025-07-24 19:04:10 +02:00
Viktor Lofgren
0d6cd015fd
(index) Evaluate reading all spans at once
2025-07-24 18:34:11 +02:00
Viktor Lofgren
c6034efcc8
(index) Cache value of bitset cardinality for speed
2025-07-24 17:24:55 +02:00
Viktor Lofgren
76068014ad
(index) More spans optimizations
2025-07-24 15:03:43 +02:00
Viktor Lofgren
1c3ed67127
(index) Byte align document spans
2025-07-24 14:06:14 +02:00
Viktor Lofgren
fc0cb6bd9a
(index) Reserve a larger size for IntArrayList in SeqenceOperations.findIntersections
2025-07-24 14:03:44 +02:00
Viktor Lofgren
c2601bac78
(converter) Remove unnecessary allocation of a 16 KB byte buffer
2025-07-24 13:25:37 +02:00
Viktor Lofgren
f5641b72e9
(index) Fix broken test
2025-07-24 13:21:05 +02:00
Viktor Lofgren
36efe2e219
(index) Optimize PositionsFileReader for concurrent reads
...
In benchmarks this is roughly twice as fast as the previous approach. Main caveat being we need multiple file descriptors to avoid read instruction serialization by the kernel. This is undesirable since the reads are complete scattershot and can't be reordered by the kernel in a way that optimizes anything.
2025-07-24 13:20:54 +02:00
Viktor Lofgren
983fe3829e
(spans) Evaluate uncompressed spans files
...
Span decompression appears to be somewhat of a performance bottleneck. This change removes compression of the spans file. The spans are still compressed in transit between the converter and index constructor at this stage. The change is intentionally kept small to just evaluate the performance implications, change in file sizes, etc.
2025-07-23 18:10:41 +02:00
Viktor Lofgren
668c87aa86
(ssr) Drop Executor from SSR as it no longer exists
2025-07-23 13:55:41 +02:00
Viktor Lofgren
9d3f9adb05
Force redeploy of everything
deploy-0300
2025-07-23 13:36:02 +02:00
Viktor
a43a1773f1
Merge pull request #216 from MarginaliaSearch/deprecate-executor
...
Architecture: Remove the separate executor service and roll it into the index service.
2025-07-23 13:32:42 +02:00
Viktor Lofgren
1e7a3a3c4f
(docs) Update docs to reflect the change
2025-07-23 13:18:23 +02:00
Viktor Lofgren
62b696b1c3
(architecture) Remove the separate executor service and merge it into the index service
...
The primary motivation for this is that in production, the large number of partitioned services has lead to an intermittent exhaustion of available database connections, as each service has a connection pool.
The decision to have a separate executor service dates back from when the index service was very slow to start, and the executor didn't always spin off its memory-hungry tasks into separate processes, which meant the executor would sometimes OOM and crash, and it was undesirable to bring the index down with it.
2025-07-23 12:57:13 +02:00
Viktor Lofgren
f1a900f383
(search) Clean up front page mobile design a bit
deploy-0299
2025-07-23 12:20:40 +02:00