Improve crawler performance and logging

- Optimize job scheduler to use NumOfWorkers for URL limits
- Clean up verbose logging in worker processing
- Update log messages for better clarity
This commit is contained in:
antanst
2025-06-29 22:27:20 +03:00
parent 9a09dd7735
commit db3448f448
4 changed files with 14 additions and 7 deletions

View File

@@ -0,0 +1,6 @@
select count(*) from snapshots
where last_crawled < now() - interval '30 days'
and error IS NULL
and gemtext IS NOT NULL
and mimetype='text/gemini'
and url ~ '^gemini://[^/]+/?$';