antanst/gemini-grs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12 Commits 1 Branch 0 Tags
17ef03d6212be1288dfa92028f6f5807208c80c7
Commit Graph
4 Commits
Author SHA1 Message Date
antanst 17ef03d621 feat: Batch insert links to improve database performance 2024-10-23 09:05:50 +03:00
antanst a2a6bd200a Optimize worker random snapshot selection 2024-10-22 23:37:31 +03:00
antanst 3c5206ae43 Change blacklist to comprise domains. 2024-10-22 16:43:55 +03:00
antanst cd60c1363b Lots of features, first version that reliably crawls Geminispace. 
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.

.

.
 2024-10-21 20:04:09 +03:00