Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers - [x] Concurrent connection limit per host - [x] URL Blacklist - [x] Save image/* and text/* files - [x] Configuration via environment variables - [x] Storing snapshots in PostgreSQL - [x] Proper response header & body UTF-8 and format validation . . .
This commit is contained in:
5
blacklist.txt
Normal file
5
blacklist.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
gemini://gemi.dev/cgi-bin/waffle.cgi
|
||||
gemini://alexey.shpakovsky.ru/maze
|
||||
gemini://kennedy.gemi.dev
|
||||
gemini://musicbrainz.uploadedlobster.com
|
||||
gemini://gemini.bunburya.eu/remini
|
||||
Reference in New Issue
Block a user