Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers - [x] Concurrent connection limit per host - [x] URL Blacklist - [x] Save image/* and text/* files - [x] Configuration via environment variables - [x] Storing snapshots in PostgreSQL - [x] Proper response header & body UTF-8 and format validation . .
This commit is contained in:
8
error_urls.txt
Normal file
8
error_urls.txt
Normal file
@@ -0,0 +1,8 @@
|
||||
// body with null byte
|
||||
gemini://kennedy.gemi.dev/archive/cached?url=gemini://spam.works/mirrors/textfiles/fun/consult.how&t=638427244900000000&raw=False
|
||||
|
||||
// has invalid url
|
||||
gemini://tlgs.one/known-hosts
|
||||
|
||||
// Needs SNI TLS info (our bug)
|
||||
gemini://hanzbrix.pollux.casa/gemlog/20241002.gmi
|
||||
Reference in New Issue
Block a user