antanst
fee7d3e01c
- [x] Concurrent downloading with workers - [x] Concurrent connection limit per host - [x] URL Blacklist - [x] Save image/* and text/* files - [x] Configuration via environment variables - [x] Storing snapshots in PostgreSQL - [x] Proper response header & body UTF-8 and format validation . .
32 lines
1.0 KiB
Markdown
32 lines
1.0 KiB
Markdown
# gemini-grc
|
|
|
|
A Gemini crawler.
|
|
|
|
## Done
|
|
- [x] Concurrent downloading with workers
|
|
- [x] Concurrent connection limit per host
|
|
- [x] URL Blacklist
|
|
- [x] Save image/* and text/* files
|
|
- [x] Configuration via environment variables
|
|
- [x] Storing snapshots in PostgreSQL
|
|
- [x] Proper response header & body UTF-8 and format validation
|
|
|
|
## TODO
|
|
- [ ] Follow robots.txt gemini://geminiprotocol.net/docs/companion/
|
|
- [ ] Test with gemini://alexey.shpakovsky.ru/maze
|
|
- [ ] Proper handling of all response codes
|
|
- [ ] Handle 3X redirects properly
|
|
- [ ] Handle URLs that need presentation of a TLS cert, like astrobotany
|
|
+ [ ] Probably have a common "grc" cert for all?
|
|
- [ ] Proper input and response validations:
|
|
+ [ ] When making a request, the URI MUST NOT exceed 1024 bytes
|
|
- [ ] Subscriptions to gemini pages? gemini://geminiprotocol.net/docs/companion/
|
|
|
|
## TODO for later
|
|
- [ ] Add other protocols
|
|
+ [ ] Gopher
|
|
+ [ ] Scroll gemini://auragem.letz.dev/devlog/20240316.gmi
|
|
+ [ ] Spartan
|
|
+ [ ] Nex
|
|
+ [ ] SuperTXT https://supertxt.net/00-intro.html
|