antanst/gemini-grs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f9b5fd5e7fb8178d1269792bc8882c2a78148d4c
gemini-grs/README.md
antanst cd60c1363b Lots of features, first version that reliably crawls Geminispace. 
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.

.

.
2024-10-21 20:04:09 +03:00

1.0 KiB
Raw Blame History

gemini-grc

A Gemini crawler.

Done

  • Concurrent downloading with workers
  • Concurrent connection limit per host
  • URL Blacklist
  • Save image/* and text/* files
  • Configuration via environment variables
  • Storing snapshots in PostgreSQL
  • Proper response header & body UTF-8 and format validation

TODO

  • Follow robots.txt gemini://geminiprotocol.net/docs/companion/
    • Test with gemini://alexey.shpakovsky.ru/maze
  • Proper handling of all response codes
    • Handle 3X redirects properly
  • Handle URLs that need presentation of a TLS cert, like astrobotany
    • Probably have a common "grc" cert for all?
  • Proper input and response validations:
    • When making a request, the URI MUST NOT exceed 1024 bytes
  • Subscriptions to gemini pages? gemini://geminiprotocol.net/docs/companion/

TODO for later

Reference in New Issue View Git Blame Copy Permalink