antanst/gemini-grc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20 Commits 1 Branch 0 Tags
daaa61c884ab44458d4b4fea8539e068865d33c0
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

gemini-grc

A Gemini crawler.

Done

  • Concurrent downloading with workers
  • Concurrent connection limit per host
  • URL Blacklist
  • Save image/* and text/* files
  • Configuration via environment variables
  • Storing snapshots in PostgreSQL
  • Proper response header & body UTF-8 and format validation
  • Follow robots.txt

TODO

  • Take into account gemini://geminiprotocol.net/docs/companion/robots.gmi
  • Proper handling of all response codes
    • Handle 3X redirects properly
  • Handle URLs that need presentation of a TLS cert, like astrobotany
    • Probably have a common "grc" cert for all?
  • Proper input and response validations:
    • When making a request, the URI MUST NOT exceed 1024 bytes
  • Subscriptions to gemini pages? gemini://geminiprotocol.net/docs/companion/

TODO for later

Reference in New Issue View Git Blame Copy Permalink
Description
A crawler for the Gemini network.
Readme ISC 783 KiB
Languages
Go 99.1%
Makefile 0.5%
PLpgSQL 0.4%