antanst/gemini-grc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24 Commits 1 Branch 0 Tags
4f4752140196b844e7487b3e88e23f4e3c414131
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
antanst 4f47521401 update gitignore
2025-02-26 10:37:20 +02:00
bin
Add gemget script that downloads Gemini pages
2025-02-26 10:35:54 +02:00
common
Reorganize code for more granular imports
2025-02-26 10:34:46 +02:00
config
Add mode that prints multiple worker status in console
2025-01-16 10:04:02 +02:00
db
Reorganize code for more granular imports
2025-02-26 10:34:46 +02:00
errors
Reorganize errors
2025-02-26 10:32:38 +02:00
gemini
Reorganize code for more granular imports
2025-02-26 10:34:46 +02:00
gopher
Add Gopherspace crawling!
2025-02-26 10:35:28 +02:00
hostPool
Simplify host pool
2025-02-26 10:35:11 +02:00
logging
Add first version of gemini-grc.
2024-12-27 12:09:55 +02:00
uid
Add first version of gemini-grc.
2024-12-27 12:09:55 +02:00
util
Add regex matching function to util
2025-01-16 22:37:39 +02:00
.gitignore
update gitignore
2025-02-26 10:37:20 +02:00
COPYING
Add license.
2024-12-27 12:13:05 +02:00
gemget.sh
Add gemget script that downloads Gemini pages
2025-02-26 10:35:54 +02:00
go.mod
Tidy go mod
2025-02-26 10:36:41 +02:00
go.sum
Tidy go mod
2025-02-26 10:36:41 +02:00
main.go
Improve main error handling
2025-02-26 10:37:09 +02:00
Makefile
Use Go race detector
2025-02-26 10:36:51 +02:00
README.md
Update DB init instructions & README
2025-01-04 15:39:21 +02:00

README.md

gemini-grc

A crawler for the Gemini network. Easily extendable as a "wayback machine" of Gemini.

Features done

  • URL normalization
  • Handle redirects (3X status codes)
  • Follow robots.txt, see gemini://geminiprotocol.net/docs/companion/robots.gmi
  • Save image/* and text/* files
  • Concurrent downloading with workers
  • Connection limit per host
  • URL Blacklist
  • Configuration via environment variables
  • Storing snapshots in PostgreSQL
  • Proper response header & body UTF-8 and format validation

TODO

  • Add snapshot history
  • Add a web interface
  • Provide to servers a TLS cert for sites that require it, like Astrobotany

TODO (lower priority)

Reference in New Issue View Git Blame Copy Permalink
Description
A crawler for the Gemini network.
Readme ISC 783 KiB
Languages
Go 99.1%
Makefile 0.5%
PLpgSQL 0.4%