antanst/gemini-grs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
36 Commits 1 Branch 0 Tags
main
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
antanst 43f2242558 Update README
2024-12-09 19:54:15 +02:00
config
DB scripts and migrations
2024-12-09 19:54:00 +02:00
db
DB scripts and migrations
2024-12-09 19:54:00 +02:00
gemini
Better error handling, many fixes all around
2024-12-09 19:53:15 +02:00
http
Preliminary web server
2024-12-09 19:54:08 +02:00
logging
Lots of features, first version that reliably crawls Geminispace.
2024-10-21 20:04:09 +03:00
uid
.
2024-11-18 16:28:45 +02:00
util
Add robots.txt checking
2024-10-31 16:58:08 +02:00
.gitignore
Better error handling, many fixes all around
2024-12-09 19:53:15 +02:00
blacklist.txt
.
2024-11-18 16:28:45 +02:00
debug.sh
.
2024-11-18 16:28:45 +02:00
error_urls.txt
.
2024-11-18 16:28:45 +02:00
go.mod
.
2024-11-18 16:28:45 +02:00
go.sum
.
2024-11-18 16:28:45 +02:00
main.go
Better error handling, many fixes all around
2024-12-09 19:53:15 +02:00
Makefile
.
2024-11-18 16:28:45 +02:00
README.md
Update README
2024-12-09 19:54:15 +02:00

README.md

gemini-grc

A Gemini crawler.

URLs to visit as well as data from visited URLs are stored into "snapshots" in the database.

Done

  • Concurrent downloading with workers
  • Concurrent connection limit per host
  • URL Blacklist
  • Save image/* and text/* files
  • Configuration via environment variables
  • Storing snapshots in PostgreSQL
  • Proper response header & body UTF-8 and format validation
  • Follow robots.txt, see gemini://geminiprotocol.net/docs/companion/robots.gmi
  • Handle redirects (3X status codes)

TODO

  • Better URL normalization
  • Provide a TLS cert for sites that require it, like Astrobotany

TODO for later

Reference in New Issue View Git Blame Copy Permalink
Description
A Gemini protocol server written in Rust.
Readme 158 KiB
Languages
Go 98%
PLpgSQL 0.9%
Makefile 0.6%
Shell 0.5%