Lots of features, first version that reliably crawls Geminispace.

- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.

.

db/migrate1_host.sh

@@ -0,0 +1,14 @@
+#!/bin/sh
+set -eu
+
+MAX_RESPONSE_SIZE=10485760 \
+LOG_LEVEL=info \
+ROOT_PATH=./snaps \
+RESPONSE_TIMEOUT=10 \
+NUM_OF_WORKERS=5 \
+PG_DATABASE=gemini \
+PG_HOST=127.0.0.1 \
+PG_PORT=5433 \
+PG_USER=gemini \
+PG_PASSWORD=gemini \
+go run ./migrate1_host.go