5 Commits

Include renames

Author	SHA1	Message	Date
antanst	6cf507bdc9	DB scripts and migrations	2024-12-09 19:54:00 +02:00
antanst	825c7e3391	.	2024-11-18 16:28:45 +02:00
antanst	a0563074ed	Lint fixes.	2024-11-01 10:10:42 +02:00
antanst	ef3f009709	Add robots.txt checking ... Still needs periodic cache refresh	2024-10-31 16:58:08 +02:00
antanst	cd60c1363b	Lots of features, first version that reliably crawls Geminispace. ... - [x] Concurrent downloading with workers - [x] Concurrent connection limit per host - [x] URL Blacklist - [x] Save image/* and text/* files - [x] Configuration via environment variables - [x] Storing snapshots in PostgreSQL - [x] Proper response header & body UTF-8 and format validation . . .	2024-10-21 20:04:09 +03:00