|
|
825c7e3391
|
.
|
2024-11-18 16:28:45 +02:00 |
|
|
|
6bcc7081b2
|
Fix dependency on go-nanoid
|
2024-11-05 12:43:28 +02:00 |
|
|
|
cd60c1363b
|
Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.
.
.
|
2024-10-21 20:04:09 +03:00 |
|
|
|
8278f2b204
|
Proper mimetype parsing, refactoring
|
2024-10-09 13:31:49 +03:00 |
|
|
|
74e9327b0b
|
Persist pages to file system
|
2024-10-07 13:36:20 +03:00 |
|
|
|
74be6b4d0d
|
Basic functionality
|
2024-10-04 13:15:07 +03:00 |
|
