|
|
7a36614232
|
Better error handling, many fixes all around
|
2024-12-09 19:53:15 +02:00 |
|
|
|
ef3f009709
|
Add robots.txt checking
Still needs periodic cache refresh
|
2024-10-31 16:58:08 +02:00 |
|
|
|
cd60c1363b
|
Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.
.
.
|
2024-10-21 20:04:09 +03:00 |
|
|
|
91f8e69fdf
|
Work on header parsing & saving other files
|
2024-10-08 18:16:47 +03:00 |
|
|
|
c3d6481de0
|
Add configuration via env vars
|
2024-10-08 12:42:08 +03:00 |
|
|
|
74e9327b0b
|
Persist pages to file system
|
2024-10-07 13:36:20 +03:00 |
|
|
|
eb963542b7
|
Initial commit.
|
2024-10-04 13:14:00 +03:00 |
|
