|
|
7a36614232
|
Better error handling, many fixes all around
|
2024-12-09 19:53:15 +02:00 |
|
|
|
b52d4f6532
|
refactor: Update error message and remove commented JSON conversion methods
|
2024-11-26 09:51:25 +02:00 |
|
|
|
825c7e3391
|
.
|
2024-11-18 16:28:45 +02:00 |
|
|
|
f0452ff9f7
|
fix: Change URL Value method to use value receiver for database encoding
|
2024-11-11 16:19:07 +02:00 |
|
|
|
bb49ea8565
|
fix: Update URL Value method to use pointer receiver for consistency
|
2024-11-11 16:19:06 +02:00 |
|
|
|
a19f157c80
|
fix: Implement database serialization for gemini URL type
|
2024-11-11 16:14:59 +02:00 |
|
|
|
8b341d2ac6
|
refactor: Simplify URL parsing and remove JSON-related methods
|
2024-11-11 16:14:58 +02:00 |
|
|
|
cd60c1363b
|
Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.
.
.
|
2024-10-21 20:04:09 +03:00 |
|
