|
|
825c7e3391
|
.
|
2024-11-18 16:28:45 +02:00 |
|
|
|
f0452ff9f7
|
fix: Change URL Value method to use value receiver for database encoding
|
2024-11-11 16:19:07 +02:00 |
|
|
|
bb49ea8565
|
fix: Update URL Value method to use pointer receiver for consistency
|
2024-11-11 16:19:06 +02:00 |
|
|
|
a19f157c80
|
fix: Implement database serialization for gemini URL type
|
2024-11-11 16:14:59 +02:00 |
|
|
|
8b341d2ac6
|
refactor: Simplify URL parsing and remove JSON-related methods
|
2024-11-11 16:14:58 +02:00 |
|
|
|
2bb8589eb7
|
feat: Improve error handling with custom error types and detailed messages
|
2024-11-11 15:01:20 +02:00 |
|
|
|
6346c9a829
|
refactor: Remove default configuration constants and simplify config parsing
|
2024-11-11 15:01:18 +02:00 |
|
|
|
c7b0778b77
|
refactor: Add config validation, defaults, and unit tests
|
2024-11-11 14:58:09 +02:00 |
|
|
|
bea0d22c26
|
refactor: Replace switch statement with map-based config parsing
|
2024-11-11 14:56:41 +02:00 |
|
|
|
6bcc7081b2
|
Fix dependency on go-nanoid
|
2024-11-05 12:43:28 +02:00 |
|
|
|
f34ac651b7
|
Add Makefile and remove ad-hoc lint script
|
2024-11-05 12:40:30 +02:00 |
|
|
|
d5da9ac62d
|
Better unicode conversion
|
2024-11-05 12:39:14 +02:00 |
|
|
|
a0563074ed
|
Lint fixes.
|
2024-11-01 10:10:42 +02:00 |
|
|
|
ef3f009709
|
Add robots.txt checking
Still needs periodic cache refresh
|
2024-10-31 16:58:08 +02:00 |
|
|
|
c49a69728a
|
Simplify robots.txt parsing logic
|
2024-10-23 14:28:49 +03:00 |
|
|
|
8d9ea6cdec
|
fix: use hostname instead of host in gemini network connection
|
2024-10-23 12:46:48 +03:00 |
|
|
|
f9b5fd5e7f
|
fix: Use parsedUrl.Hostname() for TLS SNI
|
2024-10-23 12:46:03 +03:00 |
|
|
|
62369d90ae
|
fix: Refactor ConnectAndGetData function to return GeminiPageData struct
|
2024-10-23 12:46:02 +03:00 |
|
|
|
e51d84cad8
|
feat: Implement robots.txt parser
|
2024-10-23 09:26:39 +03:00 |
|
|
|
17ef03d621
|
feat: Batch insert links to improve database performance
|
2024-10-23 09:05:50 +03:00 |
|
|
|
a2a6bd200a
|
Optimize worker random snapshot selection
|
2024-10-22 23:37:31 +03:00 |
|
|
|
3c5206ae43
|
Change blacklist to comprise domains.
|
2024-10-22 16:43:55 +03:00 |
|
|
|
cd60c1363b
|
Lots of features, first version that reliably crawls Geminispace.
- [x] Concurrent downloading with workers
- [x] Concurrent connection limit per host
- [x] URL Blacklist
- [x] Save image/* and text/* files
- [x] Configuration via environment variables
- [x] Storing snapshots in PostgreSQL
- [x] Proper response header & body UTF-8 and format validation
.
.
.
|
2024-10-21 20:04:09 +03:00 |
|
|
|
212345764b
|
Properly decode URLs
|
2024-10-10 18:39:27 +03:00 |
|
|
|
8278f2b204
|
Proper mimetype parsing, refactoring
|
2024-10-09 13:31:49 +03:00 |
|
|
|
91f8e69fdf
|
Work on header parsing & saving other files
|
2024-10-08 18:16:47 +03:00 |
|
|
|
7e849feffe
|
Add README.md
|
2024-10-08 17:28:10 +03:00 |
|
|
|
c3d6481de0
|
Add configuration via env vars
|
2024-10-08 12:42:08 +03:00 |
|
|
|
74e9327b0b
|
Persist pages to file system
|
2024-10-07 13:36:20 +03:00 |
|
|
|
74be6b4d0d
|
Basic functionality
|
2024-10-04 13:15:07 +03:00 |
|
|
|
eb963542b7
|
Initial commit.
|
2024-10-04 13:14:00 +03:00 |
|
