antanst

antanst pushed to main at antanst/gemini-grc

2025-01-16 09:53:53 +02:00

f26acc04fb Simplify IP pool and convert it to host pool

ea92b90c62 Break up Gemtext link parsing code and improve tests.

982fb75bd2 Add mode that prints multiple worker status in console

Compare 3 commits »

antanst pushed to main at antanst/gemini-grc

2025-01-04 15:40:07 +02:00

ccb8f6838e Update DB init instructions & README

antanst pushed to main at antanst/gemini-grc

2025-01-04 15:31:54 +02:00

4e6fad873b Break up common functions and small refactor.

b78fe00221 Add license.

90f6ecd024 Add README.md and Makefile.

b52df073e9 Add first version of gemini-grc.

93822b239e Initial commit.

Compare 5 commits »

antanst pushed to main at antanst/gemini-grs

2024-12-09 19:54:32 +02:00

43f2242558 Update README

aa4aecdc14 Preliminary web server

6cf507bdc9 DB scripts and migrations

7a36614232 Better error handling, many fixes all around

b52d4f6532 refactor: Update error message and remove commented JSON conversion methods

Compare 14 commits »

antanst pushed to main at antanst/gemini-grs

2024-11-05 12:45:33 +02:00

6bcc7081b2 Fix dependency on go-nanoid

f34ac651b7 Add Makefile and remove ad-hoc lint script

d5da9ac62d Better unicode conversion

Compare 3 commits »

antanst pushed to main at antanst/gemini-grs

2024-11-01 10:11:05 +02:00

a0563074ed Lint fixes.

antanst pushed to main at antanst/gemini-grs

2024-10-31 17:00:33 +02:00

ef3f009709 Add robots.txt checking

c49a69728a Simplify robots.txt parsing logic

8d9ea6cdec fix: use hostname instead of host in gemini network connection

f9b5fd5e7f fix: Use parsedUrl.Hostname() for TLS SNI

62369d90ae fix: Refactor ConnectAndGetData function to return GeminiPageData struct

Compare 18 commits »

antanst pushed to main at antanst/gemini-grc

2024-10-31 16:58:27 +02:00

ef3f009709 Add robots.txt checking

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:56:19 +03:00

561f83a007 Add robots.txt checking

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:30:09 +03:00

1e54df741d Add robots.txt checking

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:28:54 +03:00

02015faa81 Add robots.txt checking

c49a69728a Simplify robots.txt parsing logic

Compare 2 commits »

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:26:47 +03:00

daaa61c884 Add robots.txt checking

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:24:13 +03:00

094394afc2 Add robots.txt checking

antanst pushed to main at antanst/gemini-grc

2024-10-23 14:07:18 +03:00

1ac250ca6e Revert "refactor: Improve robots.txt parsing and caching"

6a96fb26cc refactor: Improve robots.txt parsing and caching

3e01cb1819 refactor: Simplify robots.txt parsing logic

8d9ea6cdec fix: use hostname instead of host in gemini network connection

f9b5fd5e7f fix: Use parsedUrl.Hostname() for TLS SNI

Compare 8 commits »

antanst pushed to main at antanst/gemini-grc

2024-10-22 23:37:34 +03:00

a2a6bd200a Optimize worker random snapshot selection

antanst pushed to main at antanst/gemini-grc

2024-10-22 16:44:00 +03:00

3c5206ae43 Change blacklist to comprise domains.

antanst pushed to main at antanst/gemini-grc

2024-10-22 15:50:43 +03:00

b36c117761 Change blacklist to comprise domains.

antanst pushed to main at antanst/gemini-grc

2024-10-21 20:04:15 +03:00

cd60c1363b Lots of features, first version that reliably crawls Geminispace.

antanst pushed to main at antanst/gemini-grc

2024-10-21 20:03:34 +03:00

fee7d3e01c Lots of features, first version that reliably crawls Geminispace.

antanst pushed to main at antanst/gemini-grs

2024-08-17 13:53:20 +03:00

4eed099154 Remove .idea