antanst 561f83a007 Add robots.txt checking ...

Still needs periodic cache refresh

2024-10-23 14:56:14 +03:00

..

connectionPool.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

files.go

Change blacklist to comprise domains.

2024-10-22 16:43:55 +03:00

gemini_url.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

gemini.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

ip-address-pool.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

network_test.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

network.go

fix: use hostname instead of host in gemini network connection

2024-10-23 12:46:48 +03:00

persistence.go

feat: Batch insert links to improve database performance

2024-10-23 09:05:50 +03:00

processing_test.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

processing.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

robotmatch.go

Add robots.txt checking

2024-10-23 14:56:14 +03:00

robots_test.go

Add robots.txt checking

2024-10-23 14:56:14 +03:00

robots.go

Simplify robots.txt parsing logic

2024-10-23 14:28:49 +03:00

snapshot.go

Lots of features, first version that reliably crawls Geminispace.

2024-10-21 20:04:09 +03:00

worker.go

Add robots.txt checking

2024-10-23 14:56:14 +03:00