antanst 98d3ed6707 Fix infinite recrawl loop with skip-identical-content ...

Add last_crawled timestamp tracking to fix fetchSnapshotsFromHistory()
infinite loop when SkipIdenticalContent=true. Now tracks actual crawl
attempts separately from content changes via database DEFAULT timestamps.

2025-06-29 22:38:38 +03:00

..

blackList

Enhance crawler with seed list and SQL utilities

2025-06-29 22:38:38 +03:00

contextlog

Implement structured logging with slog

2025-06-29 22:38:38 +03:00

errors

Change errors to use xerrors package.

2025-06-29 22:38:38 +03:00

linkList

Update and refactor core functionality

2025-06-29 22:38:38 +03:00

seedList

Enhance crawler with seed list and SQL utilities

2025-06-29 22:38:38 +03:00

snapshot

Fix infinite recrawl loop with skip-identical-content

2025-06-29 22:38:38 +03:00

text

Improvements in error handling & descriptions

2025-02-27 09:20:22 +02:00

url

Enhance crawler with seed list and SQL utilities

2025-06-29 22:38:38 +03:00

whiteList

Enhance crawler with seed list and SQL utilities

2025-06-29 22:38:38 +03:00

shared.go

Update and refactor core functionality

2025-06-29 22:38:38 +03:00

worker.go

Fix infinite recrawl loop with skip-identical-content

2025-06-29 22:38:38 +03:00