2 Commits

Author	SHA1	Message	Date
antanst	2357135d5a	Fix snapshot overwrite logic to preserve successful responses ... - Prevent overwriting snapshots that have valid response codes - Ensure URL is removed from queue when snapshot update is skipped - Add last_crawled timestamp tracking for better crawl scheduling - Remove SkipIdenticalContent flag, simplify content deduplication logic - Update database schema with last_crawled column and indexes	2025-06-29 22:38:38 +03:00
antanst	98d3ed6707	Fix infinite recrawl loop with skip-identical-content ... Add last_crawled timestamp tracking to fix fetchSnapshotsFromHistory() infinite loop when SkipIdenticalContent=true. Now tracks actual crawl attempts separately from content changes via database DEFAULT timestamps.	2025-06-29 22:38:38 +03:00