7 Commits

Author	SHA1	Message	Date
antanst	aa2658e61e	Fix snapshot overwrite logic to preserve successful responses ... - Prevent overwriting snapshots that have valid response codes - Ensure URL is removed from queue when snapshot update is skipped - Add last_crawled timestamp tracking for better crawl scheduling - Remove SkipIdenticalContent flag, simplify content deduplication logic - Update database schema with last_crawled column and indexes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-06-18 11:23:56 +03:00
antanst	4e225ee866	Fix infinite recrawl loop with skip-identical-content ... Add last_crawled timestamp tracking to fix fetchSnapshotsFromHistory() infinite loop when SkipIdenticalContent=true. Now tracks actual crawl attempts separately from content changes via database DEFAULT timestamps. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-06-17 10:41:17 +03:00
antanst	f9024d15aa	Refine content deduplication and improve configuration	2025-06-16 17:09:26 +03:00
antanst	330b596497	Enhance crawler with seed list and SQL utilities ... Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.	2025-06-16 12:29:33 +03:00
antanst	bfaa857fae	Update and refactor core functionality ... - Update common package utilities - Refactor network code for better error handling - Remove deprecated files and functionality - Enhance blacklist and filtering capabilities - Improve snapshot handling and processing	2025-05-22 12:47:01 +03:00
antanst	9dc008cb0f	Use go_errors library everywhere.	2025-02-26 13:31:46 +02:00
antanst	4bceb75695	Reorganize code for more granular imports	2025-02-26 10:34:46 +02:00