5 Commits

Author	SHA1	Message	Date
antanst	2357135d5a	Fix snapshot overwrite logic to preserve successful responses ... - Prevent overwriting snapshots that have valid response codes - Ensure URL is removed from queue when snapshot update is skipped - Add last_crawled timestamp tracking for better crawl scheduling - Remove SkipIdenticalContent flag, simplify content deduplication logic - Update database schema with last_crawled column and indexes	2025-06-29 22:38:38 +03:00
antanst	98d3ed6707	Fix infinite recrawl loop with skip-identical-content ... Add last_crawled timestamp tracking to fix fetchSnapshotsFromHistory() infinite loop when SkipIdenticalContent=true. Now tracks actual crawl attempts separately from content changes via database DEFAULT timestamps.	2025-06-29 22:38:38 +03:00
antanst	c54c093a10	Implement context-aware database operations ... - Add context support to database operations - Implement versioned snapshots for URL history - Update database queries to support URL timestamps - Improve transaction handling with context - Add utility functions for snapshot history	2025-06-29 22:38:38 +03:00
antanst	ca008b0796	Reorganize code for more granular imports	2025-02-26 10:34:46 +02:00
antanst	4e6fad873b	Break up common functions and small refactor.	2025-01-04 15:31:26 +02:00