2 Commits

Author	SHA1	Message	Date
antanst	37d5e7cd78	Enhance crawler with seed list and SQL utilities ... Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.	2025-06-16 12:29:33 +03:00
antanst	fe40874844	Add robots.txt parsing and matching functionality ... - Create separate robotsMatch package for robots.txt handling - Implement robots.txt parsing with support for different directives - Add support for both Allow and Disallow patterns - Include robots.txt matching with efficient pattern matching - Add test cases for robots matching	2025-05-22 12:46:21 +03:00