2 Commits

Author	SHA1	Message	Date
antanst	8588414b14	Enhance crawler with seed list and SQL utilities ... Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.	2025-06-29 22:38:38 +03:00
antanst	dc6eb610a2	Add robots.txt parsing and matching functionality ... - Create separate robotsMatch package for robots.txt handling - Implement robots.txt parsing with support for different directives - Add support for both Allow and Disallow patterns - Include robots.txt matching with efficient pattern matching - Add test cases for robots matching	2025-06-29 22:38:38 +03:00