2 Commits

Author	SHA1	Message	Date
antanst	330b596497	Enhance crawler with seed list and SQL utilities ... Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.	2025-06-16 12:29:33 +03:00
antanst	8a9ca0b2e7	Add robots.txt parsing and matching functionality ... - Create separate robotsMatch package for robots.txt handling - Implement robots.txt parsing with support for different directives - Add support for both Allow and Disallow patterns - Include robots.txt matching with efficient pattern matching - Add test cases for robots matching	2025-05-22 12:46:21 +03:00