antanst 8a9ca0b2e7 Add robots.txt parsing and matching functionality ...

- Create separate robotsMatch package for robots.txt handling
- Implement robots.txt parsing with support for different directives
- Add support for both Allow and Disallow patterns
- Include robots.txt matching with efficient pattern matching
- Add test cases for robots matching

2025-05-22 12:46:21 +03:00

1.3 KiB

Raw Blame History

View Raw