antanst dc6eb610a2 Add robots.txt parsing and matching functionality ...

- Create separate robotsMatch package for robots.txt handling
- Implement robots.txt parsing with support for different directives
- Add support for both Allow and Disallow patterns
- Include robots.txt matching with efficient pattern matching
- Add test cases for robots matching

2025-06-29 22:38:38 +03:00

1.4 KiB

Raw Blame History

View Raw