antanst dc6eb610a2 Add robots.txt parsing and matching functionality ...

- Create separate robotsMatch package for robots.txt handling
- Implement robots.txt parsing with support for different directives
- Add support for both Allow and Disallow patterns
- Include robots.txt matching with efficient pattern matching
- Add test cases for robots matching

2025-06-29 22:38:38 +03:00

..

robots_test.go

Add robots.txt parsing and matching functionality

2025-06-29 22:38:38 +03:00

robots.go

Add robots.txt parsing and matching functionality

2025-06-29 22:38:38 +03:00

robotsMatch_test.go

Add robots.txt parsing and matching functionality

2025-06-29 22:38:38 +03:00

robotsMatch.go

Add robots.txt parsing and matching functionality

2025-06-29 22:38:38 +03:00