Commit Graph

2 Commits

Author SHA1 Message Date
3e01cb1819 refactor: Simplify robots.txt parsing logic 2024-10-23 14:06:55 +03:00
e51d84cad8 feat: Implement robots.txt parser 2024-10-23 09:26:39 +03:00