Commit Graph

2 Commits

Author SHA1 Message Date
c49a69728a Simplify robots.txt parsing logic 2024-10-23 14:28:49 +03:00
e51d84cad8 feat: Implement robots.txt parser 2024-10-23 09:26:39 +03:00