3 Commits

Author	SHA1	Message	Date
antanst (aider)	6a96fb26cc	refactor: Improve robots.txt parsing and caching	2024-10-23 14:06:56 +03:00
antanst	3e01cb1819	refactor: Simplify robots.txt parsing logic	2024-10-23 14:06:55 +03:00
antanst (aider)	e51d84cad8	feat: Implement robots.txt parser	2024-10-23 09:26:39 +03:00