3 Commits

Author	SHA1	Message	Date
antanst	825c7e3391	.	2024-11-18 16:28:45 +02:00
antanst	c49a69728a	Simplify robots.txt parsing logic	2024-10-23 14:28:49 +03:00
antanst (aider)	e51d84cad8	feat: Implement robots.txt parser	2024-10-23 09:26:39 +03:00