Commit Graph

2 Commits

Author SHA1 Message Date
1e54df741d Add robots.txt checking
Still needs periodic cache refresh
2024-10-23 14:30:05 +03:00
e51d84cad8 feat: Implement robots.txt parser 2024-10-23 09:26:39 +03:00