antanst/gemini-grc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enhance crawler with seed list and SQL utilities

Browse Source 
Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.
This commit is contained in:
antanst
2025-06-16 12:29:33 +03:00
parent dfb050588c
commit 37d5e7cd78
37 changed files with 742 additions and 682 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
robotsMatch/robots.go
View File

@@ -7,7 +7,7 @@ import (
"gemini-grc/common/contextlog"
"gemini-grc/contextutil"
"gemini-grc/logging"
"git.antanst.com/antanst/logging"
)
// ParseRobotsTxt takes robots.txt content and a host, and