antanst/gemini-grc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
54 Commits 1 Branch 0 Tags
f362a1d2da323f8b608a20230a75524adae6c1a4
Commit Graph

14 Commits

Author SHA1 Message Date
antanst
57eb2555c5 Improve crawler performance and logging 
- Optimize job scheduler to use NumOfWorkers for URL limits
- Clean up verbose logging in worker processing
- Update log messages for better clarity
 2025-06-29 22:38:38 +03:00
antanst
453cf2294a Update log message for clarity 
- Change "old content" to "old snapshot" for more accurate terminology
 2025-06-29 22:38:38 +03:00
antanst
ee2076f337 Clean up logging in worker processing 
- Move response code logging to happen after successful snapshot save
- Remove error message from log output for cleaner display
- Consolidate logging logic in saveSnapshotAndRemoveURL function
 2025-06-29 22:38:38 +03:00
antanst
acbac15c20 Improve crawler performance and worker coordination 
- Add WaitGroup synchronization for workers to prevent overlapping scheduler runs
- Increase history fetch multiplier and sleep intervals for better resource usage
- Simplify error handling and logging in worker processing
- Update SQL query to exclude error snapshots from history selection
- Fix worker ID variable reference in spawning loop
- Streamline snapshot update logic and error reporting
 2025-06-29 22:38:38 +03:00
antanst
ddbe6b461b Update log message to reflect crawl date update behavior 2025-06-29 22:38:38 +03:00
antanst
55bb0d96d0 Update last_crawled timestamp when skipping duplicate content and improve error handling 2025-06-29 22:38:38 +03:00
antanst
349968d019 Improve error handling and add duplicate snapshot cleanup 2025-06-29 22:38:38 +03:00
antanst
2357135d5a Fix snapshot overwrite logic to preserve successful responses 
- Prevent overwriting snapshots that have valid response codes
- Ensure URL is removed from queue when snapshot update is skipped
- Add last_crawled timestamp tracking for better crawl scheduling
- Remove SkipIdenticalContent flag, simplify content deduplication logic
- Update database schema with last_crawled column and indexes
 2025-06-29 22:38:38 +03:00
antanst
98d3ed6707 Fix infinite recrawl loop with skip-identical-content 
Add last_crawled timestamp tracking to fix fetchSnapshotsFromHistory()
infinite loop when SkipIdenticalContent=true. Now tracks actual crawl
attempts separately from content changes via database DEFAULT timestamps.
 2025-06-29 22:38:38 +03:00
antanst
8b498a2603 Refine content deduplication and improve configuration 2025-06-29 22:38:38 +03:00
antanst
8588414b14 Enhance crawler with seed list and SQL utilities 
Add seedList module for URL initialization, comprehensive SQL utilities for database analysis, and update project configuration.
 2025-06-29 22:38:38 +03:00
antanst
a8173544e7 Update and refactor core functionality 
- Update common package utilities
- Refactor network code for better error handling
- Remove deprecated files and functionality
- Enhance blacklist and filtering capabilities
- Improve snapshot handling and processing
 2025-06-29 22:38:38 +03:00
antanst
5b84960c5a Use go_errors library everywhere. 2025-02-26 13:31:46 +02:00
antanst
ca008b0796 Reorganize code for more granular imports 2025-02-26 10:34:46 +02:00