Libraries & frameworks
NetDiscovery
fengzhizi715/NetDiscovery
A general-purpose Java crawler framework and middleware built on Vert.x and RxJava 2, offering modular downloaders, distributed crawling, message-queue integrations and proxy/cookie pools.
Topics
Related projects
spider2.8kA concurrency-first Rust crawling engine that streams pages locally or at scale for search and RAG feeds.
retrofit-adapters558A set of Retrofit call adapters for Android that models network responses with Kotlin Result, Paging3, and Arrow Either.
EasySpider44.8kA free, open-source visual web scraper that lets anyone design and run data-collection tasks by clicking page elements, with optional command-line execution for embedding into other systems.
proxy_pool23.8kA self-hosted proxy IP pool for web crawlers. It periodically collects and validates free proxies and exposes them through a small HTTP API and CLI so scraping code can fetch working proxies.
TorBot5.2kTorBot is an OSINT crawler for the dark web. It crawls .onion sites, checks link liveness, builds link-relationship trees and exports JSON, with an optional desktop app and evidence-first AI analysis.
skills1.1kA collection of portable agent skills for Kotlin and Jetpack Compose development, installable into coding assistants.