-
spider_chromiumoxide_cdp
Contains all the generated types for chromiumoxide
-
git-leave
Check for unsaved or uncommitted changes on your machine
-
firecrawl
Rust SDK for Firecrawl API
-
crawn
web crawling and scraping
-
spider-client
Spider Cloud client
-
kodegen_tools_citescrape
KODEGEN.ᴀɪ: Memory-efficient, Blazing-Fast, MCP tools for code generation agents
-
wdict
Create dictionaries by scraping webpages or crawling local files
-
spider-cloud-cli
The Spider Cloud CLI for web crawling and scraping
-
spider-pipeline
Pipeline implementations for the spider-lib web scraping framework
-
spider-lib
A Rust-based web scraping framework inspired by Scrapy (Python)
-
yps
Yggdrasil Port Scanner
-
toai
path crawler, that copies all SRC files into a singe output to send it to a ai (toai)
-
mq-crawler
Directory crawler for batch Markdown file processing
-
fav_core
Fav's core crate; A collection of traits
-
spider-core
Core functionality for the spider-lib web scraping framework
-
spider_chromiumoxide_types
Contains the essential types necessary for using chromiumoxide
-
siteprobe
CLI tool to fetch URLs from sitemap.xml, check their existence, and generate performance reports
-
spider_transformations
Transformation utils to use for spider
-
product-os-crawler
Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation
-
robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library
-
spider-middleware
Middleware implementations for the spider-lib web scraping framework
-
silkworm-rs
Async-first web scraping framework (Rust port)
-
spider_worker
The fastest web crawler as a worker or proxy
-
spider-util
Shared utility functions and types for the spider-lib ecosystem
-
iocaine
The deadliest poison known to AI
-
halldyll-core
Core scraping engine for Halldyll - high-performance async web scraper for AI agents
-
tarzi
Rust-native lite search for AI applications
-
pulsarss
RSS Aggregator for Gemini Protocol
-
spider_cli
The fastest web crawler CLI written in Rust
-
firecrawl-sdk
Rust SDK for Firecrawl API
-
crawlurls
A fast async Rust crawler that discovers and filters URLs by pattern without scraping content
-
json-crawler
Wrapper for serde_json that provides nicer errors when crawling through large json files
-
inverta
A basic search engine that downloads pages and matches your search query against their contents
-
robotxt
Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions
-
website_crawler
gRPC tokio based web crawler built with spider
-
seaward
grep-like tool for the web
-
crawly
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting
robots.txtrules -
wrake
Collect links from the given URL
-
actix_block_ai_crawling
Actix Middleware that blocks Generative AI crawlers
-
ungoliant
The pipeline for the OSCAR corpus
-
omnivore-cli
Universal web scraper and code extractor CLI - crawl websites, analyze repositories, build knowledge graphs
-
minisearchtk
Small toolkit for crawling and searching web pages
-
ferrisfetcher
A cutting-edge, high-level web scraping library crafted in Rust
-
shader-prepper
Shader include parser and crawler
-
aquatic-crawler
Crawler tool for the Aquatic BitTorrent tracker API
-
spider_utils
Spider Web Crawler
-
scoutlang
A web crawling programming language
-
capp
Common things i use to build Rust CLI tools for web crawlers
-
unobtanium-crawler
The default web-crawler for unobtanium
-
coma
lightweight command-line tool designed for crawling websites
-
unobtanium
Opinioated Web search engine library with crawler and viewer companion
-
headless_chrome_fork
Control Chrome programatically
-
voyager
Web crawler and scraper
-
mitsuba
Lightweight 4chan board archive software (like Foolfuuka), in Rust
-
recursive_scraper
Constant-frequency recursive CLI web scraper with frequency, filtering, file directory, and many other options for scraping HTML, images and other files
-
turboscraper
A high-performance, concurrent web scraping framework for Rust with built-in support for retries, storage backends, and concurrent request handling
-
scout-lexer
A web crawling programming language
-
s5_importer_http
HTTP importer for S5
-
surly-spider
A command line interface for crawling websites
-
chan-downloader
CLI to download all images/webms of a 4chan thread
-
firecrawl_rs
Rust SDK for Firecrawl API
-
wappu
fast and flexible web scraping library for Rust, designed to efficiently navigate and extract data from websites. Perfect for data mining, content aggregation, and web automation tasks.
-
indexea
OpenAPI of Indexea
-
spider_scraper
A css scraper using html5ever
-
async_job
async cron job crate for Rust
-
frangipani
Scraping framework for rust
-
quick_crawler
QuickCrawler is a Rust crate that provides a completely async, declarative web crawler with domain-specific request rate-limiting built-in
-
brchd
Data exfiltration toolkit
-
spacebar
An anti-plagiarism tool based on null width characters
-
omnivore-core
Core crawler and knowledge graph engine for Omnivore - web scraping, AI extraction, browser automation
-
maman
Rust Web Crawler
-
rsfile
operate files or web pages easily and quickly
-
spidery
Rust SDK for Spidery API
-
web-crawler
Finds every page, image, and script on a website (and downloads it)
-
ptt-crawler
A crawler for the web version of PTT, the largest online community in Taiwan
-
ntwrk
TODO
-
jsdom
javascript dom parser for web scraping
-
finde-rs
Multi-threaded filesystem crawler
-
stream_crawler
scraping web pages and extracting URLs and endpoints
-
url-crawl
URL crawler for HTML code
-
krate
Get information and metadata for published Rust crates
-
actix-prerender
Actix middleware that sends requests to Prerender.io or a custom Prerender service URL
-
od-get
recursively crawling & downloading data from open directories
-
rust-rock-rover
Concert web crawler in Rust
-
ssufid
SSU Announcement Crawler for Everyone
-
kodict
Korean Dictionary Implements and Crawler for Rust
-
doublesite
Alternative for httrack
-
wls
Easily crawl multiple sitemaps and list URLs
-
crawl
Rust crawl
-
lolchive
local liminal archiver for webpages
-
source-demo-tool-crawler
WIP: a gui tool for opening (editing planned) source engine demo files
-
spire
The flexible scraper framework powered by tokio and tower
-
emails
A web scraper to extract email addresses from websites
-
sws-crawler
Web crawler with plugable scraping logic
-
flatcrawl-crawler
set of webpage crawlers. New crawlers can be easily configured and the output can be written to an AMQP queue.
-
scraper_query
Ergonomic Query for HTML with Scraper
-
task_deport
Organize simple task queue
-
gar-crawl
High level HTML crawler with concise builder
-
ac_crawler_types
normalized types for the anti capital public data crawlers
-
labisu
implementing algorithms finding large bipartite subgraphs
-
dblp_crawler
DBLP Crawler
-
spire-macros
Macros for spire
-
crawler_data_client
client for programmatic download of crawler data
-
pop-os/apt-repo-crawler
crawling through files in an apt repo
-
scrupy
fast, modern spider framework written in and for Rust. The framework implements the functionalities of Scrapy, but is low-level and typesafe. It exposes an elegant API and uses zero unsafe code.
-
karkinos
Powerful and flexible web scraper with YAML configuration, supporting pagination, data transformations, caching, and multiple output formats
Try searching with DuckDuckGo.