Lib.rs

› Keywords #backup #encryption #compression #duplicate-file #restic #cli #logging

#deduplicate

  1. refine

    your file collections using Rust!

    v3.0.0 1.3K #file-rename #deduplicate #batch-file #batch-rename #file

  2. deduplicator

    find,filter and delete duplicate files

    v0.3.1 1.0K #duplicate-file #file-sorting #duplicates #filter #find-duplicates #deduplicate #exclude #follow-links #pwd #png

  3. rustic_core

    fast, encrypted, deduplicated backups that powers rustic-rs

    v0.9.0 650 #encryption #deduplicate #backup #restic #library

  4. dupe-krill

    An incremental file deduplicator which minimizes amount of data read. Replaces duplicate files with identical content with hardlinks.

    v1.5.0 #disk-space #deduplicate #dedupe #deduplication

  5. blobby

    Iterator over simple binary blob storage

    v0.4.0 32K #data-storage #deduplicate #iterator #file-format #binary-format

  6. dedup-cli

    An extremely fast and efficient duplicate file finder

    v0.3.1 #deduplicate #hash #file-finder #duplicate-file #finder #deduplication

  7. rustic-rs

    rustic - fast, encrypted, deduplicated backups powered by Rust

    v0.10.3 #deduplicate #encryption #restic #backup #cli

  8. rumina

    High-throughput UMI-aware deduplication of next-generation sequencing data

    v0.99.5 600 #bam #deduplicate #barcode #bioinformatics #sequencing

  9. rds2rust

    A pure Rust library for reading and writing R's RDS (R Data Serialization) files without requiring an R runtime

    v0.1.39 #serialization #rds #object #compression #gzip #string-interning #dataframe #deduplicate #amazon-s3 #s4

  10. sosorted

    A set of methods to efficiently manipulated sorted arrays

    v0.2.0 #sorting #array #methods #simd #deduplicate #dest #input-data #multiset #primitive-integer

  11. biblib

    Parse, manage, and deduplicate academic citations

    v0.3.2 #bibliography #deduplicate #doi #nbib #citation #deduplication

  12. backdown

    A smart CLI for removing thousands of duplicates on your disks

    v1.1.2 500 #deduplicate #disk #directory #staged #set #symlink

  13. vsb

    Very simple in configuring, but powerful backup tool

    v1.1.10 #cloud-storage #backup #deduplicate

  14. nostr-archive-cursor

    iterating over JSON-L archives

    v0.5.1 #archive #nostr #events #iterating #database #deduplicate #compression #zstd #sled-database #date

  15. subduction_cli

    CLI server and client for Subduction sync over WebSockets

    v0.3.0 #relay-server #metrics #sync-server #websocket #message #broadcast #logging #deduplicate #server-sockets #automerge-repo

  16. deduplicate

    caching, asynchronous, request deduplication

    v0.4.1 1.1K #cache #request #coalesce #caching #delegate

  17. reflicate

    Deduplicate data by creating reflinks between identical files

    v0.5.0 #deduplicate #reflink #filesystem #cli

  18. rustic_backend

    supporting various backends in rustic-rs

    v0.5.4 900 #deduplicate #restic #encryption #backup #library

  19. rustdupe

    Smart duplicate file finder with interactive TUI

    v0.2.0 #deduplicate #duplicate-file #tui #file-deduplication #deduplication

  20. anew

    adding new lines to files, skipping duplicates and write in Rust!

    v0.1.4 290 #deduplicate #web #security #deduplication

  21. gsm-idempotency

    Shared idempotency guard and storage interfaces for Greentic messaging workflows

    v0.4.43 #tenant #telegram #slack #idempotent #nats #ingress #greentic #deduplicate #whats-app #jetstream

  22. dset

    processing and managing dataset-related files, with a focus on machine learning datasets, captions, and safetensors files

    v0.1.12 750 #safetensors #json #caption #json-output #dataset #text-content #text-processing #deduplicate #json-processing #file-extension

  23. fhrn

    File Hash Renamer

    v0.1.1 #rename #cli #deduplicate #hash

  24. aws-sdk-keyspacesstreams

    AWS SDK for Amazon Keyspaces Streams

    v1.20.0 #aws-sdk #stream #keyspaces #change-data-capture #table #metrics #stream-data #api-reference #deduplicate #write-operations

  25. llm-optimizer-processor

    Data processing and transformation pipeline

    v0.1.1 #stream-processing #data-processing #standard-deviation #deduplicate #timestamp #event-time #performance-monitoring #llm #moving-average #sliding-window

  26. watto

    parsing and serializing Plain Old Data

    v0.2.0 867K #plain-old-data #parser #string-table #writer #serialization #parsing-and-serialization #output-buffer #deduplicate #string-serialization #aligning

  27. uniflight

    Coalesces duplicate async tasks into a single execution

    v0.1.0 #deduplicate #oxidizer #coalescing #singleflight #stempede

  28. dedups

    A fast and efficient file deduplication tool with support for media files

    v0.1.0 #deduplicate #hashing #duplicate-finder #file-deduplication #duplicate-file #hash #deduplication

  29. ivoryvalley

    A transparent deduplication proxy for Mastodon and the Fediverse

    v0.4.0 #activity-pub #deduplicate #mastodon #proxy #deduplication

  30. borgbackup

    A wapper for the borgbackup utility

    v0.10.0 #borg #backup #encryption #deduplicate #secure #en #compression #wapper #authenticated-encryption

  31. bees-prometheus-exporter

    Prometheus exporter for the bees deduplication daemon

    v2.0.0 140 #prometheus #daemon #bees #deduplicate #metrics-exporter #logging #debugging #statistics

  32. rustic_config

    configuration support in rustic-rs

    v0.2.3 220 #deduplicate #backup #restic #encryption #library

  33. bupstash

    Easy and efficient encrypted backups

    v0.12.0 #encryption #backup #decryption #deduplicate #tags #encryption-key #client-side

  34. czkawka-dupes-to-symlinks

    Safely turn Czkawka duplicate reports into space-saving symlinks

    v0.1.1 #symlink #deduplicate #czkawka #filesystem #cli

  35. fdedup

    Cross platform md5 based file deduplication tool

    v1.0.1 #deduplicate #cross-platform #dedup #file

  36. rsdupes

    A file deduplication utility

    v0.1.0 #deduplicate #file-deduplication #cli

  37. chunkfs

    An in-memory file system that can be used to compare different deduplication algorithms

    v0.1.3 190 #deduplicate #chunking #cdc #filesystem

  38. sprinter

    Run parallel queued tasks

    v0.3.0 500 #task-queue #parallel-task-execution #queued #run #concurrency #deduplicate

  39. formati

    Evaluate dot notation and arbitrary expressions in format! macros

    v0.1.4 260 #format-macro #expression #dot #string-formatting #notation #standard-formatting #deduplicate #expression-evaluation #dotted #logging

  40. backpak

    A content-addressed backup system with deduplication and compression

    v0.3.0 160 #backup #deduplicate #compression

  41. rust-gd

    Generalized Deduplication based on Error-Correcting Codes

    v0.2.3 #deduplicate #generalized #error-correcting-codes #compression #data-deduplication #hamming-code #lossless-compression

  42. qbice_storage

    The Query-Based Incremental Computation Engine

    v0.4.8 #rocksdb #wide-column #incremental-computation #sieve #interning #qbice #cache #thread-safe #eviction-algorithm #deduplicate

  43. iter-set

    Set operations on sorted, deduplicated iterators

    v2.0.2 800 #deduplicate #iterator #sorting #element #operation

  44. prestige-cli

    CLI interface for manually fetching and reading Prestige-parquet files

    v0.2.6 #parquet #command-line-interface #deduplicate #compact #fetching #compression #object-storage

  45. mcp-hub

    Fast hub for MCP tools

    v0.1.0 #mcp-tool #mcp-server #circuit-breaker #fault-tolerance #hub #blocklist #logging #hot-reloading #deduplicate #sub-processes

  46. backup-deduplicator

    deduplicate backups. It builds a hash tree of all files and folders in the target directory. Optionally also traversing into archives like zip or tar files. The hash tree is then used to find duplicate files and folders.

    v0.3.0 130 #hash-tree #deduplicate #file-deduplication #archive-management

  47. cdc-chunkers

    A collection of Content Defined Chunking algorithms

    v0.1.3 210 #deduplicate #chunking #cdc

  48. block-array-cow

    In memory array de-duplication, useful for efficient storing of a history of data versions

    v0.1.4 #storage #version #deduplicate #array #data-deduplication #data-structures #stride #in-memory #memory-data #block-size

  49. ilytix

    cli tool for images analysis, written in Rust

    v0.2.5 310 #image-analysis #command-line-tool #deduplicate #integrity-checks #checking #image-cli #incorrect #mv

  50. time-key-stream-set

    A time-keyed stream set

    v0.1.6 #deduplicate #stream-key #set-key #iot-data #timestamp #user-id #stream-data #device-id #u128 #data-structures

  51. deduplication

    efficiently store data

    v0.1.0 #deduplicate #save-file #file-deduplication #load-file #chunks #store-data #save-load #database

  52. request_coalescer

    An asynchronous request coalescing library for Rust

    v0.1.0 #concurrency #deduplicate #tokio #coalescing #async

  53. meza

    in-memory data table written in Rust

    v0.2.1 #data-table #in-memory-data #deduplicate #column #average #import-export #csv

  54. libecc

    Error-Correcting Codes for GD

    v0.2.2 #deduplicate #gd #hamming-code #codes #generalized #error-correcting-codes #reed-solomon

  55. yama

    Deduplicated, compressed and encrypted content pile manager

    v0.4.0 #deduplicate #encryption #pointers #pile #storage #store-path

  56. rmd

    An improved rm implementation able to remove duplicate files

    v0.5.3 #deduplicate #rm #fileutils #cli #remove

    57. Try searching with DuckDuckGo.

  57. vinculum

    Lock-Free Deduplication in Rust

    v0.1.0 #deduplicate #backup #deduplication

  58. dedup_signature

    implements TextProfileSignature and Lookup3 algorithms to generates a hash/signature/footprint in order to be used for detecting duplicate documents

    v0.2.1 #deduplicate #hash #lookup3 #signature #fuzzy #deduplication

  59. hld

    Hard Link Deduplicator

    v0.3.0 #deduplicate #cli #deduplication

  60. libasuran

    Deduplicating, encrypting, fast, and tamper evident archive format

    v0.0.3 #encryption #archive #deduplicate #compression #backup

  61. titan-utils

    Internal crate for the titan-family

    v0.4.2 1.6K #web-framework #titan-family #path-router #deduplicate #css-in-rust #javascript

  62. serde-intern

    A Serde addon that allows interning of strings and byte sequences behind Arcs during deserialization

    v1.0.0 #string-interning #deserialize #serde #plugin #byte-sequences #arc #data-structures #deduplicate #deserializer

  63. pyth-lazer-client

    client for Pyth Lazer

    v19.0.0 #lazer #price-feed #pyth #exponential-backoff #websocket #data-stream #deduplicate #cache #api-client #redundancy

  64. archival-dedupe

    Deduplicate read-only files on a UNIX filesystem

    v1.0.0 #unix-filesystem #deduplicate #read-only #original

  65. fastchr

    Faster memchr using SIMD intrinsics

    v0.3.0 #simd-intrinsics #deduplicate #memchr #detect #run-time-cpu-features #deduplicator #occurrence #input-file

  66. derive_aliases_proc_macro

    detail of derive_aliases crate

    v0.4.7 110 #alias #derive-alias #macro-derive #define #partial-eq #deduplicate

  67. dsc

    cli tool for finding and removing duplicate files on one or multiple file systems, while respecting your gitignore rules

    v0.1.3 #deduplicate #cmp #du #duplicates

  68. car-mirror

    CAR Mirror protocol

    v0.1.0 #ipld #car #mirror #transfer-protocol #protocols #deduplicate #sans-io

  69. multi-machine-dedup

    Deduplication tool using SQLite to allow multi-machine features

    v0.2.0 #deduplicate #cli #dedup

  70. stable-bloom-filter

    A Rust-implementation of a stable Bloom filter for filtering duplicates out of data streams

    v0.3.0 #bloom-filter #filtering #stream #sbf #deduplicate

  71. filedupes

    Deduplicate sets of files

    v0.1.1 #deduplicate #set #hashing #sub-directory #vector #closest #file-check

  72. btrfs-dedupe

    BTRFS whole-file deduplication tool

    v1.0.2 #deduplicate #btrfs #dedupe #io #hash #sha-256 #gitlab #clone-repository

  73. cargo-deduplicate-warnings

    Deduplicate warning messages in the cargo json output

    v0.1.0 #json-output #deduplicate #warnings #cargo #message #strip

  74. dedup-advanced

    Fast and accurate deduplication tool to be used with blockhash

    v1.2.0 #block-hash #deduplicate #tool #accurate #list #xargs #regex #printf #bash

  75. cleanup-cli

    Recursively find and remove duplicate files in a target directory

    v0.1.2 #deduplicate #target-directory #recursion #find-and-remove #input

  76. idar

    Image deduplication and removal tool

    v0.3.0 #deduplicate #image #directory #tool #image-cli #image-processing #image-hashing

  77. datman

    A chunked and deduplicated backup system using Yama

    v0.1.0 #deduplicate #backup #system #yama #chunked

  78. dupfind

    Duplicate Finder to identify and remove duplicate files

    v1.0.0 #deduplicate #file

  79. woopdedupe

    Aggressively deduplicate files in a directory

    v0.1.5 #directory #aggressively #dedupe #deduplicate

  80. uq

    sort | uniq alternative

    v0.1.2 #uniq #sorting #deduplicate #print #user-friendly