Lib.rs

› Keywords #async-stream #async-pipeline #pipeline #rust #stream-processing #fluvio

#data-processing

  1. block-buffer

    Buffer types for block processing of data

    v0.11.0 31.4M #data-processing #buffer #block

  2. yamlscript

    Program in YAML — Code is Data

    v0.2.8 #functional-programming #yaml-data #yaml-processing #data-processing #load-file #scripting-language #dynamic-configuration #ys #config-file

  3. frd_pu

    A high-performance, zero-dependency library for creating efficient and fast applications

    v0.2.7 1.3K #zero-dependency #data-processing #memory-efficient #high-performance #rust

  4. feagi-sensorimotor

    Sensorimotor System - Data processing, caching, and neuron voxel encoding for FEAGI agents (sensory input and motor output)

    v0.0.1-beta.9 450 #data-processing #cache #voxel #data-pipeline #feagi #neuron #image-processing #motor #sensory #processing-pipeline

  5. cano

    High-performance orchestration engine for building resilient, self-healing systems in Rust. Uses Finite State Machines (FSM) for strict, type-safe transitions.

    v0.7.2 #scheduler #data-processing #workflow

  6. data-url

    Processing of data: URL according to WHATWG’s Fetch Standard

    v0.3.2 1.0M #data-processing #url-standard #whatwg #parse-url #fetch #mime

  7. ushcn

    US Historical Climatology Network data downloader

    v0.2.5 280 #parquet #downloader #station #data-processing #daily #historical #lat-lon #climate #cache #parallel-processing

  8. neuromorphic-types

    Shared Neuromorphic data types for drivers and processing frameworks

    v0.6.0 380 #driver #data-processing #events #camera #neuromorphic #shared-data #real-time #rate-limiting #usb

  9. enc

    aids in processing encoded data

    v0.18.0 #data-processing #encoded #aids #hex-encoding #varint #base64 #urlencode

  10. fluvio-extension-common

    Fluvio extension common

    v0.50.1 2.1K #fluvio #extension #data-processing #real-time-processing #events #data-pipeline #event-driven-architecture #stream-processing #data-integration #cloud-native

  11. fluvio-stream-dispatcher

    Fluvio Event Stream access

    v0.50.1 1.2K #event-streaming #fluvio #stream-processing #data-processing #real-time-data-processing #event-stream #distributed-systems #cloud-native #stream-processing-engine #dataflow

  12. razel

    a command executor with caching for data processing pipelines

    v0.5.6 #data-processing #local-cache #command-executor #remote-execution #sandbox #task-execution #execute-command

  13. reflux

    A microservice framework aimed at scalability, flexibility and ease of use

    v6.4.2 3.2K #microservices #framework #extractor #distributed #data-processing #broadcast #funnel #distributed-computing #ensuring #messenger

  14. fluvio-types

    Fluvio common types and objects

    v0.50.1 2.0K #fluvio #distributed-systems #stream-processing #data-processing #cloud-native #real-time-data-processing #dataflow #wasm #stream-processing-engine #event-processing

  15. udataframe_rs

    A pure Rust library for data frame operations, particularly useful for processing data extracted from PDF files or OCR recognize

    v0.1.5 #dataframe #ocr #data-processing #pdf

  16. batch-processing

    A batch library for processing a list of items in parallel

    v0.1.17 1.7K #data-processing #step #parallel-data-processing #list-of-items #async-data-processing #low-memory

  17. fluvio-package-index

    Fluvio Package Index

    v0.50.1 1.3K #fluvio #latest #stream-processing #cloud-native #data-processing #distributed-systems #real-time-data-processing #dataflow #stream-processing-engine #streaming-processing

  18. ppl

    A structured parallel programming library for Rust

    v0.1.6 410 #thread-pool #pipeline #data-processing #framework #dataset #word-counter #concurrency #farm #data-parallelism

  19. veloxx

    High-performance, lightweight Rust library for in-memory data processing and analytics. Features DataFrames, Series, advanced I/O (CSV, JSON, Parquet), machine learning (linear regression…

    v0.4.0 #dataframe #data-processing #analytics #rust

  20. jsonschema-transpiler

    transpile JSON Schema into schemas for data processing

    v2.0.0 120 #json-schema #data-processing #avro-schema #json-schema-validation #big-query #sql #serialization #transpile #jq

  21. dyer-cli

    great tool created to guide you use dyer fast and at ease, helps you build a robust crawler, data processor, netwrok program fast and correctly

    v0.6.0 4.2K #web-framework #data-processing #request-response

  22. dapt

    serializing and unseralizing unstructured data

    v0.1.9 650 #serialization #dynamic-data-structure #stream-processing-engine #deserialize #data-processing #object-serialization

  23. rmrs

    parallel processing and data processing

    v0.1.0 #parallel-processing #data-processing #rf #rm #cross-platform

  24. recoco

    A Rust-only fork of CocoIndex, a modular, high-performance ETL and data transformation library

    v0.2.1 #etl-pipeline #data-processing #pipeline #cocoindex

  25. feagi_connector_core

    Adds Caching and Processing systems that are used by Feagi Connector

    v0.0.50-beta.60 #feagi #cache #connector #data-processing #collection

  26. llm-optimizer-processor

    Data processing and transformation pipeline

    v0.1.1 #stream-processing #data-processing #standard-deviation #deduplicate #timestamp #event-time #performance-monitoring #llm #moving-average #sliding-window

  27. feagi_core_data_structures_and_processing

    FEAGI libraries to use to handle data processing

    v0.0.50-beta.9 3.9K #data-processing #feagi #neuron #serialization #cortical #input-data #output-processing #sensory #centralized #deserialize

  28. minifix

    A bare essentials library for Financial Information Exchange (FIX)

    v0.3.0 #fix-protocol #message-decoder #finance-exchange #market-data #data-processing

  29. jsonkit

    A high-performance, lightweight Rust utility library engineered for robust JSON data processing

    v0.2.2 #json #json-validation #data-processing #engineered-for-robust #utility #strip-comments #prettify #json-format

  30. streamweave

    Composable, async, stream-first computation in pure Rust

    v0.9.0 #string-interning #data-processing #async-stream #data-pipeline #rust #stream-processing #async-pipeline #async-data-processing #async-stream-processing

  31. fluvio-spu-schema

    Fluvio API for SPU

    v0.50.1 1.2K #fluvio #spu #api-for-spu #distributed-systems #data-processing #stream-processing #event-driven-architecture #real-time-data-processing #data-pipeline #cloud-native

  32. yngvi

    Read, calculate and write weather data

    v0.4.3 310 #weather #station #display #data-processing #read

  33. tremor-script

    Tremor Script Interpreter

    v0.13.0-rc.33 3.1K #tremor #event-processing #script-interpreter #scripting-language #data-processing #pattern-matching #scripting-engine

  34. e9571_file_lib

    file and directory operations, including MD5 hashing and recursive copying

    v0.1.2 #md5 #data-processing #filesystem #directory

  35. fluvio-sc-schema

    Fluvio API for SC

    v0.50.1 1.2K #fluvio #sc #real-time-streaming #api-for-sc #data-processing #data-pipeline

  36. panopticon-core

    An extensible data processing and templating engine

    v0.2.1 #templating #data-processing #pipeline #polars #etl

  37. fluvio-controlplane-metadata

    Metadata definition for Fluvio control plane

    v0.50.1 1.2K #control-plane #fluvio #define #metadata #data-processing #real-time-events

  38. track-rails

    Rust Version of TrackRails Protocol

    v0.1.2 700 #data-processing #stream-processing #protocols #processing-stream

  39. gt7_server

    A server for GT7 telemetry data processing

    v1.1.1 170 #telemetry-data #gt7 #websocket #socket-server #data-processing #udp-socket

  40. ibuf

    A memory buffer pool implementation

    v0.3.1 220 #memory-pool #buffer-pool #data-processing #performance-data #mbuf

  41. e9571_json_lib

    JSON serialization, deserialization, and non-standard JSON parsing

    v0.1.1 #json-parser #json #deserialize #data-processing #serialization

  42. table-to-csv

    Parsley CSV is a command-line tool that converts SQL database dumps to CSV files

    v0.4.0 #csv #database #data-processing

  43. egml

    processing GML data

    v0.0.1-alpha.11 #data-processing #gml #early

  44. egraphics

    processing graphics formats in 3D space

    v0.0.1-alpha.9 #3d-space #format #data-processing #graphics #gltf #serialization #model-name #format-variant

  45. e9571_config-reader

    read config.json from executable directory

    v0.1.1 #config-json #config #data-processing #json

  46. ecitygml

    processing CityGML data

    v0.0.1-alpha.11 #city-gml-data #data-processing #io #model

  47. evoxel

    processing 3D point clouds

    v0.0.1-alpha.9 #point-cloud-processing #voxel #3d-point-cloud #grids #data-processing

  48. ecad-processor

    High-performance multi-metric weather data processor for European Climate Assessment & Dataset (ECA&D) archives with Parquet output

    v2.0.1 #data-processing #parquet #climate #weather #ecad

  49. streamweave-error

    Error handling system for StreamWeave

    v0.6.0 #data-processing #async-stream #async #rust #pipeline #async-pipeline

  50. work_dispatcher

    concurrent data processing framework

    v0.1.1 #data-processing #actor #actor-concurrency #producer-consumer

  51. ecitygml-cli

    CLI tool for processing CityGML data

    v0.0.1-alpha.11 #data-processing #city-gml-data #tool-for-processing #command-line-tool

  52. par-stream

    Asynchronous parallel streams analogous to rayon

    v0.10.2 3.0K #async-stream #stream-processing #parallel-worker #stream-combinator #data-processing #tokio-runtime #async-std-runtime #broadcast #analogous

  53. suma_core

    A core library for suma_ulsa, used for academics methods and data processing

    v0.1.18 #data-processing #methods #academics #suma-ulsa

  54. reifydb-sub-flow

    Flow subsystem for stream processing and data flows

    v0.2.0 #database #reify-db #flows #data-processing #stream #processing-stream #web-server

  55. decode

    A High-Performance Query Tool for JSON, YAML and TOML

    v0.1.0 #json-query #json #json-path #data-processing

  56. egml-cli

    CLI tool for processing GML data

    v0.0.1-alpha.11 #data-processing #gml #command-line-tool #tool-for-processing

  57. etiles-cli

    CLI tool for processing 3D Tiles data

    v0.0.1-alpha.9 #data-processing #tile #3d #tool-for-processing #command-line-tool

  58. dyer

    designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed

    v3.3.2 53K #web-scraping #data-processing #request-response #web-crawling

  59. scbl-utils

    A command-line utility for data processing and delivery at the Single Cell Biology Lab at the Jackson Laboratory

    v0.1.8 #data-processing #single-cell #biology #laboratory #lab #jackson #cache #command-line-tool

  60. byte-tools

    Bytes related utility functions

    v0.3.1 405K #cryptography #byte #data-processing

  61. resol-vbus

    processing RESOL VBus data

    v0.2.1 #v-bus #data-processing #resol

  62. etiles

    processing 3D Tiles data

    v0.0.1-alpha.9 #tile #3d #data-processing

  63. epiq

    Laboratory for pipeline construction with feedback

    v0.1.0 #pipeline #data-processing #construction-with-feedback #laboratory #stream #jq #data-analysis

  64. mongo-collector

    CLI tool for extracting values of a specific field from a MongoDB collection and saving them into a target collection

    v0.1.2 #mongo-db #collector #data-processing #aggregation #cli

  65. ngs

    Command line tool for processing next-generation sequencing data

    v0.4.0 #next-generation-sequencing #quality-control #bam-file #data-processing #genome #data-analysis #metrics #index-file #command-line-tool #plot

  66. e9571_base64_pro_lib

    base64 encoding/decoding with special character handling and file conversion support

    v0.1.0 #base64 #file-conversion #data-processing

  67. jedisct1/flowgger

    fast, simple and lightweight data collector

    GitHub 0.3.3 #data-collector #kafka #logging #data-processing #validation #log-entries #graylog #redis #udp #capnp

  68. fluvio-system-util

    Fluvio System components

    v0.1.1 #fluvio #stream-processing #distributed-systems #data-processing #cloud-native #system-utilities #stream-processing-engine #real-time-data-processing #dataflow #streaming-processing

  69. egml-io

    IO operations for processing GML data

    v0.0.1-alpha.11 #data-processing #iops #gml #io #operation

  70. fluxus

    lightweight stream processing engine written in Rust, designed for efficient real-time data processing and analysis

    v0.2.0 #stream-processing-engine #processing-analysis #real-time-data-processing #data-processing #processing-stream #data-analysis

  71. ecitygml-io

    IO operations for processing CityGML data

    v0.0.1-alpha.11 #iops #city-gml-data #data-processing #io #operation

  72. ecitygml-transform

    Supplementary operations for CityGML data

    v0.0.1-alpha.11 #city-gml-data #operation #supplementary #data-processing

  73. noir-compute

    Network of Operators In Rust

    v0.2.0 #data-processing #stream-processing #operator #distributed-data-processing #block #noir #distributed-systems #dataflow-graph #dataflow-programming

  74. fluvio-runner-local

    Fluvio Engine Runner

    v0.3.0 #fluvio #stream-processing #distributed-systems #data-processing #wasm #dataflow #real-time #real-time-processing #data-pipeline

  75. rsflow

    High-performance Rust flow processing engine

    v0.1.0 #stream-processing #data-processing #artificial-intelligence #stream-processing-engine #sql #kafka #real-time-data-processing #event-processing #data-stream #redis

  76. egml-transform

    Supplementary operations for GML data

    v0.0.1-alpha.11 #gml #supplementary #operation #data-processing

  77. black-jack

    DataFrame / Series / general data processing in Rust

    v0.1.0 #dataframe #data-processing #series #principles #memory-data #long-term-goal

  78. wikidump

    parsing Mediawiki XML dumps

    v0.3.1 270 #mediawiki #xml #data-processing #language-analysis #dump #wikipedia #backup #revision #article #nlp

  79. opstream

    that involves smaller very simplified operations, normally triggered by processing data

    v0.1.0 #data-processing #operation #involve #normally #smaller

  80. tea-dyn

    that primarily defines traits for dynamic vector operations, enabling flexible data processing across various types and backends

    v0.5.1 230 #back-ends #traits #tevec #financial-analysis #data-processing #finance #polars #vector-operations #multi-dimensional-array #vec-deque

  81. hyperlog_simd

    A SIMD optimized implementation of the HLL and HLL++ algorithms

    v0.2.0 #simd-optimized #hyper-log-log #hll #serialization #data-processing #serde #distinct

  82. rusty-talib

    using pure rust code for technical analysis

    v0.1.6 #technical-analysis #talib #data-processing #analysis #indicator

  83. pipeline_derive

    A procedural macro crate to simplify building monadic-like data pipelines in Rust

    v0.1.1 #macro-derive #proc-macro #monadic #data-processing #processing-pipeline #data-pipeline

  84. dr

    Command-line data file processing in Rust

    v0.7.0 #csv #parquet #data-processing #command-line-data #sql #file-processing

  85. strm-privacy-driver

    Rust API driver for STRMPrivacy

    v1.0.1 #driver #strm-privacy #data-processing #schema #api #authentication #data-privacy #data-stream #security-privacy

  86. streaming_serde_json

    streaming alternative to serde_json for processing JSON data sources that are too large to fit in memory

    v0.1.2 #streaming-json #serde-json #data-source #data-processing #key-value #json-processing #json-array

  87. sanitise

    Headache-free data clean-up

    v0.4.0 #data-processing #yaml-config

  88. j9

    Rust interface for jq-based JSON processing

    v0.1.4 290 #json-processing #jq #data-processing #interface #bindings

  89. mako

    main Sidekick AI data processing library

    v0.3.0 #artificial-intelligence #data-processing #tokenize #data-loader #machine-learning #batch-processing #dataflow #sidekick #tokenized

  90. async-pipes

    building concurrent data processing pipelines

    v0.3.1 #data-processing #data-pipeline #tokio #async-pipeline #async-data

  91. stdf

    Standard Test Data Format (STDF) serialization and data processing

    v0.2.0 #data-processing #semiconductor #manufacturing #test-equipment

  92. inexor-rgf-model-string

    Inexor - Reactive Graph Flow - Model - String

    v0.9.1 #reactive-graph-flow #reactive-programming #inexor #model #data-processing #git #dataflow

    93. Try searching with DuckDuckGo.

  93. gex_sys_build_parse_mak

    Parse mak file to build gex-sys

    v0.1.0 #parallel-processing #gex-sys #parser #data-processing #framework #apgas #address-space #udp #crayfish #process-id

  94. std-dev

    Your Swiss Army knife for swiftly processing any amount of data. Implemented for industrial and educational purposes alike.

    v0.1.0 #standard-deviation #swiss-army #linear-time #data-processing #amount #swiftly #input-data #plot #percentile #n2

  95. recoco-core

    core library of Recoco; it's nearly identical to the main ReCoco crate, which is a simple wrapper around recoco-core and other sub-crates

    v0.2.1 #etl-pipeline #data-processing #pipeline #cocoindex #etl

  96. wd2sql

    Transform a Wikidata JSON dump into an SQLite database

    v0.1.0 #sqlite #wikidata #json #dump #data-processing #entity-id #database #json-parser #allocator #simd-accelerated

  97. etiles-io

    IO operations for processing 3D Tiles data

    v0.0.1-alpha.9 #3d #data-processing #io #tile #operation #iops

  98. vehicle_parser

    Parse data about vehicles to json

    v0.1.0 #parser #data-processing #vehicle #json #car #grammar #text-input #grammar-rules #model-format #string-parser

  99. byte_array

    that supports Java-like series processing with byteorder

    v0.1.3 #byteorder #data-processing #java-like #series #format #ba

  100. rusted_pipe

    Real time processing library for developing multithreaded ML pipelines, written in Rust

    v0.0.2 #graph-node #metrics #pipeline #processing-time #data-processing #real-time #real-time-processing #ml #multi-threading #rusted

  101. flowing

    flow graph processing library

    v0.2.1 #graph #dataflow-programming #data-processing #dsp #flowgraph

  102. ts-merger

    A multi-threaded tool for processing and aggregating time series data files

    v0.2.0 #time-series-data #csv #data-processing #time-series #aggregation #processing-time

  103. shepherd-rs

    Shepherd is a resilient, non-blocking orchestrator that persistently transforms and delivers data—built for remote, compute-heavy workloads

    v0.2.0 110 #devops #orchestration #scheduler #data-processing

  104. fluxus-utils

    components for Fluxus stream processing engine

    v0.2.0 160 #data-processing #async #stream-processing #real-time-data-processing #real-time

  105. fluxus-sinks

    Sink components for Fluxus stream processing engine

    v0.2.0 110 #stream-processing #data-processing #real-time-data-processing #real-time #analytics #real-time-analytics

  106. fluxus-transformers

    Transformer components for Fluxus stream processing engine

    v0.2.0 120 #data-processing #stream-processing #async #real-time-analytics #real-time-data-processing #real-time

  107. codebake

    data processing framework and lisp

    v0.1.1 #lisp #data-processing #interpreter #recipe #framework #dish #rot13 #toy #bake #reference-documentation

  108. punch_api

    Punch WebAssembly public API

    v1.0.0 #punch #faas #log-management #data-processing

  109. fluxus-sink-telegram

    A Telegram sink component for the Fluxus stream processing framework, enabling real-time message delivery to Telegram channels and chats

    v0.1.0 #stream-processing #real-time #data-processing #telegram #analytics #real-time-data-processing

  110. rs-zips2meta2parquet

    Converts the metadata values of the zip files to a parquet

    v0.1.0 #parquet #data-processing #arrow

  111. dukascopy_rust

    Rust wrapper for Dukascopy’s free charting API

    v0.2.0 #market-data #data-processing #api-bindings #finance

  112. fluxus-sources

    Source components for Fluxus stream processing engine

    v0.2.0 120 #data-processing #real-time #stream-processing #async #analytics #real-time-data-processing

  113. openai-api-stream-rs

    that provides a simple function for interacting with the OpenAI API and performing language-based tasks. This crate focuses on streaming responses from the API, enabling real-time processing of large amounts of data.

    v0.1.1 #openai #data-processing #streaming #real-time #api-response #data-streaming #real-time-processing #top-p

  114. shr_parser_py

    Python bindings for the shr_parser Rust crate

    v0.1.0 #python-bindings #shr-parser #validation #data-processing #signature #parse-and-handle #csv

  115. tavlars

    Tabular data processing for humans

    v0.1.1 #tabular-data #data-processing #data

  116. cortex-sources

    Ready-made source implementations for various data inputs in the Cortex ecosystem

    v0.1.0 #data-processing #rag #async

  117. dataflow_nlp

    Dataflow is a data processing library, primarily for machine learning

    v0.1.1 #dataflow #machine-learning #data-processing