Lib.rs

› Keywords #stream-processing #real-time-data-processing #dataflow #real-time

#data-processing

  1. block-buffer

    Buffer type for block processing of data

    v0.11.0-rc.4 15.0M #data-processing #buffer #block

  2. aws-sdk-emr

    AWS SDK for Amazon EMR

    v1.82.0 500 #aws-sdk #emr #data-processing

  3. fluvio-stream-dispatcher

    Fluvio Event Stream access

    v0.50.1 2.2K #event-streaming #fluvio #stream-processing #distributed-systems #data-processing #event-stream #real-time-data #real-time-data-processing #dataflow #stream-processing-engine

  4. subtr-actor

    Rocket League replay transformer

    v0.1.8 260 #rocket-league #data-processing

  5. data-url

    Processing of data: URL according to WHATWG’s Fetch Standard

    v0.3.1 736K #url-standard #data-processing #parse-url #fetch #whatwg #mime

  6. trustfall

    query engine, empowering you to query everything

    v0.8.1 12K #query-engine #query-language #data-source #python #data-processing #database #query-api #data-query #database-querying #feed-parser

  7. tremor-script

    Tremor Script Interpreter

    v0.13.0-rc.33 3.1K #tremor #event-processing #script-interpreter #scripting-language #data-processing #pattern-matching #nlp

  8. gt7_server

    A server for GT7 telemetry data processing

    v1.1.1 170 #websocket-server #telemetry-data #gt7 #udp-socket #data-processing

  9. neuromorphic-types

    Shared Neuromorphic data types for drivers and processing frameworks

    v0.5.0 160 #driver #data-processing #neuromorphic #events #shared-data #dvs #rate-limiting #usb #camera

  10. batch-processing

    A batch library for processing a list of items in parallel

    v0.1.17 1.7K #data-processing #step #parallel-data-processing #list-of-items #item #async-data-processing

  11. razel

    a command executor with caching for data processing pipelines

    v0.5.4 #data-processing #local-cache #command-executor #run-command #task-execution #sandbox #remote-execution

  12. enc

    aids in processing encoded data

    v0.12.0 280 #data-processing #encoded #base64 #hex-encoding #data-encoding #varint #aids #aids-in-processing #urlencode #percent

  13. astrors

    Astronomical package to deal with FITS files (tables and images). An effort to develop a common core package for Astronomy in Rust.

    v1.0.1 #image-processing #astronomy #fits #astronomical #data-processing #data-analysis #fits-file #image-analysis #polars #hdu

  14. csv_log_cleaner

    Clean CSV files to conform to a type schema by streaming them through small memory buffers using multiple threads and logging data loss

    v0.2.1 #csv #json-schema #logging #data-processing #csv-processing #data-cleaning #conform #cleaner #multi-threading #data-streaming

  15. jsonschema-transpiler

    transpile JSON Schema into schemas for data processing

    v2.0.0 120 #json-schema #data-processing #avro-schema #big-query #json-schema-validation #data-schema #serialization #sql #transpile #schema-object

  16. dyer-cli

    great tool created to guide you use dyer fast and at ease, helps you build a robust crawler, data processor, netwrok program fast and correctly

    v0.6.0 4.2K #data-processing #web-framework #request-response

  17. ecad-processor

    High-performance multi-metric weather data processor for European Climate Assessment & Dataset (ECA&D) archives with Parquet output

    v2.0.1 240 #data-processing #parquet #climate #weather #ecad

  18. yngvi

    Read, calculate and write weather data

    v0.4.3 310 #weather #data-processing #display #station #read

  19. ppl

    A structured parallel programming library for Rust

    v0.1.6 #thread-pool #pipeline #data-processing #structured #framework #word-counter #concurrency #data-parallelism #farm #thread-management

  20. fluvio-spu-schema

    Fluvio API for SPU

    v0.50.1 2.1K #fluvio #spu #api-for-spu #distributed-systems #data-processing #stream-processing #data-pipeline #real-time-data-processing #event-driven-architecture

  21. new_york_calculate_core

    calculation some things

    v0.0.70 100 #calculate #york #finance #data-processing #candle #statistics #data-analysis #trading

  22. decode

    A High-Performance Query Tool for JSON, YAML and TOML

    v0.1.0 #json-query #json #json-path #data-processing

  23. epiq

    Laboratory for pipeline construction with feedback

    v0.1.0 #pipeline #stage #laboratory #stream #data-processing #jq #unix-command #processing-analysis

  24. flowgger

    fast, simple and lightweight data collector

    v0.3.3 #collector #logging #kafka #validation #data-processing #redis #udp #graylog #gelf #data-store

  25. feagi_core_data_structures_and_processing

    FEAGI libraries to use to handle data processing

    v0.0.45 #data-processing #neuron #feagi #handle #serialization #cortical #data-structures #deserialize #data-management #motor

  26. streaming_serde_json

    streaming alternative to serde_json for processing JSON data sources that are too large to fit in memory

    v0.1.2 #streaming-json #serde-json #data-processing #data-source #key-value #json-processing #json-array

  27. ibuf

    A memory buffer pool implementation

    v0.3.1 #memory-pool #buffer-pool #data-processing #performance-data #mbuf

  28. scbl-utils

    A command-line utility for data processing and delivery at the Single Cell Biology Lab at the Jackson Laboratory

    v0.1.8 #data-processing #single-cell #biology #laboratory #command-line-tool #lab #jackson #cache

  29. traqq

    High-performance event processing system for Redis data indexing

    v0.1.3 #event-processing #events #data-processing #redis #json #system-data #data-indexing #multi-threading #real-time-systems #concurrency

  30. feagi_core_data_structures_and_processing_nightly

    FEAGI libraries to use to handle data processing

    v0.0.45 150 #data-processing #neuron #feagi #handle #serialization #cortical #data-structures #deserialize #data-management #motor

  31. fluxus

    lightweight stream processing engine written in Rust, designed for efficient real-time data processing and analysis

    v0.2.0 #stream-processing-engine #processing-analysis #data-processing #real-time-data-processing #processing-stream #data-analysis

  32. deltoid

    calculate and apply deltas to structs and enums

    v0.12.0 140 #struct #delta #sequence #deserialize #data-processing #structs-and-enums #calculate-and-apply #keep-track-of #delta-compression

  33. round_pipers

    A way to pipe ndarrays using circular buffers

    v0.2.0 130 #circular-buffer #pipe #round #read-only #data-processing #ndarrays

  34. egml

    processing GML data

    v0.0.1-alpha.7 370 #data-processing #gml #early-stage

  35. cortex-ai

    An asynchronous flow-based processing framework for building flexible data pipelines with conditional branching and error handling

    v0.1.9 #data-processing #conditional #artificial-intelligence #flow-component #branch #data-pipeline #flow-based

  36. ecitygml-cli

    CLI tool for processing CityGML data

    v0.0.1-alpha.7 140 #data-processing #city-gml-data #command-line-tool #tool-for-processing

  37. egml-cli

    CLI tool for processing GML data

    v0.0.1-alpha.7 270 #gml #data-processing #tool-for-processing #command-line-tool #early-stage

  38. par-stream

    Asynchronous parallel streams analogous to rayon

    v0.10.2 2.3K #async-stream #parallel-worker #stream-combinator #stream-processing #data-processing #async-parallel #tokio-runtime #async-std-run-time #streaming-processing #broadcast

  39. ecitygml

    processing CityGML data

    v0.0.1-alpha.7 170 #data-processing #city-gml-data #early-stage

  40. byte-tools

    Bytes related utility functions

    v0.3.1 437K #cryptography #byte #data-processing

  41. j9

    Rust interface for jq-based JSON processing

    v0.1.4 290 #json-processing #jq #data-processing #interface #bindings #high-level-interface

  42. wikidump

    parsing Mediawiki XML dumps

    v0.3.1 270 #mediawiki #xml-parser #data-processing #dump #nlp #language-analysis #wikipedia #backup #revision #article

  43. ts-merger

    A multi-threaded tool for processing and aggregating time series data files

    v0.2.0 #data-processing #time-series-data #csv #time-series #aggregation #processing-time

  44. resol-vbus

    processing RESOL VBus data

    v0.2.1 #v-bus #data-processing #resol

  45. tea-dyn

    that primarily defines traits for dynamic vector operations, enabling flexible data processing across various types and backends

    v0.5.1 #back-ends #data-processing #traits #tevec #across #financial-analysis #finance #polars #vector-operations #multi-dimensional-array

  46. ngs

    Command line tool for processing next-generation sequencing data

    v0.4.0 #next-generation-sequencing #quality-control #data-processing #bam-file #genome #metrics #index-file #command-line-tool #plot #genome-analysis

  47. pflw

    Python Dataflow Framework in Rust

    v0.1.0 120 #dataflow #python #framework #data-processing #processing-transformation

  48. fluvio-system-util

    Fluvio System components

    v0.1.1 #fluvio #distributed-systems #stream-processing #data-processing #dataflow #cloud-native #stream-processing-engine #streaming-processing #real-time-data-processing #wasm

  49. ecitygml-io

    IO operations for processing CityGML data

    v0.0.1-alpha.7 170 #data-processing #city-gml-data #iops #io #operation #operations-for-processing

  50. async-pipes

    building concurrent data processing pipelines

    v0.3.1 #data-processing #data-pipeline #tokio #async-pipeline #async-data

  51. rusty-talib

    using pure rust code for technical analysis

    v0.1.6 #technical-analysis #indicator #talib #data-processing

  52. egml-io

    IO operations for processing GML data

    v0.0.1-alpha.7 380 #data-processing #iops #gml #io #operations-for-processing #processing-io

  53. fluxus-utils

    components for Fluxus stream processing engine

    v0.2.0 100 #data-processing #real-time #analytics #async #stream-processing #real-time-data-processing

  54. noir-compute

    Network of Operators In Rust

    v0.2.0 #data-processing #operator #distributed-data-processing #stream-processing #distributed-systems #noir #dataflow

  55. fluxus-sinks

    Sink components for Fluxus stream processing engine

    v0.2.0 #stream-processing #data-processing #real-time #analytics #async #real-time-data-processing

  56. fluxus-transformers

    Transformer components for Fluxus stream processing engine

    v0.2.0 #data-processing #real-time #stream-processing #analytics #async #real-time-data-processing

  57. ecitygml-transform

    Supplementary operations for CityGML data

    v0.0.1-alpha.7 180 #city-gml-data #data-processing #supplementary #operation #early-stage

  58. agnes

    A data wrangling library for Rust

    v0.3.2 #data-processing #data-source #wrangling #dataset #serialization #csv #data-manipulation #data-access #tabular-data #csv-processing

  59. black-jack

    DataFrame / Series / general data processing in Rust

    v0.1.0 #dataframe #data-processing #series #principles #key #long-term-goal

  60. fluxus-sources

    Source components for Fluxus stream processing engine

    v0.2.0 #data-processing #real-time #stream-processing #analytics #async #real-time-data-processing

  61. dyer

    designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed

    v3.3.2 #web-framework #web-scraping #data-processing #web-crawling

  62. archiv

    Read and write compressed, length-prefixed streams or file(s)

    v0.1.3 #compression #data-processing #reader-writer #stream #length-prefixed #parallel-processing

  63. fluvio-controlplane

    Fluvio control plane API

    v0.6.0 #fluvio #control-plane #distributed-systems #stream-processing #data-processing #dataflow #cloud-native #real-time-data-processing #wasm #control-api

  64. strm-privacy-driver

    Rust API driver for STRMPrivacy

    v1.0.1 #driver #data-processing #strm-privacy #contract #api #api-client #authentication #data-stream #security-privacy

  65. dr

    Command-line data file processing in Rust

    v0.7.0 #parquet #csv #data-processing #sql #command-line-data #file-processing

  66. sanitise

    Headache-free data clean-up

    v0.4.0 #data-processing #validation

  67. mako

    main Sidekick AI data processing library

    v0.3.0 #artificial-intelligence #data-processing #tokenize #data-loader #machine-learning #machine-learning-data #batch-processing #vec-string #dataflow #tokenized

  68. std-dev

    Your Swiss Army knife for swiftly processing any amount of data. Implemented for industrial and educational purposes alike.

    v0.1.0 450 #swiss-army-knife #standard-deviation #data-processing #linear-time #regression-model #percentile #swiftly #degree #input-data #ordinary

  69. fluxus-sink-telegram

    A Telegram sink component for the Fluxus stream processing framework, enabling real-time message delivery to Telegram channels and chats

    v0.1.0 #stream-processing #real-time #data-processing #telegram #analytics #real-time-data-processing

  70. dukascopy_rust

    Rust wrapper for Dukascopy’s free charting API

    v0.2.0 #market-data #data-processing #api-bindings #finance

  71. meteomatics

    Query weather and climate data from the Meteomatics API (https://www.meteomatics.com/en/)

    v0.2.2 #time-series-data #weather-api #data-api #connector #data-processing #climate #net-cdf

  72. gex_sys_build_parse_mak

    Parse mak file to build gex-sys

    v0.1.0 #parallel-processing #parser #programming #gex-sys #data-processing #apgas #address-space #udp #crayfish #process-id

  73. vehicle_parser

    Parse data about vehicles to json

    v0.1.0 #parser #data-processing #vehicle #grammar #car #text-input #grammar-rules #structured-data #string-parser #characteristics

  74. take_put_back

    A common abstraction of the process of taking some piece out, processing it and then putting it back

    v0.2.0 270 #back #putting #piece #graphs #data-processing

    75. Try searching with DuckDuckGo.

  75. wd2sql

    Transform a Wikidata JSON dump into an SQLite database

    v0.1.0 #sqlite #wikidata #dump #json #data-processing #database #allocator #entity-id #json-parser #simd-accelerated

  76. rusted_pipe

    Real time processing library for developing multithreaded ML pipelines, written in Rust

    v0.0.2 #metrics #pipeline #graph-node #data-processing #real-time #processing-time #real-time-processing #ml #rusted #multi-threading

  77. flowing

    flow graph processing library

    v0.2.1 #graph #data-processing #dataflow-programming #dsp #flowgraph

  78. shepherd-rs

    Shepherd is a resilient, non-blocking orchestrator that persistently transforms and delivers data—built for remote, compute-heavy workloads

    v0.1.0 #devops #data-processing #orchestration #scheduler

  79. einfach-xml-builder

    A lightweight and intuitive library for generating XML documents in Rust

    v0.1.0 #xml #xml-builder #xml-generation #text-processing #data-processing #development-tools

  80. codebake

    data processing framework and lisp

    v0.1.1 #lisp #data-processing #interpreter #recipe #framework #dish #rot13 #toy #bake #reference-documentation

  81. punch_api

    Punch WebAssembly public API

    v1.0.0 #punch #faas #log-management #data-processing

  82. openai-api-stream-rs

    that provides a simple function for interacting with the OpenAI API and performing language-based tasks. This crate focuses on streaming responses from the API, enabling real-time processing of large amounts of data.

    v0.1.1 #openai #data-processing #streaming #api-response #real-time #real-time-processing #data-streaming #top-p

  83. cortex-sources

    Ready-made source implementations for various data inputs in the Cortex ecosystem

    v0.1.0 #data-processing #rag #async

  84. querent-rs

    bridge capable of handling and executing querent workflows and be able to respond to incoming events via callbacks

    v1.1.7 230 #callback #events #workflow #respond #data-processing

  85. shr_parser_py

    Python bindings for the shr_parser Rust crate

    v0.1.0 #python-bindings #parser #shr #validation #data-processing #parse-and-handle #csv

  86. tavlars

    Tabular data processing for humans

    v0.1.1 #tabular-data #data-processing #data

  87. aoc_data

    parsing and processing data from the AoC API

    v0.1.0 #aoc #data-processing #parser

  88. dataflow_nlp

    Dataflow is a data processing library, primarily for machine learning

    v0.1.1 #dataflow #machine-learning #data-processing

  89. maelstrom-core

    Tools for processing of NGS data

    v0.1.1 #data-processing #ngs #python-packages

  90. postquet

    Stream PostgreSQL tables/queries to Parquet files seamlessly with this high-performance, Rust-based command-line tool

    v0.1.0 #parquet #postgresql #data-processing

  91. nitric

    A general-purpose data-processing library

    v0.0.1 #ecs #data-processing #entity

  92. shutter

    Image processing data structures and algorithms

    v0.0.1 #image-processing #data-processing #algorithm #image