GitHub - apache/gobblin: A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Redpanda

Redpanda 100 kafka api compatible 10x faster no zookeeper no jvm

Amazon Kinesis - Process & Analyze Streaming Data - Amazon Web Services

Collect streaming data create a real time data pipeline and perform real time clickstream analytics log analytics event analytics and io t analytics

AWS Glue | Serverless Data Integration Service | Amazon Web Services

Aws glue is a serverless data integration service that makes it easy to discover prepare and combine data for analytics machine learning and application development aws glue provides all the capabilities needed for data integration so that you can start analyzing your data and putting it to use

Census | Leader in Reverse ETL

Send data from your data warehouse to your sales marketing finance support and operational tools no engineering favors required

Chukwa - Welcome to Apache Chukwa

Welcome to Apache Flume — Apache Flume

Apache Kafka

Apache kafka a distributed streaming platform

Apache NiFi

GitHub - apache/pulsar: Apache Pulsar - distributed pub-sub messaging system

Apache pulsar distributed pub sub messaging system git hub apache pulsar apache pulsar distributed pub sub messaging system

Sqoop -

Embulk

GitHub - facebookarchive/scribe: Scribe is a server for aggregating log data streamed in real time from a large number of servers.

Scribe is a server for aggregating log data streamed in real time from a large number of servers git hub facebookarchive scribe scribe is a server for aggregating log data streamed in real tim

Fluentd | Open Source Data Collector

GitHub - gazette/core: Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Build platforms that flexibly mix sql batch and stream processing paradigms git hub gazette core build platforms that flexibly mix sql batch and stream processing paradigms

Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams – Google Research

GitHub - mozilla-services/heka: DEPRECATED: Data collection and processing made easy.

Deprecated data collection and processing made easy git hub mozilla services heka deprecated data collection and processing made easy

GitHub - sonalgoyal/hiho: Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

Hadoop data integration with various databases ftp servers salesforce incremental update dedup append merge your data on hadoop git hub sonalgoyal hiho hadoop data integration with vario

GitHub - papertrail/kestrel: simple, distributed message queue system (active — currently managed by Papertrail)

Simple distributed message queue system active currently managed by papertrail git hub papertrail kestrel simple distributed message queue system active currently managed by papertrail

Data

GitHub - LinkedInAttic/kamikaze: DocId set compression and set operation library

Doc id set compression and set operation library contribute to linked in attic kamikaze development by creating an account on git hub

GitHub - LinkedInAttic/white-elephant: Hadoop log aggregator and dashboard

Hadoop log aggregator and dashboard contribute to linked in attic white elephant development by creating an account on git hub

Logstash: Collect, Parse, Transform Logs | Elastic

Logstash part of the elastic stack integrates data from any source in any format with this flexible open source collection parsing and enrichment pipeline download for free

GitHub - Netflix/suro: Netflix’s distributed Data Pipeline

Netflix s distributed data pipeline contribute to netflix suro development by creating an account on git hub

GitHub - pinterest/secor: Secor is a service implementing Kafka log persistence

Secor is a service implementing kafka log persistence git hub pinterest secor secor is a service implementing kafka log persistence

GitHub - skizzehq/skizze: A probabilistic data structure service and storage

A probabilistic data structure service and storage git hub skizzehq skizze a probabilistic data structure service and storage

StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.

ETL MySQL to any data warehouse in minutes | Alooma

Load your my sql data to any data warehouse to run custom sql queries on your crm erp and ecommerce data and generate custom reports

GitHub - rudderlabs/rudder-server: Privacy and Security focused Segment-alternative, in Golang and React

Privacy and security focused segment alternative in golang and react git hub rudderlabs rudder server privacy and security focused segment alternative in golang and react