MySQL The world's most popular open source database.

PostgreSQL The world's most advanced open source database.

Oracle Database - object-relational database management system.

Teradata - high-performance MPP data warehouse platform.

Spark in Motion - Spark in Motion teaches you how to use Spark for batch and streaming data analytics.

Machine Learning, Data Science and Deep Learning with Python  - LiveVideo tutorial that covers machine learning, Tensorflow, artificial intelligence, and neural networks.

Data warehouse schema design - dimensional modeling and star schema - Introduction to schema design for data warehouse using the star schema method.

Elasticsearch 7 and Elastic Stack - LiveVideo tutorial that covers searching, analyzing, and visualizing big data on a cluster with Elasticsearch, Logstash, Beats, Kibana, and more.

Apache Ranger - Central security admin &amp; fine-grained authorization for Hadoop

Apache Eagle - real time monitoring solution

Apache Knox Gateway - single point of secure access for Hadoop clusters.

Apache Sentry - security module for data stored in Hadoop.

BDA - The vulnerability detector for Hadoop and Spark

Data Science at Scale with Python and Dask - Data Science at Scale with Python and Dask teaches you how to build distributed data projects that can handle huge amounts of data.

Streaming Data - Streaming Data introduces the concepts and requirements of streaming and real-time data systems.

Storm Applied - Storm Applied is a practical guide to using Apache Storm for the real-world tasks associated with processing and analyzing real-time data streams.

Fundamentals of Stream Processing: Application Design, Systems, and Analytics - This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field.

Stream Data Processing: A Quality of Service Perspective - Presents a new paradigm suitable for stream and complex event processing.

Unified Log Processing - Unified Log Processing is a practical guide to implementing a unified log of event streams (Kafka or Kinesis) in your business

Kafka Streams in Action - Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort.

Big Data - Big Data teaches you to build big data systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data.

Kafka in Action - Kafka in Action is a fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.

Fusion in Action - Fusion in Action teaches you to build a full-featured data analytics pipeline, including document and data search and distributed data clustering.

Reactive Data Handling - Reactive Data Handling is a collection of five hand-picked chapters, selected by Manuel Bernhardt, that introduce you to building reactive applications capable of handling real-time processing with large data loads--free eBook! 

Azure Data Engineering - A book about data engineering in general and the Azure platform specifically 

Grokking Streaming Systems - Grokking Streaming Systems helps you unravel what streaming systems are, how they work, and whether they’re right for your business. Written to be tool-agnostic, you’ll be able to apply what you learn no matter which framework you choose.

Bistro - general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via 

IBM Streams - platform for distributed processing and real-time analytics.  Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)

Apache Hadoop - framework for distributed processing. Integrates&nbsp;MapReduce (parallel processing), YARN (job scheduling) and HDFS (distributed file system).

Tigon - High Throughput Real-time Stream Processing Framework.

Pachyderm - Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis.

Polyaxon - A platform for reproducible and scalable machine learning and deep learning.

Smooks - An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc…) streaming applications.

Apache Airflow - a platform to programmatically author, schedule and monitor workflows.

Apache Aurora - is a service scheduler that runs on top of Apache Mesos.

Apache Falcon - data management framework.

Azure Data Factory - cloud-based pipeline orchestration for on-prem, cloud and HDInsight

Chronos - distributed and fault-tolerant scheduler.

Cronicle - Distributed, easy to install, NodeJS based, task scheduler

Dagster - a data orchestrator for machine learning, analytics, and ETL.

Linkedin Azkaban - batch workflow job scheduler.

Schedoscope - Scala DSL for agile scheduling of Hadoop jobs.

2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.    

Apache Hadoop Benchmarking - micro-benchmarks for testing Hadoop performances.

Berkeley SWIM Benchmark - real-world big data workload benchmark.

Intel HiBench - a Hadoop benchmark suite.

PUMA Benchmarking - benchmark suite for MapReduce applications.

Yahoo Gridmix3 - Hadoop cluster benchmarking from Yahoo engineer team.

411 - an web application for alert management resulting from scheduled searches into Elasticsearch.

Adobe spindle - Next-generation web analytics processing with Scala, Spark, and Parquet.

Apache Metron - a platform that integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis.

Apache OODT - capturing, processing and sharing of data for NASA's scientific archives.

Argus - Time series monitoring and alerting platform.

AthenaX - a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).

Atlas - a backend for managing dimensional time series data.

Countly - open source mobile and web analytics platform, based on Node.js &amp; MongoDB.

Domino - Run, scale, share, and deploy models — without any infrastructure.

Eclipse BIRT - Eclipse-based reporting system.

ElastAert - ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.

Eventhub - open source event analytics platform.

HASH - open source simulation and visualization platform.

Hermes - asynchronous message broker built on top of Kafka.

Imhotep - Large scale analytics platform by indeed.

Indicative - Web &amp; mobile analytics tool, with data warehouse (AWS, BigQuery) integration.

Jupyter - Notebook and project application for interactive data science and scientific computing across all programming languages.

MADlib - data-processing library of an RDBMS to analyze data.

Kapacitor - an open source framework for processing, monitoring, and alerting on time series data.

Kylin - open source Distributed Analytics Engine from eBay.

PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.

Rakam - open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB. 

Qubole - auto-scaling Hadoop cluster, built-in data connectors.

SnappyData - a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster.

Snowplow - enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.

Splunk - analyzer for machine-generated data.

Sumo Logic - cloud based analyzer for machine-generated data.

Talend - unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog &amp; Pig.

redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.

Amazon Kinesis - real-time processing of streaming data at massive scale.

Amazon Web Services Glue -  serverless fully managed extract, transform, and load (ETL) service

Census - A reverse ETL product that let you sync data from your data warehouse to SaaS Applications. No engineering favors required—just SQL.

Apache Flume - service to manage large amount of log data.

Apache Kafka - distributed publish-subscribe messaging system.

Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.

Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.

Apache Sqoop - tool to transfer data between Hadoop and a structured datastore.

Embulk - open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.

Facebook Scribe - streamed log data aggregator.

Fluentd - tool to collect events and logs.

Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.

Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.

Heka - open source stream processing software system.

HIHO - framework for connecting disparate data sources with Hadoop.

Kestrel - distributed message queue system.

LinkedIn Databus - stream of change capture events for a database.

LinkedIn Kamikaze - utility package for compressing sorted integer arrays.

LinkedIn White Elephant - log aggregator and dashboard.

Logstash - a tool for managing events and logs.

Netflix Suro - log agregattor like Storm and Samza based on Chukwa.

Pinterest Secor - is a service implementing Kafka log persistance.

Linkedin Gobblin - linkedin's universal data ingestion framework.

Skizze - sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.

StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.

Alooma - data pipeline as a service enabling moving data sources such as MySQL into data warehouses.

RudderStack - an open source customer data infrastructure (segment, mParticle  alternative) written in go.

AgensGraph - a new generation multi-model graph database for the modern complex data environment.

Apache Giraph - implementation of Pregel, based on Hadoop.

Apache Spark Bagel - implementation of Pregel, part of Spark.

ArangoDB - multi model distributed database.

DGraph - A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.

EliasDB - a lightweight graph based database that does not require any third-party libraries.

Facebook TAO - TAO is the distributed data store that is widely used at facebook to store and serve the social graph.

GCHQ Gaffer - Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.

Google Cayley - open-source graph database.

Google Pregel - graph processing framework.

GraphLab PowerGraph - a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API.

GraphX - resilient Distributed Graph System on Spark.

Infovore - RDF-centric Map/Reduce framework.

Intel GraphBuilder - tools to construct large-scale graphs on top of Hadoop.

JanusGraph - open-source, distributed graph database
with multiple options for storage backends (Bigtable, HBase, Cassandra, etc.)
and indexing backends (Elasticsearch, Solr, Lucene).

MapGraph - Massively Parallel Graph processing on GPUs.

Microsoft Graph Engine - a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine.

Neo4j - graph database written entirely in Java.

Phoebus - framework for large scale graph processing.

Titan - distributed graph database, built over Cassandra.

Twitter FlockDB - distributed graph database.

NodeXL - A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.

Actian Ingres - commercially supported, open-source SQL relational database management system.

ActorDB - a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database.

Amazon RedShift - data warehouse service, based on PostgreSQL.

BayesDB - statistic oriented SQL database.

Bedrock - a simple, modular, networked and distributed transaction layer built atop SQLite.

CitusDB - scales out PostgreSQL through sharding and replication.

Cockroach - Scalable, Geo-Replicated, Transactional Datastore.

Comdb2 - a clustered RDBMS built on optimistic concurrency control techniques.

Datomic - distributed database designed to enable scalable, flexible and intelligent applications.

FoundationDB - distributed database, inspired by&nbsp;F1.

Google F1 - distributed SQL database built on Spanner.

Google Spanner - globally distributed semi-relational database.

H-Store - is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications.

Haeinsa - linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.

HandlerSocket - NoSQL plugin for MySQL/MariaDB.

KarelDB - a relational database backed by Apache Kafka.

Map-D - GPU in-memory database, big data analysis and visualization platform.

MemSQL - in memory SQL database witho optimized columnar storage on flash.

NuoDB - SQL/ACID compliant distributed database.

Oracle TimesTen in-Memory Database - in-memory, relational database management system with persistence and recoverability.

Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.

SAP HANA - is an in-memory, column-oriented, relational database management system.

SenseiDB - distributed, realtime, semi-structured database.

Sky - database used for flexible, high performance analysis of behavioral data.

SymmetricDS - open source software for both file and database synchronization.

TiDB - TiDB is a distributed SQL database. Inspired by the design of Google F1.

VoltDB - claims to be fastest in-memory database.

yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL.

Azure ML Studio - Cloud-based AzureML, R, Python Machine Learning platform

Oryx - Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning.

Concurrent Pattern - machine learning library for Cascading.

convnetjs - Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.

DataVec - A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem. 

Deeplearning4j - Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.

Decider - Flexible and Extensible Machine Learning in Ruby.

ENCOG - machine learning framework that supports a variety of advanced algorithms, as well as support classes to normalize and process data.

etcML - text classification with machine learning.

Etsy Conjecture - scalable Machine Learning in Scalding.

Feast - A feature store for the management, discovery, and access of machine learning features. Feast provides a consistent view of feature data for both model training and model serving.

GraphLab Create - A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.

H2O - statistical, machine learning and math runtime with Hadoop. R and Python.

Karate Club - An unsupervised machine learning library for graph structured data. Python

Keras - An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow.

Lambdo - Lambdo is a workflow engine which significantly simplifies the analysis process by unifying feature engineering and machine learning operations.

Little Ball of Fur - A subsampling library for graph structured data. Python

Mahout - An Apache-backed machine learning library for Hadoop.

MLbase - distributed machine learning libraries for the BDAS stack.

MLPNeuralNet - Fast multilayer perceptron neural network library for iOS and Mac OS X.

ML Workspace - All-in-one web-based IDE specialized for machine learning and data science.

MOA - MOA performs big data stream mining in real time, and large scale machine learning.

MonkeyLearn - Text mining made easy. Extract and classify data from text.

ND4J - A matrix library for the JVM. Numpy for Java. 

nupic - Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.

PredictionIO - machine learning server buit on Hadoop, Mahout and Cascading.

PyTorch Geometric Temporal - a temporal extension library for PyTorch Geometric .

RL4J - Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem. 

SAMOA - distributed streaming machine learning framework.

scikit-learn - scikit-learn: machine learning in Python.

Shapley - A data-driven framework to quantify the value of classifiers in a machine learning ensemble. 

Spark MLlib - a Spark implementation of some common machine learning (ML) functionality.

Sibyl - System for Large Scale Machine Learning at Google.

TensorFlow - Library from Google for machine learning using data flow graphs.

Theano - A Python-focused machine learning library supported by the University of Montreal.

Torch - A deep learning library with a Lua API, supported by NYU and Facebook.

Velox - System for serving machine learning predictions.

Vowpal Wabbit - learning system sponsored by Microsoft and Yahoo!.

WEKA - suite of machine learning software.

BidMach - CPU and GPU-accelerated Machine Learning Library.

Pilosa Open source distributed bitmap index that dramatically accelerates queries across multiple, massive data sets. 

Apache Ambari - operational framework for Hadoop mangement.

Apache Bigtop - system deployment framework for the Hadoop ecosystem.

Apache Helix - cluster management framework.

Apache Slider - is a YARN application to deploy existing distributed applications on YARN.

Apache Whirr - set of libraries for running cloud services.

Brooklyn - library that simplifies application deployment and management.

Repository of BIG DATA Resources in December 2024

Select category under big-data projects and apps that you are looking to build in December 2024

Last Updated: 17 December 2024

Repository of big data Resources

Subscribe to our Newsletter

Company

Follow Us

Subscribe to our Newsletter