API Snapshots: Core, Memory, Pig, Hive, Misc

Table of Contents

Architecture

The DataSketches Library is organized into the following repository groups:

Java

sketches-core

This repository has the core sketching classes, which are leveraged by some of the other repositories.
This repository has no external dependencies outside of the DataSketches/memory repository, Java and TestNG for unit tests. This code is versioned and the latest release can be obtained from Maven Central.

High-level Repositories Structure

Sketches-core Packages. Package Description
com.yahoo.sketches Common functions and utilities
com.yahoo.sketches.cpc New Unique Counting Sketch with better accuracy per size than HLL
com.yahoo.sketches.frequencies Frequent Item Sketches, for both longs and generics
com.yahoo.sketches.hash The 128-bit MurmurHash3 and adaptors
com.yahoo.sketches.hll Unique counting HLL sketches for both heap and off-heap.
com.yahoo.sketches.hllmap The (HLL) Unique Count Map Sketch
com.yahoo.sketches.kll New quantiles sketch with better accuracy per size than the standard quantiles sketch.
com.yahoo.sketches.quantiles Sketches for quantiles, PMF and CDF functions, both doubles and generics and for heap and off-heap.
com.yahoo.sketches.sampling Weighted and uniform reservoir sampling with generics
com.yahoo.sketches.theta Unique counting Theta Sketches for both heap and off-heap
com.yahoo.sketches.tuple Tuple sketches for both primitives and generics

memory

Memory Packages Package Description
com.yahoo.memory Low level, high-performance Memory data-structure management primarily for off-heap.

sketches-android

This is a new repository dedicated to sketches designed to be run in a mobile client, such as a cell phone. It is still in development and should be considered experimental.

sketches-hive

This repository contains Hive UDFs and UDAFs for use within Hadoop grid enviornments. This code has dependencies on sketches-core as well as Hadoop and Hive. Users of this code are advised to use Maven to bring in all the required dependencies. This code is versioned and the latest release can be obtained from Maven Central.

Sketches-hive Packages Package Description
com.yahoo.sketches.hive.cpc Hive UDF and UDAFs for CPC sketches
com.yahoo.sketches.hive.frequencies Hive UDF and UDAFs for Frequent Items sketches
com.yahoo.sketches.hive.hll Hive UDF and UDAFs for HLL sketches
com.yahoo.sketches.hive.kll Hive UDF and UDAFs for KLL sketches
com.yahoo.sketches.hive.quantiles Hive UDF and UDAFs for Quantiles sketches
com.yahoo.sketches.hive.theta Hive UDF and UDAFs for Theta sketches
com.yahoo.sketches.hive.tuple Hive UDF and UDAFs for Tuple sketches

sketches-pig

This repository contains Pig User Defined Functions (UDF) for use within Hadoop grid environments. This code has dependencies on sketches-core as well as Hadoop and Pig. Users of this code are advised to use Maven to bring in all the required dependencies. This code is versioned and the latest release can be obtained from Maven Central.

Sketches-pig Packages Package Description
com.yahoo.sketches.pig.cpc Pig UDFs for CPC sketches
com.yahoo.sketches.pig.frequencies Pig UDFs for Frequent Items sketches
com.yahoo.sketches.pig.hash Pig UDFs for MurmerHash3
com.yahoo.sketches.pig.hll Pig UDFs for HLL sketches
com.yahoo.sketches.pig.kll Pig UDFs for KLL sketches
com.yahoo.sketches.pig.quantiles Pig UDFs for Quantiles sketches
com.yahoo.sketches.pig.sampling. Pig UDFs for Sampling sketches
com.yahoo.sketches.pig.theta Pig UDFs for Theta sketches
com.yahoo.sketches.pig.tuple Pig UDFs for Tuple sketches

sketches-vector

This is a new repository dedicated to sketches for vector and matrix operations. It is still somewhat experimental.

characterization

This relatively new repository is for code that we use to characterize the accuracy and speed performance of the sketches in the library and is constantly being updated. Examples of the job command files used for various tests can be found in the src/main/resources directory. Some of these tests can run for hours depending on its configuration.

Characterization Packages Package Description
com.yahoo.sketches.characterization Common functions and utilities
com.yahoo.sketches.characterization.hash Hash function performance
com.yahoo.sketches.characterization.memory Memory performance
com.yahoo.sketches.characterization.quantiles. Quantiles performance
com.yahoo.sketches.characterization.uniquecount Performance of Theta and HLL sketches

experimental

This repository is an experimental staging area for code that will eventually end up in another repository. This code is not versioned and not registered with Maven Central.

sketches-misc

Demos, command-line access, characterization testing and other code not related to production deployment.

This code is offered “as is” and primarily as a reference so that users can understand how some of the performance characterization plots were obtained. This code has few unit tests, if any, and was never intended for production use. Nonetheless, some folks have found it useful. If you find it useful, go for it. This code is versioned and the latest release can be obtained from Maven Central.

Sketches-misc Packages Package Description
com.yahoo.sketches Utility functions used by the sketches-misc packages
com.yahoo.sketches.cmd Support for Command Line functions Being Redesigned
com.yahoo.sketches.demo Simple demo for brute-force vs Theta and HLL sketches Will be superceded by Command Line functions
com.yahoo.sketches.quantiles Utility for computing & printing space table for Quantiles Sketches (only in the test branch)
com.yahoo.sketches.sampling Benchmarks and Entropy testing for sampling sketches

C++ and Python

sketches-core-cpp

This is the evolving C++ implementations of the same sketches that are available in Java. These implementations are binary compatible with their counterparts in Java. In other words, a sketch created and stored in C++ can be opened and read in Java and visa-versa.

This site also has our Python adaptors that basically wrap the C++ implementations, making the high performance C++ implementations available from Python.

sketches-postgres

This site provides the postgres-specific adaptors that wrap the C++ implementations making them available to the Postgres database users.

characterization-cpp

This is the parallel characterization repository with a parallel objective to the Java characterization repository.

experimental-cpp

This repository is an experimental staging area for C++ code that will eventually end up in another repository.

Command-Line Tool

These repositories provide a command-line tool that provides access to the following sketches:

  • Frequent Items
  • HLL
  • Quantiles
  • Reservoir Sampling
  • Theta Sketches
  • VarOpt Sampling

This tool can be installed from Homebrew.

sketches-cmd

homebrew-sketches

homebrew-sketches-cmd

Web Site

DataSketches.github.io

This is the DataSketches.github.io web site, and is constantly being updated with new material and to be current with the GitHub master. This site is not versioned and not registered with Maven Central.