Global hedge fund: tick data management and quality

Challenge

Decades of tick data, pervasive quality issues

A global market maker with decades of historical tick data needed to resolve pervasive data-quality issues: invalid or missing reference links, missing or duplicate expiry dates, missing reference fields and incorrect lot units, option chains containing unrelated instruments, incorrect intraday values requiring recalculation from raw ticks, and incorrect symbology mapping.

Solution

DataHex inside the client’s own environment

An instance of DataHex in the client’s AWS environment to download, consolidate, validate, and ingest historical and ongoing market data from multiple vendors; detection of missing or corrupt data with vendor resolution; removal of irrelevant instruments from historical option chains; creation and maintenance of a Security Master; and resolution of data-quality issues such as expiry dates, currency codes, and last trade dates.

What DataHex delivered

DataHex in the client’s AWS environment Download, consolidate, validate and ingest historical and ongoing market data from multiple vendors.
Detection of missing or corrupt data With resolution taken back to the vendors.
Security Master created and maintained Including removal of irrelevant instruments from historical option chains.
Data-quality resolution Expiry dates, currency codes and last trade dates.

Results

Analytics-ready data, with the quality problem solved

Optimized storage, search, query, and extraction for rapid discovery in the cloud
Transformation and delivery into multiple cloud platforms
Source-agnostic ingestion from multiple vendors
A published data calendar of known issues
One-minute timebars in the ingestion process
Analytics-ready data that minimizes wrangling and lowers data-management costs
Automatic validation and maintenance of the Security Master
API and GUI access to search, schedule, and extract by instrument, portfolio, asset class, or exchange