Know what the data contains.

Keep the original observations

Preserve raw files separately from cleaned tables and derived features. Record the source, collection method, instrument identifiers, observation frequency, timezone, adjustment convention and known gaps. A checksum identifies the exact bytes used in an experiment.

Validate before modelling

• Check unique instrument/time keys, ordering, duplicates and missing observations.
• Check price units, positive prices, non-negative recorded volume and consistent OHLC ranges.
• Inspect market calendars, timestamp units and whether a timestamp marks an interval's start or end.
• For equities, account for splits, dividends, delistings and membership changes.
• For fundamentals, preserve publication and revision times as well as the economic period.

Different data answers different questions

Daily adjusted prices support portfolio-return experiments. OHLCV bars support some signal studies, but cannot reconstruct the historical bid/ask queue. Order-book snapshots show displayed depth at sampled instants; trade records show executions. Neither is a complete record of every hidden order or cancellation.

Competition files are edition-specific. The data catalogue distinguishes prices, trades and observations. TMRW's legacy options exports contain serialised tables and need normalisation before they can be treated as ordinary OHLCV.

A reproducible folder structure

data/
  raw/          original files, unchanged
  processed/    validated tables with a documented schema
  features/     derived values and their availability times
experiments/
  configuration.json
  data-checks.json
  report.md

Use the workbench adapter carefully

The OHLCV loader rejects duplicate or unordered timestamps rather than silently fixing them. The Prosperity adapter extracts one product's midpoints and uses an ordinal observation index. It is suitable for a price-series experiment; evaluating an actual competition bot requires an edition-compatible exchange backtester.

When publishing data for students, include the information needed to interpret it as well as permission to reuse it. The publication guide explains the catalogue fields and download options.