Write the claim before running the test
“Large moves reverse” is a research direction. A testable claim specifies the market, observation frequency, event threshold, conditioning information, response horizon and benchmark. Record these choices before opening the evaluation sample.
For example: after a one-bar move exceeding a fixed threshold, estimate the subsequent return distribution over a specified horizon, conditional on a trailing volume measure. The threshold and horizon belong to the experimental design. They are not evidence that the effect exists.
Separate three questions
- Measurement: is the effect present in the observations, compared with an appropriate control?
- Prediction: does information available at time t improve an out-of-sample forecast?
- Implementation: can an executable decision retain value after costs, risk and constraints?
A statistically detectable pattern need not be economically useful. A successful competition strategy need not transfer to a live market with different fills and participants.
Design the comparison
Start with simple controls: a constant forecast, a historical mean fitted on training data, a no-trade policy, or an appropriate buy-and-hold allocation. Use chronological splits. If labels overlap across the split, remove training examples whose outcome windows reach into the next segment.
Report effect sizes, uncertainty, sample counts and sensitivity to reasonable design changes. Serially dependent observations need dependence-aware uncertainty estimates, such as suitable block resampling. A large grid search changes the meaning of the best-looking result; retain the full experiment history and assess selection effects.
Keep a research record
question:
data_version:
information_available_at_decision:
target_and_horizon:
training_validation_test_periods:
baseline:
cost_and_execution_model:
parameters_tried:
result_and_uncertainty:
failure_cases:
next_falsification_test:Preserve the code version, data fingerprint and configuration. A result is ready to share when another person can reproduce it and understand what could make the conclusion wrong. See research engineering for a practical project layout.