SDS Quant Research

How we test trading ideas

Most published backtests are easy to produce and hard to trust. This page describes the rules we hold ourselves to, so that you can judge any result we publish against them.

1. Hypotheses are written down first

Before running anything, we record the hypothesis, the exact rules, the parameters we will try, and how success is measured. We currently test two ideas:

Both are well-known ideas. The point is not novelty. The point is to find out, with discipline, whether they hold up after costs.

2. Every strategy is compared with random picks

A strategy that makes money in a rising market has not necessarily found anything. For each hypothesis we build matched random-selection controls: portfolios that follow the same constraints (number of positions, holding periods, universe, costs) but choose stocks at random. The strategy only counts as evidence if it beats the distribution of those random runs by a meaningful margin.

3. The data has to include the losers

Testing only on companies that are in an index today ignores the ones that were dropped, acquired at a loss or went bankrupt. We use historical index membership so that on every simulated date the universe is what an investor could actually have chosen on that date. See survivorship bias.

4. No peeking at the future

The backtest engine includes automated checks that fail the run if any decision uses information that would not have been available at the time. See look-ahead bias.

5. Three periods, and one of them is sealed

We split history into a development period (2000 to 2015), a validation period (2016 to 2020) and a final test period (2021 to 2026). We tune and debug on the first, confirm on the second, and run the final period once, at the end, with no changes afterwards. Whatever it shows gets published. See backtest overfitting.

6. We count our attempts

Every configuration we try is a trial, and the more we try, the more likely the best one is just lucky. We keep a registry of every run and report overfitting statistics that account for the number of trials, not only the winner.

7. Failures get published

If a hypothesis does not survive, we say so in the research log with the same detail as a success. A null result is still a result.

Status

The project is in development. Results will appear in the research log as each phase is completed, with the method and the data described alongside them.