Custom Trading System logo
NinjaTrader indicators, automated trading systems
Custom development and premium support
Connect with us
Historical Market Data: Do you really need it?

Historical Market Data: Do you really need it?

By CustomTradingSystem, on April 25, 2015

Historical market data traditionally is used in back-testing. Technically, it has no difference from real-time market data. However, there are nuances that may change everything.

Many data providers claim that they offer years of historical data. What they don't mention is that the data comes in a form of OHLC (Open, High, Low, Close), which is also known as a bar. Therefore, we do not know how price moved inside bars, that data is effectively lost. For example, if we have 5 minutes bars then for the whole 5 minute period we only get 4 price values:

  • Open – the price at the beginning of the 5-min interval
  • High - the highest price during the 5-min interval
  • Low - the lowest price for that 5-min interval
  • Close - the price at the end of the 5-min interval

Essentially, it means that 99% of price action has been lost, we have ony 4 price values for the whole 5 minute interval. Will a strategy be able to show any relevant results on such data? One of the biggest issues with testing on bars is the situation when both stop loss and profit target are located inside the bar: what should be filled first – stop loss or profit target?

Of course, some strategies are fine with such rough data but most are not.

OK, one may say, let's use tick-based market data, which is also offered by other market data providers. Isn't that what would give us the best precision?

Theoretically, yes, but in practice not really.

First, storing every trade requires high storage and CPU power resources on both server and client side. That's why most data providers offer only 2-4 weeks of tick data, though some popular symbols may have up to 6 months of data. If we trade ES-mini then we will almost always get 6 months of tick data. However, for the vast majority of the symbols our back-test will be as short as 2-4 weeks, which is again not enough for serious back-testing.

Second, even if we get 6-month of data we must ensure that the data has correct time resolution. What does it mean? Some data providers store time of a trade down to seconds only. So if we had 2 ticks half a second apart, one at 10:00:00.250 and another at 10:00:00.750, data provider would round the time to one second and we would get two trades at 10:00:00. Such a subtle from the first glance difference may completely break sensitive indicators and hence, strategies.

Back-testing is extremely important step but with the most of historical data available it will probably not provide relevant results. Don't rely upon historical data and if you do be aware about the pitfalls.

Conclusion

So what would be a solution one may ask? Any common back-testing solution will be a compromise between flexibility and relevance.

Nevertheless, if you use NinjaTrader platform there is a specific way to run a quality back-test. We would be happy to share our knowledge with you.