What Is Out Of Sample Testing?

One might ask what is out of sample testing? It is an important step in backtesting trading strategies.

For example, a trader may test stock data from the previous decade. Then, he will test again from the latest two years as the out of sample testing.

He can also use different years outside his main sample.

The objective is to determine whether the trading system is reliable.

Two validation is important to show real results. If there is only one test, it is not conclusive.

A trader needs result that he can actually use in the real market. He needs to have proof that the trading strategy can help him.

Consider the illustration below.

Stock price data from the years 2001 to 2010 can be used as sample size. While the years 2016 to 2017 is out of sample data.

From previous experience, backtesting requires out of sample testing too to avoid curve fitting.

Without it, it is incomplete. It is important because validating a trading strategy can help determine its effectiveness.

Let’s relate out of sample testing to backtesting.

It is important that the trader can understand its connection. Otherwise, he cannot see its value for the back tester.

People usually do not understand because nobody told them the secret of testing trading strategies.

What is backtesting?

According to Rayner, Backtesting refers to testing your trading strategy on historical data and see how it performs over time. (source)

He also shared seven questions which can help you backtest correctly.

However, he forgot to include the out of sample testing that is part of the whole process.

Of course, historical data must be divided to two parts which is the in and out of sample.

The trader must have proof that the strategy works on both time periods.

The edge in the market must show itself. It must give confidence to the trader.

Trade entry, exits and stop-loss must also be defined. Without this important information, one cannot start the test.

Optimization

Let me introduce a tool that can help anyone improve his trading.

One can use it to create a trading system/indicator and test it to see if it is reliable.

However, traders need historical data to use it. He can get it from free stock market sites or buy from vendors.

Getting it without cost may result to several errors. However Investing.com can provide historical data for free.

It is recommended to buy from premium sellers for intraday data. One needs it if he wants to become a day trader.

Time frames lower than one day is harder to get.

Trading software, such as Amibroker, can help optimize a trading strategy.

If someone do not have a strategy yet, he can always make his own AFL script.

blank

Amibroker will show the best setting(s) to get the best result. For example, for a given strategy, it will show the better setup either 10, 20 or 30 moving average crossovers.

The result is based on the historical period tested. People can use it to sell trading system that does not work.

Be careful when buying trading products because it might be just a result of optimization.

Vendors, often use Amibroker to create curved fitted trading systems. Then, they will show you how effective their backtest results.

However, it would not really help the trader in accomplishing his goals.

How to validate the trading strategy without an out of sample testing?

A trading system is valid only if it works in both the in and out of sample testing.

Actual Testing

After out of sample testing, one should test the trading system in a real trading account.

He can start with a small account and test the system in the stock market.

This is the last phase the trader should do to see if the trading method actually works.

Successful traders do this before they commit large amount of money into trading.

Most trading websites teach backtesting, but not out of sample testing.

Professional traders always discuss it to their interns or students.

That is the difference between trading educators and real traders.

Someone might ask, why nobody taught this? He must be suprised by this revelation.

Well, this website will teach real trading. It is important to share the truth.

Otherwise, what is the purpose of this website.

This free lesson will help traders succeed in the stock market.

It is not the holy grail, however, people should do it.

Please share this post if it is helpful to everyone.