Can identical EA settings produce different backtesting results?

Backtesting
#1

Assuming that the backtesting is done at a fixed spread, from date X to date Y, with tick data, and that the platform is not connected to the internet from the start to the end of the tests (to avoid the quote currency exchange rate differential at different points of the test), is there any reason other than there being some kind of bug located in the EA, that backtesting should provide different results from one test to another during the exact same period with the exact same settings?

Whilst using the open prices only method, I have been experiencing some noticeable differences from one backtest to another in terms of the net profit on the crosses. When testing the majors, I get an identical result on Test A as on Test B every time, but on the crosses, net profit changes from one test to another, sometimes albeit only by a penny on two, but sometimes much more.

It is worth noting that I haven’t downloaded historical data from the MT History Centre, I am currently in the process of downloading tick data. I am hoping that using this will give me more reliable results.

Numerous threads on here have discussed the factors that can influence backtesting results from one test to another, things such as the spread, dates, quote currency conversion are the usual culprets, but if all these factors can be eliminated, then would it be correct to say that the only reason for differing results would be some kind of bug in the EA itself?

#2

if all factors staying constant, the results should be exactly identical. but that’s a hard factor assumption i guess. (all factors staying constant)
and the part related to code is not necessarily a bug. can be how the code is supposed to work. (like those garbage EAs using random entry direction on martingale/grid and whatnot)

#3

The EA doesn’t do anything randomly. I have run tests on the MACD/Moving Average sample EA’s that MT provide, the results are similar to my experience.

Using identical settings with the internet disconnected:

MACD Sample EA showed a profit of 155.59 on EURUSD on the first test, and then a profit of 155.59 on the second test.
On EURGBP, the first test showed a profit of 32.95, and the second test a profit of 33.02.

Moving Average EA showed a profit of 1945.55 on EURUSD on the first test, and then a profit of 1945.55 on the second test.
On EURGBP, the first test showed a loss of 592.14, and then the second test a loss of 592.06.

These results seem to be consistent with my experience, but haven’t had the chance to scrutinise the logic of the sample EA’s to understand if there could be any other reason for the difference.

#4

I believe so. The only time I saw inconsistencies with backtesting results was when I found a bug in the Tester, which has since been fixed.

#6

when results are not consistent(profit wise), is the total number of trades changing as well ?

#7

No, the number of trades are equal on each test.

#8

Are you using “fixed spread” or “current”?

#9

Fixed. All the settings are identical from test to test.

#10

Are these normal single back-tests or multiple Optimisation runs?

Please note that the Strategy Tester saves data (cache files) for Optimisation runs and then uses the data in future optimisations when applicable. However these cache files (.\tester\cache), readily get corrupted for reasons unknown.

So I recommend deleting them before starting a new optimisation run so that you get clean results. This is what I do because I often get these strange results too on MT4.

#11

When I do backtests with no internet connection I get strange results. I always thought the reason is MT4 does not know some of the symbol properties like tick value, tick size, stop level

#12

Yes, that can happen! That is why you should first run one back-test while still connected in order to collect all the necessary data, then you can disconnect and run all the subsequent back-tests and optimisations using the constant values that were collected. So, even if you use “custom” for spread it will remain constant no matter what value it collected.

EDIT: In my case I don’t need to disconnect because I produce my own FXT files with real tick data and all the constants such as stops level, swaps, etc. are all coded into the FXT file which I protect by setting them to read-only so they do not get overwritten.

#13

Here are the logs from the first trade taken on either system:



1

2003.08.29 00:00

 	 	
sell

 	 	
1

 	 	
0.14

 	 	
0.68970

 	 	
0.00000

 	 	
0.00000

 	 	
 	 
2

2003.09.01 00:00

 	 	
modify

 	 	
1

 	 	
0.14

 	 	
0.68970

 	 	
0.69850

 	 	
0.00000

 	 	
 	 
3

2003.09.02 00:00

 	 	
s/l

 	 	
1

 	 	
0.14

 	 	
0.69850

 	 	
0.69850

 	 	
0.00000

 	 	
-161.95

 	 	
9838.05




1

2003.08.29 00:00

 	 	
sell

 	 	
1

 	 	
0.14

 	 	
0.68970

 	 	
0.00000

 	 	
0.00000

 	 	
 	 
2

2003.09.01 00:00

 	 	
modify

 	 	
1

 	 	
0.14

 	 	
0.68970

 	 	
0.69850

 	 	
0.00000

 	 	
 	 
3

2003.09.02 00:00

 	 	
s/l

 	 	
1

 	 	
0.14

 	 	
0.69850

 	 	
0.69850

 	 	
0.00000

 	 	
-161.94

 	 	
9838.06

How is the above possible?

Note: I was connected to the internet when running the above test, but the tests are on EURGBP (my account is denominated is GBP, so no conversion rate issue should apply), the spread is still fixed and everything else identical. The tests are run within 10 seconds of each other (on the same PC).

#14

Build 1090.
Single back tests.

#15

Your account balance may be GBP, but your test balance does not seem to be set to GBP as the values do not match, because that was a 88 pip loss and at £10 per ppip-lot, it should have been a loss of £123.20, not 161.94 as is shown.

So you are in fact suffering from a conversion rate difference, because you are probably using USD as the back-test balance deposit currency because £123.20 is approximately equivalent to $161.94 according to the current exchange rate.
21

#16

You’re right, that is the problem, I didn’t notice that.

Thanks again Fernando.