ATC2011 - Here we go again

MetaQuotes announced the 2011 Automated Trading Championship. Apart from changes in sponsors, everything looks pretty much the same.

I will be submitting the same EA as last year which was placed 56, my best perfomance so far in this series of competition. It gained a profit of $3500. The EA hardly traded because of some hiccups when the servers were reset.

I don't have much time this year and hopefully the same EA can trade more often and have its performance validated. I kept the demo version running till today and it is currently at $240K from its initial $10K in Jul 2010.

Alpari FX Contest – I’m out.

The Contest had ended. I am out of the league table; that means the overall equity had fallen below $100K. One mistake I have learnt. Running 2 EAs in the same account. I usually don’t run more than 1 EA in an account, be it live or demo. In this case, I was trying to increase trading opportunities as this contest is over a short period. The first EA is rather idle and I thought the second EA will the gaps.

I forgot to include magic numbers in the closing routine and the two EAs messed each other up; and one EA traded higher lots when there are previous losses and so cumulate up more losses.

2010G – no trades at ATC2010

This is indeed baffling. 2010G has just taken 1 trade after a server reset and went silent thereafter. In contrast, the same EA running off MetaQuotes demo servers (non-championship servers) over the same period are firing orders and progressing upwards. From the last blog which showed a forward test from 2 Jul 2010 to 29Sep 2010 and with a equity gained at $102,926 (started at $10K), the current equity is now $162,000.

But at the ATC2010, only trade (21 deals at 0.1 lot) was fired once, gaining a profit of $3252.

Alpari FX Contest – I am in!

Alpari has introduced some fun to us by offering FX contest using demo $ – which means another no risk competition. It started on 7 November 2010 at 23:00 GMT (Sunday) and will end on 10 December 2010 at 22:00 GMT (Friday). The tricky part here is the leverage – only 1:10. Trading large lot sizes will be risky because of the required margin.

I am in as well with the same id as ATC2010. I am using a channel breakout based EA. Profit/Loss is now at +0.2874 using small lot sizes at 0.1 and have increased to 0.3 a few days ago.

Efficiency of EAs at the Championship (ATC2010)

An article appeared on the MQL5 championship titled, ”Evaluating the efficiency of the Championship participants' strategies by the method of Bulashev” that analysed the performance of EAs from different perspectives. It is based on a method highlighted in the article, “Evaluation of Trade Systems - the Effectiveness of Entering, Exiting and Trades in General”.

Efficiency

This refers to how effectives is the trade entered, exit, and the trade itself. Taking just long positions, the effectiveness of the Entering is:

enter_efficiency=(max_price_trade-enter_price)/(max_price_trade-min_price_trade);

With a range 0 to 1.

Similarly, for Exiting:

exit_efficiency=(exit_price - min_price_trade)/(max_price_trade - min_price_trade);

With a range 0 to 1.

For Effectiveness of the trade itself:

trade_efficiency=(exit_price-enter_price)/(max_price_trade-min_price_trade);

Generalised:

trade_efficiency=enter_efficiency+exit_efficiency-1;

The range is –1 to 1.

From the analysis, it seems that the efficiency of entering exiting is not critical, rather it is the money management.

Weighted Efficiency

The difference here is to add the volume of the trade to the formulae. Again from the analysis of the top ten in the championship, the results were similar to the those where the lot volume were not considered.

True Strategists

Just basing on the Effectiveness of the trades yielded a different list from the top 10. It has yielded the following:

Points to note

The analysis is a short term view as the Championship has not even crossed the half way mark. Hence the true behaviour of the EAs may not have surfaced.

It is often commented that the best EAs are not necessary those at the top. The risk management is such that if you make it, the profits will rise meteorically. But if it fails, the fall is downright hard. So it is the moderate risk EAs that are better suited to most of our temperament. Hence it may be better to judge EAs for the effectiveness of their trades.

Pre-Championship Peformance

Here are the backtest and forward tests results of EA 2010g.


Backtest from 1 Jan 2010 to 1 Aug 2010


Forward test from 2 Jul 2010 to 29 Sep 2010



The forward test is not a true reflection of the final set of parameters chosen and of course the errors that were still existing while the EA was developed.

ATC2010 - I am in!

After 3 months of learning and working at MQL5, I have got my EA codename "2010g" accepted into ATC2010. MetaQuotes' 2010 Automated Trading Championship is based on EAs written in MQL5, the successor to MQL4.

This year, there are about 1700 registered participants, but only a mere 284 EAs passed the championship rules to compete. This figure was subsequently revised to 314 when the organiser had more time to do a manual test of the submitted EAs. So 2010g is amongst the 314 ready to compete on 4 Oct 2010, Monday. The competition will end on 24 Dec 2010.

2010g is a completely different EA from that submitted in 2008. wetoea is a weekly trend-based EA, which managed to stand at rank 95 out of 705 other EAs. Let's see where this new EA will go, especially when there are fewer EAs in the competition. Anyway, there are many levels to survive, especially when we cannot tweak the EA or correct any mistakes:

1. not making a loss
2. not getting margin call

2009 Summary

2009 is ending. This one year marks the greatest change I have seen in the world of Metatrader platform. Let's reflect on this.

Metatrader Success
MT4 is now one of the most succesful trading platform for retail forex traders. The platform is simple to use, easy to understand and with a host of EAs and indicators from the Internet. You can download the software easily and there are many brokers providing free demo accounts. Reputable brokers who did not implement metatrader had jumped in - Gain, MB Trading, FXCM and Citibank to name some. This is a Russian product and they have got it right!

NFA disruptions
Lots of new rules from US's NFA seems to target metatrader because it has become a platform that is too lax eg no more hedging with the same currency pair. But it is partially this that makes metatrader successful. Brokers have implemented different strategies to counter the change of rules: from moving to a location outside US to developing a middleware layer to mitigate the changes.

No more universal EAs
So with the NFA disruption and different mitigating strategies used, it is no longer possible to implement an EA that can be used on different brokers. One must now know the idiosyncrasies of each broker.

MQL5
MQL4 is based on functional programming. This is considered a yesterday technology. But it is simple and many can learn it quickly, including traders who have no programming background. MQL5 is object-oriented and MetaQuotes has decided to "upgrade" to this newer in the name of execution speed and of course to bring it in line with today's programming technology. Will MQL5 succeed? A lot of EAs and indicators need to be re-written. The platform has been released to the public for beta testing.

No Automated Forex Trading Chanmpionship
The yearly championship run by MetaQuotes did not occur this year. MetaQuotes explained in the early part of 2009 that they are working hard to release MQL5 and will instead have the competition in 2010 but based on MQL5 so that they can promote their new platform. The Russians had a head start in this as the Russian platform had been made available long before the English version. Well, more importantly, will MQL5 enjoy the same success as MQL4 and will MQL4 ever be replaced, even though the championship will now be focussed on using MQL5 and not MQL4?

Tips for asking someone else to write your EAs or Indicators

I have been coding quite a bit of EAs and indicators for some people who I converse over the Net but have never met or know them personally. Interesting, there are common tips that I like to share so that you can make the transformation of a strategy to an EA or indicator easier.

STARTUP PHASE

1. Write down all the steps. Yes, if you can't pen down your strategy, it will never be clear to anyone except yourself. You might even discover that you are not clear about it yourself. Of course, if it is a "commercial" strategy, your job is all done up, except for some clarifications. And, yes include charts with clear annotation.

2. Write down your steps in English and unless you know MQL4 coding a fair bit, do not mix the English description with the code. For example, if you want a trade entry at a start of a new day, then say so. Don't mix in a code that say start trade at Hour[0]=0 unless you really mean the 0 hour.

3. Be prepared for lengthy clarifications. Coding needs precision and if the coder is uncertain, he will check back with you. For example, is the start of a new day, the broker's server "new day" or is it some time zone reference that you need?

4. Language is still a problem. We are all in the forex world, but still we can mean differently although we are using the same words. For example, MT4 has stops and limits and a buystop means placing an order above the current Ask price at some acceptable distance and a buylimit means placing an order below the current Ask price at some acceptable distance. But when someone said there will be a limit order or stop order, he could mean:

• "a limit order is used to start a trade if the price hits a fixed level"
• "a stop order is used to close a trade if the price hits a fixed level "

Or another person may say

• "STOP is to do with LOSSES and LIMIT is to do with PROFITS"; with a stop order you decide how much you are prepared to lose ie stoploss; with a limit order you can enter a trade at a specified level and you can specify when to exit (takeprofit)

So why is there so much confusion. Well it is because MT4 allows one entry order with stoploss and takeprofit taken simultaneously. For brokers with ECN, you can only place an entry with no stoploss and takeprofit. You can do your takeprofit and stoploss at the next step. As a result buystop or buylimit is really peculiar to MT4 and hence no commonly understood across the forex market. Of course, the orders under MT4 platform are a lot easier to understand and hence one of the reason for its popularity.

5. Where are you trading from? Sometimes it matters where and when you are trading from, so specify your city and country and the local time you are trading from.

TESTING PHASE

6. With your strategy coded, it is time to test out the EA/indicator. If there are errors, go to the expert and journal tabs and copied out the error messages. It is easier to debug with these messages.

7. For clarity, always specify which broker you are using and better still note the following:

• use of fractional pips ie 2/4 digit or 3/5 digit; note whether this is also true for the live server - sometimes the demo server is on 3/5 digit system and the live system is on 2/4 digit system
• note the server time in GMT
• does your broker use suffix like EURUSDm instead of EURUSD?
• is it an ECN broker?

EVALUATION PHASE

8. You will be surprised that most of the time, your EA will not do well. There are many reasons for this. Be prepared to discuss the strategy with your coder, who is preferably a trader himself otherwise there will be little value-add.

9. Hidden intuition. A lot of traders use intuition based on experience and it is a challenge to capture this hidden intuition. This is the secret to a successful EA. Think hard what you are looking for that is not documented in the steps. Sometimes the answers may not be even found on the screen.

10. Please do extensive testing before going live.

Hope you have found this useful and hope you can translate the gems in your head into a program that can make trading easier for you. Make that a goal for 2010!

Me at SGBarcamp4

I have put my thoughts together to finalise the talk, "Automated forex trading and the world of MQL4", for the coming Barcamp at Suntec Tower 3, Singapore this Saturday. The people who attend such barcamps are broadly two groups: the the techies and the social media/networking enthusiasts. So we have people who are hard core coders and those who are knowledgeable about Web2.0 stuff.

I hope to cover the "101" of automated forex trading using MQL4; that's alot for a half hour session that is not supposed to be one-way but interactive. On the other hand, I hope to introduce interdisciplinary mashups of talents into the world of forex trading - so i will cover the 1st to 5th derivative of linear displacement, price distribution with skewness and kurtosis and astro finance. I don't have time to put together the frequency domain trading via cycles and bandpass filters. Not sure if I have time to cover so much, we will see. Let's see if we can surface more talents into this difficult field, otherwise it is good to let programmers know that perhaps this is one area and minefield to explore, with caution.

What you plan is not what you get

I have an EA that runs well on demo account but fails miserably when used on a live account. Why?

I am sure you have encountered this and the first thing you probably did is to say the strategy fails, then drop the EA and look for the next holy grail.

The truth is - there are many differences using a live account as opposed to a demo account. Let's explore a few of them:

1. Complex-previous-trades-dependent strategies - this is a curse of automation; with the ability to execute complex trading strategies because of automation, strategies that hinge on a previous or many previous trades are doomed to fail. Why? When the sequence is broken, you will not be able to recover, even though you may have put in alot of mitgating actions to counter this break.

2. Someone at the other end will figure out your strategy and plan for your failure. You can't trust anyone, no, not even your broker. Do a google and find many horror stories. Someone will claim to have been cheated by broker A. Another may say that for all the years with Broker A, he has no problem.

3. Have you encounter many error conditions like lost connection, trade with wrong prices, etc? How genuine are these error conditions? If your EA does not forsee this, and if you do not incorporate the "right" mitigation strategies, your account can be wiped out either slowly or quickly, but surely.

4. You won't believe the things that can happen. For example. you execute a trade. Your EA thought the trade was executed (even though you have put in extra checks) and the other side happily drop your request and not even informed your EA - in the name of an error condition.

5. Really the truth may be that your EA was not written properly and not doing what you wanted it to do.

Just be aware that real trading account is different from a demo account. There will be some winners, and some losers. You need to lose so that someone else will gain. Tread carefully...

Me at SG Barcamp

My friends are organising a barcamp in Singapore on 21 Nov 2009. I have decided to join in the fun and hope to present the MQL4 automated forex trading to a group of fellow developers in other areas.

What is a barcamp? Essentially a conference where the contents are user-generated. It is usually open, participatory and contents used to be technical but nowadays there are more fun topics like "How to date a Japanese girl" (presented by a Japanese girl). Attendance can range from a few hundreds to a thousand. Barcamps are getting popular with a group of like-minded people hustling together to explore the changing world advanced through new IT technologies and ideas.

One has to fight for a slot so hopefully I can share something. Otherwise, there will always be a next.

MQL5 ready for puiblic testing

It is going to be a busy period down the road now, with MQL5 being ready for public beta testing tomorrow (12 Oct 2009). MQL4 has become so popular that a lot of big time brokers have joined in. I am not sure if MQL5 will enjoy the same success.

First, MQL5 is object-oriented, so it will be a steeper learning curve. Most retail forex traders will find it harder to learn.

Second, the compatibilty with MQL4 is not maintained because the creators said the language concepts are just too different. They tried to keep the compatibility but they couldn't. So all EAs and indicators in MQL4 have to be rewritten in MQL5. We can only hope for someone to write a tool to do a conversion for us.

Third, there are just too much success in MQL4 to motivate people to migrate. The best thing happened so far is NFA's new rules which have rendered MQL4 useless so much so that some US brokers just gave up, or encourage traders to use the non-US operations.

We shall see. For me, it just means spending some time to study MQL5 and prepare for the day when brokers indeed stop using the MQL4 platform and force us to move on to MQL5.

More Japanese indicators

Japanese are very innovative people, and you can see this from the progress of the nation. Step into Tokyo and you will discover that they are so willing to use IT and electronics in their everyday processes. It is not a "should I" but instead you will find IT/electronics quite embedded. Take for example the $10 haircut idea copied from Japan which is commonplace in Sinagpore now. Using a machine to collect the $10 notes in exchange for a token card before having your haircut solves the money collection issue by the bosses. No fraud, no miscounting and best of all the money is safe in a machine and no robber can come to clear the till.

OK, I digress. I just wanted to express my admiration of the Japanese and share their innovativeness even in trading. I talked about the Ichimoku cloud chart just a while back. Take a look at other Japanese invented indicators like Kagi charts, or Renko charts. Yet another way of looking at the prices without having to worry about the time factor. Renko candles are just bricks and boxes with no regard to time - just price action reactions. Apparently you can spot more trading opportunities using Renko than candlesticks, afterall the inclusion of open times, closing times, high and low are arbitrary formulated based on time frames. Change the time frame, and information from candlesticks change. Oh by the way, the often used candlesticks is also a Japanese invention.

Kagi charts are similar to Renko charts, paying no regard to time. Kagi charts are joining lines that look like steps depiciting highs and lows and changing of higher highs and lower lows.

Not that these indicators are important to me. I use EAs and hence don't even look at charts. So if we can have charts with no time factor, there must something to all these that is worth investigating more,

Clouds watching

Since I am now in Tokyo, Japan, basking in the beauty of the Land of the Rising Sun, it is only appropriate to write about a Japanese indicator called the Ichimoku Kinko Hyo developed in 1969.

When I first came across this indicator, I was not only fascinated to find out what the name means, but the colourful lines and meshes that seem to make this indicator mysterious.So what does Ichimoku Kinko Hyo means? Translated literally it means "one-glance balanced bars/chart" or non-literally, "chart equilibrium at a glance". It comprises of moving averages over a determined period and shifted moving averages. In short, it is really a trend indicator with support and resistance lines. Ichimoku Kinko Hyo or Ichimoku in short (or some like to it cloud charts) is commonly recommended for longer time frame trading such as daily or weekly. But some have shared success even at lower time frames. But I still think that one should stick a minimum of daily time frame.

It used to be difficult to find information about this indicator, but do a google now and you will find tons of them including books. So I shan't write much about this.

Operating in a different domain

It has been a while since I last blogged. I have been investigating a few interesting ideas. One of them is digital filters. Filters are what it says it is – filter off unwanted signals. In the world of digital filters, we are operating on the frequency domain and not time domain. They are related – after all, time is the inverse of frequency.

So what is the advantage of operating in the frequency domain? One clear advantage is the reduction in lag time. A low pass filter lets low frequency components pass through but filter off high frequency components. One way to achieve this is to use a moving average algorithm. Therefore most low pass filters are smoothers and hence the spiky signals get attenuated and a smooth curve can be obtained. Unfortunately this introduced lag, and most indicators are laggards. The more we filter off, the more lag it introduces.

A high pass filter filters off low frequency components, the exact opposite of the low pass filter. Add the high pass filter to the low pass filter and wala, you have a bandpass filter. A bandpass filter determines the range of frequencies you want to let through.

Sounds like a bunch of electronics engineering. Yes, it is and google "John F. Ehlers" and discover a world of trading equivalent indicators that are derived from electronics engineering.

And here is a demonstration of the power of digital filters.

EAs are not universal anymore

It used to be when you write a metatrader EA, it can be used across all brokers who has this platform, and the only thing to take care of is the server time as each broker has different GMT settings and that is provided the trading strategy is time dependent.

Today it is fragmented, in just a matter of months. The first is fractional pips, some brokers have switched to offer this, others remain. But this is easily solved. Next are those US brokers who are NFA registered. In this category, there are different implementations. One broker said everything remain the same, just make sure your lot sizes are not the same so that some backend system will handle the no stoploss, no takeprofit, no trailing stops that NFA has forbidden. Of course all NFA brokers cannot offer hedging now. Then on the other side where brokers are not US based, there are those who offer direct ECN connectivity. That means, you cannot send orders with TP and SL. These two parameters must be zero. But at the next tick, you can send in a modify order command.

Gone are the days where we can have more universal settings and code. So even with one platform, there are so many permutations of implementation that is getting tedious to write a universal EA.

Book Review - Mechanical Trading Systems

I came across this book that is worth mentioning only because there are not many books covering automated trading systems. The book is "Mechanical Trading Systems" by Richard L. Weissman, John Wiley & Sons, 2005.


It is a book about mechanical trading system, that is, systems that can be automated or expert advisers (EAs), a term I have been using frequently. It is interesting because it explores the psychology of different trading systems that a trader needs to know in order to match the system to his own trading psychology, something I had talked about in here. But there is no holy grail, it appears the author has some successful automted trading systems which he openly said he will not reveal. His purpose is to share the usefulness of automated trading and how it can be applied as follows:


1. Trend-Following Systems: patience is the key and not getting out of a profitable trade too quickly. The psychological make-up to trade such systems are patience to wait for trade entries, long drawn trades (could be months), ability to give back a large portion of profits.


2. Mean-Reversion Systems: these systems are intermediate term trades and needs discipline; discipline to overcome crowd psychology, media hype and action price reading. There will be more losing trades than winning trades, but the losing trades are small in value but consecutively large.


3. Short-Term Systems: there could be many trades in a day and require quick thinking and fixed formulas for entry, exit and money management. These type of trading is intensive and require the trader to stay seated to the screen (but this is where automated trading comes in handy).


Most of the examples of trading systems are based on technical indicators. I can only infer these are probably not the systems to use since they have been revealed!


The author has also covered the usefulness of automated systems for backtesting, forward testing and optimisation of trade parameters. At risk is the convenience of optimisation and curve fitting of a set of data that does not represent future prices.

So this is a book worth reading so that you know yourself better and therefore have better grasp in handling/using EAs.

Only FIFO orders NFA - no placing of stop loss or takeprofit

So we had a break from the implementation of the FIFO rule by NFA, deferred till 31 Jul 2009. See this note from NFA. Why do we need more time to have this new rule implemented?

Well, it is not just a simple first-in-first-out handling of orders. It also means you cannot put stop loss to your orders!! Essentially, a stop loss is an opposing order and you are not allowed to issue this order in advance.

Imagine you place two orders in the same direction (because you can't place opposing orders anymore ie no hedging). The first order you have a Takeprofit of say 100 pips. The second order you have no takeprofit or stoploss. You trail the second order such that the second order will be taken out say with a trailing of 30pips. But it does not work like this anymore. The first order must be closed first before the second ie FIFO. So as a result, you cannot place takeprofit or stoploss levels now because the first order must always be processed first.

Ok, the brokers in US are coming out with solutions like OCO (One Cancel the Other) which means you can still place stoploss or takeprofit via OCO.

What does this means for EA users? More reprogramming required, more testing needed but little time left.

So what are the brokers with MT4 platform doing:

1. Broker1 - use different lot sizes so that the orders are treated differently
2. Broker2 - drop MT4 becasue it cannot be changed to handle new ruling; so use another platform or use UK broker
3. Broker3 - still not a member of NFA

PS: Remember, all this applies to US brokers only.

Looking for the grail in commercial EAs

What is a commercial EA? It is one that you have to pay to use it, whether a one-time fee or by subscription. So which are the ones that one should go for? Well, there is a fourm you can go check out on Facebook Finding the Best Forex Expert Advisors. These people spent their time experimenting with different EAs, and many are with live accounts. So if you need a good gauge of an EA’s performance, then hear from the ground.

The truth about forex is – you can’t predict the future. You can only guess with certainty based on patterns, technical analysis, fundamentals, news, etc.

Have you ever heard someone said – if I sell, I got stopped out and should have long. But if you long, the same happens and you still get stopped out.

What am I getting into. The market does not wait to be predicted. There are many EAs with different strategies. If you lose money it is easy to blame the EA you have just bought. If you win, you think everything is fine and continue until your equity starts to ebb. Every commercial EA get to be called “spam” because either they are really spam or did not meet your profit making criteria when used at that point in time. Almost every EA you read in the forum has followers and non-followers.

Some EAs are expensive, some are cheap. Does a more expensive EA brings better performance?