Monday, 24 December 2012


Cant believe it, first post on this blog was exactly 3 years ago wow... seems like such a long time. 3 years ago I couldn`t tell you the difference between buy, sell, bid, ask, long or short yet some how the projects and code I`ve written during this time now checks or trades, millions, tens of millions, hundreds of millions and sometimes billions of dollars every single day. The depressing part? all the time and effort in finance has not translated into George Washingtion`s. in my bank account.... bugger.

So when your imagination and reality are so divergent its a good  times to step back and work out what your missing and not even consider the how or where of the equation - just focus on the what. Lets take a high level look at the game from the top down.

HFT game has 5 major components

1) Speed / Deterministic / Technology
2) Commissions / cost structure
3) Influence with the exchange
4) Capital
5) Strategy

Speed / Deterministic / Technology 

This is the classic arms race, bigger faster, better. Its very true technology is a large component of any HF strategy however the trap technologist (like myself) fall into is, they believe technology alone can beat the market. What we dont realize is our technology edge starts and ends at the cable plugged into our box/switch. Theres so many other moving parts that are out of our control that tips the balance way into the red.
In Round 1 any technology edge I had ended when it entered at the brokers pre-trade risk checks let alone entering the exchanges infra. In Round 2 have best in class access to the market but the edge is gone once the packet has left the NIC. Net result, speed/technology is important but you cant win a trade with it alone.

Commissions / cost structure

HF strategies are unique because the per trade profit is very small, which makes it extremely sensitive to the per trade and monthly recurring costs. For example if your trade makes +100 cents on a win and -100cents on a loss with say 100 shares eg. 1cent/share. Say your broker, the exchange, the SEC takes 25 cents or 1/4 of the profit.

This means you need a 15% edge (65% of the trades profitable) just to break even.

Yet if your total fees is 5 cents vs 25 cents you need only a 3% edge (53%) of trades to break even.

A delta of 12% profitable trades, means you need a really good signal, which is extremely difficult. Thats just to break even!

Influence with the exchange

This one is probably a bit surprising to many as in theory an exchange is equal access for all participants. Thats the theory, yet in practice an exchange has finite manpower and equipment which results in the dude trading $100 / day vs the dude trading $100MM / day gets vastly different levels of service.

Aside from man-power theres always political / boys club games and really is a catch-22 situation. To join the club you need to trade massive volumes, but to trade (profitably) massive volumes you need A class access into the matching engine.

Why is it important? Your technology edge ends at the BGP endpoint (your router/switch) with a shit ton of other crap between that endpoint and a trade thats in the exchanges control.


For the HF game capital is usually not a problem. The true believers are flat at the end of the day with no overnight / borrowing costs. Intraday your capital just limits the sizes of your position. The one thing capitial does give you is, market access and influence with an exchange. Any tier one prime broker will want a few million dollars deposit before they will even talk to you. Next step down requires a few $100K and so on. For now capital is not an issue.


And finally strategy, this is the true leveler. If you have excellent strategy then all the above can become noise. Yet the inverse is not true - you can have the best technology in the world but it will never make a bad strategy good.

This is by far my biggest weakness and the skill I need to develop further, but what exactly needs to be learnt? The what is not an easy question. The first 6months of this expedition was largely wasted doing essentially some form of technical analysis - if signal X crossed signal Y then buy/sell kind of strategy development. Im sure it works for someone but thats not the HF game to play.

So what`s missing? its how to mine the data and what to mine to gain a statistical edge. The how to mine is an easy technology problem, the what to mine is not. Im convinced machine learning is the way forward as its the next logical step beyond stat arb. After all isnt stat arb a form of the most basic machine learning? guess it all depends on your definite of learning.

Whats next? Bayes, Markov, Kernels, Boosting and Kaggle mmmm Kaggle lol


  1. You'd really do better to forget all the math stuff and focus on trying to get domain (i.e. trading) traction. ML is not a generator of trading ideas.

  2. thanks for the comment, interesting point. my idea is using ML to find the interesting patterns, from that you can generate the trade ideas.

    certainly is not feed all this data into a black box and it magically generates a buy/sell signal.

    your thoughts?

  3. I'm no expert, but I have been around long enough to see these patterns. Your story seems fairly typical of someone who comes at trading (of any frequency) from a technical background, in so much as they try to reduce it to an engineering problem, so what happens is that the person ends up running millions of backtests and getting nowhere (as I know from experience). You need to make the transition from programmer to trader. As a trader, you need to try and play the weakest players, but by trying to do HFT in liquid stocks you're attempting to battle with the strongest players who not only have better resources, but crucially they have more trading experience. I would go up in terms of time and just watch the markets, see if you notice any repeating behaviors, try to note unusual events etc, then try to model them using ML or whatever.

    Didn't you mention a while a go that you had hooked up with a group of prop traders? Did this not yield any useful information?

    With your infrastructure you should be easily able to get some kind of basic intra-day stat arb going.

  4. Thanks for the comments. Yup guilty as charged, been treating it as a technical problem. Last 6 months or so my perspective has changed alot. Use to think it was all about numbers, greeks, algos, code etc etc but now its behaviour and patterns.

    Problem is how to find those patterns, thus the ML angle. Think have enough experience now to know what to feed the ML e.g. garbage in/garbage out kind of thing but from the sounds of things you look at the market and find interesting things, but how? looking at charts, or crunching data or ?

    Current prop group is unwinding. Learnt alot, traded alot but net pnl just not there - trades very competitive e.g. well known trades. Will post about it shortly.

    Think I know what to look for now (patterns), just need to work more on the how part. Any advice?

  5. I'd look at highly correlated mid volume securities (ADRs for example) and look for fair value anomalies at the open. Another idea would be Yeah, these will be crowded trades, but you should be fast enough.

  6. Hmm academic SPY vs IVV arb, my favorate part is

    "We focus on mispricing of 0.2% and above (net of spreads) as we do not want the sample
    dominated by small mispricing that is offset by commissions and short sale costs. The mispricing
    we document is not a common occurrence. We find just 183 instances over the 2001 – 2010

    I mean seriously wtf were they thinking? 183 arb trades over 9 years my god. Actually had a look at SPY/IVV early this year, was highly correlated to the point where there wasnt an aggressive trade there.

    Like how you worded that "fair value anomalies" so true, thanks.

  7. Unfortunately, a little imagination is required to extrapolate it into the real world, but you sound like you have matters in hand.
    Good Luck.

  8. Yes a little :) Thanks for the advice, really appreciate it.

  9. Is there a reason you don't just play market maker using your speed to get near the front of the queue and pulling orders when it become disadvantageous? From what I read, that is basically what most HFT do more or less.

  10. Yes thats the market markets / liquidity provisioning game. Whats difficult here is

    1) infra and market access requirements to play that game are pretty high -> read expensive.

    2) having an edge in knowing what the true price is. Stated different, knowing when your order price against the true price become a disadvantage. The magic is of course knowing the true price... certainly not easy.