Trading is hard

Not quite dead yet, but close from working insane hours.... my life story lol. Its been 5 months since quitting a good job to focus full time on HF equities trading and almost 2 years since finance and trading appeared on my radar. But never imagined in any shape or form that successful, consistently profitable trading would be so hard. Its not surprising... I guess, if not painfully obvious as if everyone could extract money from the market like this, then everyone would be living on a Yacht in the Bahamas... or whatever fantasy you can conjure up.

Problem is the barrier to entry is unbelievably low, similar to "Art". Anyone can pickup a paintbrush/whatever and make something and put it on a wall, just as anyone can buy and sell stock/whatever and plot it on a chart. The difference? Unlike friends and family is with your Art, the market is a brutal, merciless, heartless bitch that tells you exactly what it thinks, blow by blow, day by day, no hesitation to kick you in the teeth, with cast iron steel capped boots, when your down and out in a pool of your own blood. Then finally, in a very polite and impersonal way gives  a quantifiable score every minute/hour/day/month and year - sounds fun eh? :P

Guess its no co-incidence some of the smartest and sharpest people I`ve met over the last year or two are HF traders and what Ive found in nearly 2 decades of writing code in some shape or form is, you typically find is one, two or maybe three if your really lucky of these kinds of truly gifted people in your standard tech company/division (*1) yet it seems every successful HF trader ive met is one of these gifted people - none of this big dumb asshole trader image that is typically in popular culture.

So as 6months draws to a close, after dozens of strategies, endless hours looking at charts, flows, stats, far too much hair pulling and about 5 completely re-written trading systems(one day will write that all up) I`m ready to jump in and start trading for real January 2012.

Maybe PnL >0
Maybe PnL < 0

But...  Enjoy the holiday break and all the best for 2012!!

(*1) - this means people who get shit done, not those who can cite text books, bullshit brilliantly or display alphabet soups after their name.

Tick Life

Someone recently asked "at what speed is the game played at now" on HN. To which one of the responses was "do milliseconds matter". The answer of course depends on what kind of strategy your talking about, but for the sake of argument presume he means an ultra high freq/low latency every tick/quote/trade matters style strategy.

So heres some plots to show how fast (or slow) the market moves, first up is the BBO for part of the morning session early this month on..... you guessed it MSFT

And the life of each tick, measured in nanoseconds. Life means, every new BBO on the bid or ask resets the timer, e.g. starts at 0ns

Above graph is scaled verticaly to 60Bn nanosecconds == 60secconds == 1minute. A quick look shows at most one BBO pair`s life maxes out at around 30secconds.

If we zoom in a bit more, the above plot is the same graph but vertical scaled to around 15 seconds. From here it becomes more apprent the vast majority of BBO pairs are under seconds, possibly less. 

.. and continuing the zoom, below is some half interesting zoom in. First plot is the BBO

and yellow plot is the tick life. Here you get a better feel for how long each BBO pair lasts. A few last 10sec, a few less than 1sec and son.

digging further, below is a zoom in of the small ant hill seen on the larger BBO above.

and the corresponding tick life below.

... where you can see the same 15sec or so life of the BBO peek. Ok so your saying 15sec eh? dosent sound so ultra high frequency? true until you dig further down and say, whats the lifespan of a BBO pair which has a spread of 2cents?

... (above plot) when the spread widens is circled in red. First one around 1M ns around 1 millisecond its fast but not that fast. The second one 0.1M ns, or roughtly 100microsecconds, to put that in perspective, the standard linux kernel scheduler slices at a granularity of every 50microsecconds.

... 100microsecconds is where the fun starts. 

Silly tricks

... and the long grind forward continues abeit slowly. This week found some weirdness with the exchange and problems with my strategies using end of day tick data aka non realtime paper trading. Was using slightly stale 3-4Month old data and figured it should be close enough but dam.... how much things have changed in a few months. Will post about this when ive got some spare cycles.

Im a huge fan of autonomic computing, which means hardware&software systems that do integrity checking and automatic recovery, with the obvious examples being RAID for disk and ECC for ram. Once upon a time I worked for and with (insert huge megacorps you all know) and had a fascinating discussion with one of their hardware engineers.

HW Dude: hmm thats a weird problem, whats the value of register at offset 0xbeefbabe?

Hacking Nasdaq: register 0xbeefbabe reads out to be ... 0x01234567.

(silence)

HW Dude: are you sure thats correct?

Hacking Nasdaq: ... yes

HW Dude: thats impossible, are you on crack?

.

.

.

What he was referring to is the MSB of that register was the logical OR of the other 30bits. Making a value of 0x01234567 impossible with the only correct value being  0x81234567. Moral of the story is encoding self integrity checks into everything makes it easy to catch and not waste time on dumb ass errors - the "oh woops the cable isnt plugged in, sorry" kind

... which leads us to OUCH and the 14byte identifier token. What Ive done is reserve the last byte as an integrity check such that the 32bit sum of the previous 13bytes modulo 26 is its value.

e.g.

u32 Sum = 0;

for (int i=0; i < 13; i++) Sum += P->Msg.OrderAdd.Token[i]

P->Msg.OrderAdd.Token[13] = 'A' + Sum % 26;

This way its trivial to check if the OrderToken you looking at is actually real or corrupt without any effort. So far ive caught this a few times, usually when some bit of code has gone rouge and pissing all over memory.

...and yes thats only in development, no errors in prod... yet :)

stuffing the turkey

Been a bit crazy busy of late so not so many posts, but keeping in the spirit of things here is an interesting POV on quote stuffing and some of the basic microstructure games that get played.

Assume the following plots

Best Ask (Above)

Best Bid (Above)

Open Qty @ Best Bid (above)

As you can see(hilighted in red) what looks like a neat pristinely manicured bit of astroturf among the wild grass, weeds and occasional dead spot. What someone is doing is adding to the BB qty, then immediately canceling it all in rapid fire.

Above is the time delta between changes, where its toggling it every say 100usec. Clearly some algo messing with the BBO. Now what happens if you use a dumb ass  simple moving average of say the last 1024 open qty levels of the BB - plot below.

Above between the yellow lines is the average open qty @ BB when on the astroturf.. and that is how you can exploit stupidityweakness in someones algo. Here it is again with a more aggressive vertical scaling

Again, a beautiful manipulation of a simple moving average. So whats worse? Spaming the market with a bit of noise OR using a simple moving average as a signal? Personally Id say the latter as we`re no longer in the school yard where there`s no "special needs play area"... However the former is far easier to bitch and complain about.

"The Wall"

Found a just classic example of the problem currently fighting with, an uptick which all signals says GO but has the crap beaten out of it by some freight train.

Below is the best bid

Red circle is where the strategy correctly predicts an uptick, but as you can see it promptly gets hammered down. So lets look at the qty open at the best bid, first thing you notice is this massive bid qty spike, kind of like a train heading towards a brick wall. 

Above is the open qty at the best bid, where its clearly visible at the previous valley(hmm whats the lingo for this?) someone adds another 10K shares at that price level adding a nice amount of support. Then as expected market uses that support level pushing the price up again.

Above is open bid at levels 1, 2, 3 away from the best bid. As the best bid price moves up can clearly see "The Wall" moving away then returning until it becomes the best bid again.

At this point would expect the price to bounce back up as there`s support, but what actually happens is this.

Above plots are 4 signals, best bid price(one close + really closeup) with red circle where expect the price to bounce back up. 2nd is the open quantity at the best bid, and 3rd is the cumulative execution qty at that price level.

This is whats frustrating, whats happening is "The Wall" returns (in red), price jumps 1 tick then.. someone decides to hit the entire best bid, knocking the price down again, then continues to eat chunks of 10K shares at that price level (red, blue purple) - the price I thought no one was interested in... Probably the most interesting thing is not that someone hit the bid, but whoever is buying continues to replenish with 10k shares! And thus the reason half my orders just beat the spread, with gross PnL 0/net PnL neagtive hmm...

If you read all the trading books and what not, the less technical ones really push "whos on the other side of your trade" e.g. who are you trading against and whats their psychology/motives/brand of coffee/star sign/phase of the moon or whatever. So with that said, probably the most important question here is, is it man or machine? e.g. is this some crazy ass trader thumping 10k shares buy/sell button on a workstation? or is this algos at play.

Above is the delta time between quotes for the above sequence, where vertical axis is time in nanosecconds. As you can see theres this huge chunk of black in the middle with some fairly high peeks, one  goes all the way to around 500msec. But this whole sequence is on the order of 1seccond or so. I guess its possible the seller is human but would have to be really quick to pull this off, i mean *really* fast, which is possible but I think unlikely seller is clicking some GUI button here. The buyer on the otherhand is clearly an algo.

Above is zommed in time delta between quotes/trades in the same time series as the above, 100K on vertical is 100usec. Where we can see the bid qty for "The Wall Part2 & Part3" happens within in a few MSec of the executed order... which has to be a machine. 

... hmm so how to model this...

Not loosing, is not winning - transaction costs.

Been a few days since the last post as just hammering away at microstructure strategies which is dam fun but frustrating at times - starting from 0 with nothing/nobody to learn from. Found some nice trigger happy strategies that does *not loose* 90% of the time with Sharpes in the double digits but... there lies the problem, theres 3 states(win, loss, draw) not 2. A "draw" e.g you correctly predict a 1 tick up/down swing where the spread is 1tick happens 45% of the time with win (> 1tick) the remaining 45%. Meaning you end up with gross 0 PnL for 45% of trades as your just beating the spread. Then accounting for transaction costs puts it into net negative PnL 65% of the time (draw + loose) and thus unprofitable - teh sucks dude.

For reference transaction costs are:
- $0.0045/share (broker fee)
- $0.0030/share (nasdaq remove liquidity fee)

Whats happening? its predicting correct movement very well except that movement is rejected (as in basketball reject) half the time as everyone jumps on the first uptick aka mean-revision and knocks the price down 1 tick. So... need to better classify the rejected jump shots from the high Sharpe slam dunks.

...the expedition continues.

the opening spam

Been poking around looking at how the market moves and thought would share a "how to spam the open" session.

First the plots, where everything is in "tick time" e.g one X unit is 1 tick. .. we have the best bid

.. and best ask

Next up is how far away from the best bid the tick is (zero if tick is a sell)

.. and the other side

finally the time delta between ticks (in nanosecconds)

As you can see the closer it gets to the opening bell the harder it bursts, and the further away from the best bid/ask it bursts. But... is it adding or removing shares at that price level?

Above is the share delta for ticks on the buy side, where positive is adding shares to the book at that price level - clearly its adding.

... and the same for the sell side, also clearly adding (positive share delta).

So whats it doing? clearly its layering orders vertically across the entire book, presumably at prices that would be very attractive should they get matched - e.g. highly passive strategy. But... why so close to the opening bell? Guessing theres also a spamming component to this. Where if theres 50K ticks in the queue to process, all at different price levels your orderbook has no choice but to update everything and thus, quite possibly, delay your snapshot of the world by 100`s of ticks after the first execution. 

Fantasy? not sure but there`s a very clear and distinct pattern/strategy going on here.