Too Many Hats

Another week with consistent alpha proving elusive. Got most of the tech problems got sorted out which left only math problems. The math unfortunately is not so easy to fix. Was running a few incarnations of a sub tick arbitrage strategy that unfortunately the combination of comissions/rebates/latency/law of large numbers means it will never make money/inconsistent or not enough to cover the fixed infra costs. So hit the puke point, taken the stop loss and building a new strategy.

Whats interesting(and frustrating) is I can now build a variety of profitable back tested strategies fairly easily, which was certainly not the case a few months ago. Yet when running it live it fails to match the model. The reason? many. but to a large extent I didnt know alot of the characteristics/behaviour when interacting with a real market/real broker which significantly eroded the edge.

My general process for strategy development goes like

Stage 1) "back of the envelope mode" this is quickly test some basic assumptions with level 1 data or medium granularity bars or both. At this point don`t care about latency or even the queue and commissions/rebates arent priced in. The goal here is to check the "edge" is there and if so, it needs to be sufficiently strong  & consistent.

Stage 2) Add the queue/comissions/rebates/tax and then trade & quote data. At this point everything is run in lock step. e.g. one step == one message. Find this is where I most of the time. Whats good about this stage is, its really easy to change the strategy add new things, remove things do all sorts of random stuff to it and not have to worry about any of the order entry/market data details, e.g. order/cancel reject handling, re-priced, latency, etc etc

Stage 3) Code it up as a a real strategy. This means real async order entry/market data and you now have a time constraint - if a tick takes 1sec to crunch then you will get run-over. The environment here is a fully simulated exchange both order entry & market data that is replaying historical. This can be fiddly but your just re-coding up Stage 2) which is fairly straight forward and the environment you already coded thus dosent take very long.

Stage 4) Run it live against the real exchange. My setup makes launching a strategy simply executing a program on the command line, theres no monolithic gui/master program/program within program. All a strategy does is talk to shared memory. Thus can run it one shot at a time, or continuously, have scripts calling scripts call the strat, etc etc the usual linux scriptfu. From here its easy to monitor and watch, tune and fix/debug. The key here is to make it match the theoretical/backtested model or change the theoretical model to match the observed behaviour.

... and this is where im at. Sucks pretty badly spent so much time and effort so far without any real success. Had a few highs tho, turning my position over a few times was really cool! as was sending half a million dollars worth of orders to the market but sadly, still missing something, still in the red and the  20H days are catching up.

the quest continues...

Week Number 2

Wow my 2nd week live has finished, pretty much just testing the break, accelerator, warming up the tiers, a few test laps, getting a feel for at what speed/rpm to shift gears. how it corners and most importantly how fast does it go and how fast can it stop.

so far no trip to the emergency room.... yet lol

This pretty much sums up the experience

http://www.youtube.com/watch?v=VaFP1yZykuk

... and yes getting wacked happens a bit, tho only down $20 so far. Whats weird is there`s this completely irrational fear, similar to the fear of talking to that girl... or hitting that asshole on the nose...  and now a new addition...  putting 100shrs at the BBO. Theres no logical explanation for it. Sure, she might ignore you, or you might get some nasty bruise or you might get executed with a 100shrs of an incredibly liquid stock, all 3 of which are of little consequence.

So just like the above examples, after you`ve done it a few times and realize its not so bad, the irrational fear dissipates and the analytical part of your brain takes over from more primal/emotional.

Basic breakdown has been

Week 1

- code order entry gateway

- code market data feed handler

- code risk managmenet

- cron jobs

- scripts of all sorts

- various system infra stuff.. pissed you cant monitor ECC failures

- various silly bug fixes

Week2

- recon/SOD/EOD automation

- performance/latency tests

- debugging latency problems with market data

- re-wrote feed handler (latency probs)

(most of the week spent on market data latency)

Whats gone wrong so far?

- incorrectly sent massive burst of cancels (5000+) ... opps

- VPN connection into colo dropped out while trading! holy fark!

- a few run-away terminal sessions, flood of printf`s makes ssh un-responsive

- desktop PC`s deciding to update and reboot by themself

- pretty much most of my worst fears have happened

Murphy I really do hate you.

Next up? bit more latency/performance tuning and now finally digging deeper into strategy development, and as they say.

Gentlemen, please start your engines!

h4cking HASDAQ 1U High Performance Server Model 1337

Busy as hell.. seems to be a the never ending story. Original plan for colo was teaming up with a small HFT group that would have been really cool but.. alas fell thru at the last minute, thus had to re-group both infra and strategies. So as of now im finally plugged in hooked up and almost ready to let that first strategy run loose to generate those millions of dollars ... ho ho ho ha :P

When you get ready to put servers into colo theres a sudden realization of how expensive enterprise grade name brand servers are. Typically HP G6 or G7, various flavors of IBM and occasionally a DELL shows up. Unfortunately these kinds of machines are out of my budget or more precisely I`d rather spend the money elsewhere so.... lets build are own 1U server on the cheap.

Reference point is

HP ProLiant DL370 G7 High Performance 2U Server
x2 6 core 3.4Ghz X5690
16GB ECC memory
5TB of disk space
Dual 10Gb Port NIC

http://h71016.www7.hp.com/dstore/MiddleFrame.asp?view=all&oi=E9CED&BEID=19701&SBLID=&AirTime=False&BaseId=35620&FamilyID=3180&ProductLineID=431

For a total of $14,421.00 ..... OUCH!

Brand new h4cking NASDAQ 1U High Performance Server Model Number 1337

... lol

So whats in the box?

x1 4 Core 8 thread 3.4Ghz E3-1270 (Sandy Bridge)
16GB ECC memory
4.5TB HDD space
120GB Intel SSD
x1 SolarFlare Dual 10Gb NIC

... and yes its a tiny 1U as you can see from the $20 bill. Typically this kind of chassis is used for low end systems as the aiflow is pretty messed up ... but it fits my style, lean, mean and a hell of alot cheaper! Only major concern is airfow as cold air intake from the side and the blower pushing the air  out the back (pic is a little old) yet so far no problems.

To break the costs

Motherboard is SuperMicro X9SCM-F (IPMI very important) - $200
Processor E3-1270 - $340
16 GB ECC Memory - $200
WD Green 2TB - $120
WD Green 2.5TB $120 (disk in the picture is a 1TB that was replaced)
120GB Intel SSD (thing between PSW & HDD) - $200
Chassis, imported directly from Taiwan - $100 (including shipping + PSW + Blower + Riser)
SolarFlare Dual Port 10Gb NIC - $1000

Total all up $2,280

That`s a saving of $12,000 not bad. Of course the reliability of the server, comparability, SLA is non existent, HDD are slow.. uses some "Desktop Grade" parts as well as a power supply of unknown origin. Not exactly the fairest of fair compairson to your decked out HP G7 box but spending money where it counts, CPU frez, Memory, and NIC. To sum it up nicely would be to say, if it fuck ups its your own fault and theres no support line to call/sob/pass the buck to. 

All up haven`t had any problems and more than happy with the performance and yes DO NOT mess with the punk ass pink bunny!

Trading is hard

Not quite dead yet, but close from working insane hours.... my life story lol. Its been 5 months since quitting a good job to focus full time on HF equities trading and almost 2 years since finance and trading appeared on my radar. But never imagined in any shape or form that successful, consistently profitable trading would be so hard. Its not surprising... I guess, if not painfully obvious as if everyone could extract money from the market like this, then everyone would be living on a Yacht in the Bahamas... or whatever fantasy you can conjure up.

Problem is the barrier to entry is unbelievably low, similar to "Art". Anyone can pickup a paintbrush/whatever and make something and put it on a wall, just as anyone can buy and sell stock/whatever and plot it on a chart. The difference? Unlike friends and family is with your Art, the market is a brutal, merciless, heartless bitch that tells you exactly what it thinks, blow by blow, day by day, no hesitation to kick you in the teeth, with cast iron steel capped boots, when your down and out in a pool of your own blood. Then finally, in a very polite and impersonal way gives  a quantifiable score every minute/hour/day/month and year - sounds fun eh? :P

Guess its no co-incidence some of the smartest and sharpest people I`ve met over the last year or two are HF traders and what Ive found in nearly 2 decades of writing code in some shape or form is, you typically find is one, two or maybe three if your really lucky of these kinds of truly gifted people in your standard tech company/division (*1) yet it seems every successful HF trader ive met is one of these gifted people - none of this big dumb asshole trader image that is typically in popular culture.

So as 6months draws to a close, after dozens of strategies, endless hours looking at charts, flows, stats, far too much hair pulling and about 5 completely re-written trading systems(one day will write that all up) I`m ready to jump in and start trading for real January 2012.

Maybe PnL >0
Maybe PnL < 0

But...  Enjoy the holiday break and all the best for 2012!!

(*1) - this means people who get shit done, not those who can cite text books, bullshit brilliantly or display alphabet soups after their name.

Tick Life

Someone recently asked "at what speed is the game played at now" on HN. To which one of the responses was "do milliseconds matter". The answer of course depends on what kind of strategy your talking about, but for the sake of argument presume he means an ultra high freq/low latency every tick/quote/trade matters style strategy.

So heres some plots to show how fast (or slow) the market moves, first up is the BBO for part of the morning session early this month on..... you guessed it MSFT

And the life of each tick, measured in nanoseconds. Life means, every new BBO on the bid or ask resets the timer, e.g. starts at 0ns

Above graph is scaled verticaly to 60Bn nanosecconds == 60secconds == 1minute. A quick look shows at most one BBO pair`s life maxes out at around 30secconds.

If we zoom in a bit more, the above plot is the same graph but vertical scaled to around 15 seconds. From here it becomes more apprent the vast majority of BBO pairs are under seconds, possibly less. 

.. and continuing the zoom, below is some half interesting zoom in. First plot is the BBO

and yellow plot is the tick life. Here you get a better feel for how long each BBO pair lasts. A few last 10sec, a few less than 1sec and son.

digging further, below is a zoom in of the small ant hill seen on the larger BBO above.

and the corresponding tick life below.

... where you can see the same 15sec or so life of the BBO peek. Ok so your saying 15sec eh? dosent sound so ultra high frequency? true until you dig further down and say, whats the lifespan of a BBO pair which has a spread of 2cents?

... (above plot) when the spread widens is circled in red. First one around 1M ns around 1 millisecond its fast but not that fast. The second one 0.1M ns, or roughtly 100microsecconds, to put that in perspective, the standard linux kernel scheduler slices at a granularity of every 50microsecconds.

... 100microsecconds is where the fun starts. 

Silly tricks

... and the long grind forward continues abeit slowly. This week found some weirdness with the exchange and problems with my strategies using end of day tick data aka non realtime paper trading. Was using slightly stale 3-4Month old data and figured it should be close enough but dam.... how much things have changed in a few months. Will post about this when ive got some spare cycles.

Im a huge fan of autonomic computing, which means hardware&software systems that do integrity checking and automatic recovery, with the obvious examples being RAID for disk and ECC for ram. Once upon a time I worked for and with (insert huge megacorps you all know) and had a fascinating discussion with one of their hardware engineers.

HW Dude: hmm thats a weird problem, whats the value of register at offset 0xbeefbabe?

Hacking Nasdaq: register 0xbeefbabe reads out to be ... 0x01234567.

(silence)

HW Dude: are you sure thats correct?

Hacking Nasdaq: ... yes

HW Dude: thats impossible, are you on crack?

.

.

.

What he was referring to is the MSB of that register was the logical OR of the other 30bits. Making a value of 0x01234567 impossible with the only correct value being  0x81234567. Moral of the story is encoding self integrity checks into everything makes it easy to catch and not waste time on dumb ass errors - the "oh woops the cable isnt plugged in, sorry" kind

... which leads us to OUCH and the 14byte identifier token. What Ive done is reserve the last byte as an integrity check such that the 32bit sum of the previous 13bytes modulo 26 is its value.

e.g.

u32 Sum = 0;

for (int i=0; i < 13; i++) Sum += P->Msg.OrderAdd.Token[i]

P->Msg.OrderAdd.Token[13] = 'A' + Sum % 26;

This way its trivial to check if the OrderToken you looking at is actually real or corrupt without any effort. So far ive caught this a few times, usually when some bit of code has gone rouge and pissing all over memory.

...and yes thats only in development, no errors in prod... yet :)

stuffing the turkey

Been a bit crazy busy of late so not so many posts, but keeping in the spirit of things here is an interesting POV on quote stuffing and some of the basic microstructure games that get played.

Assume the following plots

Best Ask (Above)

Best Bid (Above)

Open Qty @ Best Bid (above)

As you can see(hilighted in red) what looks like a neat pristinely manicured bit of astroturf among the wild grass, weeds and occasional dead spot. What someone is doing is adding to the BB qty, then immediately canceling it all in rapid fire.

Above is the time delta between changes, where its toggling it every say 100usec. Clearly some algo messing with the BBO. Now what happens if you use a dumb ass  simple moving average of say the last 1024 open qty levels of the BB - plot below.

Above between the yellow lines is the average open qty @ BB when on the astroturf.. and that is how you can exploit stupidityweakness in someones algo. Here it is again with a more aggressive vertical scaling

Again, a beautiful manipulation of a simple moving average. So whats worse? Spaming the market with a bit of noise OR using a simple moving average as a signal? Personally Id say the latter as we`re no longer in the school yard where there`s no "special needs play area"... However the former is far easier to bitch and complain about.