SVD of HFT

Cant believe it, first post on this blog was exactly 3 years ago wow... seems like such a long time. 3 years ago I couldn`t tell you the difference between buy, sell, bid, ask, long or short yet some how the projects and code I`ve written during this time now checks or trades, millions, tens of millions, hundreds of millions and sometimes billions of dollars every single day. The depressing part? all the time and effort in finance has not translated into George Washingtion`s. in my bank account.... bugger.

So when your imagination and reality are so divergent its a good  times to step back and work out what your missing and not even consider the how or where of the equation - just focus on the what. Lets take a high level look at the game from the top down.

HFT game has 5 major components

1) Speed / Deterministic / Technology

2) Commissions / cost structure

3) Influence with the exchange

4) Capital

5) Strategy

Speed / Deterministic / Technology 

This is the classic arms race, bigger faster, better. Its very true technology is a large component of any HF strategy however the trap technologist (like myself) fall into is, they believe technology alone can beat the market. What we dont realize is our technology edge starts and ends at the cable plugged into our box/switch. Theres so many other moving parts that are out of our control that tips the balance way into the red.

In Round 1 any technology edge I had ended when it entered at the brokers pre-trade risk checks let alone entering the exchanges infra. In Round 2 have best in class access to the market but the edge is gone once the packet has left the NIC. Net result, speed/technology is important but you cant win a trade with it alone.

Commissions / cost structure

HF strategies are unique because the per trade profit is very small, which makes it extremely sensitive to the per trade and monthly recurring costs. For example if your trade makes +100 cents on a win and -100cents on a loss with say 100 shares eg. 1cent/share. Say your broker, the exchange, the SEC takes 25 cents or 1/4 of the profit.

This means you need a 15% edge (65% of the trades profitable) just to break even.

Yet if your total fees is 5 cents vs 25 cents you need only a 3% edge (53%) of trades to break even.

A delta of 12% profitable trades, means you need a really good signal, which is extremely difficult. Thats just to break even!

Influence with the exchange

This one is probably a bit surprising to many as in theory an exchange is equal access for all participants. Thats the theory, yet in practice an exchange has finite manpower and equipment which results in the dude trading $100 / day vs the dude trading $100MM / day gets vastly different levels of service.

Aside from man-power theres always political / boys club games and really is a catch-22 situation. To join the club you need to trade massive volumes, but to trade (profitably) massive volumes you need A class access into the matching engine.

Why is it important? Your technology edge ends at the BGP endpoint (your router/switch) with a shit ton of other crap between that endpoint and a trade thats in the exchanges control.

Capital

For the HF game capital is usually not a problem. The true believers are flat at the end of the day with no overnight / borrowing costs. Intraday your capital just limits the sizes of your position. The one thing capitial does give you is, market access and influence with an exchange. Any tier one prime broker will want a few million dollars deposit before they will even talk to you. Next step down requires a few $100K and so on. For now capital is not an issue.

Strategy

And finally strategy, this is the true leveler. If you have excellent strategy then all the above can become noise. Yet the inverse is not true - you can have the best technology in the world but it will never make a bad strategy good.

This is by far my biggest weakness and the skill I need to develop further, but what exactly needs to be learnt? The what is not an easy question. The first 6months of this expedition was largely wasted doing essentially some form of technical analysis - if signal X crossed signal Y then buy/sell kind of strategy development. Im sure it works for someone but thats not the HF game to play.

So what`s missing? its how to mine the data and what to mine to gain a statistical edge. The how to mine is an easy technology problem, the what to mine is not. Im convinced machine learning is the way forward as its the next logical step beyond stat arb. After all isnt stat arb a form of the most basic machine learning? guess it all depends on your definite of learning.

Whats next? Bayes, Markov, Kernels, Boosting and Kaggle mmmm Kaggle lol

Top 5 reasons my (your) backtest is bullshit

Alot of the time when building/backtesting strategies you get a result thats just too-good-to-be-true. When this shows up the best way test your brand spanking new million *cough* dollar *cough* strategy a day is to

1) flip the sides/signals - if possible e..g not capturing bid/ask spread..

2) random() is your friend, replace random() with your secret sauce signal gen and see how it goes.

My preference is to use a random() strategy or an "always trade" strategy as its less intrusive. If it generates million dollar backtested profits... you know its complete and utter bullshit. 

So in dave letternman top 10 style..

top 5 reasons my (your) backtested results are bullshit

number 5...  your cost structures are wrong

Part of the shock and horror of HF trading is the realization your being screwed on all angles in all holes by everyone. When ppl think about trading its all about the (sell price - buy price) == money in your bank account. Sadly because the gross margins are so low & with so many fees and taxs along the way, money in the bank isnt always that good..

Be sure to include

1) exchange fees / rebates

2) SEC fees/taxs

2) brokers fees

3) your (not) friendly capitial gains tax man

4) [optional] exchange rates

Got totally screwed by 4) this year - Japanese Yen / USD really hurt

number 4 ....   You have built a time machine, cool can I buy one?

Need to seriously change my trade idea scratchpad / research framework soon to completely eliminate this. For now using essentially arrays of data, with each index being a time/event sample. Problem is its all too easy to do a if (Price[t] > Price[t+1] ) kind of compare which, takes a peek into the future and can heavily bias your results. The more painful and subtle variation is using multiple datasets/signals that are not perfectly synchronized e.g. A signal is one time/event ahead of the others.

The only way around this is to make this impossible, say strategy has no access to the array/precalced data or be very very careful. My approach for the moment is the latter. When I started would make this mistake more often than I want to admit but rarely screw it up these days - too many hours lost chasing wtf is this random() strategy profitable

number 3.... EventSeries != TimeSeries

Most exchanges will pack multiple market data events into a single UDP packet making the market data event stream appear as a time series, when it is not. Simple passive example strategy would be, if BBO price level has less than X orders/qty then replace away/deeper into the book. If you equate the event stream for a time series then the backtest will always avoid deep, aggressive trades. Why? because when each order gets wacked by some massive aggressive trade, each and every passive order that gets filled, an individual market data update is generated.

Now. if you treat each of market data update as a discrete event, your strat/backtest can see each update individually until it reaches the minimum X level and you replace away. aka a perfect 0 latency system.

Whats wrong? In real-time all the orders get wacked at the same time including yours! with a group of market data events actually being an atomic operation.

number 2.... resetting position at start-of-day

when doing multi-day backtesting its easy to forget the strategy is still open at the end of the day. e.g. strategy exit condition has not triggered as its usually in a loss state. Simplest example of this is, for every 1minute buy SPY aggressively then recycle it passively at +X ticks and start the day with no position. What happens is you have 100% winning trades! Why? because when the market moves against you the strategy just sits there drifting away from the bbo. (in a 10ft pool of red) waiting for that passive exit. Then when the market closes, your backtest resets the strat for the next day that loss is simply ignored, disappeared and never existed. If there were a wizard of oz, would certainly ask for such a button.

Always update the position/pnl after every trade - not just at the exit, and never reset between days.

number 1.... aggressively buying at the bid and aggressively selling at the ask.

Its so easy.. just aggressively buy at the bid, and aggressively sell at the ask for every tick, on every instrument for every exchange in the world and you, yourself, can bail Greece, Spain and probably half of the world no worries.

the problem is, in real life your aggressively buying at the *ask* aka paying the spread to fill now. I still screw this up all the time when testing out an idea. Usually by fiddling with different styles to enter/exit a trade then forgetting to revert the code back and preso.... $1M / day in pnl.

--------

 By now your either shaking your head saying how badly I suck, or quietly nodding about those indiscretions you never told anyone, or laughing your ass off saying yeah been there done that, will never do again.

oh the shame... lol

fpga paper

Ran into this advertisement^H^H^H^H^H peer reviewed IEEE paper today

http://algo-logic.com/~jwlockwd/publications/Low_Latency_Library_for_HFT-Algo-Logic-4831a009.pdf

Its quite well written and covers alot of ground yet contains some pretty bad assumptions. Lets deconstruct

In the abstract:

"The application sustains 10Gb/s Ethernet line rate with a fixed end-toend latency of 1µs  – up to two orders of magnitude lower than comparable software implementations."

What they are saying is software latency for a brain dead strategy is 100usec? This is complete and utter bullshit, yes if you write shitty code your wire-wire latency will be 100usec +/- 25usec jitter. If you write tight code using standard hardware and know what your doing... can get under 10usec wire-to-wire no problem.

Next point:

"An initial line handling stage uses redundancy in the streams to replace any information temporarily missed due to packet losses in order to reconstruct the necessary parts of the  order book of the exchange."

This is pure magic, have nfi what their talking about here. How is it possible when you drop a market data packet you can reconstruct its symbol/price/side/qty/operation, in less than 1usec, in RTL on an fpga. Will concede given say 100msec or more AND significant delay in the stream e.g. you dropped a new quote X, but received the cancel quote X msg, its possible to take a good guess what was in the packet.... maybe they are talking about using the *cough cough* market data recovery channels.

Next point:

"Prices for all 8000 securities traded on U.S. Exchanges fit within the FPGA’s on-chip memory. The table size is scalable to support more symbols as required."

This is great but wtf? Guessing they are talking about a 8000 * 4bytes(32bit fixed point price only) * 2 (bid/ask) * number of venues to store the best bid/offer for a symbol. That easily fits in the block ram of a Xilinx V5 240T (note: block ram is memory on the die of the fpga and not to be confused with external memory e.g. DDR/QDR/SRAM)

Great but... how do they calculate what the BBO is? You need to store the entire book, which has to be stored in external memory because the working set is atleast 1GB. 

Why store the entire book? 

1) When you get an quote cancel message exchanges rarely have the price/qty/side/symbol in the cancel message. Typically its just the order id and/or quantity canceled, so you need to keep price/qty/side/symbol/orderid and more around to correctly process e.g. what qty is currently open for this order after Y quote cancel messages.

2) when a quote is deleted you need to remove it from the book, then iterate through the book to find the best price - read you need to keep the book in memory somewhere. The trick here is being smart in how and when you iterate to find the best price, e.g. sorting a mostly sorted list can be pretty dam fast.

3) .... and more but wasted too much time on this today.

... so are saying, we can store 8000 bid/ask prices ondie in the fpga for the rtl coded algo to use? hmm..

Its a great well written technical paper, probably one of the best papers on the subject however glosses over some more important details... as doe all academic papers on the subject.. aka yes just sit and hold that 30% draw down for 3 months and the pnl is beautiful.

/rant

First Year in the Jungle - Part Two

Changing career from Technology to Trading is a bit odd, most tech guys I`ve known have no interest in trading or only romanticize about it and others slowly inch towards it at a prop desk. For the few (fools) who take the plunge without a clue, we get bitten on the ass by the bitch that is reality.

For some they are in a geographically good area, e.g. NY, Chicago, London for me, I didnt have that luxury. One of my options was to call up the clients I worked with and asked for a job which would have caused all sorts of weirdness/problems. But would also have to go thru the same process of start in technology, find a way into research, then strategy development, then eventually some how prove myself enough to trade, and only after all that would I get my own PnL - we`re talking years here. And if you apply conditional probability to each of these steps then the chances of getting to the end is quite low.

The other option is to trade your own capital where the only thing stopping you is the market but your completely on your own, no one to learn from or help you - anything but a walk in the park and what I chose, simply due to a lack of any other option.

Have been working professionally since age 17, moved continent age 21, moved continent again at age 27. Have been part of at least 3 fairly famous yet grueling projects, been off into the woods as solo startup founder and a ton of other shit. Out of all of this HF trading is by far the most challenging and difficult work so far. Why? because you have no one to learn from, learning material is sketchy at best and the only real way to learn is by trial and error in the market with real money. If you compound trial & error with the stress of say, there`s a bug and your holding $250K of inventory when the market moves against you, its what either breaks you or further detaches you from the emotion.

A few skills have gained this year

- Comfortable losing money

Its the old saying, need to spend money to make money. Prior to now, spending money and loosing money was extremely painful. These days it sucks but its ok to loose money. One of the traps traders get into is having to be right 100% of the time e.g. 100% profitable trades. Some trades are positive, some are negative, some days are positive, some days are negative and thats ok.

- linux/big data skills moved to a new level

Been using linux since 1995 (or 96 memory bit fuzzy) professionally its been my development environment since 2001 yet only this year when dealing with TerraBytes of data have I been forced to dig really deep into the unix philosophy. Prior to that have known, hacked and understood unix/linux technically but lacked the deep philosophy of its simplicity and how it all connects to make it run silky smooth.

- intuition with statistics

Have known basic stats for a while, used doing linear algebra for what seems forever, calculus a little but this year finally gained deeper intuition about statistics. Everyone has heard of or used min, max, mean, median, arithmetic, geometric, standard deviation, correlation, variance, t values, p values, linear regression, non-linear regression, autoregressive, eigenvalues, eigenvectors random walks, monte carlo..... blah blah blah and so on. You can even call this function / library and it spits out some result without any idea how it works, yet only recently have the basic foundations become completely natural and not forced.


Some things I miss:

- regular cheque each month

Goes hand in hand with losing money. In previous lives had say in budget allocations so knew well about the need to spend money, however when its *your* money its certainly different. The beauty of an employer/employee relationship is the comfort and (perceived) stability you get of regular cheques. Now days the amount and sign of the cheque changes all the time, not an easy transition.

- Flexible schedule

These days I live and die by the clock of the exchange. The exchange dosent care if you have something else to do, certainly dosent care if your bladder is full and most certainly does not give a shit if you have caught the flu. In previous jobs you can take days off, arrive late, leave early go to the bathroom any time and it dosent cause much disruption. For trading even automated systems you (or someone) needs to be there watching it, monitoring it so when (not if) it fucks up your around to clean up..

- 100% precision, 100% of the time stress.

Automated trading is one of those few professions that has such horrific consequences if you screw up (read Knights recent meltdown), what worse is the speed systems must be developed and changed is frantic compared to say medical/aviation which are given a very long times to build/verify. Trading changes fast and things always go wrong, as such have built systems and an approach thats different and so far has worked well. In a regular development role you can do sloppy code and it really dosent matter, or you can always come back to it when you have more time to fill in the blanks.


Some new enjoyable things

- feel alive / connected

Having such a direct connection to money some how makes it all real. In reality its still bits in a computer, statements are still bits, the money sits as bits at a bank, its all just bits, but feels mortal. Guess we get programmed as a child about money, its value how central it is thus touching this much money even virtually shakes us to the core. In other lives the closet ive got to this kind of buzz is for mischievous things.

- sit back and watch it print money

Its great when you can sit back, and watch the pnl pop up, and up, and up, and up and not even touch the keyboard. Awesome feeling, probably similar to what sales ppl get when they close deals / sell items. Dosen`t happen all the time but when its a good day, its a great feeling.

- respect from non geeks

Kinda surprising, my friends and family are impressed when we talk about trading. Seems because its real money it gets respected. In previous lines of work no one really understood, "shaved 1micro second off the latency today" vs "traded $XXXX million dollars today" the latter gets oohs and ahhs, with the former a nod and "thats nice".


Guess the main question is, after 1 year have I become successfully? Hardly. When I left my last job the goal was to consistently make $100/day which might sound ridiculous, but is orders of magnitude harder than it sounds. Thankfully I`m making more than $100/day but after everyone takes their cut, the ridiculously strong yen, financially I`m barely scraping by, Overall certainly a net loss if you amortize it over the year, rent bills food etc etc. and so far this is easily the lowest paying job ive had since turning 20yo.

20/20 hindsight would I do it all again?  ..... tough question.

First Year in the Jungle - Part 1

Its been a little over 1 year since leaving a good job in finance, 1 year and 1 month exactly since my employment ended and dam... so much has changed since then. To give some insight, back in 2010 I joined a front office connectivity group for a large bank building some cool tech. The plan was work my ass off, starting in technology, and slowly migrate towards the "trading" side of things - how ever you define trading. How green was I ? so green that I thought had joined a HF prop fund inside a bank... yes bright florescent green with blinking lights.. holy cow. It took about a month and half before finally admitting to myself what connectivity means, what a broker does, how we make money, our place in the food chain and most importantly its the *clients* who do all the trading - was a jarring week.

Fast forward 18months, the project built, rolled into prod, making money all is good and im there thinking how do I get to "trading". By this stage had acquired enough info to know trading at that point means UHF/HF stat arb prop trading and for some reason thought market making had no relation.. hmmm.. yes green. Worse yet there wasn`t anything remotely close available at the bank and thus took an interview at a well known, well respected quantitative fund in the area.

The interview was the worst ive ever had, like ever - my failed teenage interview for a gas station attendant went far better. Problem was I insisted on applying for a trading position (no interest in pure tech) with my tech credentials getting me the interview and my complete lack of skills flunking it in spectacular fashion. What do you mean add some numbers and take a percentage of it, without paper, without a calculator, no computer, no nothing just use your grey matter? ... oh and we are timing you. It was a skill last practiced when I was 10 years old memorizing multiplication tables for a school test and a gold star, after all that's what computers are good at? right? .... maybe.

This should have been a dude wtf are you doing slap-in-the-face, as translated into coding skills, I couldn`t even write hello world in BASIC let alone C or .

Since then after many weeks, months of banging my head against a wall staring into the abyss is, trading, even at the UHF level which in my opinion is the most demanding on the tech side vs the "trading" side is still a 100% completely different profession. What I some how missed is the mirage it presented being so similar to pure tech, because the tools are the same.

Think about that for a second, maybe its blindingly obvious to everyone and I`m a naive chump (rhetorical question lol) Yet how many professions use a screwdriver and a hammer? Hell I use one from time to time, however the difference between a plummer and electrician is very large but they both use a hammer and screwdriver.


Trading is an art, don`t confuse it with mathematics, computer science or the ability to use a hammer.

.... to be continued

fpgaz

Been really flat out, not many posts lately. Teamed up with a small prop group of really cool guys, having a blast but more on that later.

The amount of fudzilla that`s out there about HFT and FPGA is pretty immense, yet stumbled across a fairly decent set of articles on the subject.

http://www.hftreview.com/pg/blog/mike/read/55950/fpga-hardware-accelerated-trading-part-one-who-what-where-and-why

http://www.hftreview.com/pg/blog/mike/read/56720/fpga-hardware-accelerated-trading-part-two-alternative-approaches

http://www.hftreview.com/pg/blog/mike/read/57349/fpga-hardware-accelerated-trading-part-three-programming

http://www.hftreview.com/pg/blog/mike/read/58189/fpga-hardware-accelerated-trading-part-four-challenges-and-constraints

Its more an assortment of comments from various vendors selling stuff/in the space.. SEO bait? maybe but does give a bit more realistic view.

fpga dev is the ultimate culture clash, you`ve got hw guys vs sw guys vs biz guys, everyone has their own vastly different view of reality.

hw guys say to the sw guys -> wtf there`s no reference model to code the rtl against?

sw guys says to hw guys -> wtf is this timing violation nonsense, just hit compile let it crunch and you have a binary?

biz guys says to hw guys -> wtf you cant fix that bug and deploy a new version of the fpga before lunch? .. its 9am now?

hw guys say to biz guys -> wtf theres 0 rtl written and you want all rtl finished in 1-2 weeks?

.... and so it goes on.

HFT Postmortem - Round 1

Its always good to reflect and understand the positive and negative points of a project even for a negative result as you learn the most from your failures than successes... or at least you should.

The Round 1 Score

• 1 : NASDAQ
• 0 : Hacking NASDAQ

No KO or TKO with the decision based entirely on points. Kinda sucks ass things haven`t gone as planned but such is life. Lets start with the positive

What Went Right

1) JFDI

Which stands for "Just Fucking Do It". Also known as Analysis Paralysis meaning over thinking and over analyzing the situation to the point where whichever direction you want to go there`s a ton of arguments for it but also against it. 

What happened is I reached the point where the behaviour of the exchange or more precisely the behaviour of my orders to/from/on the exchange became a significant factor in profitability. Thus one of the goals was to test my assumptions on this vs what happens in the real world... which turned out to be very wrong. None the less it is extremely valuable info to know.

2) Real Trading. 

Its all to easy to code up some system, back test it, do whatever feel smug about how much paper money you`ve made...  but is completely different to trading with real money on the real exchange with real consequences if you seriously fuck up. First few weeks were seriously terrifying, what if X happens, what if Y happens what if X causes Y which in turn means I need to liquidate everything I own, move into a tent and beg for change. The reality is if you have multiple independent saftey valves both aggressive and passive there`s not a whole lot that can go wrong and when it does, its not that bad. At this point, I`m quite comfortable running fully automated strategies on the exchange that puts 1000`s of orders  into the book each day.

3) Building Real Trading System.

Learnt alot when building my system. Its one thing to build tools & simulators, completely different to have prod systems that are live and constantly evolving. Ended up writing a bunch of tools adding new components, doing new analysis and quite a few things that are simple but did not anticipate. All very valuable info for Round 2.

4) Watching the Market

Before prod, I almost never looked at what the market did day-to-day, quite oblivious infact as was focusing on historical data. I think your perspective & pattern matching radically changes when your no longer in back test time. When 1 day is say 5 seconds becomes 1 day is 6.5 Hours of up/down/up/down I gained some intuition about how the market moves in the mid-freq time frame. Guess you could say, had no connection to the  market, as it was just numbers and stats - it had no personality.

5) Bite sized Risk

Had a choice of going all out with a significant % of net worth deposit, 5 figure monthly infra costs the hole 9 yards. Or a chunk of change deposit that's a small % of net worth and infra thats good but not on the edge. Chose the latter, my reasoning was have 0 experience trading and I should not risk my current assets on something thats unproven and un-tested. e.g. Do a test first before committing more cash, and I think this was the correct choice.

What Went Wrong

1) No Strategy

Its kind of a fib to say there were no strategies, perhaps a better wording is had no clearly profitable strategy. The strategies I wanted to run are all highly skewed by the infrastructure/comissions and behaviour of the exchange. So in hindsight the risk was actually very high as the result was highly dependent on a large unknown factor. The end result was after realizing the first batch of strategies wasn't going to work had to start from 0 on a completely different style of trading which there wasn't enough time to get up to speed on.

2) Going it Alone

One of the problems when trading in a bubble is you have no one to bounce ideas off, no one is there to point out the obvious and most importantly no one is there to slap you in the face and say 

what the fuck man? 

Self reflection and transitioning between abstraction levels is really hard, especially when you eat/breath/sleep at one level its not easy to step back and look at the wider picture and realize your spending time in the wrong place.

3) Wrong Infrastructure

The infrastructure/commissions was clearly a bad choice for the kinds of strategies I was testing. Yet its hard to say it was wrong given the cost & risk of more appropriate infra. In the end it was just an incorrect matching of strategy and infrastructure, given the goal is to make a profit. If the goal was to understand the exchange / market it was a good choice but.... in the real world one needs to make money.

Theres a bunch of smaller things but I think that summarized the main points.  Round 1 was fun, exciting, learnt an absolute ton but need to re-calibrate before entering the ring again for Round 2.