SVD of HFT

Cant believe it, first post on this blog was exactly 3 years ago wow... seems like such a long time. 3 years ago I couldn`t tell you the difference between buy, sell, bid, ask, long or short yet some how the projects and code I`ve written during this time now checks or trades, millions, tens of millions, hundreds of millions and sometimes billions of dollars every single day. The depressing part? all the time and effort in finance has not translated into George Washingtion`s. in my bank account.... bugger.

So when your imagination and reality are so divergent its a good  times to step back and work out what your missing and not even consider the how or where of the equation - just focus on the what. Lets take a high level look at the game from the top down.

HFT game has 5 major components

1) Speed / Deterministic / Technology

2) Commissions / cost structure

3) Influence with the exchange

4) Capital

5) Strategy

Speed / Deterministic / Technology 

This is the classic arms race, bigger faster, better. Its very true technology is a large component of any HF strategy however the trap technologist (like myself) fall into is, they believe technology alone can beat the market. What we dont realize is our technology edge starts and ends at the cable plugged into our box/switch. Theres so many other moving parts that are out of our control that tips the balance way into the red.

In Round 1 any technology edge I had ended when it entered at the brokers pre-trade risk checks let alone entering the exchanges infra. In Round 2 have best in class access to the market but the edge is gone once the packet has left the NIC. Net result, speed/technology is important but you cant win a trade with it alone.

Commissions / cost structure

HF strategies are unique because the per trade profit is very small, which makes it extremely sensitive to the per trade and monthly recurring costs. For example if your trade makes +100 cents on a win and -100cents on a loss with say 100 shares eg. 1cent/share. Say your broker, the exchange, the SEC takes 25 cents or 1/4 of the profit.

This means you need a 15% edge (65% of the trades profitable) just to break even.

Yet if your total fees is 5 cents vs 25 cents you need only a 3% edge (53%) of trades to break even.

A delta of 12% profitable trades, means you need a really good signal, which is extremely difficult. Thats just to break even!

Influence with the exchange

This one is probably a bit surprising to many as in theory an exchange is equal access for all participants. Thats the theory, yet in practice an exchange has finite manpower and equipment which results in the dude trading $100 / day vs the dude trading $100MM / day gets vastly different levels of service.

Aside from man-power theres always political / boys club games and really is a catch-22 situation. To join the club you need to trade massive volumes, but to trade (profitably) massive volumes you need A class access into the matching engine.

Why is it important? Your technology edge ends at the BGP endpoint (your router/switch) with a shit ton of other crap between that endpoint and a trade thats in the exchanges control.

Capital

For the HF game capital is usually not a problem. The true believers are flat at the end of the day with no overnight / borrowing costs. Intraday your capital just limits the sizes of your position. The one thing capitial does give you is, market access and influence with an exchange. Any tier one prime broker will want a few million dollars deposit before they will even talk to you. Next step down requires a few $100K and so on. For now capital is not an issue.

Strategy

And finally strategy, this is the true leveler. If you have excellent strategy then all the above can become noise. Yet the inverse is not true - you can have the best technology in the world but it will never make a bad strategy good.

This is by far my biggest weakness and the skill I need to develop further, but what exactly needs to be learnt? The what is not an easy question. The first 6months of this expedition was largely wasted doing essentially some form of technical analysis - if signal X crossed signal Y then buy/sell kind of strategy development. Im sure it works for someone but thats not the HF game to play.

So what`s missing? its how to mine the data and what to mine to gain a statistical edge. The how to mine is an easy technology problem, the what to mine is not. Im convinced machine learning is the way forward as its the next logical step beyond stat arb. After all isnt stat arb a form of the most basic machine learning? guess it all depends on your definite of learning.

Whats next? Bayes, Markov, Kernels, Boosting and Kaggle mmmm Kaggle lol

Top 5 reasons my (your) backtest is bullshit

Alot of the time when building/backtesting strategies you get a result thats just too-good-to-be-true. When this shows up the best way test your brand spanking new million *cough* dollar *cough* strategy a day is to

1) flip the sides/signals - if possible e..g not capturing bid/ask spread..

2) random() is your friend, replace random() with your secret sauce signal gen and see how it goes.

My preference is to use a random() strategy or an "always trade" strategy as its less intrusive. If it generates million dollar backtested profits... you know its complete and utter bullshit. 

So in dave letternman top 10 style..

top 5 reasons my (your) backtested results are bullshit

number 5...  your cost structures are wrong

Part of the shock and horror of HF trading is the realization your being screwed on all angles in all holes by everyone. When ppl think about trading its all about the (sell price - buy price) == money in your bank account. Sadly because the gross margins are so low & with so many fees and taxs along the way, money in the bank isnt always that good..

Be sure to include

1) exchange fees / rebates

2) SEC fees/taxs

2) brokers fees

3) your (not) friendly capitial gains tax man

4) [optional] exchange rates

Got totally screwed by 4) this year - Japanese Yen / USD really hurt

number 4 ....   You have built a time machine, cool can I buy one?

Need to seriously change my trade idea scratchpad / research framework soon to completely eliminate this. For now using essentially arrays of data, with each index being a time/event sample. Problem is its all too easy to do a if (Price[t] > Price[t+1] ) kind of compare which, takes a peek into the future and can heavily bias your results. The more painful and subtle variation is using multiple datasets/signals that are not perfectly synchronized e.g. A signal is one time/event ahead of the others.

The only way around this is to make this impossible, say strategy has no access to the array/precalced data or be very very careful. My approach for the moment is the latter. When I started would make this mistake more often than I want to admit but rarely screw it up these days - too many hours lost chasing wtf is this random() strategy profitable

number 3.... EventSeries != TimeSeries

Most exchanges will pack multiple market data events into a single UDP packet making the market data event stream appear as a time series, when it is not. Simple passive example strategy would be, if BBO price level has less than X orders/qty then replace away/deeper into the book. If you equate the event stream for a time series then the backtest will always avoid deep, aggressive trades. Why? because when each order gets wacked by some massive aggressive trade, each and every passive order that gets filled, an individual market data update is generated.

Now. if you treat each of market data update as a discrete event, your strat/backtest can see each update individually until it reaches the minimum X level and you replace away. aka a perfect 0 latency system.

Whats wrong? In real-time all the orders get wacked at the same time including yours! with a group of market data events actually being an atomic operation.

number 2.... resetting position at start-of-day

when doing multi-day backtesting its easy to forget the strategy is still open at the end of the day. e.g. strategy exit condition has not triggered as its usually in a loss state. Simplest example of this is, for every 1minute buy SPY aggressively then recycle it passively at +X ticks and start the day with no position. What happens is you have 100% winning trades! Why? because when the market moves against you the strategy just sits there drifting away from the bbo. (in a 10ft pool of red) waiting for that passive exit. Then when the market closes, your backtest resets the strat for the next day that loss is simply ignored, disappeared and never existed. If there were a wizard of oz, would certainly ask for such a button.

Always update the position/pnl after every trade - not just at the exit, and never reset between days.

number 1.... aggressively buying at the bid and aggressively selling at the ask.

Its so easy.. just aggressively buy at the bid, and aggressively sell at the ask for every tick, on every instrument for every exchange in the world and you, yourself, can bail Greece, Spain and probably half of the world no worries.

the problem is, in real life your aggressively buying at the *ask* aka paying the spread to fill now. I still screw this up all the time when testing out an idea. Usually by fiddling with different styles to enter/exit a trade then forgetting to revert the code back and preso.... $1M / day in pnl.

--------

 By now your either shaking your head saying how badly I suck, or quietly nodding about those indiscretions you never told anyone, or laughing your ass off saying yeah been there done that, will never do again.

oh the shame... lol

fpga paper

Ran into this advertisement^H^H^H^H^H peer reviewed IEEE paper today

http://algo-logic.com/~jwlockwd/publications/Low_Latency_Library_for_HFT-Algo-Logic-4831a009.pdf

Its quite well written and covers alot of ground yet contains some pretty bad assumptions. Lets deconstruct

In the abstract:

"The application sustains 10Gb/s Ethernet line rate with a fixed end-toend latency of 1µs  – up to two orders of magnitude lower than comparable software implementations."

What they are saying is software latency for a brain dead strategy is 100usec? This is complete and utter bullshit, yes if you write shitty code your wire-wire latency will be 100usec +/- 25usec jitter. If you write tight code using standard hardware and know what your doing... can get under 10usec wire-to-wire no problem.

Next point:

"An initial line handling stage uses redundancy in the streams to replace any information temporarily missed due to packet losses in order to reconstruct the necessary parts of the  order book of the exchange."

This is pure magic, have nfi what their talking about here. How is it possible when you drop a market data packet you can reconstruct its symbol/price/side/qty/operation, in less than 1usec, in RTL on an fpga. Will concede given say 100msec or more AND significant delay in the stream e.g. you dropped a new quote X, but received the cancel quote X msg, its possible to take a good guess what was in the packet.... maybe they are talking about using the *cough cough* market data recovery channels.

Next point:

"Prices for all 8000 securities traded on U.S. Exchanges fit within the FPGA’s on-chip memory. The table size is scalable to support more symbols as required."

This is great but wtf? Guessing they are talking about a 8000 * 4bytes(32bit fixed point price only) * 2 (bid/ask) * number of venues to store the best bid/offer for a symbol. That easily fits in the block ram of a Xilinx V5 240T (note: block ram is memory on the die of the fpga and not to be confused with external memory e.g. DDR/QDR/SRAM)

Great but... how do they calculate what the BBO is? You need to store the entire book, which has to be stored in external memory because the working set is atleast 1GB. 

Why store the entire book? 

1) When you get an quote cancel message exchanges rarely have the price/qty/side/symbol in the cancel message. Typically its just the order id and/or quantity canceled, so you need to keep price/qty/side/symbol/orderid and more around to correctly process e.g. what qty is currently open for this order after Y quote cancel messages.

2) when a quote is deleted you need to remove it from the book, then iterate through the book to find the best price - read you need to keep the book in memory somewhere. The trick here is being smart in how and when you iterate to find the best price, e.g. sorting a mostly sorted list can be pretty dam fast.

3) .... and more but wasted too much time on this today.

... so are saying, we can store 8000 bid/ask prices ondie in the fpga for the rtl coded algo to use? hmm..

Its a great well written technical paper, probably one of the best papers on the subject however glosses over some more important details... as doe all academic papers on the subject.. aka yes just sit and hold that 30% draw down for 3 months and the pnl is beautiful.

/rant

First Year in the Jungle - Part Two

Changing career from Technology to Trading is a bit odd, most tech guys I`ve known have no interest in trading or only romanticize about it and others slowly inch towards it at a prop desk. For the few (fools) who take the plunge without a clue, we get bitten on the ass by the bitch that is reality.

For some they are in a geographically good area, e.g. NY, Chicago, London for me, I didnt have that luxury. One of my options was to call up the clients I worked with and asked for a job which would have caused all sorts of weirdness/problems. But would also have to go thru the same process of start in technology, find a way into research, then strategy development, then eventually some how prove myself enough to trade, and only after all that would I get my own PnL - we`re talking years here. And if you apply conditional probability to each of these steps then the chances of getting to the end is quite low.

The other option is to trade your own capital where the only thing stopping you is the market but your completely on your own, no one to learn from or help you - anything but a walk in the park and what I chose, simply due to a lack of any other option.

Have been working professionally since age 17, moved continent age 21, moved continent again at age 27. Have been part of at least 3 fairly famous yet grueling projects, been off into the woods as solo startup founder and a ton of other shit. Out of all of this HF trading is by far the most challenging and difficult work so far. Why? because you have no one to learn from, learning material is sketchy at best and the only real way to learn is by trial and error in the market with real money. If you compound trial & error with the stress of say, there`s a bug and your holding $250K of inventory when the market moves against you, its what either breaks you or further detaches you from the emotion.

A few skills have gained this year

- Comfortable losing money

Its the old saying, need to spend money to make money. Prior to now, spending money and loosing money was extremely painful. These days it sucks but its ok to loose money. One of the traps traders get into is having to be right 100% of the time e.g. 100% profitable trades. Some trades are positive, some are negative, some days are positive, some days are negative and thats ok.

- linux/big data skills moved to a new level

Been using linux since 1995 (or 96 memory bit fuzzy) professionally its been my development environment since 2001 yet only this year when dealing with TerraBytes of data have I been forced to dig really deep into the unix philosophy. Prior to that have known, hacked and understood unix/linux technically but lacked the deep philosophy of its simplicity and how it all connects to make it run silky smooth.

- intuition with statistics

Have known basic stats for a while, used doing linear algebra for what seems forever, calculus a little but this year finally gained deeper intuition about statistics. Everyone has heard of or used min, max, mean, median, arithmetic, geometric, standard deviation, correlation, variance, t values, p values, linear regression, non-linear regression, autoregressive, eigenvalues, eigenvectors random walks, monte carlo..... blah blah blah and so on. You can even call this function / library and it spits out some result without any idea how it works, yet only recently have the basic foundations become completely natural and not forced.


Some things I miss:

- regular cheque each month

Goes hand in hand with losing money. In previous lives had say in budget allocations so knew well about the need to spend money, however when its *your* money its certainly different. The beauty of an employer/employee relationship is the comfort and (perceived) stability you get of regular cheques. Now days the amount and sign of the cheque changes all the time, not an easy transition.

- Flexible schedule

These days I live and die by the clock of the exchange. The exchange dosent care if you have something else to do, certainly dosent care if your bladder is full and most certainly does not give a shit if you have caught the flu. In previous jobs you can take days off, arrive late, leave early go to the bathroom any time and it dosent cause much disruption. For trading even automated systems you (or someone) needs to be there watching it, monitoring it so when (not if) it fucks up your around to clean up..

- 100% precision, 100% of the time stress.

Automated trading is one of those few professions that has such horrific consequences if you screw up (read Knights recent meltdown), what worse is the speed systems must be developed and changed is frantic compared to say medical/aviation which are given a very long times to build/verify. Trading changes fast and things always go wrong, as such have built systems and an approach thats different and so far has worked well. In a regular development role you can do sloppy code and it really dosent matter, or you can always come back to it when you have more time to fill in the blanks.


Some new enjoyable things

- feel alive / connected

Having such a direct connection to money some how makes it all real. In reality its still bits in a computer, statements are still bits, the money sits as bits at a bank, its all just bits, but feels mortal. Guess we get programmed as a child about money, its value how central it is thus touching this much money even virtually shakes us to the core. In other lives the closet ive got to this kind of buzz is for mischievous things.

- sit back and watch it print money

Its great when you can sit back, and watch the pnl pop up, and up, and up, and up and not even touch the keyboard. Awesome feeling, probably similar to what sales ppl get when they close deals / sell items. Dosen`t happen all the time but when its a good day, its a great feeling.

- respect from non geeks

Kinda surprising, my friends and family are impressed when we talk about trading. Seems because its real money it gets respected. In previous lines of work no one really understood, "shaved 1micro second off the latency today" vs "traded $XXXX million dollars today" the latter gets oohs and ahhs, with the former a nod and "thats nice".


Guess the main question is, after 1 year have I become successfully? Hardly. When I left my last job the goal was to consistently make $100/day which might sound ridiculous, but is orders of magnitude harder than it sounds. Thankfully I`m making more than $100/day but after everyone takes their cut, the ridiculously strong yen, financially I`m barely scraping by, Overall certainly a net loss if you amortize it over the year, rent bills food etc etc. and so far this is easily the lowest paying job ive had since turning 20yo.

20/20 hindsight would I do it all again?  ..... tough question.

First Year in the Jungle - Part 1

Its been a little over 1 year since leaving a good job in finance, 1 year and 1 month exactly since my employment ended and dam... so much has changed since then. To give some insight, back in 2010 I joined a front office connectivity group for a large bank building some cool tech. The plan was work my ass off, starting in technology, and slowly migrate towards the "trading" side of things - how ever you define trading. How green was I ? so green that I thought had joined a HF prop fund inside a bank... yes bright florescent green with blinking lights.. holy cow. It took about a month and half before finally admitting to myself what connectivity means, what a broker does, how we make money, our place in the food chain and most importantly its the *clients* who do all the trading - was a jarring week.

Fast forward 18months, the project built, rolled into prod, making money all is good and im there thinking how do I get to "trading". By this stage had acquired enough info to know trading at that point means UHF/HF stat arb prop trading and for some reason thought market making had no relation.. hmmm.. yes green. Worse yet there wasn`t anything remotely close available at the bank and thus took an interview at a well known, well respected quantitative fund in the area.

The interview was the worst ive ever had, like ever - my failed teenage interview for a gas station attendant went far better. Problem was I insisted on applying for a trading position (no interest in pure tech) with my tech credentials getting me the interview and my complete lack of skills flunking it in spectacular fashion. What do you mean add some numbers and take a percentage of it, without paper, without a calculator, no computer, no nothing just use your grey matter? ... oh and we are timing you. It was a skill last practiced when I was 10 years old memorizing multiplication tables for a school test and a gold star, after all that's what computers are good at? right? .... maybe.

This should have been a dude wtf are you doing slap-in-the-face, as translated into coding skills, I couldn`t even write hello world in BASIC let alone C or .

Since then after many weeks, months of banging my head against a wall staring into the abyss is, trading, even at the UHF level which in my opinion is the most demanding on the tech side vs the "trading" side is still a 100% completely different profession. What I some how missed is the mirage it presented being so similar to pure tech, because the tools are the same.

Think about that for a second, maybe its blindingly obvious to everyone and I`m a naive chump (rhetorical question lol) Yet how many professions use a screwdriver and a hammer? Hell I use one from time to time, however the difference between a plummer and electrician is very large but they both use a hammer and screwdriver.


Trading is an art, don`t confuse it with mathematics, computer science or the ability to use a hammer.

.... to be continued

fpgaz

Been really flat out, not many posts lately. Teamed up with a small prop group of really cool guys, having a blast but more on that later.

The amount of fudzilla that`s out there about HFT and FPGA is pretty immense, yet stumbled across a fairly decent set of articles on the subject.

http://www.hftreview.com/pg/blog/mike/read/55950/fpga-hardware-accelerated-trading-part-one-who-what-where-and-why

http://www.hftreview.com/pg/blog/mike/read/56720/fpga-hardware-accelerated-trading-part-two-alternative-approaches

http://www.hftreview.com/pg/blog/mike/read/57349/fpga-hardware-accelerated-trading-part-three-programming

http://www.hftreview.com/pg/blog/mike/read/58189/fpga-hardware-accelerated-trading-part-four-challenges-and-constraints

Its more an assortment of comments from various vendors selling stuff/in the space.. SEO bait? maybe but does give a bit more realistic view.

fpga dev is the ultimate culture clash, you`ve got hw guys vs sw guys vs biz guys, everyone has their own vastly different view of reality.

hw guys say to the sw guys -> wtf there`s no reference model to code the rtl against?

sw guys says to hw guys -> wtf is this timing violation nonsense, just hit compile let it crunch and you have a binary?

biz guys says to hw guys -> wtf you cant fix that bug and deploy a new version of the fpga before lunch? .. its 9am now?

hw guys say to biz guys -> wtf theres 0 rtl written and you want all rtl finished in 1-2 weeks?

.... and so it goes on.

HFT Postmortem - Round 1

Its always good to reflect and understand the positive and negative points of a project even for a negative result as you learn the most from your failures than successes... or at least you should.

The Round 1 Score

• 1 : NASDAQ
• 0 : Hacking NASDAQ

No KO or TKO with the decision based entirely on points. Kinda sucks ass things haven`t gone as planned but such is life. Lets start with the positive

What Went Right

1) JFDI

Which stands for "Just Fucking Do It". Also known as Analysis Paralysis meaning over thinking and over analyzing the situation to the point where whichever direction you want to go there`s a ton of arguments for it but also against it. 

What happened is I reached the point where the behaviour of the exchange or more precisely the behaviour of my orders to/from/on the exchange became a significant factor in profitability. Thus one of the goals was to test my assumptions on this vs what happens in the real world... which turned out to be very wrong. None the less it is extremely valuable info to know.

2) Real Trading. 

Its all to easy to code up some system, back test it, do whatever feel smug about how much paper money you`ve made...  but is completely different to trading with real money on the real exchange with real consequences if you seriously fuck up. First few weeks were seriously terrifying, what if X happens, what if Y happens what if X causes Y which in turn means I need to liquidate everything I own, move into a tent and beg for change. The reality is if you have multiple independent saftey valves both aggressive and passive there`s not a whole lot that can go wrong and when it does, its not that bad. At this point, I`m quite comfortable running fully automated strategies on the exchange that puts 1000`s of orders  into the book each day.

3) Building Real Trading System.

Learnt alot when building my system. Its one thing to build tools & simulators, completely different to have prod systems that are live and constantly evolving. Ended up writing a bunch of tools adding new components, doing new analysis and quite a few things that are simple but did not anticipate. All very valuable info for Round 2.

4) Watching the Market

Before prod, I almost never looked at what the market did day-to-day, quite oblivious infact as was focusing on historical data. I think your perspective & pattern matching radically changes when your no longer in back test time. When 1 day is say 5 seconds becomes 1 day is 6.5 Hours of up/down/up/down I gained some intuition about how the market moves in the mid-freq time frame. Guess you could say, had no connection to the  market, as it was just numbers and stats - it had no personality.

5) Bite sized Risk

Had a choice of going all out with a significant % of net worth deposit, 5 figure monthly infra costs the hole 9 yards. Or a chunk of change deposit that's a small % of net worth and infra thats good but not on the edge. Chose the latter, my reasoning was have 0 experience trading and I should not risk my current assets on something thats unproven and un-tested. e.g. Do a test first before committing more cash, and I think this was the correct choice.

What Went Wrong

1) No Strategy

Its kind of a fib to say there were no strategies, perhaps a better wording is had no clearly profitable strategy. The strategies I wanted to run are all highly skewed by the infrastructure/comissions and behaviour of the exchange. So in hindsight the risk was actually very high as the result was highly dependent on a large unknown factor. The end result was after realizing the first batch of strategies wasn't going to work had to start from 0 on a completely different style of trading which there wasn't enough time to get up to speed on.

2) Going it Alone

One of the problems when trading in a bubble is you have no one to bounce ideas off, no one is there to point out the obvious and most importantly no one is there to slap you in the face and say 

what the fuck man? 

Self reflection and transitioning between abstraction levels is really hard, especially when you eat/breath/sleep at one level its not easy to step back and look at the wider picture and realize your spending time in the wrong place.

3) Wrong Infrastructure

The infrastructure/commissions was clearly a bad choice for the kinds of strategies I was testing. Yet its hard to say it was wrong given the cost & risk of more appropriate infra. In the end it was just an incorrect matching of strategy and infrastructure, given the goal is to make a profit. If the goal was to understand the exchange / market it was a good choice but.... in the real world one needs to make money.

Theres a bunch of smaller things but I think that summarized the main points.  Round 1 was fun, exciting, learnt an absolute ton but need to re-calibrate before entering the ring again for Round 2.

Looking for a Trading Crew

Have met quite a few people over the last 9 months from a wide cross-section of the trading eco-system. From stressed traders, to hardcore techies, to hardcore WOW players looking to reduce latency! Yet at this point in time, I need more experience/mentors/infrastructure to accelerate my progress.

Looking for a lean and mean group (or individual) that`s cool with remote and wants my trading to be successful.

What can I offer?

- 100% dedicated self starter, "gets shit done", no supervision needed

- Full front to back UHF/HF strategy life cycle

• Idea
• Research
• Backtest
• Implementation
• Trading

- Code almost anything

• CPU codez  (assembler/C/C++/scripting)
• GPU codez (CUDA/OpenCL/DirectX/OpenGL)
• FPGA codez (Verilog)
• Anything Embedded ...fixed point eh?
• Anything Network Layer1 - Layer7
• Anything high performance

- Maintain/hack/own anything with electricity

• Network Infrastructure (cisco/arista config)
• Server Infrastructure (linux anything)
• Trading Systems (Front/Back/Middle)
• Historical System (Market Data/BackTest/Analysis)
• Strategies (Trade/Research/Tweek)
• Playstation3 (Design/Build/Deploy/Config/Hack)
• Coffee Pot (Order/Ack/Fill/Cancel/Panic)
• etc etc .. 

I learn like a sponge, work like a dog and looking for a group to trade with long term.

Hit me up, hacking.nasdaq@gmail.com

NOTE: Can only do remote or remote+travel

HFT Strategy

Two posts in one day... yes, have too much free time. Colo box is heading back, lost this round but just one of many to come. Decided to target pure FPGA strategies as it plays to my strengths  - hardcore tech (yes can code verilog) & need team up for that,  but more on that later.

So what does the return on a HFT strategy look like? Somethings like this.

Above is March 2012 worth of accumulated PnL, horizontal axis is # trades, vertical axis is accumulated dollars. In this case it makes about $4K / month on a single symbol on roughly 4M shares / month of volume, which is about $80MM/month of trades. On the day-to-day basis 90% of the time the PnL is flat or positive doing about 100-200 trades per day - keep in mind this is only 1 symbol out of many. Its simulated with full trade & quote data + liquidity rebates, commissions and latency discussed later.

How real is it? Its not live thus 100% BS :) In reality my model of the nasdaq queue is fairly simple and there`s some weirdness in the matching engine I haven`t worked out thus isn`t modeled. The only way to know is trade it for real... which I dont have the infra for.

The theory goes $4K/month for 1 symbol,  find 10 symbols makes it $40K/month, 20 symbols $80K/month etc all is good. Thus go out, signup for an eTrade/IB account and trade your way to $100M return in a year... or NOT.

HFT strategies are either highly sensitive to latency, commissions or both, so lets explore. First up latency.

The above plots show the monthly PnL with a -1.25 commission at various levels of simulated latency. From top to bottom its, 0usec, 100usec, 250usec, 500usec, 1msec, 2msec, 5msec, 10msec, 100msec. Thats a pretty broad spectrum. Whats clear is the variance dramatically changes from a 100usec latency to a 100msec latency aka the return gets very "lumpy" something you really don`t want.

In pure dollar terms, 0usec latency returns $3,942 with 100usec $3,179, 1msec $2,341, 100msec $1,978.

Or put another way, 

100usec  $3942 - $3179 =  $763/mo @ 10 symbols thats  $7,630/mo

+900usec $3942 - $2341 = $1601/mo @ 10 symbols thats $16,010/mo

+90msec  $3942 - $1978 = $1964/mo @ 10 symbols thats $19,640/mo

... 90msec costs $20,000/mo. Which can be interpreted in a variety of ways. e.g. dont trade on 100msec of latency!

If you look closely the number of trades slightly decreases then increases with the higher latency, the reason? Its a passive strategy thus if your too slow when the world moves you get hit by a bus because your sleeping in the middle of the road... instead of running to the other side. Resulting in a alot of unfavorable trades. In tennis I guess this is called "unforced errors".

So even @ 100msec latency its profitable making a hypothetical Tech setup of IB account hooked up to say Amazon EC2 & even if the everything is Java(.. ducks..) we should be well under 100msec tick to trade.. all good.

The other axis is per trade commission's as discussed in a previous post. Whats the PnL look like? ......

... and there`s alot of red ink on the plot. The above plot has commission's from top to bottom of 0, -1, -5, -10, -15, -20 per share. As you can see @ -1 myri / share its a solid green, -5 its a bit shaky yet still firmly in the green.  At -10 myri/share its in trouble and it just goes down hill from there.

So whats a reasonable commission/latency estimate? It depends at what level of the game your playing. For IB something like 100msec & -20myri / share which gives...

... a really red and nasty picture.. it loses about -$10,000/month. Same strategy just different latency & commissions.

Whats do you need and what is realistic ? 100usec & -1.25myri/share if you have $250K - $1MM to invest and doing alot of volume every day... which looks like.

... a nice picture indeed.

As has been said by many people before HFT is half strategies, half technology, you need both or you will lose.

Tax Hike

Didn`t realize the SEC constantly changes their "securities transaction fee" aka SEC Section 31. My understanding was it was a constant fixed rate, seems not.

http://www.sec.gov/news/press/2012/2012-35.htm

So the previously stated  -0.0000192 * Notional Value is no longer correct and its now

$22.40 / 1e6 -> -0.00002240 * Notional Value

There`s a nice history of the rate here.

http://www.traderstatus.com/secfeerates.htm

Not sure why it constantly changes, so lets dig into the docs. Presumably SEC needs hit a specified $$$ value each quarter and if the volumes drop enough they need to bump the fee,

... and thats pretty much what happens

http://www.sec.gov/rules/other/2012/34-66495.pdf

"Specifically, the Commission must now adjust the  fee rates to a uniform adjusted rate that is reasonably likely to produce aggregate fee collections  (including assessments on security futures transactions) equal to the regular appropriation to the Commission for the applicable fiscal year."

Whats interesting is the number for  "regular appropriation to the Commission" which is.... $1,321,000,000

So the SEC`s revenue from this tax alone is $1.3Bn wow! But is pretty small when compared to the aggregate Personal & Corporate tax revenue each year.

The Sub Tick HFT Game

Been ranting a bit about math problems so its time for another 6am post - please excuse the occasional incoherence  By math I mean what probabilities a strategy must hit to be profitable given the rules of the game.

Basic math to NASDAQ is as follows  (in units of $0.0001 per share)

-30 remove liquidity

+21 add liquidity

-0.0000192 * dollar value of sold symbol

0 to -20 for commission 

+ gross realized value of the trade

(its a variable commission as depending who you clear with, it could be 0 e.g. mega fund with its own clearing division, or say -20 for entry level algo brokers)

Lets simplify this somewhat and say, its a pair of buy/sell at different price but same qty, making the net realized gain

Net = Qty * (SellPrice - BuyPrice + Rebate + Commission + Tax)

First structure classification is what kind of entry/exit points you choose. Options are

Aggressive + Aggressive  (AA) : -30 - 30 = -60

Aggressive + Passive     (AP) : -30 + 21 = -9

Passive + Agressive      (PA) : +21 - 30 = -9

Passive + Passive        (PP) : +21 + 21 = +42

(Passive means your orders was visible on the book when it got executed)

(Aggressive means your order hit something that was visible on the book e.g. crossed the spread)

Simplify this a little into 3 classes

AA ( -60) AP (-9) PP(+42)

Note: AA your crossing the spread thus its really -160 but for back of the envelope purposes lets ignore that and enjoy Disney Land.

Let say our strategy has 2 discrete poles, +100 win and -100 loss, so 1 tick each side with no 0 draw result.

Below is a plot of the the trades winning probability (most left column) and the realized gain/loss for 1 share @ the specified probability for each entry/exit styles.

Rebate Only

0.000000 : AA:    -160 AP:    -109 PP:     -58

0.040000 : AA:    -152 AP:    -101 PP:     -50

0.080000 : AA:    -144 AP:     -93 PP:     -42

0.120000 : AA:    -136 AP:     -85 PP:     -34

0.160000 : AA:    -128 AP:     -77 PP:     -26

0.200000 : AA:    -120 AP:     -69 PP:     -18

0.240000 : AA:    -112 AP:     -61 PP:     -10

0.280000 : AA:    -104 AP:     -53 PP:      -2

0.320000 : AA:     -96 AP:     -45 PP:       6

0.360000 : AA:     -88 AP:     -37 PP:      14

0.400000 : AA:     -80 AP:     -29 PP:      22

0.440000 : AA:     -72 AP:     -21 PP:      29

0.480000 : AA:     -64 AP:     -13 PP:      38

0.520000 : AA:     -56 AP:      -5 PP:      46

0.560000 : AA:     -47 AP:       3 PP:      54

0.600000 : AA:     -40 AP:      11 PP:      62

0.640000 : AA:     -32 AP:      19 PP:      70

0.680000 : AA:     -24 AP:      27 PP:      78

0.720000 : AA:     -16 AP:      35 PP:      86

0.760000 : AA:      -8 AP:      43 PP:      94

0.800000 : AA:       0 AP:      51 PP:     102

0.840000 : AA:       8 AP:      59 PP:     110

0.880000 : AA:      16 AP:      67 PP:     118

0.920000 : AA:      24 AP:      75 PP:     126

0.960000 : AA:      32 AP:      83 PP:     134

1.000000 : AA:      40 AP:      91 PP:     142

Result here is, you need a strategy > 80% wins with an AA model > 56% wins for a AP model, and > 32% for a PP model. 

That makes sense, so lets add the tax @ say a $20symbol

Rebate + Tax @ $20

0.000000 : AA:    -163 AP:    -112 PP:     -61

0.040000 : AA:    -155 AP:    -104 PP:     -53

0.080000 : AA:    -147 AP:     -96 PP:     -45

0.120000 : AA:    -139 AP:     -88 PP:     -37

0.160000 : AA:    -131 AP:     -80 PP:     -29

0.200000 : AA:    -123 AP:     -72 PP:     -21

0.240000 : AA:    -115 AP:     -64 PP:     -13

0.280000 : AA:    -107 AP:     -56 PP:      -5

0.320000 : AA:     -99 AP:     -48 PP:       2

0.360000 : AA:     -91 AP:     -40 PP:      10

0.400000 : AA:     -83 AP:     -32 PP:      18

0.440000 : AA:     -75 AP:     -24 PP:      26

0.480000 : AA:     -67 AP:     -16 PP:      34

0.520000 : AA:     -59 AP:      -8 PP:      42

0.560000 : AA:     -51 AP:       0 PP:      50

0.600000 : AA:     -43 AP:       7 PP:      58

0.640000 : AA:     -35 AP:      15 PP:      66

0.680000 : AA:     -27 AP:      23 PP:      74

0.720000 : AA:     -19 AP:      31 PP:      82

0.760000 : AA:     -11 AP:      39 PP:      90

0.800000 : AA:      -3 AP:      47 PP:      98

0.840000 : AA:       4 AP:      55 PP:     106

0.880000 : AA:      12 AP:      63 PP:     114

0.920000 : AA:      20 AP:      71 PP:     122

0.960000 : AA:      28 AP:      79 PP:     130

1.000000 : AA:      36 AP:      87 PP:     138

Not a huge difference, yet even $20 in tax shifts everything by a few %, which could easily move your strategy into the red.

Lets up it a notch and say you want to trade SPY @ say $139 (2012/04/17)

Rebate + Tax @ $139

0.000000 : AA:    -186 AP:    -135 PP:     -84

0.040000 : AA:    -178 AP:    -127 PP:     -76

0.080000 : AA:    -170 AP:    -119 PP:     -68

0.120000 : AA:    -162 AP:    -111 PP:     -60

0.160000 : AA:    -154 AP:    -103 PP:     -52

0.200000 : AA:    -146 AP:     -95 PP:     -44

0.240000 : AA:    -138 AP:     -87 PP:     -36

0.280000 : AA:    -130 AP:     -79 PP:     -28

0.320000 : AA:    -122 AP:     -71 PP:     -20

0.360000 : AA:    -114 AP:     -63 PP:     -12

0.400000 : AA:    -106 AP:     -55 PP:      -4

0.440000 : AA:     -98 AP:     -47 PP:       3

0.480000 : AA:     -90 AP:     -39 PP:      11

0.520000 : AA:     -82 AP:     -31 PP:      19

0.560000 : AA:     -74 AP:     -23 PP:      27

0.600000 : AA:     -66 AP:     -15 PP:      35

0.640000 : AA:     -58 AP:      -7 PP:      43

0.680000 : AA:     -50 AP:       0 PP:      51

0.720000 : AA:     -42 AP:       8 PP:      59

0.760000 : AA:     -34 AP:      16 PP:      67

0.800000 : AA:     -26 AP:      24 PP:      75

0.840000 : AA:     -18 AP:      32 PP:      83

0.880000 : AA:     -10 AP:      40 PP:      91

0.920000 : AA:      -2 AP:      48 PP:      99

0.960000 : AA:       5 AP:      56 PP:     107

1.000000 : AA:      13 AP:      64 PP:     115

Things look a bit more "difficult", with an AA structure you need to hit >= 96% accuracy, PA >= 68% and PP >= 44%. Pretty clear AA for a +100/-100 trade is getting near impossible and if you consider the additional -100 spread cost... it is impossible.

The above has a commission of 0, so lets add say -5 commission on a $20 symbol

Rebate + Commission (-5) + Tax @ $20

0.000000 : AA:    -173 AP:    -122 PP:     -71

0.040000 : AA:    -165 AP:    -114 PP:     -63

0.080000 : AA:    -157 AP:    -106 PP:     -55

0.120000 : AA:    -149 AP:     -98 PP:     -47

0.160000 : AA:    -141 AP:     -90 PP:     -39

0.200000 : AA:    -133 AP:     -82 PP:     -31

0.240000 : AA:    -125 AP:     -74 PP:     -23

0.280000 : AA:    -117 AP:     -66 PP:     -15

0.320000 : AA:    -109 AP:     -58 PP:      -7

0.360000 : AA:    -101 AP:     -50 PP:       0

0.400000 : AA:     -93 AP:     -42 PP:       8

0.440000 : AA:     -85 AP:     -34 PP:      16

0.480000 : AA:     -77 AP:     -26 PP:      24

0.520000 : AA:     -69 AP:     -18 PP:      32

0.560000 : AA:     -61 AP:     -10 PP:      40

0.600000 : AA:     -53 AP:      -2 PP:      48

0.640000 : AA:     -45 AP:       5 PP:      56

0.680000 : AA:     -37 AP:      13 PP:      64

0.720000 : AA:     -29 AP:      21 PP:      72

0.760000 : AA:     -21 AP:      29 PP:      80

0.800000 : AA:     -13 AP:      37 PP:      88

0.840000 : AA:      -5 AP:      45 PP:      96

0.880000 : AA:       2 AP:      53 PP:     104

0.920000 : AA:      10 AP:      61 PP:     112

0.960000 : AA:      18 AP:      69 PP:     120

1.000000 : AA:      26 AP:      77 PP:     128

At -5 / share (remember its on both sides) AA requires nearly 90% accuracy, that`s a tough number to hit.

Rebate + Commission @ -10 / shares + Tax @ $20

0.000000 : AA:    -183 AP:    -132 PP:     -81

0.040000 : AA:    -175 AP:    -124 PP:     -73

0.080000 : AA:    -167 AP:    -116 PP:     -65

0.120000 : AA:    -159 AP:    -108 PP:     -57

0.160000 : AA:    -151 AP:    -100 PP:     -49

0.200000 : AA:    -143 AP:     -92 PP:     -41

0.240000 : AA:    -135 AP:     -84 PP:     -33

0.280000 : AA:    -127 AP:     -76 PP:     -25

0.320000 : AA:    -119 AP:     -68 PP:     -17

0.360000 : AA:    -111 AP:     -60 PP:      -9

0.400000 : AA:    -103 AP:     -52 PP:      -1

0.440000 : AA:     -95 AP:     -44 PP:       6

0.480000 : AA:     -87 AP:     -36 PP:      14

0.520000 : AA:     -79 AP:     -28 PP:      22

0.560000 : AA:     -71 AP:     -20 PP:      30

0.600000 : AA:     -63 AP:     -12 PP:      38

0.640000 : AA:     -55 AP:      -4 PP:      46

0.680000 : AA:     -47 AP:       3 PP:      54

0.720000 : AA:     -39 AP:      11 PP:      62

0.760000 : AA:     -31 AP:      19 PP:      70

0.800000 : AA:     -23 AP:      27 PP:      78

0.840000 : AA:     -15 AP:      35 PP:      86

0.880000 : AA:      -7 AP:      43 PP:      94

0.920000 : AA:       0 AP:      51 PP:     102

0.960000 : AA:       8 AP:      59 PP:     110

1.000000 : AA:      16 AP:      67 PP:     118

Commissions at -10/share with a +100/-100 2 pole result, AA nearing impossible requiring 95% accuracy for any return e.g. you need a time machine to make money with this structure.

Finally lets really push it to -20/share

0.000000 : AA:    -203 AP:    -152 PP:    -101

0.040000 : AA:    -195 AP:    -144 PP:     -93

0.080000 : AA:    -187 AP:    -136 PP:     -85

0.120000 : AA:    -179 AP:    -128 PP:     -77

0.160000 : AA:    -171 AP:    -120 PP:     -69

0.200000 : AA:    -163 AP:    -112 PP:     -61

0.240000 : AA:    -155 AP:    -104 PP:     -53

0.280000 : AA:    -147 AP:     -96 PP:     -45

0.320000 : AA:    -139 AP:     -88 PP:     -37

0.360000 : AA:    -131 AP:     -80 PP:     -29

0.400000 : AA:    -123 AP:     -72 PP:     -21

0.440000 : AA:    -115 AP:     -64 PP:     -13

0.480000 : AA:    -107 AP:     -56 PP:      -5

0.520000 : AA:     -99 AP:     -48 PP:       2

0.560000 : AA:     -91 AP:     -40 PP:      10

0.600000 : AA:     -83 AP:     -32 PP:      18

0.640000 : AA:     -75 AP:     -24 PP:      26

0.680000 : AA:     -67 AP:     -16 PP:      34

0.720000 : AA:     -59 AP:      -8 PP:      42

0.760000 : AA:     -51 AP:       0 PP:      50

0.800000 : AA:     -43 AP:       7 PP:      58

0.840000 : AA:     -35 AP:      15 PP:      66

0.880000 : AA:     -27 AP:      23 PP:      74

0.920000 : AA:     -19 AP:      31 PP:      82

0.960000 : AA:     -11 AP:      39 PP:      90

1.000000 : AA:      -3 AP:      47 PP:      98

At this point AA is theoretically impossible with PA needing 76% of trades to be correct just to break even.

Obviously its far more involved with a ton of subtle yet major pain-in-the-ass-problems but gives you an idea of the level of accuracy & commissions needed to play the sub-tick game.

Also this says nothing about what kind of latency/tech needed for passively executed orders... or  how low the real-world probabilities are but that`s for some other time.... if only that order passively unwound at that price, there would be peace joy and happiness in the world :P

... the grind continues