fpga paper

Ran into this advertisement^H^H^H^H^H peer reviewed IEEE paper today

http://algo-logic.com/~jwlockwd/publications/Low_Latency_Library_for_HFT-Algo-Logic-4831a009.pdf

Its quite well written and covers alot of ground yet contains some pretty bad assumptions. Lets deconstruct

In the abstract:

"The application sustains 10Gb/s Ethernet line rate with a fixed end-toend latency of 1µs  – up to two orders of magnitude lower than comparable software implementations."

What they are saying is software latency for a brain dead strategy is 100usec? This is complete and utter bullshit, yes if you write shitty code your wire-wire latency will be 100usec +/- 25usec jitter. If you write tight code using standard hardware and know what your doing... can get under 10usec wire-to-wire no problem.

Next point:

"An initial line handling stage uses redundancy in the streams to replace any information temporarily missed due to packet losses in order to reconstruct the necessary parts of the  order book of the exchange."

This is pure magic, have nfi what their talking about here. How is it possible when you drop a market data packet you can reconstruct its symbol/price/side/qty/operation, in less than 1usec, in RTL on an fpga. Will concede given say 100msec or more AND significant delay in the stream e.g. you dropped a new quote X, but received the cancel quote X msg, its possible to take a good guess what was in the packet.... maybe they are talking about using the *cough cough* market data recovery channels.

Next point:

"Prices for all 8000 securities traded on U.S. Exchanges fit within the FPGA’s on-chip memory. The table size is scalable to support more symbols as required."

This is great but wtf? Guessing they are talking about a 8000 * 4bytes(32bit fixed point price only) * 2 (bid/ask) * number of venues to store the best bid/offer for a symbol. That easily fits in the block ram of a Xilinx V5 240T (note: block ram is memory on the die of the fpga and not to be confused with external memory e.g. DDR/QDR/SRAM)

Great but... how do they calculate what the BBO is? You need to store the entire book, which has to be stored in external memory because the working set is atleast 1GB. 

Why store the entire book? 

1) When you get an quote cancel message exchanges rarely have the price/qty/side/symbol in the cancel message. Typically its just the order id and/or quantity canceled, so you need to keep price/qty/side/symbol/orderid and more around to correctly process e.g. what qty is currently open for this order after Y quote cancel messages.

2) when a quote is deleted you need to remove it from the book, then iterate through the book to find the best price - read you need to keep the book in memory somewhere. The trick here is being smart in how and when you iterate to find the best price, e.g. sorting a mostly sorted list can be pretty dam fast.

3) .... and more but wasted too much time on this today.

... so are saying, we can store 8000 bid/ask prices ondie in the fpga for the rtl coded algo to use? hmm..

Its a great well written technical paper, probably one of the best papers on the subject however glosses over some more important details... as doe all academic papers on the subject.. aka yes just sit and hold that 30% draw down for 3 months and the pnl is beautiful.

/rant

First Year in the Jungle - Part Two

Changing career from Technology to Trading is a bit odd, most tech guys I`ve known have no interest in trading or only romanticize about it and others slowly inch towards it at a prop desk. For the few (fools) who take the plunge without a clue, we get bitten on the ass by the bitch that is reality.

For some they are in a geographically good area, e.g. NY, Chicago, London for me, I didnt have that luxury. One of my options was to call up the clients I worked with and asked for a job which would have caused all sorts of weirdness/problems. But would also have to go thru the same process of start in technology, find a way into research, then strategy development, then eventually some how prove myself enough to trade, and only after all that would I get my own PnL - we`re talking years here. And if you apply conditional probability to each of these steps then the chances of getting to the end is quite low.

The other option is to trade your own capital where the only thing stopping you is the market but your completely on your own, no one to learn from or help you - anything but a walk in the park and what I chose, simply due to a lack of any other option.

Have been working professionally since age 17, moved continent age 21, moved continent again at age 27. Have been part of at least 3 fairly famous yet grueling projects, been off into the woods as solo startup founder and a ton of other shit. Out of all of this HF trading is by far the most challenging and difficult work so far. Why? because you have no one to learn from, learning material is sketchy at best and the only real way to learn is by trial and error in the market with real money. If you compound trial & error with the stress of say, there`s a bug and your holding $250K of inventory when the market moves against you, its what either breaks you or further detaches you from the emotion.

A few skills have gained this year

- Comfortable losing money

Its the old saying, need to spend money to make money. Prior to now, spending money and loosing money was extremely painful. These days it sucks but its ok to loose money. One of the traps traders get into is having to be right 100% of the time e.g. 100% profitable trades. Some trades are positive, some are negative, some days are positive, some days are negative and thats ok.

- linux/big data skills moved to a new level

Been using linux since 1995 (or 96 memory bit fuzzy) professionally its been my development environment since 2001 yet only this year when dealing with TerraBytes of data have I been forced to dig really deep into the unix philosophy. Prior to that have known, hacked and understood unix/linux technically but lacked the deep philosophy of its simplicity and how it all connects to make it run silky smooth.

- intuition with statistics

Have known basic stats for a while, used doing linear algebra for what seems forever, calculus a little but this year finally gained deeper intuition about statistics. Everyone has heard of or used min, max, mean, median, arithmetic, geometric, standard deviation, correlation, variance, t values, p values, linear regression, non-linear regression, autoregressive, eigenvalues, eigenvectors random walks, monte carlo..... blah blah blah and so on. You can even call this function / library and it spits out some result without any idea how it works, yet only recently have the basic foundations become completely natural and not forced.


Some things I miss:

- regular cheque each month

Goes hand in hand with losing money. In previous lives had say in budget allocations so knew well about the need to spend money, however when its *your* money its certainly different. The beauty of an employer/employee relationship is the comfort and (perceived) stability you get of regular cheques. Now days the amount and sign of the cheque changes all the time, not an easy transition.

- Flexible schedule

These days I live and die by the clock of the exchange. The exchange dosent care if you have something else to do, certainly dosent care if your bladder is full and most certainly does not give a shit if you have caught the flu. In previous jobs you can take days off, arrive late, leave early go to the bathroom any time and it dosent cause much disruption. For trading even automated systems you (or someone) needs to be there watching it, monitoring it so when (not if) it fucks up your around to clean up..

- 100% precision, 100% of the time stress.

Automated trading is one of those few professions that has such horrific consequences if you screw up (read Knights recent meltdown), what worse is the speed systems must be developed and changed is frantic compared to say medical/aviation which are given a very long times to build/verify. Trading changes fast and things always go wrong, as such have built systems and an approach thats different and so far has worked well. In a regular development role you can do sloppy code and it really dosent matter, or you can always come back to it when you have more time to fill in the blanks.


Some new enjoyable things

- feel alive / connected

Having such a direct connection to money some how makes it all real. In reality its still bits in a computer, statements are still bits, the money sits as bits at a bank, its all just bits, but feels mortal. Guess we get programmed as a child about money, its value how central it is thus touching this much money even virtually shakes us to the core. In other lives the closet ive got to this kind of buzz is for mischievous things.

- sit back and watch it print money

Its great when you can sit back, and watch the pnl pop up, and up, and up, and up and not even touch the keyboard. Awesome feeling, probably similar to what sales ppl get when they close deals / sell items. Dosen`t happen all the time but when its a good day, its a great feeling.

- respect from non geeks

Kinda surprising, my friends and family are impressed when we talk about trading. Seems because its real money it gets respected. In previous lines of work no one really understood, "shaved 1micro second off the latency today" vs "traded $XXXX million dollars today" the latter gets oohs and ahhs, with the former a nod and "thats nice".


Guess the main question is, after 1 year have I become successfully? Hardly. When I left my last job the goal was to consistently make $100/day which might sound ridiculous, but is orders of magnitude harder than it sounds. Thankfully I`m making more than $100/day but after everyone takes their cut, the ridiculously strong yen, financially I`m barely scraping by, Overall certainly a net loss if you amortize it over the year, rent bills food etc etc. and so far this is easily the lowest paying job ive had since turning 20yo.

20/20 hindsight would I do it all again?  ..... tough question.