HFT Strategy

Two posts in one day... yes, have too much free time. Colo box is heading back, lost this round but just one of many to come. Decided to target pure FPGA strategies as it plays to my strengths  - hardcore tech (yes can code verilog) & need team up for that,  but more on that later.

So what does the return on a HFT strategy look like? Somethings like this.

Above is March 2012 worth of accumulated PnL, horizontal axis is # trades, vertical axis is accumulated dollars. In this case it makes about $4K / month on a single symbol on roughly 4M shares / month of volume, which is about $80MM/month of trades. On the day-to-day basis 90% of the time the PnL is flat or positive doing about 100-200 trades per day - keep in mind this is only 1 symbol out of many. Its simulated with full trade & quote data + liquidity rebates, commissions and latency discussed later.

How real is it? Its not live thus 100% BS :) In reality my model of the nasdaq queue is fairly simple and there`s some weirdness in the matching engine I haven`t worked out thus isn`t modeled. The only way to know is trade it for real... which I dont have the infra for.

The theory goes $4K/month for 1 symbol,  find 10 symbols makes it $40K/month, 20 symbols $80K/month etc all is good. Thus go out, signup for an eTrade/IB account and trade your way to $100M return in a year... or NOT.

HFT strategies are either highly sensitive to latency, commissions or both, so lets explore. First up latency.

The above plots show the monthly PnL with a -1.25 commission at various levels of simulated latency. From top to bottom its, 0usec, 100usec, 250usec, 500usec, 1msec, 2msec, 5msec, 10msec, 100msec. Thats a pretty broad spectrum. Whats clear is the variance dramatically changes from a 100usec latency to a 100msec latency aka the return gets very "lumpy" something you really don`t want.

In pure dollar terms, 0usec latency returns $3,942 with 100usec $3,179, 1msec $2,341, 100msec $1,978.

Or put another way, 

100usec  $3942 - $3179 =  $763/mo @ 10 symbols thats  $7,630/mo

+900usec $3942 - $2341 = $1601/mo @ 10 symbols thats $16,010/mo

+90msec  $3942 - $1978 = $1964/mo @ 10 symbols thats $19,640/mo

... 90msec costs $20,000/mo. Which can be interpreted in a variety of ways. e.g. dont trade on 100msec of latency!

If you look closely the number of trades slightly decreases then increases with the higher latency, the reason? Its a passive strategy thus if your too slow when the world moves you get hit by a bus because your sleeping in the middle of the road... instead of running to the other side. Resulting in a alot of unfavorable trades. In tennis I guess this is called "unforced errors".

So even @ 100msec latency its profitable making a hypothetical Tech setup of IB account hooked up to say Amazon EC2 & even if the everything is Java(.. ducks..) we should be well under 100msec tick to trade.. all good.

The other axis is per trade commission's as discussed in a previous post. Whats the PnL look like? ......

... and there`s alot of red ink on the plot. The above plot has commission's from top to bottom of 0, -1, -5, -10, -15, -20 per share. As you can see @ -1 myri / share its a solid green, -5 its a bit shaky yet still firmly in the green.  At -10 myri/share its in trouble and it just goes down hill from there.

So whats a reasonable commission/latency estimate? It depends at what level of the game your playing. For IB something like 100msec & -20myri / share which gives...

... a really red and nasty picture.. it loses about -$10,000/month. Same strategy just different latency & commissions.

Whats do you need and what is realistic ? 100usec & -1.25myri/share if you have $250K - $1MM to invest and doing alot of volume every day... which looks like.

... a nice picture indeed.

As has been said by many people before HFT is half strategies, half technology, you need both or you will lose.

Tax Hike

Didn`t realize the SEC constantly changes their "securities transaction fee" aka SEC Section 31. My understanding was it was a constant fixed rate, seems not.

http://www.sec.gov/news/press/2012/2012-35.htm

So the previously stated  -0.0000192 * Notional Value is no longer correct and its now

$22.40 / 1e6 -> -0.00002240 * Notional Value

There`s a nice history of the rate here.

http://www.traderstatus.com/secfeerates.htm

Not sure why it constantly changes, so lets dig into the docs. Presumably SEC needs hit a specified $$$ value each quarter and if the volumes drop enough they need to bump the fee,

... and thats pretty much what happens

http://www.sec.gov/rules/other/2012/34-66495.pdf

"Specifically, the Commission must now adjust the  fee rates to a uniform adjusted rate that is reasonably likely to produce aggregate fee collections  (including assessments on security futures transactions) equal to the regular appropriation to the Commission for the applicable fiscal year."

Whats interesting is the number for  "regular appropriation to the Commission" which is.... $1,321,000,000

So the SEC`s revenue from this tax alone is $1.3Bn wow! But is pretty small when compared to the aggregate Personal & Corporate tax revenue each year.

The Sub Tick HFT Game

Been ranting a bit about math problems so its time for another 6am post - please excuse the occasional incoherence  By math I mean what probabilities a strategy must hit to be profitable given the rules of the game.

Basic math to NASDAQ is as follows  (in units of $0.0001 per share)

-30 remove liquidity

+21 add liquidity

-0.0000192 * dollar value of sold symbol

0 to -20 for commission 

+ gross realized value of the trade

(its a variable commission as depending who you clear with, it could be 0 e.g. mega fund with its own clearing division, or say -20 for entry level algo brokers)

Lets simplify this somewhat and say, its a pair of buy/sell at different price but same qty, making the net realized gain

Net = Qty * (SellPrice - BuyPrice + Rebate + Commission + Tax)

First structure classification is what kind of entry/exit points you choose. Options are

Aggressive + Aggressive  (AA) : -30 - 30 = -60

Aggressive + Passive     (AP) : -30 + 21 = -9

Passive + Agressive      (PA) : +21 - 30 = -9

Passive + Passive        (PP) : +21 + 21 = +42

(Passive means your orders was visible on the book when it got executed)

(Aggressive means your order hit something that was visible on the book e.g. crossed the spread)

Simplify this a little into 3 classes

AA ( -60) AP (-9) PP(+42)

Note: AA your crossing the spread thus its really -160 but for back of the envelope purposes lets ignore that and enjoy Disney Land.

Let say our strategy has 2 discrete poles, +100 win and -100 loss, so 1 tick each side with no 0 draw result.

Below is a plot of the the trades winning probability (most left column) and the realized gain/loss for 1 share @ the specified probability for each entry/exit styles.

Rebate Only

0.000000 : AA:    -160 AP:    -109 PP:     -58

0.040000 : AA:    -152 AP:    -101 PP:     -50

0.080000 : AA:    -144 AP:     -93 PP:     -42

0.120000 : AA:    -136 AP:     -85 PP:     -34

0.160000 : AA:    -128 AP:     -77 PP:     -26

0.200000 : AA:    -120 AP:     -69 PP:     -18

0.240000 : AA:    -112 AP:     -61 PP:     -10

0.280000 : AA:    -104 AP:     -53 PP:      -2

0.320000 : AA:     -96 AP:     -45 PP:       6

0.360000 : AA:     -88 AP:     -37 PP:      14

0.400000 : AA:     -80 AP:     -29 PP:      22

0.440000 : AA:     -72 AP:     -21 PP:      29

0.480000 : AA:     -64 AP:     -13 PP:      38

0.520000 : AA:     -56 AP:      -5 PP:      46

0.560000 : AA:     -47 AP:       3 PP:      54

0.600000 : AA:     -40 AP:      11 PP:      62

0.640000 : AA:     -32 AP:      19 PP:      70

0.680000 : AA:     -24 AP:      27 PP:      78

0.720000 : AA:     -16 AP:      35 PP:      86

0.760000 : AA:      -8 AP:      43 PP:      94

0.800000 : AA:       0 AP:      51 PP:     102

0.840000 : AA:       8 AP:      59 PP:     110

0.880000 : AA:      16 AP:      67 PP:     118

0.920000 : AA:      24 AP:      75 PP:     126

0.960000 : AA:      32 AP:      83 PP:     134

1.000000 : AA:      40 AP:      91 PP:     142

Result here is, you need a strategy > 80% wins with an AA model > 56% wins for a AP model, and > 32% for a PP model. 

That makes sense, so lets add the tax @ say a $20symbol

Rebate + Tax @ $20

0.000000 : AA:    -163 AP:    -112 PP:     -61

0.040000 : AA:    -155 AP:    -104 PP:     -53

0.080000 : AA:    -147 AP:     -96 PP:     -45

0.120000 : AA:    -139 AP:     -88 PP:     -37

0.160000 : AA:    -131 AP:     -80 PP:     -29

0.200000 : AA:    -123 AP:     -72 PP:     -21

0.240000 : AA:    -115 AP:     -64 PP:     -13

0.280000 : AA:    -107 AP:     -56 PP:      -5

0.320000 : AA:     -99 AP:     -48 PP:       2

0.360000 : AA:     -91 AP:     -40 PP:      10

0.400000 : AA:     -83 AP:     -32 PP:      18

0.440000 : AA:     -75 AP:     -24 PP:      26

0.480000 : AA:     -67 AP:     -16 PP:      34

0.520000 : AA:     -59 AP:      -8 PP:      42

0.560000 : AA:     -51 AP:       0 PP:      50

0.600000 : AA:     -43 AP:       7 PP:      58

0.640000 : AA:     -35 AP:      15 PP:      66

0.680000 : AA:     -27 AP:      23 PP:      74

0.720000 : AA:     -19 AP:      31 PP:      82

0.760000 : AA:     -11 AP:      39 PP:      90

0.800000 : AA:      -3 AP:      47 PP:      98

0.840000 : AA:       4 AP:      55 PP:     106

0.880000 : AA:      12 AP:      63 PP:     114

0.920000 : AA:      20 AP:      71 PP:     122

0.960000 : AA:      28 AP:      79 PP:     130

1.000000 : AA:      36 AP:      87 PP:     138

Not a huge difference, yet even $20 in tax shifts everything by a few %, which could easily move your strategy into the red.

Lets up it a notch and say you want to trade SPY @ say $139 (2012/04/17)

Rebate + Tax @ $139

0.000000 : AA:    -186 AP:    -135 PP:     -84

0.040000 : AA:    -178 AP:    -127 PP:     -76

0.080000 : AA:    -170 AP:    -119 PP:     -68

0.120000 : AA:    -162 AP:    -111 PP:     -60

0.160000 : AA:    -154 AP:    -103 PP:     -52

0.200000 : AA:    -146 AP:     -95 PP:     -44

0.240000 : AA:    -138 AP:     -87 PP:     -36

0.280000 : AA:    -130 AP:     -79 PP:     -28

0.320000 : AA:    -122 AP:     -71 PP:     -20

0.360000 : AA:    -114 AP:     -63 PP:     -12

0.400000 : AA:    -106 AP:     -55 PP:      -4

0.440000 : AA:     -98 AP:     -47 PP:       3

0.480000 : AA:     -90 AP:     -39 PP:      11

0.520000 : AA:     -82 AP:     -31 PP:      19

0.560000 : AA:     -74 AP:     -23 PP:      27

0.600000 : AA:     -66 AP:     -15 PP:      35

0.640000 : AA:     -58 AP:      -7 PP:      43

0.680000 : AA:     -50 AP:       0 PP:      51

0.720000 : AA:     -42 AP:       8 PP:      59

0.760000 : AA:     -34 AP:      16 PP:      67

0.800000 : AA:     -26 AP:      24 PP:      75

0.840000 : AA:     -18 AP:      32 PP:      83

0.880000 : AA:     -10 AP:      40 PP:      91

0.920000 : AA:      -2 AP:      48 PP:      99

0.960000 : AA:       5 AP:      56 PP:     107

1.000000 : AA:      13 AP:      64 PP:     115

Things look a bit more "difficult", with an AA structure you need to hit >= 96% accuracy, PA >= 68% and PP >= 44%. Pretty clear AA for a +100/-100 trade is getting near impossible and if you consider the additional -100 spread cost... it is impossible.

The above has a commission of 0, so lets add say -5 commission on a $20 symbol

Rebate + Commission (-5) + Tax @ $20

0.000000 : AA:    -173 AP:    -122 PP:     -71

0.040000 : AA:    -165 AP:    -114 PP:     -63

0.080000 : AA:    -157 AP:    -106 PP:     -55

0.120000 : AA:    -149 AP:     -98 PP:     -47

0.160000 : AA:    -141 AP:     -90 PP:     -39

0.200000 : AA:    -133 AP:     -82 PP:     -31

0.240000 : AA:    -125 AP:     -74 PP:     -23

0.280000 : AA:    -117 AP:     -66 PP:     -15

0.320000 : AA:    -109 AP:     -58 PP:      -7

0.360000 : AA:    -101 AP:     -50 PP:       0

0.400000 : AA:     -93 AP:     -42 PP:       8

0.440000 : AA:     -85 AP:     -34 PP:      16

0.480000 : AA:     -77 AP:     -26 PP:      24

0.520000 : AA:     -69 AP:     -18 PP:      32

0.560000 : AA:     -61 AP:     -10 PP:      40

0.600000 : AA:     -53 AP:      -2 PP:      48

0.640000 : AA:     -45 AP:       5 PP:      56

0.680000 : AA:     -37 AP:      13 PP:      64

0.720000 : AA:     -29 AP:      21 PP:      72

0.760000 : AA:     -21 AP:      29 PP:      80

0.800000 : AA:     -13 AP:      37 PP:      88

0.840000 : AA:      -5 AP:      45 PP:      96

0.880000 : AA:       2 AP:      53 PP:     104

0.920000 : AA:      10 AP:      61 PP:     112

0.960000 : AA:      18 AP:      69 PP:     120

1.000000 : AA:      26 AP:      77 PP:     128

At -5 / share (remember its on both sides) AA requires nearly 90% accuracy, that`s a tough number to hit.

Rebate + Commission @ -10 / shares + Tax @ $20

0.000000 : AA:    -183 AP:    -132 PP:     -81

0.040000 : AA:    -175 AP:    -124 PP:     -73

0.080000 : AA:    -167 AP:    -116 PP:     -65

0.120000 : AA:    -159 AP:    -108 PP:     -57

0.160000 : AA:    -151 AP:    -100 PP:     -49

0.200000 : AA:    -143 AP:     -92 PP:     -41

0.240000 : AA:    -135 AP:     -84 PP:     -33

0.280000 : AA:    -127 AP:     -76 PP:     -25

0.320000 : AA:    -119 AP:     -68 PP:     -17

0.360000 : AA:    -111 AP:     -60 PP:      -9

0.400000 : AA:    -103 AP:     -52 PP:      -1

0.440000 : AA:     -95 AP:     -44 PP:       6

0.480000 : AA:     -87 AP:     -36 PP:      14

0.520000 : AA:     -79 AP:     -28 PP:      22

0.560000 : AA:     -71 AP:     -20 PP:      30

0.600000 : AA:     -63 AP:     -12 PP:      38

0.640000 : AA:     -55 AP:      -4 PP:      46

0.680000 : AA:     -47 AP:       3 PP:      54

0.720000 : AA:     -39 AP:      11 PP:      62

0.760000 : AA:     -31 AP:      19 PP:      70

0.800000 : AA:     -23 AP:      27 PP:      78

0.840000 : AA:     -15 AP:      35 PP:      86

0.880000 : AA:      -7 AP:      43 PP:      94

0.920000 : AA:       0 AP:      51 PP:     102

0.960000 : AA:       8 AP:      59 PP:     110

1.000000 : AA:      16 AP:      67 PP:     118

Commissions at -10/share with a +100/-100 2 pole result, AA nearing impossible requiring 95% accuracy for any return e.g. you need a time machine to make money with this structure.

Finally lets really push it to -20/share

0.000000 : AA:    -203 AP:    -152 PP:    -101

0.040000 : AA:    -195 AP:    -144 PP:     -93

0.080000 : AA:    -187 AP:    -136 PP:     -85

0.120000 : AA:    -179 AP:    -128 PP:     -77

0.160000 : AA:    -171 AP:    -120 PP:     -69

0.200000 : AA:    -163 AP:    -112 PP:     -61

0.240000 : AA:    -155 AP:    -104 PP:     -53

0.280000 : AA:    -147 AP:     -96 PP:     -45

0.320000 : AA:    -139 AP:     -88 PP:     -37

0.360000 : AA:    -131 AP:     -80 PP:     -29

0.400000 : AA:    -123 AP:     -72 PP:     -21

0.440000 : AA:    -115 AP:     -64 PP:     -13

0.480000 : AA:    -107 AP:     -56 PP:      -5

0.520000 : AA:     -99 AP:     -48 PP:       2

0.560000 : AA:     -91 AP:     -40 PP:      10

0.600000 : AA:     -83 AP:     -32 PP:      18

0.640000 : AA:     -75 AP:     -24 PP:      26

0.680000 : AA:     -67 AP:     -16 PP:      34

0.720000 : AA:     -59 AP:      -8 PP:      42

0.760000 : AA:     -51 AP:       0 PP:      50

0.800000 : AA:     -43 AP:       7 PP:      58

0.840000 : AA:     -35 AP:      15 PP:      66

0.880000 : AA:     -27 AP:      23 PP:      74

0.920000 : AA:     -19 AP:      31 PP:      82

0.960000 : AA:     -11 AP:      39 PP:      90

1.000000 : AA:      -3 AP:      47 PP:      98

At this point AA is theoretically impossible with PA needing 76% of trades to be correct just to break even.

Obviously its far more involved with a ton of subtle yet major pain-in-the-ass-problems but gives you an idea of the level of accuracy & commissions needed to play the sub-tick game.

Also this says nothing about what kind of latency/tech needed for passively executed orders... or  how low the real-world probabilities are but that`s for some other time.... if only that order passively unwound at that price, there would be peace joy and happiness in the world :P

... the grind continues

Under the hood of a Black Box

Building Mid-freq strategies is alot more involved than HF/UHF strategies, atleast so far. With HF/UHF your looking for a simple pattern with a simple data transform that`s consistent enough you can build a strategy to exploit. With the mid-freq strategies its still a simple pattern the difference is the abstraction level and greeks to the trade are more complicated - at least thats what it seems. e.g no longer looking at the very short term micro inventory/information/latency arbitrage opportunities and instead stationary patterns in highly abstracted and transformed data sets.

My mid freq strategies are not going well, it takes alot of time and I put a hard deadline of end-of-the-month to have something... which gets closer each day. Thus its almost certain will be pulling the machine from colo, to regroup, lick my wounds, get some sleep and then march forward with a modified approach... just cant do this alone starting from 0.

So less ranting about my (lack of) PnL and more on the tech side. Here`s a logical block diagram of my system.

Pretty simple eh?

First up NIC`s

NIC0 -> ssh / management interface
NIC1 -> Order Enter
NIC2 -> not used
NIC3 -> Market Data

Next is HWT`s

Conventional wisdom says you should disable hyper threading as it has significant impact on the performance of the other HWT for the core, which can be true. Hyper threading works by having separate "contexts" e.g. register block & program counter in hardware but sharing the same execution pipeline. Similar in concept to a time sliced operating system where each process/thread has its own copy of all user-land registers which are swapped in at the start of the threads time slice, the thread executes for a set period of time, the registers are copied back into memory, and a new thread is swapped in. This allows multiple programs to share a single CPU and maximize the utilization of the CPU. HW threads work in a similar way but at the ISA (instruction set architecture) level ontop of a processors micro architecture.

The theory for both time sliced OS and hyper threading is, for a significant % of time the execution units are idle as the program is blocked waiting for IO. Thus some other program can utilize the hw resource while waiting for the blocked IO to complete and you get higher execution occupancy & more throughput... but at the cost of increased latency.

OS Example:

while a thread is blocked waiting for Keyboard/Disk/Network input, some other thread runs

HW Example:

a memory read missed L1/L2/L3 and has to be fetched from DDR (100cycles), some other program runs for those cycles.

Have been coding for 8 core asynchronous systems since 2001 so designing for wide processing is quite natural these days - have suffered that transition pain. Thus have plenty of process/threads but not enough cores, so have to eat the latency cost and enable hyperthreading.

Short description of each HWT. All processes/threads are locked to their respective HWT.

HWT0

This is the general purpose, everything runs on this. To linux the system looks like a 1 HWT machine. bash sshd etc etc.

HWT1:

nothing pre-defined. depends what im doing with the machine for what is assigned to this. e.g. live strategies, back testing, backup/crunching.

HWT2:

FIFO scheduled (e.g. not time sliced) for all strategies to run. Processing is setup so 1 cycle of a strategy is run, the round robbin(via linux scheduler) to the next strategy and so. Can be dangerous as the strategies can effect each other but the core strategy logic is usually very simple and light.

HWT3:

For HF/UHF the amount of brute force number crunching is not so high, thus a single HWT is sufficient. The thread has a job queue where anyone can submit something to be crunched.

HWT4:

Market Data Feed handler. You might ask why only 1 HWT for this? The answer is the more queue`s you add the higher the latency. My system only has 1 Queue and thats the Socket`s Rx Buffer which is massive. The 2nd answer is, i`m not keeping a book for all ~6.5K symbols on nasdaq thus dont need the additional throughput. As mentioned way back in 2010, the key here is extremely fast trivial reject`s to filter out all the crap you dont need.

HWT5:

This is the disk io core who`s sole purpose in life is to copy blocks of shared memory to the SSD.

HWT6:

The Gateway + Active Risk checks. This translates internal order requests (new/mod/can/exe) into native protocol versions and performs basic risk check / position management / fat finger checks before sending it into the market. Gateway or OMS as some call it has hooks to external programs which can enable/disable the sending of orders. The risk checks are minimal as its on the critical latency path, thus the more elaborate checks are done passively post trade.

HWT7:

Networking utils / passive risk checking. Part one of this HWT is capture and logs everything on the wire in all directions going everywhere, think NSA style layer2 snooping - yes I see you knocking on the door 192.168.42.1. The other part is to digest and analyze the captured data pseudo realtime. There are soft latency limits here, ideally all these functions would be running on an independent machine but... didnt want to spend the cash for that.

.. and so the quest continues as digging thru terrabytes of data, racing a 500HP golf cart on the screaming edge of technology down some sketchy back alleyway in Hong Kong... is so my thing :P

Work Life Balance

Another week finished and still unprofitable.. haven`t really traded much last week or two as busy in research/quant mode. My edge gets better by the day, but sadly the comission`s and latency do not. Current problem is I keep building profitable strategies but for some reason always make them passive ... which require extremely good queue positions which means... it wont work with my setup... meaning wasting more time.

To have an excellent queue position on say SPY requires a direct feed from nasdaq, 10Gb, ultra low latency pre trade risk checks and fpga/extremely tight code for market data processing. e.g. the ability to filter out all the crap when your about to get hit by a bus. Unfortunately this is really expensive (for myself atleast) so dont have it and thus have finally made the conscious decision to absolutely under no situation build a passive strategy.

For those who dont know, I`m based out of Tokyo which means really weird and utterly brutal hours. All the movies of dudes sitting on Yachts in the Bahamas spending 10min/day adjusting their positions via laptop over a satilite connection? ... is well ... not my case and certainly not the life of anyone I know who trades.

Imagine this schedule


Mon:  5:00 - 18:30 : 
Mon: 30min - 1H siesta
Mon: 19:30 - 00:00 : Market Open 9:30 EST (22:30 Tokyo)
Tue:  0:00 -  7:00 : Market close 16:00 EST (5:00 Tokyo)
--- sleep ---
Tue: 11:00-   0:00
Wed:  0:00 -  7:00
-- sleep ---
Wed: 11:00 -  0:00
Thu:  0:00 -  7:00
-- sleep --
Thu: 11:00 -  0:00

Fri:  0:00 -  7:00
-- sleep --
Fri: 11:00 -  0:00
Sun:  0:00 -  7:00  <--- now!
-- sleep --
Sun: 13:00 -  0:00
-- sleep -- 

And yes Monday is a 24H work day. All up think its easier to use 24H * 7 - SleepTime... which is

24*7 - (6 * 4 + 1 * 5) = 168 - 19 = 149H week

Obviously this aint sustainable long term and pile onto that 6days/week * 12-16H/day for 6months and you get some sort of idea on how many hours its taken me to get to this point.. and still not profitable (holy fark!) For those lucky enough that are in the right location and can learn from co-workers/existing code bases it makes a *huge* difference... ive wasted so much time on the wrong things, but on the plus side what I do understand is burned into my cortex forever as have paid the price via trial & error.

.....dont try this at home kids :P

Too Many Hats

Another week with consistent alpha proving elusive. Got most of the tech problems got sorted out which left only math problems. The math unfortunately is not so easy to fix. Was running a few incarnations of a sub tick arbitrage strategy that unfortunately the combination of comissions/rebates/latency/law of large numbers means it will never make money/inconsistent or not enough to cover the fixed infra costs. So hit the puke point, taken the stop loss and building a new strategy.

Whats interesting(and frustrating) is I can now build a variety of profitable back tested strategies fairly easily, which was certainly not the case a few months ago. Yet when running it live it fails to match the model. The reason? many. but to a large extent I didnt know alot of the characteristics/behaviour when interacting with a real market/real broker which significantly eroded the edge.

My general process for strategy development goes like

Stage 1) "back of the envelope mode" this is quickly test some basic assumptions with level 1 data or medium granularity bars or both. At this point don`t care about latency or even the queue and commissions/rebates arent priced in. The goal here is to check the "edge" is there and if so, it needs to be sufficiently strong  & consistent.

Stage 2) Add the queue/comissions/rebates/tax and then trade & quote data. At this point everything is run in lock step. e.g. one step == one message. Find this is where I most of the time. Whats good about this stage is, its really easy to change the strategy add new things, remove things do all sorts of random stuff to it and not have to worry about any of the order entry/market data details, e.g. order/cancel reject handling, re-priced, latency, etc etc

Stage 3) Code it up as a a real strategy. This means real async order entry/market data and you now have a time constraint - if a tick takes 1sec to crunch then you will get run-over. The environment here is a fully simulated exchange both order entry & market data that is replaying historical. This can be fiddly but your just re-coding up Stage 2) which is fairly straight forward and the environment you already coded thus dosent take very long.

Stage 4) Run it live against the real exchange. My setup makes launching a strategy simply executing a program on the command line, theres no monolithic gui/master program/program within program. All a strategy does is talk to shared memory. Thus can run it one shot at a time, or continuously, have scripts calling scripts call the strat, etc etc the usual linux scriptfu. From here its easy to monitor and watch, tune and fix/debug. The key here is to make it match the theoretical/backtested model or change the theoretical model to match the observed behaviour.

... and this is where im at. Sucks pretty badly spent so much time and effort so far without any real success. Had a few highs tho, turning my position over a few times was really cool! as was sending half a million dollars worth of orders to the market but sadly, still missing something, still in the red and the  20H days are catching up.

the quest continues...

Week Number 2

Wow my 2nd week live has finished, pretty much just testing the break, accelerator, warming up the tiers, a few test laps, getting a feel for at what speed/rpm to shift gears. how it corners and most importantly how fast does it go and how fast can it stop.

so far no trip to the emergency room.... yet lol

This pretty much sums up the experience

http://www.youtube.com/watch?v=VaFP1yZykuk

... and yes getting wacked happens a bit, tho only down $20 so far. Whats weird is there`s this completely irrational fear, similar to the fear of talking to that girl... or hitting that asshole on the nose...  and now a new addition...  putting 100shrs at the BBO. Theres no logical explanation for it. Sure, she might ignore you, or you might get some nasty bruise or you might get executed with a 100shrs of an incredibly liquid stock, all 3 of which are of little consequence.

So just like the above examples, after you`ve done it a few times and realize its not so bad, the irrational fear dissipates and the analytical part of your brain takes over from more primal/emotional.

Basic breakdown has been

Week 1

- code order entry gateway

- code market data feed handler

- code risk managmenet

- cron jobs

- scripts of all sorts

- various system infra stuff.. pissed you cant monitor ECC failures

- various silly bug fixes

Week2

- recon/SOD/EOD automation

- performance/latency tests

- debugging latency problems with market data

- re-wrote feed handler (latency probs)

(most of the week spent on market data latency)

Whats gone wrong so far?

- incorrectly sent massive burst of cancels (5000+) ... opps

- VPN connection into colo dropped out while trading! holy fark!

- a few run-away terminal sessions, flood of printf`s makes ssh un-responsive

- desktop PC`s deciding to update and reboot by themself

- pretty much most of my worst fears have happened

Murphy I really do hate you.

Next up? bit more latency/performance tuning and now finally digging deeper into strategy development, and as they say.

Gentlemen, please start your engines!