b-b-b-b-b-bursting

As we saw in the previous post, the tick datarates appear to be pretty low end, to the point where - why cant you subscribe to nasdaq level2 marketdata over your home DSL? hmm.. the answer? its all about bursting.

Take an ITCH new order tick without MPID of say, 31Bytes and lets say double it to 64Bytes, and use a time period of 1second. Resulting in

1000e6(1Gbit) / 8 (bits->bytes) / 64 (our tick) ~= 1.9M new orders per second.

hypothetically say, each order is 100shares @ $5USD  resulting in

1.9M * 100 *5 ~= $950M usd / seccond

 ... so around $1Bn usd / second of new orders... which is crazy for any sustained period of time.

6.5H * 60min * 60 sec * $1Bn ~= $23,400Bn USD / day
...just pocket change.

Thus enter the burstyness of trading.

If we change the timebin to count burst rate over a sliding window of 100usec instead of 1second, then the picture becomes much clearer. Keep in mind the graph samples @ 10msec taking the peek burst rate of that slice (10,000usec/100usec) e.g 100 samples and plots it.

.. and it starts to get more interesting. Here we can see during trading hours its typically bursting at around a more expected 500Mbit/sec

occasionally peeking at the 1000Mbit/sec

Market open also becomes more intense, with a boatload of 500Mbit/sec bursts at the start

the next step is to reduce the timebin by x10, calculating the max bandwidth of a sliding window of 10usec, over the 10msec sample point

... and see a peek of around 38GBit/sec at market close! hmm.... meaning... I need to confess that im using the nanosec timestamp in the ITCH packet for all these calculations. e.g its software time stamped by NASDAQ and isnt hardware timestamped as it comes down the wire. Yet its still a good estimate of what is being sent.

market open also looks more interesting, with a consistent burst rate in the 5GBit/sec rate. Interestingly its exactly x10 higher than our 100usec time bin.

finally what happens if we use the duration of the last 256packets sent, instead of fixed time window?

Above is maket open, showing the familiar cyclic lumps 30sec/kaboom/relax opening pattern. However if we drill down into the samples theres  some rather interesting behaviour  observed.

... which looks extremely artificial. First thoughts are, this is the garbage collector running on some of the nasdaq servers. Heres a closeup

no idea what kind of software nasdaq runs, but would guess a managed language like C# or java. None the less, as its pretty unlikely all participants on the exchange would suddenly stop sending orders, or send at such a consistent rate that would result in such a plot.

.... and the exercise left to you, the reader, is how do you turn garbage into alpha :P