I was looking through some of the raw data in the platform I use and I noticed that sometimes (fairly rarely) that some of my backfill data has ticks with timestamps that are out of order. For example:

@ES#
2011-07-11 15:58:34,1315.00,4573299  <-- tick id
2011-07-11 15:58:34,1315.00,4573298
2011-07-11 15:58:33,1315.00,4573297
2011-07-11 15:58:33,1315.00,4573296
2011-07-11 15:58:34,1315.00,4573295
2011-07-11 15:58:33,1315.00,4573294
2011-07-11 15:58:33,1315.00,4573293

You can see the third tick from the bottom has a timestamp that is out of order. The tick ID seems to suggest that the timestamp is incorrect (either that or the tick id is incorrect). I refreshed the data and it stays the same so I believe it's stored on the server this way. It doesn't happen too often but it always happens the same way with the timestamp slightly off but the tick id is in order.

Even though it seems to not happen often, I'm a bit fanatical about having accurate data so I'm wondering if I should manually correct the timestamp or move the ticks to their correct position in the stream?
Edited by programmer on Jul 31, 2011 at 09:25 PM

unfortunately the exchange doesn't always send data in order. What you are seeing is data you would have seen on the stream as well. It would be up to you if you want to re-order the data, but if you backtest on data you have made adjustments to, just be aware that in the stream you will be seeing it differently which likely would affect your live trading results. Unfortunately "accurate" data and "orderly" data don't always go hand in hand.

---

Jay Froscheiser
Telvent DTN

OK thanks. If that's the way it would show up in the live stream then I'll just leave it as is.

what does it mean when tick ids are non-consecutive? ie example below 2558081 jumps to 2558117, is there any significance?


2011-08-09 01:55:07 1.0121 0 C 1.0119 0 1.0121 1 2558117 49518
2011-08-09 01:54:59 1.012 0 C 1.0119 0 1.012 1 2558081 49517
2011-08-09 01:54:57 1.0121 0 C 1.012 0 1.012 5 2558058 49516
2011-08-09 01:54:57 1.0121 0 C 1.012 0 1.012 1 2558057 49511

There is no significance to the tickid. It is a unique identifier for the tick and you may not always see them in order.

---

Jay Froscheiser
Telvent DTN