Hi Steve,

I reported a similar problem earlier this year, but it seemed like the problem was on my end. However, I seem to be running into this problem again, and with more recent data, so I was wondering if you could verify if the problem is on my end, or if you see the same data that I'm seeing.

Some sample data is:

2012-09-06 15:59:28,836.20,1,102940,836.10,836.20,9328080,0,0,C,
2012-09-06 15:59:28,836.10,1,102939,836.10,836.20,9328061,0,0,C,
2012-09-06 15:59:28,836.10,1,102938,836.10,836.20,9328013,0,0,C,
2012-09-06 15:59:28,836.10,1,102937,836.10,836.20,9328013,0,0,C,
2012-09-06 15:59:28,836.10,1,102936,836.10,836.20,9328013,0,0,C,
2012-09-06 15:59:28,836.10,1,102935,836.10,836.20,9328013,0,0,C,
2012-09-06 15:59:28,836.10,1,102934,836.10,836.20,9328013,0,0,C,
2012-09-06 15:59:28,836.20,1,102933,836.10,836.20,9328002,0,0,C,
2012-09-06 15:59:28,836.10,1,102932,836.10,836.30,9327939,0,0,C,
2012-09-06 15:59:28,836.10,1,102931,836.10,836.30,9327939,0,0,C,
2012-09-06 15:59:28,836.10,1,102930,836.10,836.30,9327939,0,0,C,
2012-09-06 15:59:28,836.10,1,102929,836.10,836.30,9327939,0,0,C,
2012-09-06 15:59:28,836.10,1,102928,836.10,836.30,9327939,0,0,C,
2012-09-06 15:59:28,836.20,1,102927,836.10,836.30,9327895,0,0,C,
2012-09-06 15:59:28,836.20,1,102926,836.20,836.30,9327895,0,0,C,
2012-09-06 15:59:28,836.20,2,102925,836.20,836.30,9327895,0,0,C,
2012-09-06 15:59:28,836.30,1,102923,836.20,836.30,9327871,0,0,C,
2012-09-06 15:59:28,836.20,1,102922,836.10,836.30,9327823,0,0,C,
2012-09-06 15:59:28,836.20,1,102921,836.10,836.30,9327776,0,0,C,
2012-09-06 15:59:28,836.20,1,102920,836.10,836.30,9327760,0,0,C,
2012-09-06 15:59:28,836.20,1,102919,836.10,836.20,9327760,0,0,C,
2012-09-06 15:59:28,836.20,1,102918,836.10,836.30,9327682,0,0,C,
2012-09-06 15:59:28,836.20,1,102917,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,1,102916,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,2,102915,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,1,102913,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,3,102912,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,1,102909,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,4,102908,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.20,1,102904,836.20,836.30,9327671,0,0,C,
2012-09-06 15:59:28,836.30,1,102903,836.20,836.30,9327614,0,0,C,

I'm taking this data straight off the data stream from IQFeed, so I believe this data is correct. You can see that the tickId is being repeated many times across this same second.

Some other weird data is:

2012-08-29 20:55:00,812.40,1,9,812.10,812.90,5662,0,0,C,
2012-08-29 20:55:00,812.80,1,8,812.40,812.90,5662,0,0,C,
2012-08-29 20:55:00,812.90,1,7,0.00,0.00,5389,0,0,C,
2012-08-29 20:55:00,812.90,1,6,0.00,0.00,5389,0,0,C,
2012-08-29 20:55:00,812.90,1,5,0.00,0.00,5389,0,0,C,
2012-08-29 20:55:00,812.90,1,4,0.00,0.00,5389,0,0,C,
2012-08-29 20:55:00,812.90,2,3,0.00,0.00,5389,0,0,C,
2012-08-29 20:55:00,812.90,1,1,0.00,0.00,5389,0,0,C,

It appears that the tickId is the same for many ticks, and the bid/ask is 0.00. Is this what you see on your end?

In fact, the problem is pretty pervasive, I'm seeing literally tens of thousands of duplicate tickIds on each second, from 8/19/2012 to today.

If you can confirm that you're seeing the same data as me, I can give you the list of all the seconds that I'm seeing with duplicate tickIds, if that helps you.

thanks,

Steve

Just to clarify the above data is from @TFS#.

One more data point, the data appears to go weird as of 8/19/2012. I'm not sure if this date is relevant or not on your side.

Thanks for your help,

Steve

Hi Steve1986,

I see the duplicate tick IDs that you mention and the 0's for bid and ask prices on 8-29-12. I will ask our Market Data Services to take a look and get back to you here with a response. Thanks.

Hello Steve1986,

Thank you for bringing this to our attention. Our developers are looking at the way the data comes to us from the ICE and tell me that this situation looks more complicated than expected. We appreciate your patience and will keep you posted here.

Just a slight update to this.

We have verified that the only information that is incorrect is the TickID field. All of the rest of the data is correct.

Okay great, thanks for the update Steve.

Do you know if this problem started 8/19/2012, or is this something that has been happening for a while?

Sorry, I mean thanks Todd and Steve :)

Yes, this started on the weekend of the 19th and would affect all ICE exchanges (not just @TFS). We have identified the cause and are working on resolving it (hopefully this weekend).

The zero bid/ask issue on 8/29 is unrelated and is still being investigated.

Steve1986,

The ICE exchange tells us that the 0's for bid and ask prices on 8-29 was correct.
They further added:
"During Pre-open the orders are not sent to the market. When the state change from pre-open to open you would receive the trades during the matching period and you would start to see bid and asks when the market transition to open state. This is as designed."

We will keep working on the other issue and report back here.

Thanks Todd, so if I understand it right, the ticks don't have bid or ask because they were pre-market orders that were matched on Open?

Would you happen to know if there's any progress with the duplicate tick ids?

thanks,

Steve

Hi Steve1986,

Yes, that is correct.
And we are still working on the issue of duplicate tick ids and hope to have it
resolved this weekend.

I will keep you posted here.

Great, thanks for the update!

Hello Steve1986,

I am glad to report that the issue of duplicate tick ids has been corrected.
Thank you for your patience.

Great, thanks so much Todd!

Hi Todd,

Unfortunately, I still see the same duplicate tickids for @TFS#. I just downloaded a fresh set of data going back to pre-8/19/2012, and I still see hundreds of thousands of instances of tickids being duplicated.

Can you double check to see if this problem was actually fixed?

thanks,

Steve

Hi Steve,

The issue of duplicate tick ids has been fixed going forward, but we do not have a fix for the past tick data. If you see problems in the current data, we would be interested in knowing about that. Thanks again.

Thanks Todd. Is there a plan to fix the past tick data, or is there no intention to fix it?

Hi Steve1986,

There is no fix for the past tick data in this case.
And, as Steve said above: We have verified that the only information that is incorrect is the TickID field. All of the rest of the data is correct.

Thanks.

The same problem @CCZ12:
20.08.2012,040000,2442,1,1,0,0,714949,0,0,C,
20.08.2012,040000,2442,1,2,0,0,714949,0,0,C,
20.08.2012,040000,2442,1,3,0,0,714949,0,0,C,
20.08.2012,040000,2442,1,4,0,0,714949,0,0,C,
20.08.2012,040000,2442,1,5,0,0,714949,0,0,C

Is it bunch of numerous duplicates? should I delete\ignore them except first one?

14.09.2012,134904,2637,3,10127,2636,2639,6570900,0,0,C,
14.09.2012,134904,2636,2,10129,2636,2639,6570900,0,0,C,
14.09.2012,134904,2636,1,10130,2636,2639,6570900,0,0,C, <------------
14.09.2012,134904,2636,1,10131,2636,2639,6570900,0,0,C, <------------
14.09.2012,134904,2636,1,10132,2636,2639,6570900,0,0,C, <------------

Edited by mvvcorp on Oct 19, 2012 at 12:51 AM

Hello mvvcorp,

You do not need to delete or ignore the above trades.
The trades are legitimate - only the tick ids were in error. Thanks.

Thx

Next question:
I found multiple unordered ID's. Should I sort lines by ID or not ?

EPIX12 (raw data):
19.10.2012,161219,983.50,1,57462,982.75,983.50,12622342,0,0,C,
19.10.2012,161219,983.50,1,57463,982.75,983.50,12622585,0,0,C,
19.10.2012,161219,983.50,1,57464,982.75,983.50,12622587,0,0,C,
19.10.2012,161219,983.50,1,57465,983.50,983.75,12622584,0,0,C,
19.10.2012,161219,983.50,1,57466,983.50,983.75,12622600,0,0,C,

OR


EBK13 (raw data):
19.10.2012,115656,108.50,1,6332,108.50,108.53,17846918,0,0,C,/
19.10.2012,115656,108.50,1,6333,108.50,108.53,17890254,0,0,C,\ --- --------------- should be here
19.10.2012,115656,108.50,3,6336,108.50,108.53,17890261,0,0,C, |
19.10.2012,115656,108.50,4,6340,108.50,108.53,17890267,0,0,C, |
19.10.2012,115656,108.50,1,6341,108.50,108.53,17890273,0,0,C, |
19.10.2012,115656,108.50,2,6343,108.50,108.53,17890279,0,0,C, |
19.10.2012,115656,108.50,3,6346,108.50,108.53,17890285,0,0,C, |
19.10.2012,115656,108.50,1,6347,108.50,108.53,17847488,0,0,C, -> unordered ID
19.10.2012,115701,108.41,1,6348,108.37,108.41,17899340,0,0,C,
19.10.2012,115710,108.40,1,6349,108.38,108.40,17903487,0,0,C,

But if I sort lines by ID, I'll get unordered cumulative volume...

So more important question:
What is of greater priority: native line number OR tick ID OR cumulative volume? What should I do if they have conflict with each other?

thx

Edited by mvvcorp on Oct 22, 2012 at 12:20 PM

Hi mvvcorp,

We are digging into this and expect to have an answer shortly. Thanks.

@DXZ12:
26.09.2012,082229,80.000,1,4740,79.995,80.000,7592703,0,0,C,
26.09.2012,082229,80.000,1,4741,79.995,80.005,7656031,0,0,C,
26.09.2012,082230,80.000,2,4743,80.000,80.005,7668917,0,0,C,
26.09.2012,082243,80.005,3,4746,80.000,80.005,7585461,0,0,C,
26.09.2012,082243,80.005,1,4747,80.000,80.005,7666640,0,0,C,
26.09.2012,082243,80.005,2,4749,80.000,80.005,7668825,0,0,C,
26.09.2012,082243,80.005,2,4751,80.000,80.005,7669010,0,0,C,
26.09.2012,082243,80.005,1,4752,80.000,80.010,7669012,0,0,C,
26.09.2012,082243,80.010,1,4753,80.000,80.010,7233582,0,0,C,
26.09.2012,082243,80.010,1,4754,80.005,80.010,7665989,0,0,C,
26.09.2012,082243,80.010,1,4755,80.005,80.010,7670630,0,0,C,
26.09.2012,082243,80.015,1,4756,80.005,80.015,7500844,0,0,C,
26.09.2012,082243,80.015,1,4757,80.005,80.015,7579144,0,0,C,
26.09.2012,082243,80.015,2,4759,80.005,80.015,7584981,0,0,C,
26.09.2012,082243,80.015,2,4761,80.005,80.015,7585802,0,0,C,
26.09.2012,082243,80.015,3,4764,80.005,80.015,7606130,0,0,C,
26.09.2012,082243,80.015,1,4765,80.005,80.015,7631035,0,0,C,
26.09.2012,082243,80.015,1,4766,80.005,80.015,7666282,0,0,C,
26.09.2012,082243,80.015,1,4767,80.005,80.015,7666307,0,0,C,
26.09.2012,082243,80.015,2,4769,80.005,80.015,7670678,0,0,C,
26.09.2012,082243,80.015,2,4771,80.005,80.015,7670679,0,0,C,
26.09.2012,082243,80.015,2,4773,80.005,80.015,7670692,0,0,C,
26.09.2012,082243,80.015,3,4776,80.005,80.020,7670693,0,0,C,
26.09.2012,082243,80.020,1,4777,80.005,80.020,7579151,0,0,C,
26.09.2012,082243,80.020,2,4779,80.005,80.020,7585779,0,0,C,
26.09.2012,082243,80.020,2,4781,80.005,80.020,7606150,0,0,C,
26.09.2012,082243,80.020,3,4784,80.005,80.020,7624726,0,0,C,
26.09.2012,082243,80.020,1,4785,80.005,80.020,7666409,0,0,C,
26.09.2012,082243,80.020,1,4786,80.005,80.020,7668682,0,0,C,
26.09.2012,082243,80.020,19,4805,80.005,80.025,7670690,0,0,C,
26.09.2012,082243,80.010,1,4806,80.005,80.020,7670720,0,0,C,

Well, and finally what should I do in this case? :

@CTZ12_0:
14.08.2012,211325,72.91,1,233,72.84,72.91,215704,0,0,C,
14.08.2012,211347,72.92,1,234,72.84,72.92,215718,0,0,C,
14.08.2012,211408,72.86,1,235,72.86,72.94,215737,0,0,C,
14.08.2012,211630,72.84,1,236,72.84,72.89,215821,0,0,C,
14.08.2012,210000,73.15,1,236,72.75,72.84,215922,0,0,C, <--------------!!!
14.08.2012,212115,72.78,1,237,72.78,72.84,216104,0,0,C,
14.08.2012,212156,72.77,1,238,72.77,72.82,216140,0,0,C,
14.08.2012,212241,72.82,1,239,72.78,72.82,216159,0,0,C,
14.08.2012,212247,72.84,6,245,72.82,72.84,216168,0,0,C,
14.08.2012,212254,72.84,8,253,72.82,72.84,216172,0,0,C,

Yes, we have ordered IDs but obviously there is wrong ID for the line starting with "14.08.2012,210000..."!
Apparently this line should be moved to another place (much more earlier).

one more case:
@ESZ12:
08.10.2012,104715,1451.00,1,461593,1450.75,1451.00,1817180,0,0,C,
08.10.2012,104715,1451.00,3,461596,1450.75,1451.00,1817181,0,0,C,
08.10.2012,104715,1451.00,3,461599,1450.75,1451.00,1817182,0,0,C,
08.10.2012,104716,1451.00,3,461602,1450.75,1451.00,1817183,0,0,C,<----------- ???
08.10.2012,104715,1451.00,2,461604,1450.75,1451.00,1817184,0,0,C,
08.10.2012,104715,1451.00,1,461605,1450.75,1451.00,1817185,0,0,C,
08.10.2012,104715,1451.00,2,461607,1450.75,1451.00,1817186,0,0,C,
08.10.2012,104715,1451.00,1,461605,1450.75,1451.00,1817185,0,0,C,


M.NI3=LX:
03.04.2012,080346,18379.00,1,1416,18362.00,18382.00,42747,0,0,C,
03.04.2012,080346,18379.00,2,1418,18362.00,18382.00,42750,0,0,C,
03.04.2012,080346,18379.00,2,1420,18362.00,18382.00,42755,0,0,C,
03.04.2012,090414,18375.00,0,1420,18364.00,18379.00,42807,0,0,C,<----------- ???
03.04.2012,080420,18379.00,1,1421,18363.00,18381.00,42815,0,0,C,
03.04.2012,080420,18380.00,1,1422,18363.00,18381.00,42816,0,0,C,
03.04.2012,080439,18381.00,1,1424,18362.00,18390.00,42883,0,0,C,
03.04.2012,080439,18382.00,3,1427,18362.00,18390.00,42884,0,0,C,
03.04.2012,080456,18390.00,2,1429,18370.00,18390.00,42896,0,0,C,
03.04.2012,090457,18380.00,0,1429,18371.00,18380.00,42901,0,0,C,<----------- ???
03.04.2012,080531,18387.00,1,1430,18377.00,18390.00,42970,0,0,C,
03.04.2012,080551,18367.00,1,1432,18367.00,18390.00,42987,0,0,C,
03.04.2012,080951,18387.00,1,1433,18381.00,18394.00,43036,0,0,C,
03.04.2012,081000,18392.00,1,1434,18371.00,18394.00,43043,0,0,C,

MTV12:
17.09.2012,103007,3552.00,4,54306,3552.00,3552.50,491456,0,0,C,
17.09.2012,103010,3552.50,1,54307,3552.00,3553.00,491902,0,0,C,
17.09.2012,103010,3552.50,1,54304,3552.00,3553.00,491903,0,0,C,
17.09.2012,103033,3552.50,1,52531,3552.00,3553.00,492427,0,0,C,
17.09.2012,103033,3552.00,4,52535,3552.00,3553.00,492428,0,0,C,
17.09.2012,103106,3552.00,1,52536,3551.50,3552.50,493270,0,0,C, <-----------
17.09.2012,103053,3546.50,1773,54309,3552.00,3552.50,493291,0,0,C,<-----------
17.09.2012,103145,3553.00,5,50768,3553.00,3553.50,494112,0,0,C,
17.09.2012,103145,3553.00,5,50773,3552.50,3553.00,494114,0,0,C,
17.09.2012,103149,3552.00,1,50774,3551.00,3552.50,494449,0,0,C,
17.09.2012,103149,3552.00,1,50770,3552.00,3552.50,494451,0,0,C,
17.09.2012,103158,3552.00,1,50771,3551.50,3552.50,495049,0,0,C,
17.09.2012,103158,3552.00,1,50771,3552.00,3552.50,495051,0,0,C,
17.09.2012,103202,3552.50,1,50772,3552.00,3553.00,495169,0,0,C,


QCLJ12:
13.02.2012,140110,100.73,1,74293,100.73,100.75,1333152,0,0,C,
13.02.2012,140110,100.73,1,74294,100.71,100.73,1333261,0,0,C,
13.02.2012,140110,100.73,1,74295,100.71,100.73,1333275,0,0,C,
13.02.2012,140111,100.73,1,74296,100.77,100.79,1333374,0,0,C,
13.02.2012,140110,100.73,1,74297,100.77,100.79,1333394,0,0,C,
13.02.2012,140006,100.77,1,74296,100.78,100.80,1333706,0,0,C,<----------???
13.02.2012,140014,100.77,1,74233,100.76,100.77,1333856,0,0,C,
13.02.2012,140025,100.76,1,74234,100.74,100.76,1334045,0,0,C,
13.02.2012,140110,100.73,1,74235,100.74,100.77,1334085,0,0,C,
13.02.2012,140111,100.73,1,74296,100.76,100.78,1334177,0,0,C,
Edited by mvvcorp on Oct 23, 2012 at 08:34 AM

mvvcorp,

Trades should not be arranged by Tick id as the tick ids are supposed to be unique to each trade for each day, but are not necessarily sequential or incremental.

thx!