Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"If you are serious about your trading I would not rely on IB data for serious daytrading. Took me a while to justify the cost of IQ Feed and in the end, it's just a 2 point stop on ES. Better safe than sorry" - Comment from Public Forum
"I am a hedge fund manager here. It’s funny, I have a Bloomberg terminal and a Bridge feed, but I still like having my DTN feed!" - Comment from Feras
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
"This beats the pants off CQG, I am definitely switching to the ProphetX 3.0!" - Comment from Stephen
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
"IQ feed is brilliant. The support is mind-bending. What service!" - Comment from Public Forum Post
"Thanks for all of your help. Great customer service deserves to be recognized which one the reasons I've been a customer of DTN for over 10 years!" - Comment from Stuart
"Just a thank you for the very helpful and prompt assistance and services. You provided me with noticeably superior service in my setup compared to a couple of other options I had looked at." - Comment from John
Home  Search  Register  Login  Blogs Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Corrupted L1 data-in recent past on multiple products at 'random' times
Author Topic: Corrupted L1 data-in recent past on multiple products at 'random' times (23 messages, Page 1 of 1)

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Oct 11, 2018 08:53 PM          Msg. 1 of 23
I have been seeing invalid and stale market data in various insturments (CL, ES) in the last month or so (could be slightly longer). It occurs at seemingly random times and usually lasts about 10 minutes sometimes up to an hour.
Example from today: 10:08-10:20am, 3:05-3:15pm CST in ES. Here is a snapshot of L1 updates from one of the periods:

Q,@ESZ18,2738.25,3859981,2,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859982,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859983,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859984,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@EDU20,96.7650,270309,2,96.7650,96.7700,8665,15942,,16:07:28.306149,9268532,16:08:31.734942,16:08:31.768035,a,
Q,@EDM19,96.9800,405656,2,96.9800,96.9850,7411,30599,,16:00:00.036589,9268244,16:08:31.753186,16:08:31.768255,a,
Q,@EDU20,96.7650,270309,2,96.7650,96.7700,8665,15824,,16:07:28.306149,9268532,16:08:31.734942,16:08:31.768255,a,
Q,@EDH20,96.7650,342864,2,96.7650,96.7700,4985,27436,,16:07:28.306149,9268541,16:08:31.734935,16:08:31.768255,a,
Q,@EDM20,96.7600,348765,3,96.7600,96.7650,6401,20909,,16:07:28.306149,9268536,16:08:31.733403,16:08:31.768255,a,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,13,1,,16:08:31.779471,271740673,16:08:31.779481,16:05:00.164499,b,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,14,1,,16:08:31.779471,271740673,16:08:31.779658,16:05:00.164499,b,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,15,1,,16:08:31.779471,271740673,16:08:31.779676,16:05:00.164499,b,

In the above (and with my message structure) the best bid/ask data are crossed and indicating 2738.25,2733.75 for bid and ask for ES, clearly nonsensical. This same stale/invalid bid/ask info stays such for the period in question. Notice that ED above (eurodollar) is just fine as are most other contracts. This has been happening for over a month and I alerted iqfeed support to it. Initially they expressed it was due to testing MBO feed. When I contended those reasons they claimed they had some issues with CME feed. Whatever it is, this is not getting fixed and I am concerned this is not getting the priority it deserves. Any advanced developer depends on full/complete/correct data-as the feed proclaims to provide. Please correct this issue. I can provide more details if needed.

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Oct 12, 2018 06:52 AM          Msg. 2 of 23
Good morning,

I apologize that this continues to happen and for my poor communication of our concern over it. We are very concerned with the quality of our data. To that end, here we have added the CME trade aggressor flag to our level one data. This flag tells if the trade occurred at the bid or the ask, so not only will you save on the processing, but you will have a definite answer on every trade of where it occurred. We hope to have this available to our developers later this month as a beta, but 4th quarter for sure. We have also developed better ways to identify when these situations occur, so that we can ensure quality and strive towards a continuous improvement to our data.

I know that the people that are needed to insure the continued priority of this work actively watch the forums and I know that your concerns will only help to keep the focus where it needs to be. I will try to keep you more in the loop in the future when related updates occur.

Tim

developer_hm
-Interested User-
Posts: 2
Joined: Oct 26, 2018


Posted: Oct 26, 2018 03:05 PM          Msg. 3 of 23
I am also still seeing this issue. Sometimes restarting the instance of iqfeed resolves the issue, but often times it does not. I've been noticing that for many instruments that trade on the CME, a bid or ask will get stuck at a particular price for several hours.

Constantly having these bad prices is making the data effectively unusable. Do you have an ETA on when the CME data will become usable again?

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Oct 26, 2018 08:28 PM          Msg. 4 of 23
Today was particularly bad, CL and currency data (CL, 6E, 6A,6J) all had this issue for hours after 9:31am central time.

developer_hm
-Interested User-
Posts: 2
Joined: Oct 26, 2018


Posted: Oct 29, 2018 08:01 AM          Msg. 5 of 23
I agree that Friday was particularly bad. The entire strip November expiring ESZ8 options was giving me bad prices, which was throwing off calculations we use to drive our model.

DTN_Steve_S
-DTN Guru-
Posts: 2080
Joined: Nov 21, 2005


Posted: Oct 29, 2018 09:32 AM          Msg. 6 of 23
Hello, we are continuing to monitor and make changes to address this issue. Friday was particularly bad since we had several different issues contributing to make the problem worse. We made some configuration changes on our servers over the weekend and continue to actively work with our providers to get the issue resolved.

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 7, 2019 11:38 PM          Msg. 7 of 23
Has this been resolved?

MAC

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Feb 11, 2019 07:12 PM          Msg. 8 of 23
It hasn't, for example today (2/11/2019) it occurred in CL, ES, 6E between 10:18-10:23am CST.

quickTick
-Interested User-
Posts: 43
Joined: Nov 17, 2013


Posted: Feb 12, 2019 12:11 AM          Msg. 9 of 23
I believe I saw it at about the same time today, for ES.

quickTick
-Interested User-
Posts: 43
Joined: Nov 17, 2013


Posted: Feb 12, 2019 12:12 AM          Msg. 10 of 23
Today as in 2019-02-11 CST, that is.

quickTick
-Interested User-
Posts: 43
Joined: Nov 17, 2013


Posted: Feb 12, 2019 12:18 AM          Msg. 11 of 23
As of right, now, I also see the gap in the historical tick data. Or maybe that's where I saw it in the first place.

The quotes got stuck at 2707.50, trades do continue (in the historical tick data).

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 19, 2019 08:01 AM          Msg. 12 of 23
aQuant or quickTick,

Could you explain how you detect this? I want to start monitoring.

Is there a complete lack of Q messages arriving with 'b' or 'a' updates?

or

Are the Q messages coming in for 'b' or 'a' but the prices don't change?

I have disabled IQFeed from my production deployment and will run it in the background for monitoring only.

I was about to add NYMEX, COMEX, CBOT and EUREX to my subscriptions. I will wait until this has been resolved.

MAC

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Feb 19, 2019 05:04 PM          Msg. 13 of 23
Based on the raw level1 messages it appears that one side stops updating, in the below case it was bid that got 'stuck' at 51.51, there may be other ways this happens, I didn't study them all:

Q,QCLH19,51.36,465114,2,51.51,51.37,1,46,183,11:19:10.391894,11321247,11:17:37.737313,11:19:10.391882,C,
Q,QCLH19,51.36,465114,2,51.51,51.37,1,47,,11:19:10.391894,11321247,11:17:37.737313,11:19:10.392009,a,
Q,QCLH19,51.36,465115,1,51.51,51.37,1,47,183,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392009,C,
Q,QCLH19,51.36,465115,1,51.51,51.37,1,48,,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392418,a,
Q,QCLH19,51.36,465116,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,49,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392538,a,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,50,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392633,a,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,51,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393159,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,6,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393887,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,7,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395510,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,8,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395553,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,10,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395590,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,13,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395627,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,14,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395727,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,15,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395840,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,18,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395916,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,20,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396142,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,22,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396215,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,23,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396250,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,24,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396328,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,25,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396725,a,

quickTick
-Interested User-
Posts: 43
Joined: Nov 17, 2013


Posted: Feb 20, 2019 04:38 PM          Msg. 14 of 23
@ aQuant

I'm not sure about which field is which, but it doesn't seem to show the beginning.
Do quotes get crossed because they are stuck, or do they get stuck because they are crossed?

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 20, 2019 08:14 PM          Msg. 15 of 23
aQuant,

I am now monitoring the following CME symbols (AD,BP,CD,JY,EC,NG,ES).

I didn't see anything out of the ordinary for trader date 2018-02-20. However, my monitoring is only gather stats and not saving the raw messages. There is a possibility I have a bug in my stats.

If it's not too much to ask, could you post here when you see one of these symbols get stuck or cross? I can then verify with my monitoring.

If I can verify, we can all push for a solution.

Thanks in advance.

MAC

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Feb 20, 2019 09:53 PM          Msg. 16 of 23
In the post above yours I didn't show where it began, I would have to look for that. I can try to locate it later.
I am definitely willing to supply a lot more information to anyone who will as a result help find a solution. But I would like to see some involvement of IQFeed support in this. I can help them nail this if they really want to solve it...

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 21, 2019 06:18 AM          Msg. 17 of 23
aQuant,

I spoke too soon. I see the bid frozen from 2018-02-21 02:00:00 to 02:17:00 ET on all the products I'm monitoring except JY. It looks like my simple stats are able to catch this.

I haven't coded up the HT* command (request tick history). Will this show the same behavior? If so, surely IQFeed won't be able to refute the problem.

I will call them this morning.

MAC

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 21, 2019 07:20 AM          Msg. 18 of 23
Attached is the HTT response for @NQH19.

Note that the bids do not change for 17 minutes.

MAC



File Attached: nq_trades.txt.zip (downloaded 19 times)

DTN_Steve_S
-DTN Guru-
Posts: 2080
Joined: Nov 21, 2005


Posted: Feb 21, 2019 10:44 AM          Msg. 19 of 23
We continue to actively monitor this thread and the information you are providing is being utilized by our network, development and operations teams. aQuant, regarding your comment “I would like to see some involvement of IQFeed support in this… if they really want to solve it” – the answer is absolutely yes. We are here to support our customers and continually improve our products. Feedback from each of you is critical to our continued success.

There is a lot of action going on behind the scenes at DTN to further optimize our service and increase resiliency. This involves not only our staff, but teams in the middle and all the way through to the exchanges. In fact, this morning we received word that the CME is performing maintenance on fiber connections that we believe could be responsible sporadic dropped packets that may result in the reported stuck bids. Please continue to share your findings by sending them directly to developer support, and we will continue to work with you to resolve any issues you are experiencing.

quickTick
-Interested User-
Posts: 43
Joined: Nov 17, 2013


Posted: Feb 21, 2019 03:07 PM          Msg. 20 of 23
Quote: Attached is the HTT response for @NQH19.

Note that the bids do not change for 17 minutes.

--- Original message by mac on Feb 21, 2019 07:20 AM
@ mac

In your text file, the bid keeps the value it first acquires at 02:01:12.834878, at the value of 7091.75.
That's about a screen down from the beginning.
Right in the next line, the bid gets crossed by both ask and trade. Perhaps that is not a coincidence that it happens so close.

Perhaps the bid gets stuck because bid and ask cross. For example, somewhere might be code like "if ask < bid", perhaps even as a safety guard, and that code causes it to get stuck at that value. If so, bid and ask probably shouldn't cross in the first place, but that might be just a delay in the delivery of one vs the other.

Just a suggestion, since I don't see that on my own screen displaying the historical tick data for NQ, as far as I can tell.

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 21, 2019 08:33 PM          Msg. 21 of 23
@quickTick,

It's interesting that you don't see this. The attached file is the raw text data from the socket. I've attached the full response to the HTT request run at 2018-02-21 21:20:00 ET. It's a bit longer but still shows the frozen bid.

I should note, I have yet to upgrade to IQConnect v6. I will do that now and see if it fixes the problem.

Below is the HTT command I issued.

HTT,@NQH19,20190221 015000,20190221 022000,0005000,000000,235959,1,REQ_HST_TCK0000000,0000500

Same result without including BeginFilterTime and EndFilterTime in the request

HTT,@NQH19,20190221 015000,20190221 022000,0005000,,,1,REQ_HST_TCK0000000,0000500

MAC



File Attached: nq_trades_full.txt.zip (downloaded 16 times)

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 21, 2019 08:54 PM          Msg. 22 of 23
I just upgraded to IQConnect v6 and the bid is no longer frozen. I am still using 5.2 protocol. (see attached file)

@aQuant, which version of IQConnect are you running? If v6, I will assume the frozen bid I saw 2am on Feb 21 was version related.

I will continue to monitor with IQConnect v6.

I hope this finding will shed some light on the issue.

MAC



File Attached: nq_trades_full_v6.txt.zip (downloaded 18 times)

aQuant
-Interested User-
Posts: 37
Joined: Jul 20, 2012


Posted: Feb 21, 2019 09:40 PM          Msg. 23 of 23
I am using 5.2.5.0.
 

 

Time: Mon March 18, 2019 8:50 AM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003