Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"DTN has never given me problems. It is incredibly stable. In fact I've occasionally lost the data feed from Interactive Brokers, but still been able to trade because I'm getting good data from DTN." - Comment from Leighton
"Everything is working great with the API. I love it." - Comment from Calvin
"IQ feed works very well, does not have all of the normal interruptions I have grown used to on *******" - Comment from Mark
"I use IQ Feed, Great stuff as far as data analysis information, storage and retrieval is concerned." - Comment from Public Forum
"I "bracket trade" all major news releases and I have not found one lag or glitch with DTN.IQ feed. I am very comfortable with their feed under all typical news conditions (Fed releases, employment numbers, etc)." - Comment from Public Forum
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
"I just wanted to let u know that your data feed/service is by far the best!!! Your unfiltered tick data is excellent for reading order flow and none of your competitors delivers this quality of data!" - Comment from Peter via Email
"The service is great, I see a noticeable improvement in my volume profiles over [broker]'s data feed" - Comment from Larry
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
Home  Search  Register  Login  Blogs Recent Posts

Information on Various DTN Products:
DTN IQFeed | DTN ProphetX | DTN Ag | NxCore
Follow DTN_IQFeed on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Delays fom IQFeed 66.112.148.x data servers
Author Topic: Delays fom IQFeed 66.112.148.x data servers (28 messages, Page 1 of 1)

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Jun 21, 2011 08:59 AM          Msg. 1 of 28
Hi,

Since May 23 we have noticed that our data feeds from the IQFeed 66.112.148.x data servers are experiencing data delays of up to 10 seconds throughout the trading day. We use the IQFeed (server side) generated timestamps to determine delay by comparing them with the actual time. This has proven to be a reliable method and the difference is usually less than 500ms throughout the day except during activity spikes.

Unfortunately we only generate alerts when delays exceed 10 seconds so we were a little slow in noticing the problem. Since the delays weren't experienced every day we trawled through our logs and identified the 66.112.148.x farm as the culprit (the 66.112.156.x farm is behaving normally). Additionally we have separate IQ accounts used for NASDAQ and NYSE feeds and the NYSE feed is significantly worse...probably reflecting the larger rate of updates.

Was some kind of change implemented on the weekend of 21/22 May?

I have attached some charts (delay is the pink, measured in ms)



File Attached: 20110617_delay.jpg (downloaded 594 times)

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 21, 2011 09:15 AM          Msg. 2 of 28
Dennis, we are going to look into this. Just for clarification when you say "We use the IQFeed (server side) generated timestamps to determine delay by comparing them with the actual time."

Which timestamp are you using? Are you using the actual Timestamp messages that are sent in the feed (T,CCYYMMDD HH:MM:SS) or are you using the trade timestamps in the update messages (field 18)?

Also, are you comparing that timestamp to your local PC time or some other "actual time"? I'm going to assume that if you are comparing to local PC time that you have something that frequently sync's your PC time with a time server.

One thing we did notice is that you are using an extremely old version of IQFeed. I'm not willing to blame what you are seeing on that old version just yet but you should seriously consider upgrading to a newer version. Not only are the newer versions more efficient at processing the feed but they also support compression which will greatly reduce your bandwidth required for the feed.

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 21, 2011 03:15 PM          Msg. 3 of 28
I don't know if this helps, but I also see bursts of delay from 66.112.148.x.
This is by comparing the T message time to the local clock, which is time server sync'd every hour. I just assumed it was my internet connection.

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Jun 21, 2011 04:59 PM          Msg. 4 of 28
Steve,

We do the same thing as Craig. Compare local clock with T message and local clock is regularly synced to a time server.

Dennis

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 21, 2011 05:08 PM          Msg. 5 of 28
Thanks for the reports guys. We are continuing to look into this.

There weren't any hardware or software changes that we know of which could have affected this occurring at that timeframe.

Dennis, is there any way you can upgrade to a newer version of IQFeed and test again? Looking at that server farm, users on older versions do appear to be getting reduced performance compared to users on newer versions. It looks as though the older version is not able to keep up with the feed to the extent that you need it to any longer. Of course, that doesn't explain why you don't see the issue when connected to the other server farm but we are fairly confident that it might resolve (or at least greatly aleviate) the issue for you until we can determine if there are other issues and resolve anything that arises.

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Jun 21, 2011 06:57 PM          Msg. 6 of 28
Steve,

While I appreciate that upgrading the client will eliminate it as the source of the problem, doing so would require some development work at our end which we cannot do in the short term.
However, I should point out that the problem did not exist prior to May 23 and it came on suddenly. Also, as you pointed out, we don't have a problem with the .156 farm which suggests a server side issue. The fact that Craig is also seeing delays supports this.

As a workaround, we will reset our alert threshold to 5 sec which will cause our software to reconnect and hopefully we will reconnect to the .156 farm (from memory we can force a connection to the .156 farm but we won't do that for now).

Dennis

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 22, 2011 09:00 AM          Msg. 7 of 28
Dennis, as I said, we will continue looking into potential problems on our end. However, I would recommend that you make upgrading to a newer version a priority. You are almost 6 versions out of date at this point (4.8 is in beta as you should know) and we don't do any testing with that old of a version internally so there is no guarantee it will continue to work. At any point, we may release server side code that unknowingly will break that old version. At that point, you will be without your feed until you make the changes to upgrade.

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 22, 2011 11:46 AM          Msg. 8 of 28
Dennis/Craig, we are not finding anything internal that could explain this. Nor are we finding anything consistent across all users on the system that would show a pattern of what you are seeing .

Can you send us a ping and traceroute to both of our server farms? of course, feel free to email develiper support with this info if you don't want to post it publicly on the forums.

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Jun 22, 2011 12:57 PM          Msg. 9 of 28
Steve,

I've looked at the pings & trace routes and I don't believe they reveal anything but here they are. I also ran the pings for around 15 minutes and had no anomalies.

/home/trader-> ping 66.112.148.114
PING 66.112.148.114 (66.112.148.114) 56(84) bytes of data.
64 bytes from 66.112.148.114: icmp_seq=1 ttl=54 time=45.4 ms
64 bytes from 66.112.148.114: icmp_seq=2 ttl=54 time=45.7 ms
64 bytes from 66.112.148.114: icmp_seq=3 ttl=54 time=46.0 ms
64 bytes from 66.112.148.114: icmp_seq=4 ttl=54 time=45.5 ms
64 bytes from 66.112.148.114: icmp_seq=5 ttl=54 time=45.6 ms
64 bytes from 66.112.148.114: icmp_seq=6 ttl=54 time=45.3 ms

--- 66.112.148.114 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5004ms
rtt min/avg/max/mdev = 45.302/45.626/46.084/0.365 ms


/home/trader-> traceroute 66.112.148.114
traceroute to 66.112.148.114 (66.112.148.114), 30 hops max, 40 byte packets
1 69.10.58.1 (69.10.58.1) 0.845 ms 1.374 ms 1.343 ms
2 66.45.224.33 (66.45.224.33) 0.718 ms 0.710 ms 0.688 ms
3 64.20.47.17 (64.20.47.17) 0.663 ms 0.873 ms 0.851 ms
4 nyrkbprj02-ae1.303.rd.ny.cox.net (68.105.31.93) 0.815 ms 0.792 ms 1.009 ms
5 mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 43.692 ms mtc1dsrj02-ae2.0.rd.om.cox.net (68.1.0.127) 43.654 ms 43.653 ms
6 66.37.238.194 (66.37.238.194) 44.304 ms 66.37.238.202 (66.37.238.202) 44.188 ms 66.37.238.194 (66.37.238.194) 44.289 ms
7 69.63.125.106 (69.63.125.106) 44.895 ms 44.969 ms 44.805 ms
8 (98.142.95.133) 45.064 ms 45.021 ms 45.001 ms
9 (98.142.95.129) 80.018 ms 79.985 ms 79.933 ms
10 216.58.227.114 (216.58.227.114) 45.117 ms 45.398 ms 45.061 ms
11 (66.112.151.18) 46.523 ms 46.510 ms 46.484 ms
12 66.112.148.114 (66.112.148.114) 45.275 ms !X 45.434 ms !X 45.305 ms !X


/home/trader-> ping 66.112.156.220
PING 66.112.156.220 (66.112.156.220) 56(84) bytes of data.
64 bytes from 66.112.156.220: icmp_seq=1 ttl=52 time=43.9 ms
64 bytes from 66.112.156.220: icmp_seq=2 ttl=52 time=45.3 ms
64 bytes from 66.112.156.220: icmp_seq=3 ttl=52 time=42.4 ms
64 bytes from 66.112.156.220: icmp_seq=4 ttl=52 time=42.5 ms
64 bytes from 66.112.156.220: icmp_seq=5 ttl=52 time=42.4 ms
64 bytes from 66.112.156.220: icmp_seq=6 ttl=52 time=43.6 ms
64 bytes from 66.112.156.220: icmp_seq=7 ttl=52 time=42.4 ms

--- 66.112.156.220 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6006ms
rtt min/avg/max/mdev = 42.452/43.264/45.336/1.053 ms


/home/trader-> traceroute 66.112.156.220
traceroute to 66.112.156.220 (66.112.156.220), 30 hops max, 40 byte packets
1 69.10.58.1 (69.10.58.1) 0.975 ms 0.939 ms 1.115 ms
2 66.45.224.33 (66.45.224.33) 0.553 ms 0.858 ms 0.846 ms
3 207.239.51.85 (207.239.51.85) 0.806 ms 0.794 ms 0.772 ms
4 vb1010.rar3.nyc-ny.us.xo.net (216.156.0.17) 3.621 ms 2.260 ms 2.882 ms
5 * * *
6 206.111.13.198.ptr.us.xo.net (206.111.13.198) 0.600 ms 0.794 ms 0.871 ms
7 omh-core-02.inet.qwest.net (205.171.5.241) 41.770 ms 41.800 ms 41.776 ms
8 omh-edge-09.inet.qwest.net (205.171.132.66) 41.398 ms 51.287 ms 51.279 ms
9 65.116.176.82 (65.116.176.82) 42.890 ms 43.224 ms 43.202 ms
10 66.112.152.50 (66.112.152.50) 42.816 ms 43.141 ms 43.124 ms
11 solar0.interquote.com (66.112.156.220) 42.411 ms !X 43.386 ms !X 42.559 ms !X

Neil Brennan
-Interested User-
Posts: 38
Joined: Apr 13, 2007


Posted: Jun 22, 2011 06:09 PM          Msg. 10 of 28
Hi Steve,

As you may recall, I work with Dennis. One of the things that has slowed down any attempt by us to upgrade our client is that (as far as we can see) there is no step-wise way to do it. Once you have a new version, old versions seem to be removed from your site.

Is there anywhere we can get some of the older versions of the client that you think are stable and clean, so that we can upgrade in an orderly fashion?

The best version for us would be one where we aren't forced to change our code, of course. :-)

Thanks,

Neil

Neil Brennan
-Interested User-
Posts: 38
Joined: Apr 13, 2007


Posted: Jun 22, 2011 06:09 PM          Msg. 11 of 28
Hi Steve,

As you may recall, I work with Dennis. One of the things that has slowed down any attempt by us to upgrade our client is that (as far as we can see) there is no step-wise way to do it. Once you have a new version, old versions seem to be removed from your site.

Is there anywhere we can get some of the older versions of the client that you think are stable and clean, so that we can upgrade in an orderly fashion?

The best version for us would be one where we aren't forced to change our code, of course. :-)

Thanks,

Neil

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 27, 2011 01:14 PM          Msg. 12 of 28
I'm having another day of large delays (up to half a minute).
CPU usage on box is 30-40%.

:~$ ping 66.112.148.113
PING 66.112.148.113 (66.112.148.113) 56(84) bytes of data.
64 bytes from 66.112.148.113: icmp_req=1 ttl=48 time=207 ms
64 bytes from 66.112.148.113: icmp_req=2 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=3 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=4 ttl=48 time=207 ms
64 bytes from 66.112.148.113: icmp_req=5 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=6 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=7 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=8 ttl=48 time=207 ms

~$ traceroute 66.112.148.114
traceroute to 66.112.148.114 (66.112.148.114), 30 hops max, 60 byte packets
1 () 0.593 ms 0.832 ms 1.135 ms
2 .jetstream.xtra.co.nz () 60.229 ms 71.474 ms 71.641 ms
3 * * *
4 ae4-10.akbr5.global-gateway.net.nz (202.37.244.221) 25.405 ms 25.534 ms 25.661 ms
5 ae1-2.akbr4.global-gateway.net.nz (202.50.232.77) 25.918 ms 26.102 ms 26.269 ms
6 ae1-10.tkbr9.global-gateway.net.nz (202.50.232.37) 27.267 ms 17.957 ms 18.945 ms
7 so7-0-1.labr5.global-gateway.net.nz (122.56.127.38) 143.181 ms so6-1-3.labr5.global-gateway.net.nz (202.50.232.42) 143.910 ms so6-0-3.labr5.global-gateway.net.nz (122.56.127.34) 145.077 ms
8 ae0-3.lebr6.global-gateway.net.nz (203.96.120.86) 145.254 ms 146.206 ms 147.174 ms
9 LANGBBPC01GEX0200A001.R2.LA.COX.NET (206.223.123.42) 179.592 ms 179.712 ms 166.048 ms
10 mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 184.227 ms mtc1dsrj02-ae2.0.rd.om.cox.net (68.1.0.127) 184.668 ms mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 185.858 ms
11 66.37.238.202 (66.37.238.202) 192.268 ms 66.37.238.194 (66.37.238.194) 191.780 ms 192.653 ms
12 69.63.125.106 (69.63.125.106) 210.996 ms 205.209 ms 205.147 ms
13 98.142.95.133 (98.142.95.133) 180.526 ms 181.126 ms 181.603 ms
14 98.142.95.129 (98.142.95.129) 208.744 ms 205.341 ms 206.487 ms
15 216.58.227.114 (216.58.227.114) 182.080 ms 181.492 ms 182.788 ms
16 66.112.151.18 (66.112.151.18) 210.879 ms 211.679 ms 211.753 ms
17 66.112.148.114 (66.112.148.114) 212.004 ms !X 207.400 ms !X 207.563 ms !X

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 27, 2011 01:48 PM          Msg. 13 of 28
Craig, is your situation like Dennis where you only sees this on the 148 server farm?

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 27, 2011 01:58 PM          Msg. 14 of 28
I hadn't noted the remote address in previous cases, but I shall do so in future.

Neil Brennan
-Interested User-
Posts: 38
Joined: Apr 13, 2007


Posted: Jun 27, 2011 09:31 PM          Msg. 15 of 28
Just quickly,

Since we have limited our connections to the .156 farm we have had service as normal, which for us is over 99% of data delivered with a lag of less than 100ms. Today's latency graph is attached.

Steve, we will work with you offline to see what we can do about upgrading our client.

Thanks,

Neil



File Attached: 20110627.png (downloaded 625 times)

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 27, 2011 10:14 PM          Msg. 16 of 28
How does one limit the connection address?

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Jun 28, 2011 07:44 AM          Msg. 17 of 28
Craig,

From your trace route it looks like you're in NZ. We originally (until 4-5 years ago) ran our servers from Australia and had a lot of trouble with the feed during busy periods due to the high network latency (250ms ping times). At the time this resulted in data corruption but may manifest itself differently with the later versions of IQFeed. This was resolved when we relocated to the East coast USA.

Also, as IQFeed is single threaded, look at the process CPU usage to see if it's maxing out a single core (25% on a quad core, 50% on a dual core, etc) rather than overall CPU usage.

Dennis

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 28, 2011 01:36 PM          Msg. 18 of 28
Running against a 156 address today, seeing no perceptible delay, also running a lower volume watch list.

Dennis: Thanks for the tips, living in NZ I do expect some latency of course, just not 30 seconds of it. IQFeed was at about 20% on one core and there is plenty of B/W.
Edited by Craig on Jun 28, 2011 at 01:37 PM

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Jun 28, 2011 01:55 PM          Msg. 19 of 28
Craig, At this point, I believe the difference in volume of your watchlist today is a larger factor than switching server farms. Comparatively today you are getting roughtly 8 times less data than you would have been getting with your symbol list from yesterday.

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Jun 28, 2011 02:33 PM          Msg. 20 of 28
I suspect you are right, though it's hard to know if that's the whole story without switching watch-lists again. However, I've seen IQFeed balloon in memory, use 100% CPU etc when the client application can't keep up, and that wasn't happening yesterday.

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Aug 10, 2011 03:19 PM          Msg. 21 of 28
I had terrible lag today...yesterday I processed nearly twice as much data with only sporadic lag...

IQFeed reports that it's using around 20KB/s bandwidth, the link bandwidth is 6MB/s.
The server applications are all C++ applications using async. processing, the CPU is perhaps maxing at 30%. IQFeed is not growing in memory so I know I'm processing the data quick enough. Any other suggestions on how I can troubleshoot this?

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Aug 11, 2011 04:55 AM          Msg. 22 of 28
Craig,

If it makes you feel any better we have been experiencing larger feed delays recently (again using timestamps as a reference). Normally these occur during bursts of market activity such as open/close and after major news announcements. Of course, with recent events these have occurred more often. Tick counts have more than doubled over the last few days for the same watchlists on both the NYSE and NASDAQ.

Dennis

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Aug 11, 2011 05:25 PM          Msg. 23 of 28
Thanks Dennis.

...of course today I had a much larger tick count but with no delay.
Go figure.

Gremlin123
-Interested User-
Posts: 6
Joined: Aug 18, 2010


Posted: Aug 16, 2011 07:09 PM          Msg. 24 of 28
I am having latency problems too.

I have traded using IQFeed for more than a year. I am seeing the "time" messages become very eratic, delay 20-30s, and many go missing.

I scalled back the number of symbols, but the it had no improvment.

I also changed the code only request the fields need, but no help.

The CPU load on both processes is very low, only about 1%

I measured how much time the tick reader spends waiting on the socket and it spends very little time processing messages and almost all of the time waiting for messages.

It is connected to 66.112.148.225.

Not sure what else I can do.

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Aug 16, 2011 07:14 PM          Msg. 25 of 28
Today was another latency disaster for me as well.

ers811
-Interested User-
Posts: 13
Joined: Mar 4, 2010


Posted: Aug 22, 2011 08:30 AM          Msg. 26 of 28
Pretty good delay here last week too. 10-20 seconds in some cases.

I don't know if this will help, but I went to the IQ Connection Manager and watched the "Market time" field. It would actually lag it's update to the next minute by the same delay as I was seeing with quotes.

Not sure what server I was on last week. Today one machine is on 66.112.156.228

DTN_Steve_S
-DTN Guru-
Posts: 2029
Joined: Nov 21, 2005


Posted: Aug 22, 2011 09:27 AM          Msg. 27 of 28
ers811, it sounds like your issue is a different issue since it wasn't on the same server farm that this thread applies to. I'm happy to work with you outside this thread (in order to keep issues separated) to work towards a resolution.

Craig/Neil/Denis/Gremlin, there were some changes made by the network provider at our 148 server farm Thursday night that we believe should have aleviated the issues that you guys were seeing with that server farm. Please let us know if you continue to see issues.

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Aug 22, 2011 01:27 PM          Msg. 28 of 28
Better today, I'm still seeing the odd burst of lag, but nothing like last Friday.
 

 

Time: Mon December 11, 2017 1:21 AM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003