Hi,

Since May 23 we have noticed that our data feeds from the IQFeed 66.112.148.x data servers are experiencing data delays of up to 10 seconds throughout the trading day. We use the IQFeed (server side) generated timestamps to determine delay by comparing them with the actual time. This has proven to be a reliable method and the difference is usually less than 500ms throughout the day except during activity spikes.

Unfortunately we only generate alerts when delays exceed 10 seconds so we were a little slow in noticing the problem. Since the delays weren't experienced every day we trawled through our logs and identified the 66.112.148.x farm as the culprit (the 66.112.156.x farm is behaving normally). Additionally we have separate IQ accounts used for NASDAQ and NYSE feeds and the NYSE feed is significantly worse...probably reflecting the larger rate of updates.

Was some kind of change implemented on the weekend of 21/22 May?

I have attached some charts (delay is the pink, measured in ms)

Dennis, we are going to look into this. Just for clarification when you say "We use the IQFeed (server side) generated timestamps to determine delay by comparing them with the actual time."

Which timestamp are you using? Are you using the actual Timestamp messages that are sent in the feed (T,CCYYMMDD HH:MM:SS) or are you using the trade timestamps in the update messages (field 18)?

Also, are you comparing that timestamp to your local PC time or some other "actual time"? I'm going to assume that if you are comparing to local PC time that you have something that frequently sync's your PC time with a time server.

One thing we did notice is that you are using an extremely old version of IQFeed. I'm not willing to blame what you are seeing on that old version just yet but you should seriously consider upgrading to a newer version. Not only are the newer versions more efficient at processing the feed but they also support compression which will greatly reduce your bandwidth required for the feed.

I don't know if this helps, but I also see bursts of delay from 66.112.148.x.
This is by comparing the T message time to the local clock, which is time server sync'd every hour. I just assumed it was my internet connection.

Steve,

We do the same thing as Craig. Compare local clock with T message and local clock is regularly synced to a time server.

Dennis

Thanks for the reports guys. We are continuing to look into this.

There weren't any hardware or software changes that we know of which could have affected this occurring at that timeframe.

Dennis, is there any way you can upgrade to a newer version of IQFeed and test again? Looking at that server farm, users on older versions do appear to be getting reduced performance compared to users on newer versions. It looks as though the older version is not able to keep up with the feed to the extent that you need it to any longer. Of course, that doesn't explain why you don't see the issue when connected to the other server farm but we are fairly confident that it might resolve (or at least greatly aleviate) the issue for you until we can determine if there are other issues and resolve anything that arises.

Steve,

While I appreciate that upgrading the client will eliminate it as the source of the problem, doing so would require some development work at our end which we cannot do in the short term.
However, I should point out that the problem did not exist prior to May 23 and it came on suddenly. Also, as you pointed out, we don't have a problem with the .156 farm which suggests a server side issue. The fact that Craig is also seeing delays supports this.

As a workaround, we will reset our alert threshold to 5 sec which will cause our software to reconnect and hopefully we will reconnect to the .156 farm (from memory we can force a connection to the .156 farm but we won't do that for now).

Dennis

Dennis, as I said, we will continue looking into potential problems on our end. However, I would recommend that you make upgrading to a newer version a priority. You are almost 6 versions out of date at this point (4.8 is in beta as you should know) and we don't do any testing with that old of a version internally so there is no guarantee it will continue to work. At any point, we may release server side code that unknowingly will break that old version. At that point, you will be without your feed until you make the changes to upgrade.

Dennis/Craig, we are not finding anything internal that could explain this. Nor are we finding anything consistent across all users on the system that would show a pattern of what you are seeing .

Can you send us a ping and traceroute to both of our server farms? of course, feel free to email develiper support with this info if you don't want to post it publicly on the forums.

Steve,

I've looked at the pings & trace routes and I don't believe they reveal anything but here they are. I also ran the pings for around 15 minutes and had no anomalies.

/home/trader-> ping 66.112.148.114
PING 66.112.148.114 (66.112.148.114) 56(84) bytes of data.
64 bytes from 66.112.148.114: icmp_seq=1 ttl=54 time=45.4 ms
64 bytes from 66.112.148.114: icmp_seq=2 ttl=54 time=45.7 ms
64 bytes from 66.112.148.114: icmp_seq=3 ttl=54 time=46.0 ms
64 bytes from 66.112.148.114: icmp_seq=4 ttl=54 time=45.5 ms
64 bytes from 66.112.148.114: icmp_seq=5 ttl=54 time=45.6 ms
64 bytes from 66.112.148.114: icmp_seq=6 ttl=54 time=45.3 ms

--- 66.112.148.114 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5004ms
rtt min/avg/max/mdev = 45.302/45.626/46.084/0.365 ms


/home/trader-> traceroute 66.112.148.114
traceroute to 66.112.148.114 (66.112.148.114), 30 hops max, 40 byte packets
1 69.10.58.1 (69.10.58.1) 0.845 ms 1.374 ms 1.343 ms
2 66.45.224.33 (66.45.224.33) 0.718 ms 0.710 ms 0.688 ms
3 64.20.47.17 (64.20.47.17) 0.663 ms 0.873 ms 0.851 ms
4 nyrkbprj02-ae1.303.rd.ny.cox.net (68.105.31.93) 0.815 ms 0.792 ms 1.009 ms
5 mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 43.692 ms mtc1dsrj02-ae2.0.rd.om.cox.net (68.1.0.127) 43.654 ms 43.653 ms
6 66.37.238.194 (66.37.238.194) 44.304 ms 66.37.238.202 (66.37.238.202) 44.188 ms 66.37.238.194 (66.37.238.194) 44.289 ms
7 69.63.125.106 (69.63.125.106) 44.895 ms 44.969 ms 44.805 ms
8 (98.142.95.133) 45.064 ms 45.021 ms 45.001 ms
9 (98.142.95.129) 80.018 ms 79.985 ms 79.933 ms
10 216.58.227.114 (216.58.227.114) 45.117 ms 45.398 ms 45.061 ms
11 (66.112.151.18) 46.523 ms 46.510 ms 46.484 ms
12 66.112.148.114 (66.112.148.114) 45.275 ms !X 45.434 ms !X 45.305 ms !X


/home/trader-> ping 66.112.156.220
PING 66.112.156.220 (66.112.156.220) 56(84) bytes of data.
64 bytes from 66.112.156.220: icmp_seq=1 ttl=52 time=43.9 ms
64 bytes from 66.112.156.220: icmp_seq=2 ttl=52 time=45.3 ms
64 bytes from 66.112.156.220: icmp_seq=3 ttl=52 time=42.4 ms
64 bytes from 66.112.156.220: icmp_seq=4 ttl=52 time=42.5 ms
64 bytes from 66.112.156.220: icmp_seq=5 ttl=52 time=42.4 ms
64 bytes from 66.112.156.220: icmp_seq=6 ttl=52 time=43.6 ms
64 bytes from 66.112.156.220: icmp_seq=7 ttl=52 time=42.4 ms

--- 66.112.156.220 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6006ms
rtt min/avg/max/mdev = 42.452/43.264/45.336/1.053 ms


/home/trader-> traceroute 66.112.156.220
traceroute to 66.112.156.220 (66.112.156.220), 30 hops max, 40 byte packets
1 69.10.58.1 (69.10.58.1) 0.975 ms 0.939 ms 1.115 ms
2 66.45.224.33 (66.45.224.33) 0.553 ms 0.858 ms 0.846 ms
3 207.239.51.85 (207.239.51.85) 0.806 ms 0.794 ms 0.772 ms
4 vb1010.rar3.nyc-ny.us.xo.net (216.156.0.17) 3.621 ms 2.260 ms 2.882 ms
5 * * *
6 206.111.13.198.ptr.us.xo.net (206.111.13.198) 0.600 ms 0.794 ms 0.871 ms
7 omh-core-02.inet.qwest.net (205.171.5.241) 41.770 ms 41.800 ms 41.776 ms
8 omh-edge-09.inet.qwest.net (205.171.132.66) 41.398 ms 51.287 ms 51.279 ms
9 65.116.176.82 (65.116.176.82) 42.890 ms 43.224 ms 43.202 ms
10 66.112.152.50 (66.112.152.50) 42.816 ms 43.141 ms 43.124 ms
11 solar0.interquote.com (66.112.156.220) 42.411 ms !X 43.386 ms !X 42.559 ms !X

Hi Steve,

As you may recall, I work with Dennis. One of the things that has slowed down any attempt by us to upgrade our client is that (as far as we can see) there is no step-wise way to do it. Once you have a new version, old versions seem to be removed from your site.

Is there anywhere we can get some of the older versions of the client that you think are stable and clean, so that we can upgrade in an orderly fashion?

The best version for us would be one where we aren't forced to change our code, of course. :-)

Thanks,

Neil

Hi Steve,

As you may recall, I work with Dennis. One of the things that has slowed down any attempt by us to upgrade our client is that (as far as we can see) there is no step-wise way to do it. Once you have a new version, old versions seem to be removed from your site.

Is there anywhere we can get some of the older versions of the client that you think are stable and clean, so that we can upgrade in an orderly fashion?

The best version for us would be one where we aren't forced to change our code, of course. :-)

Thanks,

Neil

I'm having another day of large delays (up to half a minute).
CPU usage on box is 30-40%.

:~$ ping 66.112.148.113
PING 66.112.148.113 (66.112.148.113) 56(84) bytes of data.
64 bytes from 66.112.148.113: icmp_req=1 ttl=48 time=207 ms
64 bytes from 66.112.148.113: icmp_req=2 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=3 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=4 ttl=48 time=207 ms
64 bytes from 66.112.148.113: icmp_req=5 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=6 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=7 ttl=48 time=206 ms
64 bytes from 66.112.148.113: icmp_req=8 ttl=48 time=207 ms

~$ traceroute 66.112.148.114
traceroute to 66.112.148.114 (66.112.148.114), 30 hops max, 60 byte packets
1 () 0.593 ms 0.832 ms 1.135 ms
2 .jetstream.xtra.co.nz () 60.229 ms 71.474 ms 71.641 ms
3 * * *
4 ae4-10.akbr5.global-gateway.net.nz (202.37.244.221) 25.405 ms 25.534 ms 25.661 ms
5 ae1-2.akbr4.global-gateway.net.nz (202.50.232.77) 25.918 ms 26.102 ms 26.269 ms
6 ae1-10.tkbr9.global-gateway.net.nz (202.50.232.37) 27.267 ms 17.957 ms 18.945 ms
7 so7-0-1.labr5.global-gateway.net.nz (122.56.127.38) 143.181 ms so6-1-3.labr5.global-gateway.net.nz (202.50.232.42) 143.910 ms so6-0-3.labr5.global-gateway.net.nz (122.56.127.34) 145.077 ms
8 ae0-3.lebr6.global-gateway.net.nz (203.96.120.86) 145.254 ms 146.206 ms 147.174 ms
9 LANGBBPC01GEX0200A001.R2.LA.COX.NET (206.223.123.42) 179.592 ms 179.712 ms 166.048 ms
10 mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 184.227 ms mtc1dsrj02-ae2.0.rd.om.cox.net (68.1.0.127) 184.668 ms mtc1dsrj01-ae2.0.rd.om.cox.net (68.1.0.125) 185.858 ms
11 66.37.238.202 (66.37.238.202) 192.268 ms 66.37.238.194 (66.37.238.194) 191.780 ms 192.653 ms
12 69.63.125.106 (69.63.125.106) 210.996 ms 205.209 ms 205.147 ms
13 98.142.95.133 (98.142.95.133) 180.526 ms 181.126 ms 181.603 ms
14 98.142.95.129 (98.142.95.129) 208.744 ms 205.341 ms 206.487 ms
15 216.58.227.114 (216.58.227.114) 182.080 ms 181.492 ms 182.788 ms
16 66.112.151.18 (66.112.151.18) 210.879 ms 211.679 ms 211.753 ms
17 66.112.148.114 (66.112.148.114) 212.004 ms !X 207.400 ms !X 207.563 ms !X

Craig, is your situation like Dennis where you only sees this on the 148 server farm?

I hadn't noted the remote address in previous cases, but I shall do so in future.

Just quickly,

Since we have limited our connections to the .156 farm we have had service as normal, which for us is over 99% of data delivered with a lag of less than 100ms. Today's latency graph is attached.

Steve, we will work with you offline to see what we can do about upgrading our client.

Thanks,

Neil

How does one limit the connection address?

Craig,

From your trace route it looks like you're in NZ. We originally (until 4-5 years ago) ran our servers from Australia and had a lot of trouble with the feed during busy periods due to the high network latency (250ms ping times). At the time this resulted in data corruption but may manifest itself differently with the later versions of IQFeed. This was resolved when we relocated to the East coast USA.

Also, as IQFeed is single threaded, look at the process CPU usage to see if it's maxing out a single core (25% on a quad core, 50% on a dual core, etc) rather than overall CPU usage.

Dennis

Running against a 156 address today, seeing no perceptible delay, also running a lower volume watch list.

Dennis: Thanks for the tips, living in NZ I do expect some latency of course, just not 30 seconds of it. IQFeed was at about 20% on one core and there is plenty of B/W.
Edited by Craig on Jun 28, 2011 at 01:37 PM

Craig, At this point, I believe the difference in volume of your watchlist today is a larger factor than switching server farms. Comparatively today you are getting roughtly 8 times less data than you would have been getting with your symbol list from yesterday.

I suspect you are right, though it's hard to know if that's the whole story without switching watch-lists again. However, I've seen IQFeed balloon in memory, use 100% CPU etc when the client application can't keep up, and that wasn't happening yesterday.

I had terrible lag today...yesterday I processed nearly twice as much data with only sporadic lag...

IQFeed reports that it's using around 20KB/s bandwidth, the link bandwidth is 6MB/s.
The server applications are all C++ applications using async. processing, the CPU is perhaps maxing at 30%. IQFeed is not growing in memory so I know I'm processing the data quick enough. Any other suggestions on how I can troubleshoot this?

Craig,

If it makes you feel any better we have been experiencing larger feed delays recently (again using timestamps as a reference). Normally these occur during bursts of market activity such as open/close and after major news announcements. Of course, with recent events these have occurred more often. Tick counts have more than doubled over the last few days for the same watchlists on both the NYSE and NASDAQ.

Dennis

Thanks Dennis.

...of course today I had a much larger tick count but with no delay.
Go figure.

I am having latency problems too.

I have traded using IQFeed for more than a year. I am seeing the "time" messages become very eratic, delay 20-30s, and many go missing.

I scalled back the number of symbols, but the it had no improvment.

I also changed the code only request the fields need, but no help.

The CPU load on both processes is very low, only about 1%

I measured how much time the tick reader spends waiting on the socket and it spends very little time processing messages and almost all of the time waiting for messages.

It is connected to 66.112.148.225.

Not sure what else I can do.

Today was another latency disaster for me as well.

Pretty good delay here last week too. 10-20 seconds in some cases.

I don't know if this will help, but I went to the IQ Connection Manager and watched the "Market time" field. It would actually lag it's update to the next minute by the same delay as I was seeing with quotes.

Not sure what server I was on last week. Today one machine is on 66.112.156.228

ers811, it sounds like your issue is a different issue since it wasn't on the same server farm that this thread applies to. I'm happy to work with you outside this thread (in order to keep issues separated) to work towards a resolution.

Craig/Neil/Denis/Gremlin, there were some changes made by the network provider at our 148 server farm Thursday night that we believe should have aleviated the issues that you guys were seeing with that server farm. Please let us know if you continue to see issues.

Better today, I'm still seeing the odd burst of lag, but nothing like last Friday.