Over the last two weeks I've noticed that our nasdaq feed has been far more erratic than usual. We watch 1300 symbols (the busiest) and normally on market open we get between 800-900 kbps solid and it then settles down to 400-500 kbps. Lately, there seems to be an artficial bandwidth limit of 600kbps and the bandwidth utilised is very erratic, sometimes going down to 10-20 kbps for seconds at a time. When we see this, we disconnect and reconnect and the problem goes away and the feed is back to normal (sometimes it takes a few goes). I'm assuming that we are connecting to a different server the second time.

We have a second computer watching 1300 nyse stocks over the same feed and it has been perfectly normal. I will attempt to preempt any questions with the following info:

- Our servers are running dual xeon 3GHz with 2GB RAM, w2K, CPU is < 50% at all times
- Internet connection is ADSL2 20Mbps download, 1Mbps upload
- same thing happens on our backup cable 10 Mbps connection
- identical software on both machines

Is there something wrong with the nasdaq feed? Has a bandwidth limit been placed per client?
What's up with the nasdaq feed?

Hello Dennis, to my knowledge, there hasnt been any kind of bandwidth limit put in place on a per user basis.

I would be interested in seeing if this is something I can duplicate on this end. Is it possible for you to email the list of symbols you are watching (or at least a snapshot of your watchlist if it changes) to our developer support?

Also, a few questions.

How much traffic does your NYSE watchlist generate compared to the NASDAQ one?
You mentioned the posibility of connecting to different servers to temporarily resolve this. Can you verify this by checking the server you are connected to in the IQConnection Manager?

I've emailed our current watchlist.

The NYSE generates signficantly less traffic at the start of day as it doesn't really have a large burst (~400kbps). However, once the day settles the NASDAQ and NYSE are comparable (~150-300kbps) with the NASDAQ usually slightly higher. The NASDAQ must have better compression as the NYSE is larger when uncompressed.

I started this thread because I tried 4 restarts today without success. The IP of our server at the moment is 66.112.156.221. I will keep track of the server IPs from now on.

Steve,
Our nasdaq feed today seems to be reasonably good (of course!). But it still seems to be capped at around 600kbps which only affects the first 30 seconds or so.
For the record, the server we are connected to is 66.112.148.222.

Dennis

Hello Dennis, I was connected to the same server this morning at market open and my experience differed somewhat.

The numbers you are posting for your watchlist you emailed in are similar to what I observed this morning but it didnt seem at all capped.

I'm not sure how to explain this compared to the numbers you say you have historicly gotten.

After market open (when the activity died down) I loaded a watchlist of about 2000 of the most actively trading symbols across all markets. The bandwidth measured at 1.7Mbps as as I loaded the watchlist and after settling down it would peak the bandwith above 800kbps frequently with an average of 650kbps.

I will continue looking into this.

Steve,

We just got disconnected (not sure why) but after reconnecting the feed was again choppy and after 10 minutes there were sections where the download stopped completely. The server IP was 66.112.156.227.

We then forced a disconnect but connected to the same server. The feed continued to be choppy until just the last minute (11:32 EST) when it seemed to sort itself out. I have captured a bandwidth graph if you want to see a visual representation.

I am interested if you are connected to your servers through a separate, internal interface or through the same interface as the Internet clients.

Dennis, over the past 2 days I have done more testing on this both internally and from an external location. I still havent been able to observe a cap on the bandwidth or the behavior concerning the connection dwindling to the point of having to reconnect.

have you had any more issues with this in the past couple days?

Steve,
Nothing has changed from our perspective. Today we never saw bandwidth above 500kbps and in fact a graph of bandwidth usage at the start of the day was very flat at 500kbps which looks like capping. We checked our internet connection with various sites including downloading software from DTN and are confident the network is not the problem. Since all counters are generated by the IQFeed client and there are no exchange timestamps or counters there's no way for us to tell if we are losing ticks which is our real concern.

Have you used our exact symbol watchlist? It would be interesting to compare our and your bandwidth for a given day.

Dennis.

Yes. both days at market open I was using the watchlist you sent me.

Today (from the remote location) after the market open, I downloaded the NASDAQ securities list from http://quotes.nasdaq.com/reference/comlookup.stm (at the bottom) and added all 3335 symbols on the list.

while watching this list, the connection would average around 450-500kbps with frequent spikes up past 800kbps to as high at times as 1.7mbps

As I said previously. I am not showing the 900kbps with your watchlist at market open that you were previously getting. My testing is showing that the connection averages around the same 500-600 kbps that you are experiencing. But it doesnt appear to be capped there since the connection has quite a bit of fluxuation.

I will test tomorrow using the full nasdaq watchlist to see what it looks like. If the connection is getting capped it should show up much more visibly with the larger watchlist.

WHO FIXED THE PROBLEM???

And please don't say nothing has changed at your end.

This problem had degenerated further and restarting the feed didn't fix it. We couldn't get a decent feed from any of the IQ servers. Even our NYSE trading machine was unusable.

For those following this thread, we sent IQFeed a sample of our app via email in the hope it would help in solving this problem. I don't know if they have used it but on Tuesday afternoon some time the problem disappeared! Here is a picture showing 3 charts of the streaming data. The first shows the bad feed, while the 2nd and 3rd show what a good feed looks like. The 3rd also demonstrates the data capping at ~600kbps that is occuring.
http://members.iinet.net.au/~nellob/iqfeed.jpg


So I ask again, who/what fixed the problem? Is it permanent?
The problem was occuring regardless of which of our three ISPs we used and they all have very different routes to the IQ servers (verified by trace route).

Dennis

Glad to hear all is well once again. I can't say what would have changed, since when we tested with your app, we weren't able to see the bottleneck when tested via an outside ISP.

---

Jay Froscheiser
DTN - Trading Markets

Mmmmmm.

It was fine at the end of trading on Tuesday but given that no one is claiming responsibility for the fix it would be presumptious (ie stupid) to assume that the it's actually fixed. From my perspective we were running the same software all day and only switch around our ISPs trying to get a better feed. Looking at trace route output from two of our ISPs they differ completely in the routes they take with ADSL going via asia and cable going direct to the US east coast. They only converge at the very last few hops:

ADSL
11 253 ms 253 ms 252 ms p1-0-0.nycec-ar1.ix.singtel.com [203.208.154.102]
12 258 ms 256 ms 258 ms 203.208.169.2
13 256 ms 255 ms 266 ms 216.140.10.221
14 258 ms 258 ms 259 ms 216.140.15.70
15 270 ms 270 ms 271 ms 67.98.17.18
16 273 ms 272 ms 272 ms solar7.interquote.com [66.112.156.227]

CABLE
14 233 ms 228 ms 227 ms ge-2-1-0.a1.chcg.broadwing.net [216.140.15.17]
15 238 ms 233 ms 232 ms p5-0.gnwd.broadwing.net [216.140.15.141]
16 233 ms 234 ms 234 ms 216.140.15.66
17 233 ms 233 ms 235 ms 216.140.15.174
18 253 ms 255 ms 255 ms 67.98.17.18
19 253 ms 255 ms 255 ms solar7.interquote.com [66.112.156.227]

To me, this rules out everything up to the point of your ISP. If IQ did not change any equipment (routers, firewalls, servers, etc) or software then suspicion naturally falls on your ISP.
When you say you tested via an outside ISP, I assume you meant either from the Internet side of your firewall or from a service provided by the same ISP. If this is so then it is inadequate.

Can anyone guess what happens when a trader acts on delayed information (ie a bad feed)? They buy rising shares later and sell falling shares later and LOSE LOTS OF MONEY! Our review of the last two days show we lost over $2000 due to feed delays.

Shrugging our collective shoulders and declaring the problem solved is not going to cut it. Can you please post a trace route output for the external ISP testing you performed?

Ooops, almost forgot, what about the very obvious bandwidth capping at the end of a nasdaq trading day? We suspected this at the start of the day as well and it is unlikely to be an ISP issue but a firewall/server limitiation deliberately imposed.

All I can say is there was no cap, deliberate or not when we tested.

"I assume you meant either from the Internet side of your firewall or from a service provided by the same ISP. If this is so then it is inadequate."

Not sure why you assume this. We have been in this business since the early 80's and are aware of how to troubleshoot problems. We tested with a residential cable modem ISP which is completely isolated from any network or ISP DTN uses.

"Shrugging our collective shoulders and declaring the problem solved is not going to cut it. Can you please post a trace route output for the external ISP testing you performed?"

Nobody is shrugging shoulders here. We tested for the problem you saw, and couldn't reproduce it. Because we didn't see the problem, we didn't save a copy of the tracert. It is obvious this isn't what you want to hear, but there is nothing more that can be said from our end.

---

Jay Froscheiser
DTN - Trading Markets

Jay,

I didn't mean to offend but I admit I am frustrated. You couldn't replicate our problem, but with 80,000+ customers I'm sure the complaints would be loud and clear if your servers were failing in some obvious way.

However, I am left with a non functioning trading system and with very few leads left to follow, which is why I asked for the trace route output. If the problem is related to our Internet connections, which seems likely, then I wanted to determine the likelihood of fixing the problem by using a co-lo nearer to the DTN server farm.

If you are agreeable, I’d like to send you an updated application that has some additional logging. If you could run this app and send us the log then we will have enough information to reach a final conclusion.

I’d like to thank you (and your colleagues) for your past assistance and patience in resolving technical issues.

Dennis

In the interest of closure on this issue for any developers that are interested, it was finally determined that the cause of our problems was in the return route (from IQ to us) where an upstream provider was having problems. We are located in Australia and have three different ISP connections and while the route from us to IQ was different, they all shared one specific international link in the return path. There was no way to determine the return path using tools like tracert.

We have now relocated one of our servers in a colo in the USA and the ping times have been reduced from ~300ms to 45ms. Interestingly, the number of corrupt packets we receive from the IQFeed client during peak times has reduced significantly. Some days we don’t get any corruption.
The corrupt packets have been extensively documented in this forum, but this adds a slight twist. Previously, bad programming (single vs multi thread), poor PC performance and insufficient bandwidth have all been blamed. I would like to add latency to that list.

A summary of the outcomes:

1) Problems like this are very difficult to analyse without assistance from IQFeed
2) There is no way to determine if data is missing from IQFeed
3) There is no way to determine if there is a delay in the feed
4) Latency causes the IQFeed client to corrupt data

Points 1 and 2 can only be solved through exchange generated time stamps and tick IDs (or comparing to another feed). These have long been promised and it would appear that they are coming soon (again).
Point 4 cannot be disputed. A powerful server (dual processor, etc) with a 20Mbps connection and 300ms latency received corrupt packets every trading day for over a year when watching 1300 NASDAQ symbols. When relocated to a lower latency location (45ms) corruption virtually stopped.