Over the past 1 year, DTN support has repeatedly promised that their Real Time internet
feed would be real time in "fast" or busy markets. This has not happened. Despite all
promises of changes, iqconnect still shows tens or 100s of millions of page faults in
process monitor resulting in delays in feed times of up to 20 minutes. Iqconnect uses all of the
system resources. Sometimes for an hour at a time.
After all types of upgrades to my computers recommended by DTN, these delays have not
gone away.
I am now of the opinion that DTN cannot deliver real time and may not know what it will
take to deliver.

We have had zero delays on our servers today. We are able to "deliver" real time data, the problem is for users to be able to "receive" real time data. I assume you are running the data into Tradestation, an application that has been unsupported for years. It was built when the markets were 1/100th the volume they are today. You are also using a piece of middleware to process the IQFeed into it. There are a lot of variables and passing of data going on. If either of those apps can't keep up with the data IQFeed is throwing at it, you will queue data (and thus the delays you are seeing), causing IQFeed to use 100% cpu. The only solution is more efficient client software and/or faster hardware (which you said you already have). Beyond that, the only suggestion we can make is to reduce the number of symbols you are watching or reduce the analytical calcuations you are performing. Reducing things to the minimum you require to trade should help. We have thousands of customers, many watching 1000 symbols without any delays.

---

Jay Froscheiser
DTN - Trading Markets

Sorry Jay, but I have to post a voice of dissent.

"The only solution is more efficient client software and/or faster hardware" is just plain wrong in that it addresses the SYMPTOM, not the PROBLEM.

The problem is that IQConnect and Client apps canot keep up with market data during heavy trading periods due to fundamental inefficiencies related to the design of Q messages. For example:

1. Field 6, percent change should be truncated to 4 decimals. I'd actually take the position this never should have been included in a Q message, but rather it should be a client side calculation.
2. Fields 9 & 10 (daily hi & low) should only be sent when a change occurs, otherwise a single character should be used indicating the prior value should be used. The same concept should be used for many other fields.
3. Field 31, last trade date. Does this need to be sent with every Q message, especially if it traded in the current day?
4. Field 43, change from open. You provide open & last. Should be client side calculation.
5. Field 51 - Regions is a very long string which need not be sent every Q message. Should probably be in the F message anyway. If iqconnect could only send it once for each ticker that would be a significant savings.

I'm sure there are many other items which would make a much more efficient Q message and reduce CPU needed by both apps.

Then there's the matter of duplicate messages and blocks of messages which contain very little useful info. I've provided samples of these to Steve. Examples include blocks of consecutive Q messages which only report bid/ask size changes, which when reported in a series really provide no useful info, but heavily contribute to the problem of loading.

Finally, and Steve & I don't exaclty see eye to eye on this, there are many instances of consecutive trade reports, which I believe are "broken" fills. For example, an order to buy 1,000 shares gets filled in lots of 100,200, 300, etc. IQ clients often receive multiple Q messages which I feel can and should be consolidated into a single Q message provided the last, bid and ask price is the same for all consecutive trade messages.

The SOLUTION is a more efficient message structure combined with eliminating messages not needed by client apps.

Let's hear what others have to say. What's more important? True tick by tick data, or a feed that can stay up to speed during fast market conditions?

Additional fields which are inefficient because either the info is duplicative, should be client side calculation, should only be sent once, extended trade info being sent during regular hours, etc.

17 Range float Trading range for the current day (high - low)
19 Open Interest integer IEOptions, Futures, FutureOptions, and SSFutures only
20 Open float The opening price of the day.
21 Close float The closing price of the day.
22 Spread float The difference between Bid and Ask prices
24 Settle float Futures, FutureOptions, and SSFutures only - Is this different than Field 21?
33 Extended trading last - Suppress during normal market hours
34 Expiration date MM/DD/YYYY Deprecated, use field 53 in the Fundamental message
37 Extended trading change - Suppress during normal market hours
38 Extended trading difference - Suppress during normal market hours
39 Price-Earnings Ratio
40 % Off Average Volume
41 Bid Change float Change in Bid since last offer
42 Ask Change float Change in Ask since last offer
43 Change From Open float Change in last since open
46 Market Capitalization float Real-time calculated market cap
45 Volatility float Real-time calculated volatility (Today's High - Today's Low) / Last
49 Days to Expiration string Number of days to contract expiration
50 PVol Integer Previous Day's Volume
51 Regions String The available regions for regional data
62 Settlement Date MM/DD/YYYY The date that the Settle (field 24) is valid for


And Field 46, Market cap, I retreive the following data points from a recent log file (August 9):

WMT 191797757.03999999
XOM 466440634.19999999

First, why is this even a float?
Second, the data is simply wrong. WMT cap is 191 Billion, not Million. XOM is $464 Billion. These should be sent as integer types, and Client app should have to multiple by 1 million, so data sent would be 191797 and 466440. Personally, I don't use this field and wish it could be supressed altogether, but I can see how others might want it.

You have good points. Some easily arguable because we are serving several hundred products with thousands of users (I won't argue with you or others becuase there will be as many arguments as there are opinions). I can see where having an option to receive a stripped down feed that doesn't provide any value-add can be useful (IQconnect not calculating change, or sending other information some clients DO need). Maybe this would be a flag you can send into IQconnect that results in you getting a different message type.
I am sure some will agree that an aggregated feed would be just fine. However, we have many more users who come to IQFeed because they need every tick for their analytics.

Again, I am more than willing to hear opinions in the hopes they can be turned into a better product for all.

---

Jay Froscheiser
DTN - Trading Markets

Another item I just noticed. Nasdaq stocks are getting 4 digits of precision while Listed stocks only two. Is that intentional? Couldn't the trailing zeros be trimmed similar to absolute price change (field 5)?

Samples:

Q,BAC,D,48.72,-0.98,-0.01971831,20355615,200,49.20,47.92,48.70,48.70 ,3200,9300,183,,1.28,13:04t,,48.30,49.70,0.,,,,t,,,,,08/09/2007,,48.72,,,,-0.98,0,10.4,-0.368426466,0.,0.,0.42,,0.026272578,216208203.12,12,2,,35709908,BSE-CSE-CHX-NYSE-PSE-NMS-PHLX,,,,,68624,,,48.65,,N,,
Q,AAPL,F,129.9000,-4.11,-0.030669353,17100279,1000,133.0000,129.8800,129.9000,129.9100 ,100,300,175,173,3.12,13:04t,,131.1100,134.0100,0.01,,,,t,N,,,,08/09/2007,,129.9000,,,,-4.11,0,37.7,-0.561744817,0.01,0.,-1.21,,0.024018476,112966365.90000001,14,4,,28883504,BSE-CSE-CHX-NYSE-PSE-NMS-PHLX,,,,,73182,,,131.5834,,N,,

Jay, the inefficiency of the IQClient is the biggest problem we have in receiving a true real time feed when watching 1300 of the busiest symbols on NYSE. We recently moved to using the fastest available Intel processor but the IQClient still ate up 100% cpu (on its own) with the recent high volumes.
By comparison, we wrote our own client for another data provider and had it create the same fields that the IQ client has (so we didn't have to change our application) and for the same data feed our client used 1/5th the cpu compared to the IQFeed client for EXACTLY the same data. Our client was written in Java which isn't exactly the fastest language to use and was also performing CPU intensive tick correction.

Moreover, we have an automated system that will disconnect/reconnect us from DTN when the delay becomes unacceptable. Usually if we reconnect to the same server the delay is still present but if we are connected to a different server the delay often disappears. This suggests that some of your servers are NOT keeping up with the data flows (but some servers are). This kind of problem has been far less prevalent since the additional servers were installed.

Additionally, we ran tests this week with a heavily overclocked PC provided by a friend. This machine runs at 4.4GHz (almost 50% faster than any available CPU) and it managed to keep up with the feed but we saw clipping of the data stream at around 2.5mbit/s suggesting that the servers had issues at that point. Unfortunately we cannot use such a PC in a data center as it is water-cooled.

One more thing in response to the original post in this thread:

Quote: "...iqconnect still shows tens or 100s of millions of page faults in
process monitor resulting in delays in feed times of up to 20 minutes."

Page faults have nothing to do with bad programming and are certainly not responsible for such long delays. It is most likely that your PC is running too low on RAM and Windows is forced to Page memory. This will certainly result in very slow processing.

Hi dhakme, Who is the other data provider? Do you have good experience with it?

I'm using the data feed provided by Interactive Brokers as a second feed. Theirs has some limitations compared to IQ in that it will only allow a limited number of tickers (I think 100, but they keep changing it), and they make no attempt to provide tick by tick data. Theirs is a "snapshot" service which provides bid/ask/last/volume a couple times a second or so for each ticker being watched. There's some downside to this, however their feed doesn't seem to experience delays as they purposely omit data.

Also, they do provide market depth for many securities.

My app is watching the e-minis (ES, NQ, YM), the underlying ETFs, and a handful of individual stoocks from both feeds, and so long as they match I assume data is reliable and accurate.

Hi Ned,

It's probably worth pointing out that the creation of the immensely unwieldy Q messages is done by the IQFeed client, and is not reflective of the data that is coming down from the DTN servers. The server data is indeed fairly optimized (but NOT compressed as claimed by DTN), and about one tenth the size of the massively-expanded data dished up by their local client.

As you have so rightly pointed out, much of the data that is continually resent by the IQFeed client is actually sent just ONCE by the servers, but a decision was obviously made somewhere in the dim, dark past to supply this information to clients over and over again.

Of course, it doesn't help that the local client is a terrible, terrible piece of software that totally hogs CPU and corrupts data as a matter of course. As Dennis wrote earlier, a properly written Java client takes less than 20% of the machine resources currently overwhelmed by the latest IQFeed offering.

Regards,

Neil