Hi,

I just got some "Q" messages on port 5009 like this one:

Q,LLTC,F,.LNCR,-0.07,,3121758,197,38.8300,.LNCR,37.4700,37.5000,700,200,175,173,38.83,14:24t,,38.5500,38.2400,0.03,,,,p,N,,,,11/05/2004,,.LNCR,,,,-0.07,0,33.5,-0.446889086,-0.68,-0.66,-0.38,1,0.018600996,11737427.68,14,4,,4576541,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,14414,,,38.5777,

Obviously the ".LNCR" in field 10 is not correct, since this field should hold the daily low.

Here are my questions:
- What does ".LNCR" mean, where I expect to get a number ?
- Which non-number (expect the empty string) may I experience in number fields ?

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best

Here is more strange data:

Q,LNCR,F,.CSCO,-0.01,,710115,100,37.8200,.CSCO,19.9700,19.9800,41200,89200,175,173,37.82,15:30t,,37.7900,37.5500,0.01,,,,p,N,,,,11/05/2004,,.CSCO,,,,-0.01,0,14.5,-0.545380922,-17.57,-17.57,-0.25,1,0.013585509,3751710.06,14,4,,1106633,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,2899,,,37.4692,'

Q,GENZ,F,.INTC,0.84,,1878108,100,56.2700,.INTC,23.3300,23.3400,65400,95600,175,173,56.27,15:30t,,55.5400,54.8700,0.01,,,,t,N,,,,11/05/2004,,.INTC,,,,0.84,0,68.6,-0.14006044,-32.38,-32.38,0.17,1,0.021719619,12701322.9,14,4,,2075059,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,8819,,,55.6850,

Q,LRCX,F,.MRVL,0.72,,2632057,100,26.4000,.MRVL,28.2400,28.2700,100,1000,175,173,26.4,15:30t,,25.9100,25.3800,0.03,,,,t,N,,,,11/05/2004,,.MRVL,,,,0.72,0,21.3,-0.136747458,2.16,2.17,0.19,1,0.026819923,3537019.8,14,4,,4110642,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,9714,,,26.1137,

Q,QCOM,F,.AMGN,0.06,,20035175,100,38.8000,.AMGN,61.0400,61.0400,1100,500,175,173,38.8,15:30t,,38.4400,38.0700,0.,,,,t,N,,,,11/05/2004,,.AMGN,,,,0.06,0,39.2,0.315766402,22.92,22.91,-0.31,1,0.022292158,62066526.930000007,14,4,,40502115,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,50890,,,38.3189,

Somehow, symbol names are in place of the price for a trade. How come ? Am I missing something ?

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best

And here is more garbage data:

Q,CECO,F,38.1200,7.22,0.233656958,1128708,150,38.1200,30.4100,30.6100,30U,F|,600,500,173,173,7.71,15:30t,,30.9100,30.9000,-0.61,,,,c,N,,,,11/05/2004,,38.1200,,,,7.22,0,18.0,-0.602007052,0.,-0.64,7.21,1,0.202256034,3907185.64,14,-1,,3179301,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,4545,,,30.7194,

The marked field should be at position 18 (1 based), but is at 19 ?

What's going wrong there ?

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best
Edited by Dierk Droth on Nov 5, 2004 at 03:21 PM

Dierk,

I'll have to look into these. Thanks for reporting them!

---

Natalie Hannan DTN Market Access, LLC.

It looks like message corruption between the IQFeed component and the server. When the server cannot transmit a full message it currently discards the unsent portion. This means the next full message sent is appended to the partial message that was sent last. Corruption in a field will remain until the field is overwritten with good data.

Reasons the server could not transmit: slow network or problems in the network can cause corruption in the server TCP buffer; not reading data fast enough off of the receive TCP buffer also causes corruption. if IQFeed does not have enough CPU available to service the TCP socket it will backlog in the socket buffer and eventually corruption will occur. There are quite a few variables that affect this: number of symbols, market activity, anything else the computer may be doing.

We are aware of these problems and are currently developing fixes to prevent message corruption. In the future we will not discard the unsent portion of the message, we will send it when the socket becomes writable, this guarantees that complete messages will be sent. Until that remainder of the message is sent any new messages will be dropped for that connection.

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS

Lonnie,

Thanks for letting me know.

Will this issue be fixed with the upcoming next release (which should be available soon, AFAIK) ?

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best

Oh boy, I just let my test driver collecting market data for some 30 minutes now and got ~3 MB of screwed up data.

Although the processor load was low, I had a skype session on the same internet connection. There must be an issue with bandwidth screwing up the communication.

This really is a serious issue, which should be fixed with priority.

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best
Edited by Dierk Droth on Nov 10, 2004 at 09:53 AM

I just ran another test session to track ~100 NASDAQ stocks.

There is tons of corrupted data coming in holding unexpected values. Plus some news coming in on 9100 are garbage bytes too.

Could somebody pls provide a feedback when this issue will be fixed ?

Thanks

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best

Dierk,

The degree to which you're seeing this concerns me. We're not seeing this as a systemic problem - it's just an occasional problem, usually under heavy load (100 NASDAQ symbols isn't a heavy load).
We are working on a fix for the corruption issue, but if the speed of processing at the client connection falls behind, which is what causes the corruption, the result will still be dropped data.
I would check cpu load, connection speed, NIC performance, etc.

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS

Lonnie,

Thanks for letting me know.

However, I disagree to some of your statements: The data feed should - in no event - be corrupted.

Here is my take: If the internet connection (CPU ...) just is too slow to recieve all requested data, IQFeed might drop/skip complete data messages and discard "incomplete" messages, but not pass corrupted message to the client.

Regards

---

Dierk Droth
www.trademagic.net
TradeMagic - Trading at its best

Dierk,

We agree 100% on this one - data should never get corrupted. It's one of the bigger fixes I've got planned for going to QA testing in about 2 weeks. No promises yet, but that's the plan.

---

Tim Russell
Software Engineer
DTN IQ & FinWin