It appears that I'm getting a handful of corrupted messages. Most often in the "last" field, I see a ticker like this:

Q,QQQQ,F,S.PY,0.164,,64713694,1000,37.2800,S.PY,3727.00,3728.00,69500,50000,175,173,37.28,13:30t,,37.2600,37.1100,1.,,,,p,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,3689.73,3690.72,0.014,0,0.010731341,20705707.,14,-1,,93432520,AME,,,,,52908,,,,
Q,QQQQ,F,S.PY,0.164,,64713694,1000,37.2800,S.PY,3727.00,3728.00,69500,50000,,173,37.28,13:30o,,37.2600,37.1100,1.,,,,t,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,3689.73,3690.72,0.014,0,0.010731341,20705707.,14,4,,93432520,AME,,,,,52908,,,,
Q,QQQQ,F,S.PY,0.164,,64713694,1000,37.2800,S.PY,3727.00,3728.00,69500,50000,,173,37.28,13:30o,,37.2600,37.1100,1.,,,,t,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,3689.73,3690.72,0.014,0,0.010731341,20705707.,14,4,,93432520,AME,,,,,52908,,,,
Q,QQQQ,F,.QQQQ,0.164,,64713694,1000,37.2800,S.PY,37.2700,37.2800,69500,50300,183,173,37.28,13:30t,,37.2600,37.1100,0.01,,,,p,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,-3689.73,-3690.72,0.014,0,0.010731341,20705707.,14,4,,93432520,AME,,,,,52909,,,,
Q,QQQQ,F,.QQQQ,0.164,,64713694,1000,37.2800,S.PY,37.2700,37.2800,69500,50300,,173,37.28,13:30o,,37.2600,37.1100,0.01,,,,t,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,-3689.73,-3690.72,0.014,0,0.010731341,20705707.,14,4,,93432520,AME,,,,,52909,,,,
Q,QQQQ,F,.QQQQ,0.164,,64713694,1000,37.2800,S.PY,37.2700,37.2800,61500,76500,,173,37.28,13:30b,,37.2600,37.1100,0.01,,,,p,N,,,,02/10/2005,,37.2390,,,,0.129,0.129,,-0.31660196,0.,0.,0.014,0,0.010731341,20705707.,14,4,,93432520,AME,,,,,52909,,,,


Less often, I get a last which is obviously wrong like this:

Q,QQQQ,F,84.8500,46.981,1.240618976,88079317,400,84.8500,.QQQQ,38.0800,38.0900,65400,99900,173,175,84.85,13:29t,,37.9000,37.8690,0.01,,,,p,N,,,,02/15/2005,,37.9100,,,,0.041,0.041,,-0.034355662,0.,0.,46.95,0,1.,47134175.,14,4,,55056970,AME,,,,,59930,,,,

I ran a filter last night and got a few hundred of these in February. Will do the same tonight to see if this was happening in January.

The affected dates in Feb were 10, 15, 22, & 28. I can email these messages if it's helpful.

I ran all my Janury data last night and see similar errors on Jan 4 & Jan 20.

Also added a trap for alpha in the "Low" field, like the last sample in my prior post where the low field contains ".QQQQ". I reran Feb 28 data log and see thousands of these errors. Here's a sample of a few where the low field contains ".SBUX":

Q,QQQQ,F,37.1500,-0.47,-0.01249335,76067608,400,37.6000,.SBUX,51.4400,51.4600,500,2800,173,175,37.6,14:50t,,37.4600,37.6200,0.02,,,,p,N,,,,02/28/2005,,37.4700,,,,-0.15,-0.15,,-0.142745644,14.29,14.31,-0.31,0,1.012113055,20636825.,14,4,,80143675,AME,,,,,66418,,,, - Low Error
Q,QQQQ,F,37.1500,-0.47,-0.01249335,76068108,500,37.6000,.SBUX,51.4400,51.4600,500,2800,183,175,37.6,14:50t,,37.4600,37.6200,0.02,,,,p,N,,,,02/28/2005,,37.4700,,,,-0.15,-0.15,,-0.142740009,14.29,14.31,-0.31,0,1.012113055,20636825.,14,4,,80143675,AME,,,,,66419,,,, - Low Error
Q,QQQQ,F,37.1500,-0.47,-0.01249335,76069608,1500,37.6000,.SBUX,51.4400,51.4600,500,2800,183,175,37.6,14:50t,,37.4600,37.6200,0.02,,,,p,N,,,,02/28/2005,,37.4700,,,,-0.15,-0.15,,-0.142723105,14.29,14.31,-0.31,0,1.012113055,20636825.,14,4,,80143675,AME,,,,,66420,,,, - Low Error
Q,QQQQ,F,37.1500,-0.47,-0.01249335,76071108,1500,37.6000,.SBUX,51.4400,51.4600,500,2800,183,175,37.6,14:50t,,37.4600,37.6200,0.02,,,,p,N,,,,02/28/2005,,37.4700,,,,-0.15,-0.15,,-0.142706201,14.29,14.31,-0.31,0,1.012113055,20636825.,14,4,,80143675,AME,,,,,66421,,,, - Low Error
Q,QQQQ,F,37.1500,-0.47,-0.01249335,76073108,2000,37.6000,.SBUX,51.4400,51.4600,500,2800,183,175,37.6,14:50t,,37.4600,37.6200,0.02,,,,p,N,,,,02/28/2005,,37.4700,,,,-0.15,-0.15,,-0.142683661,14.29,14.31,-0.31,0,1.012113055,20636825.,14,4,,80143675,AME,,,,,66422,,,, - Low Error

My March 1 data log does not exhibit these errors.

This is a recurring topic and we are addressing it. However, be aware that this problem is caused by the server not being able to send data to the client. Currently when this happens the remaining data in the message is dropped, the next message is appended to the partial message in the client and voila, data corruption.
Our solution will ensure messages are completed but if the client cannot keep up the server will drop, not queue, additional messages until the client is again able to receive.

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS

Corrupted Messages - empirical observations:

A good way to 'provoke' this bug is to run the system on a dial-up connection. Once a symbol has bad data, a bad low for example, it never gets cleaned up by the quote stream - even if you request an update on it, the erroneous data is still there. The only way (outside of exiting and re-starting the program) to get it corrected is to 'un-watch' the symbol and then 're-watch' it again. This deletes the record in the client and then it gets a new full update from the server.

It appears to me that the IQConnection Manager only gets partial record updates and the other fields that are relatively static never get touched again. I have seen no evidence that any refresh cycle (after market hours) cleans this up either. I use a special 'ALtU' key combination in my program that the user can use to force a full record update from the server. If you look at the re-request number on the IQConnection Manager it can be an indicator that you will have data errors if the re-request number is high.

The server / client non-communication may be the trigger for the problem, however it is the client that screws up and creates the bad record. Lost packets and data interruptions are a part of the internet - the interface between the client and server should be designed as though it were a UDP connection - no guarantee of packet delivery and they can also be out of sequence into the client. That forces identification tracking of packet records to make sure that the record gets built correctly, or dropped if something is missing. TCP/IP appears to me to be a poor choice to me for the stream because if there is a bandwidth/lost packet issue the server starts dropping quote updates and does not queue them any way - we see in the current implementation it does not guarantee delivery of the data, the main reason for TCP/IP. UDP has a much lower overhead and helps bandwidth issues. As to error checking, I assume that there is some sort of data integrity checking on the data packets, however there appears to be no checking on the building of the record update, letting garbage in.

I know that you are doing a major rewrite of the IQConnection Manager and I am looking forwrd to it, however I wish that there was an intermediate step that fixes some of the issues in the DTN IQFeed 2.3.0.2 client first - it was an incremental update to 2.3.0.1, and did fix several issues, however it never came out of beta. This would help in the transition to the totally new client which will have its own teething issues when it is released - I suspect it will take a few months to get all working correctly once released.

Sorry for the diatribe, but this is an important issue with me as data integrity is the number one parameter that is needed in a data communication system. Garbage from the exchanges is another issue, at least additional errors should not be added during delivery.

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Thanks David!

It certinaly isn't difficult to "un-watch" and "re-watch" a ticker once I've detected that things have gone off the rails. Do you know if there's a way to programatically monitor the Connection manager static for re-request attempts?

I think we're talking about the same problem, but on closer examination, I'm seeing what appears to be attempts to correct messages. For example, in my Feb 28 error log first I see 8 messages with bad last & low info:

Q,QQQQ,F,.SBUX,-0.47,,76067208,800,37.6000,.SBUX,51.4400,51.4600,500,2800,175,175,37.6,14:50t

(notice it also appears to have picked up the bid/ask for SBUX also)

Then I received 29 messages where the last appears to be corrected, but the bid/ask still for SBUX:

Q,QQQQ,F,37.1500,-0.47,-0.01249335,76067608,400,37.6000,.SBUX,51.4400,51.4600,500,2800,173,175,37.6,14:50t

Then I get a message where the bid ask appear correct, but the low still contains ".SBUX"

Q,QQQQ,F,37.1500,-0.47,-0.01249335,76095108,1000,37.6000,.SBUX,37.1500,37.1500,8700,100,,175,37.6,14:50b

and several thousand more messages which follow the final format.

The last error I have from this data set is time stamped 15:27o, so it would appear that somehow it fixed itself up (unless I lost quotes for QQQQ althogether).

Today, March 2, QCOM derailed at 11:15t, and the last error was time stamped 14:37b. A dump of the corrupted messages is 29.7 MB so I'm sure it's several thousand.

Ned

Lonnie:

Do you agree that kick starting the corrupted ticker is the best action to deal with this situation?

Would it be better to send a "S,DISCONNECT\n" and let my app reconnect and submit all the tickers?

Or is there another recommended action to put things back on track?

Thanks,

Ned

David & Ned,

I too am seeing corrupted data messages (e.g. letters in the last field, like Ned originally reports, among many other errors).

IQ, if you are reading this thread, please correct this!

I disagree with you, David, however, that going to UDP is a good idea. It is absolutely essential that I get messages in time sequential order as they occurred in the market, and so if you do no use TCP then somone (either IQ in their client, or even worse, you in your custom code) would need to write all that code which is a pain and bug prone. You may as well stick with TCP which does that for you.

But what I do not understand is the apparent current behavior with IQ's client code.

Why, if the iq client is having probelms keeping up with the server's data, does it send me corrupted messages? My code cannot proceed if it fails to parse it, so I have to drop the message anyways, so why does their client not drop it in the first place?

As near as I can tell, IQ currently does an extremely bad job of doing any data integrity and syntax checks on the messages that they send out. They need too. And they should drop messages that are corrupt due to being partially filled.

No there is not. The IQFeed client has no real status queries that you can get. The Re-request count is sort of a bellwether that grows mainly on initial connection. With a really bad connection the count increases during operation. Normally, once connected it remains constant. Mine is at 3 now and has been running overnight, over 15 hours since last connection. I have seen it in the 1000's on a bad wireless connection - there was a lot of data errors too from IQ too during this period.

As to your questions on the data in QQQQ I suggest that you implement the unwatch/rewatch first and then see if the subsequent data is indeed cleared up. I have no idea as to once a symbol is corrupted if it can clean-up its own fields. One would think so for the active fields like last, volume, etc. and you see this in your example. However, the data you see in LOW does not go away - I have seen similar messages where both alpha and numeric data were in wrong places. I have no knowledge of the logic of how the client updates records. My guess is that it is elementary at best. DTN has been aware of this corruption and the new re-write of the client is what is required to put it to bed. Evidently there is no simple work around for the existing client software.

Take a look at your internet connection. I use a program called PingPlotter - there is a free version available, and $15 for the full one. ( http://www.pingplotter.com/) It shows the paths and intermediate hops to any URL, latency, and packet loss. The chart it produces is great as it is not only a monitor of connections status but show when packets get lost. If you are using wireless, ping first to the gateway address on your router. I had some RF interference the other day that caused packet loss in my local wireless connection. The internet connection was fine, I did not know that my own setup was the culprit. Once fixed, no data errors from the DTN Client. If you have packet loss (use PingPlotter or similar to check) then go after the culprit. At the moment this may be the only solution for you. You might also want to trim the number of symbols in you watch list too - I have to do this for dial-up connections. It just adds to the bandwidth burden and the issue that causes corrupted data.


Good luck!

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

I appreciate the feedback.

I didn't think there were any hooks into the Connection Manager, but one never know about "undocumented" features, eh?

I don't think it's too many tickers. Only watching 59. 28 are stocks, 3 are e-mini futures, and the rest are indices.

Also doubt it's the internet connection, which is SDSL 1.5x1.5 but I can't rule it out completely. I can flip it to a "real" T1 I have access to in my computer room. No wireless in this setup.

I suspect I've received corrupted data today, but won't know for sure until I do the evening run on the data file. Currently, both Reconnections & Rerequests show 0. Avg KB/sec is 47.11, but we really want to know what the peak is, don't we?

jazzfusionb



Just to clarify a little, what I said:


I don't advocate either - just the treatment of packets coming in a manner that discards bad records. UDP does have advantages however. I do not expect or want DTN to do a redesign.

The discussion we have been having relates to the connection between the DTN client and the DTN server, not the connection (TCP/IP or COM) to the 3rd party application connected to the DTNClient. I happen to like the TCP/IP internal connection - an ideal use of TCP as the internal network is high band width and practically errorless, and I don't think prone to buffer size issues that can cause packet fragmentation over a network.

But then again... what do I know!

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Hi, just thought I'd put my 2 cents worth on this recurring topic of corruption.
The best way to handle it is definitely to unwatch/rewatch the specific symbol. When DTN/IQFeed talk about corruption occuring when the client side can't keep up with the server you have to understand where the bottleneck is actually occuring.
For example, I have a very fast (>10Mbit/s) Internet connection and an equally fast PC (dual xeon 3.2GHz) but still got many corrupt packets when watching 1300 symbols. At market open, the CPU utilisation was <50% and download speed was around 800kbps so why the corrupt packets?
The reason is the IQFeed client has very small buffers available and, it would appear, that each symbol has its own buffer. If any of these overflow, that specific symbol becomes corrupted permanently until it is unwatched/rewatched. In my case, I had to write a separate thread to read the streaming socket data from the IQFeed client very quickly and then to make this data available to my application in a buffered fashion. By quickly, I mean no more than 50ms of data should be queued by the IQFeed client when watching a large number of symbols (1300 NASDAQ symbols is approx 2000 messages per second at open). This has solved 99% of my corruption problems.

There are other types of corruption that can occur, however they are less common. For example, when I had some intermittent internet problems, I started receiving updates that had mixed data from two symbols! Fortunately this doesn't occur in normal circumstances.

cheers,
Dennis

Wow, I sure do appeciate all the feedback on this topic.

What's most disturbing to me is the trend I see where things gets de-railed:

Jan 4 & Jan 20.
Feb 10, 15, 22, & 28.
Mar 2 & 3

Last recompile on my app is Feb 8. I don't recall seeing any instances of this happening in the first few minutes of trading, when volumes are usually high, or even in the first hour or so when various data reports (Home sales, PMI, Sentiment, Consumer Confidence, etc.) often cause a spike in volume. I'd have to do further analysis to see if it seems to be related to other volume spikes.

I'll add the un-watch/re-watch code this weekend, which seems like a reasonable corrective action.

Dennis:

Thanks for the additional insight. I had not considered fully the interface to the clients that attach to the connection manager as an additional source of bad data. I can understand that delay in processing the quote stream could cause momentary data corruption due to a buffer overflow, however I would expect it should not reflect backwards (so to speak) and corrupt the internal record of the DTNClient. That would imply no buffering or spooling of the quote data then. Some how the record is the buffer. I suspect that what 'really' happens (just guessing...) is that the processing delay can cause internal sequence issues and records are not built/updated correctly, causing corruption in records that may have nothing to do with the symbol that was being sent.

I do know that the history data is spooled. The major issue I found with history is that it will interleave data if the first request is not completed and another given. While the data record format is not corrupted during this simultaneous operation you cannot correctly construct the file as there are no record ID's to separate the data so the result looks like 'bad data'.
Thanks!

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Hello All!

I have a situation where a DDE link into excel and the DTNIQ SnapQuote are refreshing substantially faster than the application that I have written on the same symbol. Has anyone seen a similar problem?

Is it reasonable that by not emptying the buffer from the DTN client fast enough I am causing this delay and thus by reducing the overhead in the routine I can cure the "delay problem"?

The code is currently in VB.Net using the IQFeedY. Any opinions on whether I should switch to the dll (I think IQ.dll) with VB.Net? with C++?

Thanks for any help or suggestions,

John