dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 19, 2004 01:30 AM
Msg. 1 of 17
I have noticed that the historical tick data does not match the ticks I have collected in real time. I know that the real time ticks are complete because there is a running aggregated volume per symbol as well as a counter (per symbol). However, there are often additional trades in the historical data (never the other way round).
Where are theses trades coming from? They don't seem to exist in any form in the streaming data.
Dennis
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 19, 2004 08:34 AM
Msg. 2 of 17
Dennis,
If you could give me an example of a symbol you see this happening on, I can look into this.
Thanks,
Tim Russell Software Engineer DTN IQ & FinWin
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 19, 2004 10:08 AM
Msg. 3 of 17
Tim, An example, Date: October 12, 2004 symbol: IACI timestamp: 13:00
The historical trades include three trades in a row of 2200, 2460 and 2800 shares. These trades do not exist in the real time data. In the data below, the first three trades are okay and then things go wrong:
REAL TIME DATA 2004-10-12 13:00:25,IACI,20.60,20.60,20.60,600,3046534,4663 2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046634,4664 2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046734,4665 2004-10-12 13:00:30,IACI,20.59,20.60,20.60,100,3046834,4666 2004-10-12 13:00:33,IACI,20.57,20.58,20.58,100,3046934,4667 2004-10-12 13:00:38,IACI,20.57,20.58,20.57,885,3047819,4668
DOWNLOADED HISTORICAL IACI,0,20041012,13:00,20.600000,600 IACI,0,20041012,13:00,20.600000,100 IACI,0,20041012,13:00,20.600000,100 IACI,0,20041012,13:00,20.600000,2200 IACI,0,20041012,13:00,20.600000,2460 IACI,0,20041012,13:00,20.600000,2800 IACI,0,20041012,13:00,20.590000,300 IACI,0,20041012,13:00,20.590000,200 IACI,0,20041012,13:00,20.590000,885
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 19, 2004 10:28 AM
Msg. 4 of 17
Dennis,
Looking at our stored historical data vs what you sent, I see that those trades received were regular trades. Moreover, the volume on these trades (which is part of the historcal retrieval) is higher than the volume you have.
The obvious reason for this is that data is being dropped passing between our servers and you. The "why" of the dropped data is what we need to pinpoint now; I'll need a bit more information from you.
- What type of connection are you using to the Internet?
- What type of machine is your application running on?
- If possible, can you send me a private message with the list of symbols you're watching? This will give me a better idea of the bandwidth involved.
Now, some background on why these things can impact your data reception.
Our quote servers, after upgrades last month, have proven to be more than capable of keeping up with our data feed and transmitting it to customers in a timely manner. However, memory on these machines is not unlimited - we cannot queue data on our end waiting for the remote clients to be ready to receive it. Doing this would put our servers at the mercy of any client who wanted to break our servers just by not receiving data for an extended period of time. Timely (and complete) delivery of data is entirely dependant on the remote end's ability to read the data fast enough.
While I'm not certain what symbols you are watching, with 1300 symbols being watched, I'm pretty sure this is generating a significant amount of data.
Get back to me with your information, and we'll try to work from there.
Thanx,
Tim Russell Software Engineer DTN IQ & FinWin
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 19, 2004 08:06 PM
Msg. 5 of 17
Tim, My trading depends on a high level of liquidity so the 1300 NASDAQ symbols I am watching are the top 1300 ordered by average traded value excluding the top two symbols (INTC & MSFT). This still leaves me with plenty of very busy symbols. (I will send my list via email)
In answer to your questions: - Internet connection is high speed cable (in Australia). I can easily sustain 7mbps downloads. Ping times to your login servers are around 200ms - I am running the app on an HP xw8000 workstation with dual 3.2GHz Xeons, 4GB RAM, 15,000rpm SCSI drives, etc Windows XP SP2. IQFeed client is v2.3.0.2
I am intrigued by your response. The real time feed had no missing counts. Does that mean that these are generated by the IQFeed client rather than your server? BTW, the historical data was downloaded by a separate application called Dynastore rather than our application.
At present, my Java application is simply receiving the data stream and writing out the data to disk. At market open the download peaks at 5.5Mbps and then settles down to around 2Mbps. The strange thing here is that the amount of data written to disk is only slightly larger than what is being received. This implies minimal compression from the servers. I have also had problems with data corruption (missing fields, truncated records, and things like "U,F" in some fields) which may be related. There is a separate thread on this.
Dennis
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 20, 2004 09:15 AM
Msg. 6 of 17
- 7MB sustained should be more than enough - at 9am EST, I'm seeing output at just under .2Mb on your watchlist.
- Nice machine =) As I'm sure you guessed, no problems there.
- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)
Btw, as far as compression goes: I'm seeing 21KBytes/second on our server output, vs 160Kbytes/second (again, with your watchlist).
I'm actually suprised about the "U,F" you're seeing in some fields - this is something you should NEVER see - it's part of our server feed before it's converted to the text feed that IQFeed puts out. I'll pass this on to the IQFeed guys to look at.
Could be this message corruption is the root of your problem, though.
Thanks for the input,
Tim Russell Software Engineer DTN IQ & FinWin
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 20, 2004 02:08 PM
Msg. 7 of 17
Quote: Tim, Some of your comments concern me. I am monitoring the bandwidth between your servers and my PC and it is very high. After the initial burst (625KBytes/sec) I am getting around 300KBytes/sec sustained for the first hour. Last week I had an almost identical watch list and was receiving around 1/8th of this bandwidth which is closer to the figures you mentioned. The ratio of data downloaded to that output by IQFeed is around 1.2:1 which is much worse than last week. Clearly something is wrong. Even now (10:38am your time) I am getting ~180KB/sec from your servers. Is there any way you can determine what is being sent to me? (ID=200643) Dennis PS I'm still getting the "U,F|" Dennis - I've reposted your email here so that everybody can benefit. I was under a slight delusion as to how IQFeed handled outgoing data - basically I looked at code that has NOT been completed yet, that fixed a problem that is most likely causing your symptoms. Basically (and I hope you understand the basics of TCP/IP communication): - IQFeed sets TCP send / recv buffers to 256 Kbytes. - When incoming data is processed, a message is created for the client. - This message is sent to the client. No checking is (currently) being done to ensure that this message is sent in its entirety. This is fine so long as the 256k TCP buffer doesn't get full. - After the TCP buffer gets full, attempts to write to the socket will fail. - You will get corrupted data / lose data. The obvious problem here is that your code may not be processing messages from IQFeed fast enough. A couple of solutions: - Make your TCP recv buffer bigger. In java.net.Socket, I believe this is accomplished via the setReceiveBufferSize() method. - Read data from the incoming socket in one thread, and pass this data to another thread - via a queue or whatever - to be processed. Sorry for failing to notice this earlier. The recv buffer size fix is often offered, but I wasn't aware of why until I got one of our client guys to help me find the problem in the code. If this helps you, let me know. Now, for the amount of data you're seeing - are you getting this via the IQConnectionManager? If not, where are you getting these values? Only curious, as what I'm seeing going to your computer doesn't account for that much data. Thanks, Tim Russell Software Engineer DTN IQ & FinWin
|
skunk
-DTN Evangelist-
Posts: 249
Joined: May 7, 2004
|
Posted: Oct 21, 2004 08:26 AM
Msg. 8 of 17
"- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)"
Without having seen his code, I imagine thet he is using the field "Number of Trades Today" (field 56 in the tcp buffer) to indicate the sequence number of the print.
Where does this field come from? From the description of this problem it appears that it is generated by the IQFeed client, not the server. This makes it significantly less useful, if not useless. Edited by skunk on Oct 21, 2004 at 08:26 AM
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 21, 2004 08:32 AM
Msg. 9 of 17
Ah - trades today. Okay.
That value's maintained by the server - it's returned to you when you request it. However, it's not transmitted on a tick-by-tick basis, since it's easily calculable on the remote end.
Tim Russell Software Engineer DTN IQ & FinWin
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 21, 2004 08:54 AM
Msg. 10 of 17
Okay......you can forget about all my concerns to do with high bandwidth as it was my fault. My application logs just about everything and I recently changed the location of the log to a networked drive. The bandwidth measurements were being skewed by this.
Tim, I have to agree with skunk on the value of the counters. If you can't tell when you've missed a trade then what's the point. If you look at the real time data example I gave earlier then it appears that I haven't missed anything. Clearly the IQFeed client wasn't aware either.
I've made the changes to the way I read from the socket connection and increased the java buffer sizes as suggested. I now have a separate thread that takes the data stream from IQfeed and place it in a separate buffer for the application to read. It seems to be working very well with only a small number of error packets so far. I'll analyse it better at the end of trading. I am somewhat stunned that a buffer overflow can result in corrupt data. I can understand missed message or even truncated messages, but corrupt individual fields within messages???
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Oct 21, 2004 09:26 AM
Msg. 11 of 17
The trade counters were originally for internal use, and used by our FinWin product - since it's a web service, it doesn't do streaming data, and this information is very useful to those customers. The reason we made this available in the streaming feed was because there seemed to be a bit of interest in it.
However, it is NOT designed to be a method of letting you know data has been dropped in the feed.
As for the corrupt messages - let me know what you see.
Tim Russell Software Engineer DTN IQ & FinWin
|
skunk
-DTN Evangelist-
Posts: 249
Joined: May 7, 2004
|
Posted: Oct 21, 2004 10:59 AM
Msg. 12 of 17
So, to summarize (and please correct any misunderstandings)
As of 2.3.0.2 and earlier there is NO flow control between IQConnect and API socket clients. The assumption is that the client will empty the receive buffer fast enough so there are no buffer overruns.
The "num trades today" field is useless to detect missed update messages beween your server farm and IQConnect. It might be useful to detect lost messages between IQConnect and the client app though.
Is there any way to detect update messages lost beween your servers and IQConnect?
And yes, this is a thinly veiled request for server (or even exchange) generated timestamps and sequence numbers to be transmitted to API clients.
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 21, 2004 10:02 PM
Msg. 13 of 17
Okay, I've now done the analysis of all data received from IQFeed (thank god for Linux utilities) and have mostly bad news. At the start of market open, I received an average of 1400 messages (trades,bids,etc) per second and had no problems at all. This is the highest data rate for the day and suggests that there are no buffer overrun problems. However, I received a large number of invalid messages starting at EXACTLY 12:00pm. Now if the corruption was due to buffer overruns or lack of bandwidth or CPU then you would expect all symbols to have corrupt messages. The below analysis of these invalid messages is very telling: Entries containing the string "U,F|" in place of expected data: 1 ADEX 1 AMZN 14821 BMET 1 BRCM 2001 CMVT 1 CNCT 2 COCO 186 EBAY 1 EMMS 6 FAST 1 FHRX 1 FRED 785 FXEN 1 JAKK 1 MLNM 2 MRCY 1 NCRX 4149 PMCS 1038 PWAV 1 PZZA 2 QLGC 13641 RSAS 1 RTIX 1 SAFC 2620 SAPE 1 TSAI 34868 VRTS 53309 YHOO ======= 127444 Trades containing more than 59 commas (invalid): 1 ADEX 1 AMZN 3391 BMET 1 BRCM 654 CMVT 1 CNCT 1 COCO 1 CSCO 169 EBAY 1 EMMS 2 FAST 1 FHRX 1 FRED 116 FXEN 1 JAKK 1 MLNM 2 MRCY 1 NCRX 1 PCAR 1510 PMCS 422 PWAV 1 PZZA 2 QLGC 4752 RSAS 1 RTIX 1 SAFC 816 SAPE 1 TSAI 13609 VRTS 18306 YHOO ======= 43768 As you can see, the 170,000+ errors affected only a handful of the 1300 symbols I'm watching. Some very busy symbols such as CSCO are not represented. Futhermore, the error in the message is in excatly the same position for a given symbol. eg Q,YHOO,F,35.3800,0.89,0.025804581,16679769,100,3537U,F|,34.9000,35.3700,35.3800,3400,600,,175,3502.1,12:56b ,,35.3700,34.4900,0.01,,,,p,N,,,,10/21/2004,,35.4000,,,,0.91,-0.01,90.8,-0.212661364,0.,0.,0.01,0,98.985302 431,48137214.260000005,14,4,,16009207,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,37928,,,35.3464, The above has a U,F| which makes it invalid. All YHOO corrupt messages have this in the same position. This seems to point the finger directly at the IQFeed client software. Perhaps it has separate buffers for each symbol........it's just speculation. TIM: you mentioned that you're more of a server developer. Can we get some input from a client side developer please. I am now heading off to the "IQFeed Data Wishlist" forum to ask Santa for my early Christmas present. http://forums.iqfeed.net/index.cfm?page=forum&forumID=15
|
DTN_Steve_G
-DTN IQFeed-
Posts: 28
Joined: Oct 1, 2004
|
Posted: Oct 22, 2004 08:58 AM
Msg. 14 of 17
Dennis,
Thank you for the analysis. I will forward this on to our software engineering staff who are already investigating the "U,F|" issue.
Steve Grunberg IQFeed Developer Support DTN Market Access
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 22, 2004 09:18 AM
Msg. 15 of 17
Steve,
You can also send them this one which I received today (never seen this before):
TL|=8800|>18.85|?1.46|@VOZ,YYT|B585.6|C28.9|D06/01/04|E172.5|F69586|H307000,20040423|I144400,20040830|J307000,20040423|K144400,20040830|k0.50,10/13 /99|}
It looks like the output to the log.txt file from the IQFeed client.
Dennis
|
dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004
|
Posted: Oct 26, 2004 07:49 PM
Msg. 16 of 17
Steve, Have you heard anything from the software developers yet? I'm keen to get acknowledgement that the problem is a bug with the IQFeed client and some indication of when a fix can be expected. To blame this problem on an application that is too slow or has insufficiently large buffers would be unacceptable as that should result in lost messages, not corrupt messages.
Dennis
|
DTN_Steve_G
-DTN IQFeed-
Posts: 28
Joined: Oct 1, 2004
|
Posted: Oct 27, 2004 12:57 PM
Msg. 17 of 17
Dennis,
The reported issue of the character string "U,F|" occasionally appearing in streaming data has been logged in our IQFeed API Bug Database as item #2750. You can monitor its status by going out to the IQFeed Developer website and selecting the menu item "View Current IQFeed API Bug Database.
Our software support team will need to investigate it to see if they can replicate its occurence, prior to resolving it. No estimated resolution time has been set.
Steve Grunberg IQFeed Developer Support DTN Market Access
|
|
|
|