About 8 of my 13 symbols are quoting 0.0000 and are not changing. Please fix immediately. I'm on the 148 servers.

Best regards,
MK
Edited by midknight on Sep 3, 2007 at 11:06 PM

Is anyone working in DTN's 24 hour data integrity team, or are you guys asleep at the wheel? 1 hour after reporting and data remains the same as when it was reported.

Fix the large spikes down to 0.0000 in the affected pairs please. Daily and intraday data is affected.

As requested by Jay in the thread 'DTN Reliability' from memory some of the affected symbols were BUSDCAD, BAUDUSD and more which I cannot remember. I did a complete data refresh yesterday morning when I made the previous post and told neoticker to force an overwrite of the previous days data. Look at the intraday charts, 60 minute timeframe would suffice, but it should show up on any timeframe.

I've manually edited out the 0.0000 spikes in my local data, that is why I cannot remember exactly all the symbols.

This is where the problem is... we can't replicate it. I checked both server farms, and here is the chart of the data direct out of the server. I know there has been an issue in the past with NT not truly clearing the data when you tell it to. Is it possible that is still happening? I will need to check, but I don't think we record zero prices to history. Thus, if Barclays sent zero prices in the feed, and NT recorded that on the fly, that is why you are seeing it but we don't have it in our history files.

Earlier today, I brought up intraday charts on every barclays spot price and didn't see any zero prices over the past 10 days.

---

Jay Froscheiser
DTN - Trading Markets

If the zeros are still being seen, would it be of value to turn on local logging to see what the local iqfeed client sees? Then the data record can be evaluated to see if neoticker is tripping over something else in inbound data.

It's possible this could be a neoticker issue then if you cannot see it on your end. I won't be able to try it again until tomorrow when I restart the computers before starting my session. I'll purge my intraday cache on BUSDCAD prior to starting neoticker.

Frustrating to debug for me. The first assumption I make is neoticker stores the data it is given, thus I always assume data problems are the fault of data vendor. My sincere apologies to DTN if this turns out to not be the case.

If this turns out to be a neoticker problem then it means a simple restart might have fixed my problem when it first occurred. It also means I wasted a perfectly good evening of trading :(

Thanks for the comments stargrazer, I'll try this tomorrow if the problem persists.

I cleaned out my BUSDCAD minute cache before starting up this morning and requested a full historical data backfill for the weeks data. Low and behold, the 0.0000 data point was not there. My sincere apologies DTN for ranting about it without checking neoticker first. Neoticker does indeed store the data it is given, as those 0.0000 quotes must have been received by it somehow, however when I requested that it overwrite the previously collected data it wasn't doing that. I suspect this could be a problem with the RAM cache vs. the disk cache. Not sure, I'll contact them on this point later today. I just wanted to express my apologies for venting frustrations here.

Best regards,
MK

Midnight,

You don't need to apologize. We are here to support our users. Unfortunately, some issues are out of our control, and it is our job to point those issues out. In the end, you received data with zero prices in the live feed. These obviously aren't valid prices and we need to be better about filtering them. Luckily, we do filter zero prices as we write it to history so users can reload with clean data. If we were to filter the data on the fly, we run the risk of adding latency to the feed, which we don't want to do. Believe it or not, we actually started filtering zero prices a few years ago due to stock market data the exchanges sent with zero prices. Data is a "dirty" business and we are constantly challenged with finding ways to efficiently clean it up.

As we have discussed, the best scenario is to find a feed that provides cleaner data from the start...

---

Jay Froscheiser
DTN - Trading Markets

Data is a "dirty" business, I like it Jay.

Yes, finding a quality feed is the best solution to find. Until then though, I can filter the data to not receive 0.0000 quotes from the live feed I think. I'll look into that this weekend.

Best regards,
MK