Our Telco provider was able to isolate a bad network that is causing our Eurex delay issues. They are planning to re-route our connections around this while they repair it. The re-route will cause our connection to the exchange to drop for a few seconds and is currently scheduled to take place early this afternoon. You may notice a brief interruption in the Eurex feed as this route change occurs.

More to come...

---

Jay Froscheiser
Telvent DTN

Thanks for the update, I really appreciate the fact that you guys are working hard on it. I know you were before but with the frequent posts and twitter updates it really lets me know you're doing your best.

Hi Jay, Agree with last comment. The frustration I and alot of others felt yest. was that we had raised this issue last thurs morning and 4 working days later it appeared nothing much was being done to resolve it. Also I was unaware of the forum and updates here. The response on e-mail was minimal, just saying their is a problem. Since yest the updates on here and the fact you are posting the link on twitter has helped keep everyone up to date. Some tech issues are out of your control, I for one understand this can happen. As long as you are working hard on getting it resolved and making progress is what we need to hear. Hope to get everything back 100% very soon, Tks.

I agree, I share any word

Hi Jay, thanks for keeping us updated as this issue unfolds. Can you please provide confirmation as to whether the re-routing of your connection went ahead this afternoon? Thanks.

The telco provider was able to reroute our connection to Eurex and things are looking good for market open. Unfortunately there isn't a good way to test without the high volume of market data coming through the feed. Our staff, along with engineers from the telco provider will be monitoring things (as I will) in case there are continued problems.

I will post again once the market opens with results of what we are seeing in our monitoring.

---

Jay Froscheiser
Telvent DTN

Hi Jay,
unfortunately problems are still continuing this morning

Hello,

right now lag is +4 minutes approx. and growing. It gets worse as the day goes on, kinda of accumulative lag. So, it seems that the problem is still there.

Yes, we are seeing the delays still coming in on our Eurex feed. All hands on deck here with the Eurex and telco company to dig further. I will post again as we learn more. Obviously we are as disappointed as you all are and will continue the pressure.

---

Jay Froscheiser
Telvent DTN

Yes, the delay is 4 minutes by now and growing. I'm losing my patience.

Ye, very frustrating this proposed temporary solution didnt work, Need this resolved soon guys.

Are already working 6 days with the problem, the unfortunate thing is that monthly bill to be paid, have not delayed

---

el sevillano

One of my two accounts and I have real-time Eurex, the other continues to lag

---

el sevillano

Sorry for the delay in updating (elsevillano is a step ahead of me). Further configuration changes were made on only one of our sites a few minutes ago and as of now it appears this site is not seeing the delays. I was waiting for a few verifications before we point customers to this site. While this site is not seeing the delays, it does have a hole in tick and minute data for approximately 20 minutes. We will work to patch that.

More to come shortly...

---

Jay Froscheiser
Telvent DTN

I confirm since 20 minutes real time feed

works well one of the two accounts, thanks

---

el sevillano

How do we get on the working server? Is it random?

we are still verifying some things on our side. Definitely not an all clear on that site yet. The telco company is still running tests and likely will make further config changes which will cause hiccups in the data (and thus holes). Of course, I would rather be sending real time data and deal with filling in the holes later.

NOTE: Filling in holes will have to be done the next day in order to not corrupt in-memory databases.

---

Jay Froscheiser
Telvent DTN

I, for one, will wait until the whole thing is fixed; my ATS relies heavily on the feed's consistency, and a hole-filled, gruyere cheese-style feed is perhaps not the best thing to feed my ATS with...

I'll definitely wait for the feed to get fixed, yes .

I disconnected and reconnected and now I don't see a lag. So apparently I "fell" on the working server.

The hole is only a slight nuisance and quite acceptable IMHO.

We have made changes to the login systems of IQFeed to direct new logins to the site which isn't showing delays on Eurex due to the configuration changes. If you still have delays, please log out and back in.

Please note that telco will be making additional configuration changes as they monitor things, and we are told to expect at least one more drop of the Eurex feed to both sites. This will cause the feed to stop and restart automatically for you, but will create a hole in historical tick and minute data. Unfortunately we don't have a way to fill in the holes in that data until the following day due to the nature of how we store current day's tick and minute data.

I fully understand that holes in the data is unacceptable, but it is a necessary evil in order to resolve the delays.

---

Jay Froscheiser
Telvent DTN
Edited by DTN_Jay_Froscheiser on Feb 2, 2012 at 03:59 AM

Even on the "good" site, we have seen delays build, then clear. Nothing like what we are seeing on the other site, but still not perfect. Configuration changes will continue so you will likely see some sporatic stoppage of data on Eurex as these are put into effect.

---

Jay Froscheiser
Telvent DTN

Hi Jay,

I'm noticing differences in the 1 minute chart especially when the market moves quite fast, are these the holes you are talking about ? Please keep up the pressure, its still not good enough.

Tomorrow is non farm payroll and ISM, so very fast day, i hope the config changes will be done by then and no more lag

---

MD

Agreed mishka. Trust me, there is no such thing as "good enough". We will be working on this until it is back to normal!

The holes I mention will be in the backfill data/history.

I am currently waiting to hear from the telco company as to when they will be making further config changes, but I have given the go ahead to do so "immediately" even though it will create additional holes in history (and user will see a temporary stoppage in stream). This gap in the stream I am told could last up to 5 minutes.

We will be working to fix the holes, but we won't be able to do so until the following day.

---

Jay Froscheiser
Telvent DTN

the minimum volatility ... Eurex data is delayed again

---

el sevillano

elsevillano,

Yes, we saw the delay build with the spike in volume for the last couple minutes. Telco is making additional changes and you will likely see a drop in the Eurex feed while this occurs.

---

Jay Froscheiser
Telvent DTN

additional configuration changes were made, and so far things look promising as the queue is dropping (on the site that was still delaying data). This is not an all clear, but things are looking better.

---

Jay Froscheiser
Telvent DTN

I have not logged out/in and the feed has caught up for now.

A little volume seems to be missing compared to IB's feed.

All delays should be gone from Eurex, but we are still seeing signs of very slight packet loss in a connection on one of the sites. The telco provider continues to work on making sure everything is clean throughout but data should be flowing real time now.

---

Jay Froscheiser
Telvent DTN

Hi Jay, Are we anticipating any problems for the European cash open tomorrow or do we just have to wait and see. Tks.