Sometimes when requesting historical data it does not return anything - or at least doesn't appear to return anything. Maybe its still processing a very long query or simply rejected the request based on invalid parameters, but there is no immediate indication that the request is either in progress or abandoned.

I think any delay is most likely a very long running query such as 8 days of ticks for QQQ for example. But a few times I got the feeling the that it had ignored the request and maybe because it was running this after hours during some backend data processing period.

Will the !ENDMSG! tag always be sent back - regardless of whether the history request was valid/invalid or completed successfuly/unsuccessfuly?

Otherwise, what are the alternatives?

I wouldn't consider a timeout to be a valid option - this is so subjective and only puts more load on your end after getting frustrated and re-requesting until either something comes back, the DTN server gets upset or the user gives up and calls DTN tech support.

Lets say a 60sec timeout is long enough in most cases but today it takes on average 80sec to complete the request. So I keep aborting and re-requesting because the system is almost ready to send data back not knowing I just had to wait a few more seconds.

After how many failed re-tries should this become a problem warranting a call to DTN tech support?

Hopefully you can see how subjective this is without any immediate response - if it is even possible to provide an immediate response?

I'm just asking for info in understanding if I will always be guaranteed a !ENDMSG! tag regardless? In other words, sit tight and keep waiting....

Hmmm, now what is a sensible timeout...How can I distinguish between queries that take 60sec to return and queries that take 600sec to return something?
Edited by sasha on Oct 9, 2004 at 08:24 AM

This issue has been a recurring one for me too. I do have time-outs, or did have until I discovered another anomaly the knowledge of which may help others.

There are times that the return data comes back much delayed. If you have a time out and do a re-request there is a very good chance that you will see some 'interesting' data - IQConnect will return all requests interleaved if you use the same socket for the requests.

This behavior has caused me to waste a lot of tem in debugging an overnight backfill of historical data - I schedule an update around 3:OAK if the program runs overnight. (in an attempt to minimize server load) In my case, I did not do a re-request if no response for a request, but went to the next issue that needed historical data updates. Once one issue got out of the queue, then almost all of the subsequent retrievals came back with the data interleaved. Not a pretty picture... and not a systematic one so took a long time to catch the problem.

While I have not proven to myself that ENDMSG does not come back at all times, I am highly suspicious that it does not. I do not think IQConnect is 'smart' enough to make sure that the loop is closed for requests that are supposed to return data in some form.

Three issues here:
1.0 There is no confirmation from the client that a request has occurred. This is true for all requests except for the initial login sequence which has the appearance of a dialog There is not an abort or time-out message from the client if there is a time-out of the data server-side, unless one gets the 'data not available' with the first query. (that does comes back fast)

2.0 In the returned historical data, there is no ID in the data identifying it with the request. I have no issues with interleaved data as long as each record has an ID. It does not have to be the stock symbol, a simple token would be fine, maybe one that is embedded in the request so that the caller can track his own entries.

3.0 Historical data should come back in a serial sequence - if there happens to be a second requests that get sent before the first finishes the data for it should queue and follow the first (for the same port)

David

---

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

Thank you David for offering your experiences. This is very helpful to know.

I agree with resolving all the issues you mentioned.

In the mean time I can only think of opening up something like a pool of 50 socket connections and keeping state of successful and unseccessful requests/sockets (one request per socket) and booting the sockets that timeout/hang.