Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"Can I get another account from you? I am tired of ******* going down so often" - Comment from George
"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"I just wanted to let u know that your data feed/service is by far the best!!! Your unfiltered tick data is excellent for reading order flow and none of your competitors delivers this quality of data!" - Comment from Peter via Email
"This beats the pants off CQG, I am definitely switching to the ProphetX 3.0!" - Comment from Stephen
"I'm satisfied with IQFeed. It's the most reliable and fastest quote feed I have ever used. Although I'm a resident in China, it's still very fast!" - Comment from Xiaofei
"I just wanted to let you know how fast and easy I found it to integrate IQFeed into our existing Java code using your JNI client. In my experience, such things almost never go so smoothly - great job!" - Comment from Nate
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
"The people at Nirvana have very nice things to say about your company and I can see why! Price and service is a potent combination." - Comment from Ed
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
Home  Search  Register  Login  Blogs Recent Posts

Information on Various DTN Products:
DTN IQFeed | DTN ProphetX | DTN Ag | NxCore
Follow DTN_IQFeed on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »Product Support »POLLS & SURVEYS »Feature Survey Part 1 - Level 1 Data Feed
Author Topic: Feature Survey Part 1 - Level 1 Data Feed (25 messages, Page 1 of 1)

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 4, 2004 03:51 PM          Msg. 1 of 25
This is to all the 3rd party developers, and any users who want to make suggestions or comments.

Over the past several months, we've noticed that we have IQFeed clients that are watching a large number of symbols, unwatching them, and then watching another large set of symbols. This is done for entire markets, then repeated. After some research, we've found that some 3rd party API's from other data vendors have built-in throttles on this type of behavior.

We're decidedly reluctant to limit our data feed in any way, but we also want to provide consistent service to all of our users while minimizing the impact from one or two clients who put a lot of stress on the servers.

I'd like your feedback on the following, especially if your software meets the above description:

1. Why are you cycling through markets this way?

2. If a snapshot service were available without the streaming quotes, would you be interested in it?

3. If IQFeed had a pricing structure that allowed streaming data of increasingly more symbols, would you be interested in that?

4. What are your thoughts on introducing throttles on the number of symbols watched and unwatched in a given time period? Say, per 5 minutes, or 10 minutes, etc?

The more feedback we can get from this, the less likely it will be that changes we make will have an adverse effect on what you're currently doing. The goal here to make the service as effecient as possible while meeting the needs of as many of our customers as possible.

Thanks,


Tim Russell
Software Engineer
DTN IQ & FinWin

Edited by DTN_Tim_Russell on Oct 4, 2004 at 04:07 PM

skunk
-DTN Evangelist-
Posts: 249
Joined: May 7, 2004


Posted: Oct 5, 2004 04:37 PM          Msg. 2 of 25
"3. If IQFeed had a pricing structure that allowed streaming data of increasingly more symbols, would you be interested in that?"

Would the proposed pricing structure change also cater for clients that only watched say 100 symbols?

David
-DTN Evangelist-
Posts: 113
Joined: May 7, 2004

I'd rather be...


Posted: Oct 5, 2004 04:55 PM          Msg. 3 of 25
1.0 Cycling - While I do not do this in my application, I have been tempted to add it as I find many users wanting extended filtering / scanning of a greater section of the market than allowed by the 500/1300 symbol limit.

In my program I have added a link to your exchange stat pages (most active, gainers, losers, vwap, new 52 week high etc.) that displays the top twenty five from each exchange. They can then insert the lists into the program or cherry pick issues that pick their interest. This has helped bring a broader view of the market into the program without having to do it locally.

2.0 Snapshot - I highly recommend this addition, so that one can get a single full quote without it cutting into the symbol limit. I have a Quote Window in my application that lets one get a quote with the most relavant information displayed in a compact form that also lets them see charts of the symbol and retrieve headlines and news stories. This window also works on the 'Exchange activity' window mentioned in 1.0 above. Gives quick analysis on the spot with only one key stroke. I do have to play games on registering the symbol and then unwatching it if the program is near the symbol limit.

3.0 Streaming Data - I would love to see this implemented as this is the way the DTN satellite feed functions. In one of my programs (using the DTN real-time satellite) I have over 200,000 symbols that have come through and are in the database. The program is capable of handling the whole market and creates history files on the fly. A higher-end version can do real-time filtering of the whole market.

As a minimum, if you were to implement a steam after the market closes of ALL of the symbols sent alphabetically, like the recap on the satellite feed, this would go a long way to helping those that want to do market screening and tracking of a large symbol database. It also would provide a end-of-day data feed that would rival TC2000 as it is limited to 9000 symbols or so.

Also, I'd like to see the current streaming data from registered symbols send only the dynamic changes, and not the full record for each quote as the parsing does take time and it is repeated for non-relavant data that is static. Also, no junk out to the client if the data is garbage coming into IQ. This happens on slower dial-up connections.

4.0 Throttles - If this would help the feed then it should be implemented! I need what you have to work well - that is the major issue that I am faced with as most customers I work with use several feeds and hold my feet to fire if something is a few seconds late compared to another feed, or different in content. The throttle should be based on a group sequence of 'unwatched' in a row. I assume that most using this 'feature' are trying to update a large database and page through the symbol lists. You must have stats there that show patterns relating to this. In my case, I only use the watch/unwatch if near the symbol limit and the user is looking at pop-quotes on symbols not in the local database. The rep rate could be several per minute as one looks at the programs 'Exchange Activity' Window for example and wants more details.

PART II - History - In my current application I will be building history bars on the fly for interday data. However, I do populate the chart initially with a download from minute history for each new symbol charted. I do keep local history, but my biggest sequence/update issue is the market beginning end times for the data so I can create correct time-calendars that overlay/synch with the data. Market holidays are an issue too.

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393

IQXP Software
http://www.iqxp.com

sasha
-Interested User-
Posts: 54
Joined: Jul 21, 2004


Posted: Oct 7, 2004 05:29 PM          Msg. 4 of 25
Issue: Cycling (watching->unwatching) within 500/1300 symbol limit
Impact: Lots of stress on the servers
Culprit: One, two or a few clients

Do you mean just the real-time servers?

Is this a technical problem to do with locking overhead maintaining thousands of real-time subscriptions while pumping several megabytes per seond?


1. Why are you cycling through markets this way?
I'm not doing this but it would have to be because of the symbol limit when people are doing large live market scans.

For equities, I don't know why you'd want full Level 2 real-time depth bid/ask updates if you are wanting to scan the market.

Providing a new seperate simplified Level 1 streaming data feed and fundamental snapshots is the way to efficiently support market scanning. By Level 1, I mean a lean-mean top level message (no Level 2 updates) stripping out all the extra repeating fields in the TCP stream update message. You could later give this diet to your current Level 2 Update messages.

You should ask those people cycling how many symbols is enough? Surely a 2000 symbol limit would be enough or even 5000 if the problem on your end isn't bandwidth but rather subscription management/locking overhead.

This would be a killer service and get you even more business!


2. Snapshot service
I suggested this in a previous post http://forums.dtniq.com/index.cfm?page=topic&topicID=157. Efficient implementation would be on another server with support for optional batch requests (single or multiple symbols).

If you keep the symbol limits in force than this using snapshots is the only other alternative to get access to more symbols.

The proper use of this service is realy for instant refreshes of Level 1/Fundamental data and EOD batch summary updates.

Since the real problem is people wanting access to more symbols, then you should address this issue. (See point 1 & 2).

This feature is another killer feature to help you sell more subscriptions! Like David said, it would rival/exceed TC2000.


3. Pricing structure
How would this solve your unspecified server "stress" problem? It might stop people subscribing/ unsubscribing hundreds of symbols at a time, but if people want to scan the market than you should provide an additional service to efficiently address this - and blow away the competition with it!

Even if you had an internet 2000, 3000 symbol limit, maybe its just going to increase your overhead and "stress" your servers anyway because your now pumping more data than before. Depending on what the "stress" problem is, it might not solve anything.

What is enough symbols to stop people cycling? And what fields do they need for cycling/market scanning? Making a new more efficient Level 1 data stream would fix this.

If there are 200,000 symbols and 15,000 active equities, then what is enough - 2000, 5000?

If someone is paying for a internet 5000 symbol limit, then they are surely not using the full Level 2 bid/ask depth ticks in real-time analysis for all 5000 Level 2 symbols at once!

In most cases one would only want to have the full Level 2 depth stream after having done a top level scan using the lean-mean Level 1 data fields. These Level 1 fields would probably be limited to the obvious ones like last, bid, ask, open, high, low, close, volume, vwap, last/bid/ask size, last/bid/ask timestamp, tick id, tick type (bid/ask/ last price change and bid/ask/last size change), etc...

You can be sure that nearly all of these market scans are only superficial scans that do not concurrently subscribe to and analyse thousands of symbols and each of their tens/hundreds of bid/ask depth updates per second. Most people do not have the technical ability or hardware requirements let alone the bandwidth to support this full market depth streaming anyway.

I'm assuming this is why you limit your internet service to 1300 symbols? Do you have a limit to how much data you can pump out?

I don't have a problem easily receiving several Mbps because my cable is a full 9Mbit downstream (connected to 1GBit fiber) where most ADSL is between 256Kbit and 1Mbit and some cable is 3Mbit.


4. Throttling
If you don't provide a lean-mean Level 1 data stream, then this is your only option to limit 'abuse'.




Summary for new internet package...
Provide a lean-mean Level 1 data stream service. Make this new service the main DTN service with 500, 1500, 3000, 5000 symbol limits.

Provide Level 2 data stream service with 300 symbol limit or maybe even just 100 is enough.

Provide snapshot service with optional batch/multiple symbol requests (up to 500 per-snapshot).

$$$ Extras $$$
Pay for additional 500, 1500, 3000, 5000 Level 2 symbols. Put Level 2 update messages on a diet.


Some things that come to mind...
1. Exchange Tick Identifiers to support cancellations among other things
2. DTN Sequential Tick Identifiers to support synchronization between watch and intraday history requests (for charts, watch new updates and fill in the previous intraday ticks)
3. Exchange Bid/Ask/Last Timestamps (at least seconds)
4. Add MMID to TCPIP Level 2 Update messages


Just some thoughts

sasha
-Interested User-
Posts: 54
Joined: Jul 21, 2004


Posted: Oct 7, 2004 06:26 PM          Msg. 5 of 25
:-( Looking at one of Jay's replies to my question about MMID and Level 2 made me realize that MMID is already provided on Level 2 - Doh!

Some of the things I wrote may not make sense because I had Level 2 on the tired brain and I don't think this was your problem.

Hopefully you can pick out some bones that still have some meat on them.

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1741
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 7, 2004 10:15 PM          Msg. 6 of 25
The server capacity and bandwidth utilization isn't really an issue of "we are too stressed". It is an issue of providing an affordable service. Our services are priced so we can make a little money while providing our customers with a GREAT value. Unfortunately, some customers are getting "more" than their money's worth by circumventing the restrictions of their subscription level. We sell a top level 1300 symbol limit. This is assuming someone watches a max of 1300 streaming symbols, not watching 10000 symbols 1300 at a time. Our system is highly optimized to allow users to set a watch, and get all ticks on that watch list. It isn't meant to process thousands of watch/unwatch requests a second.

This is why we are asking. If people are wanting scans, maybe the best thing to do is ask what people are scanning. Maybe it would be better for people to be able to define a scan to us, have it put on the server, and only serve out results. We already have a most actives, gainers (% and $), Vwap, 52 wk h/l, type scans. We also have a custom scanning server which DTN.IQ users can send up to 3 criteria and our servers split out tick by tick scanning results. There currently is no IQFeed interface for this, but we plan to offer one. Unfortunately, but guess is everyone will want something different, so we may need to make modifications to make things a bit more flexible.

Hopefully this clarifies why the questions were posted by Tim. Again, our goal is to know as much about what you want as possible so we can continue to serve your needs. This information/communication is invaluable.

Jay Froscheiser
DTN Market Access, LLC.

sasha
-Interested User-
Posts: 54
Joined: Jul 21, 2004


Posted: Oct 8, 2004 08:29 AM          Msg. 7 of 25
Hi Jay,

Yes I think 99% of us would all agree that it is not in the spirit of fairness if people are watching 10,000 symbols 1300 at a time. Especially if this impacts on the service quality available to 99% of us good citizens!

And I understand that this is the main issue.

Our system is highly optimized to allow users to set a watch, and get all ticks on that watch list. It isn't meant to process thousands of watch/unwatch requests a second.

I thought this was the case because there is extreme overhead mantaining thousands of subscriptions per second.

People are wrong to assume that a symbol limit is a deficiency and try 'work around' it. If a client has a 1300 symbol maximum and they want to do market scans, then they have the wrong feed. The should pay for the Satellite feed or as you have suggested, pay for a new customizable DTN IQFeed scanning service if there are enough people wanting to do scanning to justify it.

I wouldn't have a problem with any Throttling if it was to prevent people abusing their limits. Personally I would never consider circumventing or 'working around' the 1300 symbol maximum because a) it is what I have payed for, and b) writing code to circumvent it is a waste of time increasing headaches and ambiguity about how many symbols the 3rd Party Developer actually supports.

A snapshot service would be still be an essential feature to compliment the existing service and place DTN as the best market data provider. It would still provide valuable features that when used responsibly could allow people to efficiently snapshot EOD data or instant quotes/refreshes - but not for real-time market scanning, just semi-real-time. To prevent abuse only during market open hours, you could limit this to a maximum requests per second such as 500 or six hundred (S&P500 + some).

Hypothetically if you had a seperate very lean-mean Level 1 feed, what would the upper symbol limit be if a) there are no issues with bandwidth on your end and b) it had throttling protection? Could you "rob Peter" (lean-mean data) and "pay Paul" (pump up the data volume)?

nsolot
-DTN Guru-
Posts: 273
Joined: Sep 4, 2004


Posted: Oct 8, 2004 09:00 AM          Msg. 8 of 25
I agree. You can/should do what's needed to protect the integrity of the data feed.

Perhaps what these people need is a CNBC ticker style feed which only contains ticker, last price and size?

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1741
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 8, 2004 09:10 AM          Msg. 9 of 25
As I was ranting last night (and tired I might add), I really didn't finish my thought. We have plenty of bandwidth. We have plenty of server capacity. The issue of value for the money has to do with the price we pay for that bandwidth and server capacity, as well as other fixed costs in our business. We obviously price our services so we can make a little money. This is based on an average user watching X number of symbols, that are using X number of bytes of bandwidth. We calculate a fixed cost per customer based on these "assumptions" and service levels. If the service level increases above our assumptions, the profit model breaks and we either have to raise prices or go out of business. I prefer to not do either.

You are right that the best solution for someone who wants to scan the markets or watch more than 1300 symbols needs our satellite feed. That is the best solution, and it is available today. Because this is already available, we have to determine if it is worth our time to work on allowing the same functionality on our Internet feed (because of a few people), or to put resources on other things that have more general market appeal.

It all comes down to customer damand, and that is why we setup this forum and ask the questions. Based on feedback, we will be able to decide where the resources are allocated.

Jay Froscheiser
DTN Market Access, LLC.

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 8, 2004 10:11 AM          Msg. 10 of 25
I really think sasha's on to something with the snapshot server. Based on the seeming demand for this - and talking to Jay and Lonnie - I'm integrating this new functionality into our quote server re-write. The quote servers most likely can handle the load, as there's not a lot of management involved with the snapshot request. If there's a problem, the functionality can easily be moved to a stand-alone server.

I'm not done digesting the other suggestions here yet - but I'll respond to them after I've had time to look at them and talk them over with the other developers (and Jay ).

Thanks for the feedback so far, and keep it coming!


Tim Russell
Software Engineer
DTN IQ & FinWin

Edited by DTN_Tim_Russell on Oct 8, 2004 at 10:11 AM

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 8, 2004 03:50 PM          Msg. 11 of 25
Quote: 2. Snapshot service
I suggested this in a previous post http://forums.dtniq.com/index.cfm?page=topic&topicID=157. Efficient implementation would be on another server with support for optional batch requests (single or multiple symbols).

If you keep the symbol limits in force than this using snapshots is the only other alternative to get access to more symbols.

The proper use of this service is realy for instant refreshes of Level 1/Fundamental data and EOD batch summary updates.


Sasha & David, the new snapshot requests will be through the same quote server, but won't be impacted by symbol limit. The IQFeed API, however, will need to be implemented to handle the new command to the quote server.

Quote: Also, I'd like to see the current streaming data from registered symbols send only the dynamic changes, and not the full record for each quote as the parsing does take time and it is repeated for non-relavant data that is static. Also, no junk out to the client if the data is garbage coming into IQ. This happens on slower dial-up connections.


David: Data from the IQServers to the customers is compressed. Are you talking about the data coming out of IQFeed?

Thanks for the feedback so far - this definitely helps us to know if we're heading in the right direction or are totally off base with our plans. If you think of anything else, feel free to keep this thread going or start a new one (I monitor this forum a little too much ).

Tim Russell
Software Engineer
DTN IQ & FinWin

David
-DTN Evangelist-
Posts: 113
Joined: May 7, 2004

I'd rather be...


Posted: Oct 8, 2004 04:58 PM          Msg. 12 of 25
Quote: David: Data from the IQServers to the customers is compressed. Are you talking about the data coming out of IQFeed?


Data from the servers to the IQConnect Manager are indeed compressed and from what I see in looking at the incoming packet message content also incremental in that only dynamic data updates. I am speaking about the data coming out to the client from IQ - every quote has the full record sent which obviously includes the static information (company name for example). While most of the price/volume data is contained in the beginning section of the quote record, some dynamic items like number of trades and VWAP are in the last fields so I continue parsing to get these. I'd like to see a variation on the qoute stream that lets it be as is or one that does a tokenized format so that only data changes get sent for a given symbol in a quote update. The satellite feed does this to some extent although not tokenized.

I am always looking to optimize my system relative to speed. My comment would not be number one in priority for implementation - I would much rather see a streaming recap sent at the end of the day that covers all stock market symbols. This lets me create a and maintain a database on the whole market that I can screen and filter for alerts.

Thanks for the question!

David

IQXP Software
http://www.iqxp.com

LiveWire Update Service
PO Box 1417
Fairfield, IA 52556
641-472-8393
http://www.livewire-cablesoft.com/

taa_dtn
-DTN Evangelist-
Posts: 114
Joined: May 7, 2004


Posted: Oct 8, 2004 08:51 PM          Msg. 13 of 25
(Sorry I'm coming into this late -- I've been out-of-town.)

Tim, are you specifically concerned about watchlist cycling that occurs during market hours, or at other times as well? (My app does watchlist cycling after market close in order to pick up fundamentals for screening.)

David's suggestion for an end-of-day streaming recap strikes me as a particularly good one.

Allen

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 8, 2004 11:01 PM          Msg. 14 of 25
Allen -
For the most part - and within reason, with a dash of common sense applied, and is within the scope of whatever agreements / contracts you have with DTN - we don't care too much what's done after market with regards to data retrieval as long as it's not abusive and detrimental to other customers. I haven't seen any application behavior to date that hits the system that hard after market hours.

Our big concern here is, as Jay pointed out earlier, the attempt to circumvent the symbol limitations built in to the system to get realtime data for a vastly larger set of symbols - in a way that ultimately uses MUCH more bandwidth than if we just had different symbol limit subscriptions and let them watch them that way.

Jay also mentioned customizable scanning service on our end - which, if this is what people are attempting to do, would be a better solution all around.

The problem (and the reason I brought up these questions) is that I can only see WHAT's being done, not WHY. If the WHY is good (financially and technologically feasible), I'm 100% for attempting to take some of the strain off of our 3rd party developers by doing this on our end. Otherwise, there may be other solutions we can look at to meet your needs.

Keep in mind, I don't make the sales decisions or, ultimately, the development decisions - I'm just attempting to identify the problems and the needs that you guys have for our data feed.

As for the EOD recap, I'll talk this over with Lonnie and Jay - it seems like a decent idea to me, but there are some issues that'll have to be worked out if we decide to go forward with it.

Thanks again,

Tim Russell
Software Engineer
DTN IQ & FinWin

lvarga
-Interested User-
Posts: 12
Joined: Oct 7, 2004


Posted: Oct 11, 2004 04:23 AM          Msg. 15 of 25
Came late to the discussion, and feel need to add my comments.

1) As I was testing some new ideas, I needed market scans to test the market
sentiment and did cycle through a set of symbols, but had no idea it was a
type of an abuse. I agree, you should try to keep the current level of
integrity of the feed and prevent abuse. I will look other ways to get the
data I need. (I would probably subscribe for the satelite feed if it were available here,
but I am in Europe, so it is out of my reach.) But to make it clear, the
point was not to abuse the symbol limit but solely to get a snapshot for
a sub-market.

2) A snapshoting service would solve my problem right away so I am for it.
Even if I had the possibility to watch 3000+ symbols I wouldn't do it. I
would just subscribe for a set of symbols (for an industry) and unsubscribe
after the first update -- there is no need to watch all the stocks all the
time.

Also, I would certainly subscribe for the customizable scanning/snapshotting
service Jay proposed. That is what I've implemented as subscribe/wait one
update/unsubscribe.

Regards,
Lajos.

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 11, 2004 08:16 AM          Msg. 16 of 25
Lajos,

I'm definitely not saying that all of the scanning that's going on is abusive. The real goal isn't to lay blame or point fingers, but to come up with solutions that will make the feed better, both from your point of view and ours.

The snapshot functionality is definitely going in the next release of our quote servers / databases, so that should make life a little better for everybody who needs it. The customizable scanning will most likely require a bit development time on our part, and I don't know where Jay stands on that one.

Thanks for the post!

Tim Russell
Software Engineer
DTN IQ & FinWin

kxrdart
-Interested User-
Posts: 4
Joined: Sep 9, 2004


Posted: Oct 12, 2004 02:02 PM          Msg. 17 of 25
I would like to do scanning on my local PC several times a day. It would be nice if I could retrieve only the 1 minute or 5 minute bars for the current day on a list of symbols. I do not need to establish a streaming connection to the symbol but need to get all the bars from open to current time when the request is submitted. Could a service like this be implemented ?

Thx

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 12, 2004 02:05 PM          Msg. 18 of 25
I'm pretty sure you can already do this via history lookup. If you're looking for something different, could you go into some more detail?

Thanks!

Tim Russell
Software Engineer
DTN IQ & FinWin

jfcantin
-Interested User-
Posts: 20
Joined: May 18, 2004


Posted: Oct 13, 2004 09:19 AM          Msg. 19 of 25
Hi Tim,

1. I am cycling through the market after hours to get the number of trades as my client require this information for his analysis. He also wanted to get it at 15 minutes intervals, but I guess it is a no no now. I was not thinking about the load on your server when I was cycling through the market. Sorry guys.

1.1 If I could get the number of trades on the historical server that would be great, I wouldn't have any reason for cycling through the way I am now.

2. If the snapshot service is available before the # of trades is implemented on the historical server then I would use it as it would simplify my process.

3. 1300 symbols is quite a bit. I am unsure if anyone would need more if they have the snapshot server. Technically you can watch the whole market with your Satellite feed. I am wondering if you would create duplication of your services. I guess if people are trying to scan the whole market they should get the satellite feed anyway, and for these people my guess is 5000 symbols is not going to cut it either. They are probably using the wrong tool for the job, a lot of people do.

I guess the advantage of having a greater amount of symbols on IQFeed would be that you'd be paying only once as opposed to getting the satellite feed and subscribing to IQFeed to get historical data. Do you already have in place a bundle where you can get the satellite feed and IQFeed Historical? Or Satellite with full IQFeed in order to have a redundant Feed. Just a thought as I am not too familiar with the differences between Satellite and IQFeed.

4. You have to do what you have to do to protect your business. With the snapshot server I am wondering if anyone would be affected by a throttle. And if they are still affected, unless I misunderstood what they are doing they are again using the wrong tool for the job.

Historical Server suggestions:
0. Add number of trades
1. Return Symbol/token with the historical request, so if it comes back in the wrong order then it is easy to put know what is what. Or will it always come back one after the other? Garantied?
2. Have an end of day file we could pull to update our databases (daily for sure, minute? tick?)

Should I repost my historical comments on the historical poll?

Hope this helps,

jf

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1741
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 13, 2004 09:35 AM          Msg. 20 of 25
In answer to your questions about a combo deal with satellite and IQFeed. The answer is yes. We have always offered our Internet product (DTN.IQ or IQFeed) at half price to satellite customers just for this reason. They have redundant feeds (Satellite and Internet) as well as unlimited symbols on Satellite and Backfill via Internet. Plus mobility when needed via Internet. They also have a few more news feed options on Internet that we don't broadcast on satellite.

Jay Froscheiser
DTN Market Access, LLC.

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 13, 2004 10:04 AM          Msg. 21 of 25
jf -
No need to repost them to another poll, as I'm tracking both of them.

- Again, cycling through symbols after market isn't that big a deal (with the same caveats I stated before).

- Number of trades, in my opinion, really belongs in the market data. If it was valuable enough to people for charting purposes, I think we could store it in end-of-day data in the future (this would be a way out).

- A throttle is definitely going to happen. With that said, I seriously doubt that anybody is going to notice a significant impact. We'll test it thoroughly - and probably give 3rd party folks a chance to test it, also - before it goes live.

- The symbol/token thing I think is best implemented in IQFeed, vs the historical servers (due to the nature of the IQFeed / history server communication).

- We'll have to look at an end of day file. I think having individual tick and minute files that could be pulled wouldn't lessen the load / bandwidth on our servers, and I don't think it really gives us something that requesting the data through the history server doesn't already gives us. I'm always open to convincing, though.


As always, thanks for the feedback.

Tim Russell
Software Engineer
DTN IQ & FinWin

kxrdart
-Interested User-
Posts: 4
Joined: Sep 9, 2004


Posted: Oct 13, 2004 11:28 AM          Msg. 22 of 25
Hi Tim,

I guess my requirements would be to download todays 1 or 5 minute bars on 2000 symbols in a time of less then 3 minutes. Is this realistic and currently doable ? Since I already have the previous days history local I just need a total bar snapshot for the day.

Thx


Posted: Yesterday @ 02:02 PM [Quote Post] [Reply To Post] [Edit Message] Msg. 17 of 21
I would like to do scanning on my local PC several times a day. It would be nice if I could retrieve only the 1 minute or 5 minute bars for the current day on a list of symbols. I do not need to establish a streaming connection to the symbol but need to get all the bars from open to current time when the request is submitted. Could a service like this be implemented ?

Thx

DTN_Tim_Russell
-IQ Server Developer-
Posts: 29
Joined: May 3, 2004

DTN Market Access, LLC.


NEW Posted: Yesterday @ 02:05 PM [Quote Post] [Reply To Post] Msg. 18 of 21
I'm pretty sure you can already do this via history lookup. If you're looking for something different, could you go into some more detail?

Thanks

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 13, 2004 12:19 PM          Msg. 23 of 25
Quote: Posted By: kxrdart on Today @ 11:28 AM
Message: Hi Tim,

I guess my requirements would be to download todays 1 or 5 minute bars on 2000 symbols in a time of less then 3 minutes. Is this realistic and currently doable ? Since I already have the previous days history local I just need a total bar snapshot for the day.

Thx



I really think that 2000 symbols in under 3 minutes is a bit of a stretch at this point in time. Consider even an average response time of .25 seconds - which is pretty good. .25 * 2000 / 60 = ~8.3 minutes.

Realistically, I'd say double that and that would probably be more accurate.

Our history system wasn't ever really designed with this kind of usage in mind (which is okay, and is why we're asking for your input). If you have a big requirement for this, though, I'd definitely suggest the satellite feed as an attractive alternative.

Tim Russell
Software Engineer
DTN IQ & FinWin

spersels
-Interested User-
Posts: 2
Joined: Sep 12, 2004


Posted: Oct 29, 2004 10:26 PM          Msg. 24 of 25
I am not a current customer but I an looking seriously at the satellite feed. It is very important for me to keep track of ALL the symbols on the NASDAQ in real time at at least a 5 minute bar level and preferably at 1m bar level. I am implementing an automated multi-level market scanning system that requires course-grained price and volume movement data on all symbols. As triggers are activated on the course-grained data, fine-grained data is tracked to ID the proper entry and exit points for the symbols that have had their first-level triggers activated.

In all likely-hood, I will need to keep my own (intraday) database of market price and volume history so that I can support broad scans with potentially complex calculations. As a result, I probably would not need DTN to support anything other than getting me the data in quick snap-shots on a fairly frequent basis.

/s

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 30, 2004 12:10 AM          Msg. 25 of 25
With the satellite feed, this isn't a problem.

Tim Russell
Software Engineer
DTN IQ & FinWin
 

 

Time: Tue December 12, 2017 2:09 AM CFBB v1.2.0 32 ms.
© AderSoftware 2002-2003