Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)

"Thanks for all of your help. Great customer service deserves to be recognized which one the reasons I've been a customer of DTN for over 10 years!" - Comment from Stuart
"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"I am keeping IQFeed, much better reliabilty than *******. I may refer a few other people in the office to switch as well." - Comment from Don
"I had always used ******* but for the past 2 weeks have been trying DTN IQFeed. Customer support has been extraordinary. They call just to make sure your problem hasn't recurred." - Comment from Public Forum
"I'm very glad I switched to IQFeed. It's working perfectly with no lag, even during fast market conditions." - Comment from Andy via Email
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
"Just a quick one to say I'm very impressed so far :) The documentation for developers is excellent and I've quickly managed to get an app written to do historical downloads. The system is very robust and pretty quick considering the extent of data that's available. The support guys have been very helpful too, in combination with the forums it's been plain sailing so far!" - Comment from Adam
"IQ feed is brilliant. The support is mind-bending. What service!" - Comment from Public Forum Post
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
"IQ feed works very well, does not have all of the normal interruptions I have grown used to on *******" - Comment from Mark
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
Viewing User Profile for: mkvalor
About Contact
Joined: Oct 6, 2020 09:26 PM
Last Post: Oct 14, 2023 07:13 PM
Last Visit: Oct 15, 2023 12:07 AM
Occupation: Software Engineer
Interests: Guitar, Financial Market Speculation, Technology
Keep your tools sharp.
Yahoo IM:
Post Statistics
mkvalor has contributed to 20 posts out of 21117 total posts (0.09%) in 1,150 days (0.02 posts per day).

20 Most recent posts:
Miscellaneous Messages » Support for Linux API for the datafeed? Oct 14, 2023 07:13 PM (Total replies: 33)

Glad your way is working for you, but I'm not sure what problems you encountered using Xvfb. I run Fedora with Wine and Xvfb and this configuration has been rock stable for me for nearly 3 years. So this comment is mostly for the benefit of others who might be dissuaded from trying Xvfb by your message above.

Once Wine and Xvfb are set up, you simply prepend the other commands you want to run in the virtual framebuffer with a couple of wrapper commands and be sure to pass in flags to the program which allow non-interactive gui operation. For example, to run the iqfeed installer:

$ xvfb-run wine $HOME/Downloads/iqfeed_client_6_2_0_25.exe /S

and then to connect to IQFeed such that your separate program passes in the password via the admin port shortly after:

$xvfb-run wine IQConnect.exe -product [PRODUCT_ID] -version [MADEUP_VERSION_STR] -login [LOGIN_ID]

Once you see your program running fine in terminal blocking mode, you can set up cron or systemd timer units to start and stop it like this in the background, so you don't need to keep a terminal window open.

-Mark D. Valor
Edited by mkvalor on Oct 14, 2023 at 07:14 PM
Edited by mkvalor on Oct 14, 2023 at 07:16 PM
Edited by mkvalor on Oct 14, 2023 at 07:18 PM

Great info; thanks Gary. Yes, I was talking about L2 Order ID in Q 3. Hmm, yes, I feel that data type indicates it would not be formatted with leading zeros.

I've definitely seen that, for CME ES, Tick ID typically repeats between trades.

I can certainly test pulling data at different times for CME futures to try to discover the daily(?) volume reset. Just thought I'd ask in case this was well known. If you learn the answer please do post it here.

-Mark D. Valor

Context: the point of my question is to figure out which of these fields is either fixed-length or at least reasonably unlikely to roll over to a higher number of digits or shrink to a lower number of digits across most trading days.

I have my IQFeed client set to start watching symbols shortly before market open, so for futures contract ES (for example) the first lines of my captured L1 feed will show values like this for tick id and total volume (respectively below):


and the L2 MBO feed for ES will show order ids like this (final summary line and first delete, add and update lines):


1) I imagine the L1 total volume resets to zero when ES off-hours trading starts at 6:00 pm ET the night before each trading day. Is that accurate?

1A) When it resets, does the field length shrink to the number of digits needed (such as "1" vs "000001")?

2) The L1 tick id, which is enormous in comparison to the daily total volume (since it includes bid/ask) doesn't look like it contains only a single day's L1 transactions. Does this only increase (starting from some fixed time in the past) or does it reset to zero (for example on the first trading session of each year)?

2A) If it resets, does the field length shrink to the number of digits needed (such as "1" vs "000000001")?

3) L2 field values are similarly orders of magnitude above the L1 tick id values (due to the much broader limit order activity). Does this only increase (starting from some fixed time in the past) or does it reset to zero (for example on the first trading session of each year)?

3A) If it resets, does the field length shrink to the number of digits needed (such as "1" vs "0000000000001")?

Thanks in advance.

-Mark D. Valor
Edited by mkvalor on Aug 21, 2023 at 10:51 PM

Well, I hope you will agree with me this is not clear from the docs, since the page for Fundamental Message Format lists the Origin of Data of this field as, "Calculated by DTN".

-Mark D. Valor

Ugh, nevermind -- I was fooled by the trailing comma at the end of the Quote lines to imagine I was missing the Total Volume field (empty value).

I can see now that the final field (which includes a trailing comma) is indeed the Total Volume field and that this value does change on the line with the "v" Message Type.

Thanks for your time.

-Mark D. Valor

EDIT: Nevermind -- see the first reply below. Got fooled by the trailing comma in the Q lines.

--- Original Message -----

In the L1 feed for a futures contract below, notice that the 9th message from the bottom has Message Type 'v'. Yet I don't see a change in anything "volume-like". The bid size is the same, the ask size is the same, the most recent trade size is the same. In what sense did a "volume change" occur which could be detected in sofware consuming an L1 feed for a futures contract?

Separate issue I'll ask in a separate new topic -- no idea why I don't receive Total Volume (final field) in my L1 feed. You can see that I request it in the custom dynamic field response a the top. The documentation for the field says, "Calculated by IQConnect when not sent from servers (or provided by the exchange)". None of my L1 feeds for futures contract receive it (and I'm not currently requesting an L1 feeds for non-futures instruments to see a difference there).

--- begin sample ---
S,CURRENT UPDATE FIELDNAMES,Symbol,TickID,Message Contents,Bid Time,Bid,Bid Size,Ask Time,Ask,Ask Size,Most Recent Trade Conditions,Most Recent Trade Time,Most Recent Trade,Most Recent Trade Size,Most Recent Trade Aggressor,Tick,Total Volume
--- end sample ---

-Mark D. Valor
Edited by mkvalor on Aug 4, 2023 at 09:30 AM

Thanks. I've been issuing a fundamentals query and then cutting off my watches a few seconds after market close. And once I realized I wasn't (yet) seeing the current day's activity reflected, I understood I could just start delaying when I ended my watches. But I figured I'd ask, in case there was specific guidance about it.

As a follow up -- would any particular fields of the L1 F "fundamentals" message get updated at the same time? For example, if the instrument hit a new 52-Week High during that day, would that get updated in the Fundamentals message around the same time that P "Summary" message gets updated? (or if not, when would such an update occur?)

-Mark D. Valor

Here is a real-time L1 'F' fundamental message for the CME ES futures contract. I requested the watch about 15 minutes before open a few days ago using the @ES# symbol which stands in for the currently active contract (in case that makes a difference):

F,@ES#,22,,,4609.25,3683.50,,,,,,,,,,,,E-MINI S&P 500 SEPTEMBER 2023,,,,,,,,,,,,12,2,,12.42,8,43,07/19/2023,10/11/2022,,,,,,09/15/2023,,,ES,,,18:00:00,17:00:00,USD,50,--H--M--U--Z,0.250000000,,,,

Even though volume is recorded through the day in L1 updates and is available in the historical data, I notice that column 5 here for Average Volume is empty. I wondery, why?

-Mark D. Valor

I noticed in this thread:
that Market Summary reports are compiled at about 5:30 pm each trading day. Would that be around the time that L1 "P" summary lines get updated with data for session end (such as close) for that day as well? And is there any difference in the time when this fresh data becomes available for US futures vs equities?

-Mark D. Valor

tl;dr: If you use L2 Market by Order messages for CME/ICE futures, do you feel your systtem greatly benefits from the broader range of updated price points and/or the ability to correlate the OrderId with the L1 messages?

I'm asking because I coded my system under an older protocol so I only have L2 Market by Price. It works fine. But I'm trying to gague whether it's worth it to go back and update my custom software system (a fairly invasive change).

---- Original message below ----
I'm an IQFeed API programmer who mostly analyzes CME/ICE futures and I joined DTN well after the older forum posts which initially requested something like the L2 MBO data. Yet I also joined before protocol 6 was out of beta. As such, I originally coded my apps using protocol 5.2 to pull in the L2 MBP data and 'it wasn't broke so I didn't fix it'. I did upgrade my IQFeed client and protocol once 6.x came out and my apps do use the latest of each (though still only pull MBP data).

Lately, out of curiosity about whether my algorithms might beneifit from more/different data, I've been wondering what the main benefits of MBO data are over MBP. Here's what I believe I understand about the differences:
  • MBO data doesn't have the same limitation on number of levels per side (buy, sell)

  • As a consequence of the first point, MBO summary messages may be of high volume while the initial state of all(?) prices are transmitted (with a consideration in the documentation to perhaps throttle these requests to avoid saturating the network connection and perhaps stalling other watches).

  • There's an OrderId with MBO messages that lets you correlate(?) these messages with the OrderIds of L1 messages

  • MBP updates replace the previous message you received at that price with a new aggregated total

  • MBO updates are only deltas and you must maintain the structure of the price levels yourself and mathmatically apply the deltas as they come in

  • Equity L2 (NASDAQ) comes through as MBO messages and includes MMID but not Order Id

If I made any errors or significant omissions above, please comment to correct them. So , focusing on futures -- I think the main benefits of MBO are the expanded price levels and using the OrderID to correlate with the L1 messages. Yet the cost would be recoding my apps to maintian the structure of the L2 data by applying the updates mathematically as well as potentially ingesting a bunch of price points far(?) from the current price activity. There is a potential fragility if MBO updates are somehow lost.

Please comment with your thoughts about switching from MBP levels to MBO updates (for markets that offer it). Do you make much use of the Order Id correlation and the broader price range (which may include prices far from the current price action)? Does an app analyzing L2 MBO data suddenly start consuming 3X the memory (vs MBP) to hold all the new price levels? If you switched, do you feel it was worth it? etc.

-Mark D. Valor
Edited by mkvalor on Jul 20, 2023 at 10:54 PM

A long time ago, someone stated in this Data and Contet Support topic: that market indexes and stats are updated frequently, perhaps as often as tick-by-tick. I could swear I've read somewhere on the web that many of the famous indexes (such as SPX) have their vlaues updated only as often as every 15 seconds. But anyway...

For those with realtime subscriptions who write their own code with the TCP IQFeed API, what is the best way to consume indexes and market stats, if we want the quickest available updates? Obviously the Market Summary snapshot call wouldn't be ideal if, for instance, the Russell "exchange" summary file for security type Index is updated on DTN servers every 5 minutes.

1. Can indexes and market stats actually be consumed via an L1 Watch API call? (clearly most of the dynamic fields such as those related to bid and ask would not apply, which is why I ask)

1a. If an L1 feed is the way to go, are there any special values for any of the dynamic fields (such as "Message Contents") which woudl have any specific or unusual meanings for indexes or market stats?

2. Do people instead request streaming interval bars for these with an interval set as low as applicable (with a floor of 1 second)? If the old thread I referenced is correct about tick-by-tick frequency, this wouldn't take advantage of the "realtime" subscription.

2a. Many market stats such as "TRIN" lend themselves to line charts rather than bars... In this caseI guess one might simply use "last" from the OHLC values and just discard the other 3 "price" fields? (moot if streaming bars wouldn't be used for live index/market stat requests)

2a. I can grab historical bars off-hours, so I'm not worried about the "8/180" day restriction for second intervals which aren't a multiple of 20; I'd mainly be consuming only the current day's "trades". But, as a good citizen in the DTN ecosystm, is there an upper limit to the number of 1-second streaming bars calls I can make (currrent day only) or is this simply included in the (base) 500-symbol limit? (moot if streaming bars wouldn't be used for live index/market stat requests)


3. If there is a better API call to stream real time data for indexes and/or market stats not covered above, just mention that and ignore the other numberd questions

-Mark D. Valor
Edited by mkvalor on Dec 1, 2022 at 12:53 AM
Edited by mkvalor on Dec 1, 2022 at 12:54 AM
Edited by mkvalor on Dec 1, 2022 at 01:04 AM

Forum Suggestions » https:// ???? Nov 18, 2022 04:20 PM (Total replies: 7)

Great news! Thanks for taking care of this.

-Mark D. Valor

Ah, I hadn't thought of the wider spread situation since I mostly trade markets where that is rare (during the hours when I am active). Thank you for your explanation.

-Mark D. Valor

New IQFeed Forum » Count of DTN Market Statistics Nov 14, 2022 11:13 AM (Total replies: 3)

Thanks for the reply. User error caused the low count on the web page. I had accidentally typed a single character into the search box which I hadn't noticed.

-Mark D. Valor

New IQFeed Forum » Count of DTN Market Statistics Nov 14, 2022 10:25 AM (Total replies: 3)

This is my first time exploring the Market Statistics available from IQFeed. Finally figured out the way to get them via API was to request a 5-minute snapshot (this is neither explained in the docs that I could see nor intuitive to someone new to the API, by the way).

Now, when I use the website search page, I find there are 752 Market Statistic codes results returned (and all of them under "Exchange" DTN). Yet, when I make a 5-minute snapshot request via API v6.2 for the security type code for Market Statistics and the exchange code for DTN, my system receives a file with 3383 lines returned. Are there actually additional types of Market Statistics available via the API? Or what else explains the difference between these two counts?

-Mark D. Valor

Quote: I have been running it for years from Sunday night to Friday night, i.e. a full trading week. No issues. The client has a very small memory footprint.
Great qQuant, just the type of reply I was looking for, thanks!

I noticed this recent reply from DTN_Steve_S in another forum thread mentioning that Saturdays are considered maintenance days [1]:
Quote: Hello all, Saturdays are our routine maintenance days. This involves server reboots and software restarts and, on occasion, can cause requests to fail. We typically try to keep it limited to the morning on Saturdays but sometimes it runs longer. Trying to avoid Saturday mornings for data downloads would be recommended if possible and keeping in mind that request failures should be expected during this time if you are working with our API.

So putting it all together, it seems that allowing IQConnect to shut down Fridays at the close of US trading hours and restarting it on Sundays (to grab historical data and begin off-hours quotes) may be ideal.


-Mark D. Valor

Forum Suggestions » https:// ???? Oct 22, 2022 11:51 PM (Total replies: 7)

I don't mind "necro"-ing a thread for an issue as important as this one. I love the product but frankly, it's a bit of an embarassment that both the developer portal and this forum pass sensitive secrets in the clear over the Internet. You guys are bulldogs about protecting the exchange data agreements from secondary clients -- lets see some of that focus and tenacity applied toward your customers' security, please.

Glad the data service migration to Chicago was a success! I can tell it was by the low ping latency I get from my colo server hosted in that area ;) Was there any big announcement about this once it was accomplished? I don't remember seeing anything. Edit: Ah, searching throught the forum I found the 2021 post explaining the email about firewall rules when this happened.

-Mark D. Valor
Edited by mkvalor on Oct 22, 2022 at 11:53 PM
Edited by mkvalor on Oct 23, 2022 at 01:43 AM

I'm on the latest stable 6.2 IQFeed client release and I'm very pleased with the software. Yet I'm also a server software engineer and I know that eventually, with heap memory fragmentation, etc, it's usually best to restart a network server such as IQConnect.exe at some reasonable interval (usually measured in days). I currently close all connections and allow it to exit automatically each night. But, for all I know, perhaps some of my fellow developers keep it running with clients connected for weeks/months at a time with no issues.

Supposing I had sufficient data subscriptions to keep IQConnect.exe L1/L2, derivative data, and streaming bars busy (but not with overloaded network, RAM, or CPU) during all available US and globex trading hours each week, is there a max recommeded uptime (in days) for keeping it running before a restart? Additional context: I do have scheduled custom clients that wake up before market hours each trading day to grab historical data, so there is definitely a certain level of TCP connection thrashing to IQConnect each day.

Thanks in advance for both IQFeed and other developer answers!

-Mark D. Valor
Edited by mkvalor on Oct 22, 2022 at 08:30 PM
Edited by mkvalor on Oct 22, 2022 at 11:45 PM

Thanks for the reply Gary. Yes, for "Tick" I meant tracking previous trades with Message Contents having trade identifier 'C'.

Would you please help my newbie mind understand what kinds of factors lead to my code not being able to infer the trade aggressor? As you said,

Quote: For other exchanges, this can usually be inferred by comparing the last price to the bid and ask price.

Do the CME and ICE exchanges have additional hidden pools of trades (or some other obscured forms of data) which do not come through the IQFeed L1 stream, making the generic strategy ineffective for them? Or is it simply that -- since those two exchanges do provide the explicit aggressor, it's less error prone to use that instead of relying on custom client code?

-Mark D. Valor
Edited by mkvalor on Oct 23, 2022 at 07:53 AM

I have been tinkering with the L1 dynamic fields my custom app consumes from IQFeed and I was curious about one thing:

If my application keeps track of the qualified trade values in the "Most Recent Trade" field, is there really any benefit to consuming the "Tick" or "Most Recent Trade Aggressor" fields in the feed? My custom system holds onto the last sent qualified "Most Recent Trade" value, so when a new Qualified Trade message comes through, my code can easily deduce whether the new trade value is higher, lower, or the same.

This seems clear enough, but I hate to assume. So, is there some additional benefit/information conveyed in these fields I am not aware of, besides the relative value of the current qualified trade vs the previous qualified trade?

-Mark D. Valor

Time: Wed November 29, 2023 1:30 PM CFBB v1.2.0 15 ms.
© AderSoftware 2002-2003