mkvalor has contributed to 26 posts out of 21251 total posts
(0.12%) in 1,462 days (0.02 posts per day).
20 Most recent posts:
Thanks Gary. The sales dept was able to take care of my needs. Mainly wanted to raise awareness about the Premium Services page being unavailable.
-Mark D. Valor
Hello,
I am an active customer. I understand there is a new payment portal at dtn dot my dot site dot com, and I am registered there. Yet the link at the top of that site for "Add or Remove service" takes me to the old sales website at myaccount dot dtn dot com. So far, so good.
However, on the "myaccount" site, the links on the left nav under Manage Services for both "Existing Accounts" and "Add Premium/Exchange" take me to the same service summary page. It used to be that the "Add Premium/Exhange" link would take me to a different page, which allowed me to choose new services to add to my account. The end result is that I am unable to add new services to my account, since I cannot navigate to the proper page with the option to them.
Summary: 1. New payment portal sends customer to old site to "Add or Remove services". 2. Old site has a link on the left nav to "Add Premium/Exchange" 3. However, the "Add Premium/Exchange" link takes us to the same service summary page as the "Existing Accounts" link there 4. End result: customer cannot add services 5. (extra: cancelling services still works fine from the summary page)
I have already sent an email with my account number to sales at iqfeed dot net. But I wanted to post here in case this is more of a technical issue (broken website link) that would not be solved by the sales depeartment.
-Mark D. Valor
I'm a user like yourself, but I've seen a few posts in the past dealing with memory issue problems.
One thing: do you happen to know if your app makes a separate connection to IQConnect.exe for each symbol? Especially using python, I could see this adding up to a lot of CPU and memory overhead. Even though this strategy can be more convenient (because each program only has to deal with data for a single symbol), it can be a lot more memory and CPU efficient to open multiple symbol watches (say, 50 at a time) on a single connection to IQConnect and then use logic in your applicatoin to send each line to the right place based on the ticker symbol at the beginning (such as to a separate data file, to a data stream queue like Kafka, or to some other logic handler to calculate moving averages, etc).
For example, you could use a python dict where the key is the IQF symbol name and the value is an open file handle for that symbol (if you don't find it in the dict, you create a new file and then add it there) -- then check the start of each line to look up the right file to write it to. This is more complex because of the logic I mentioned but also because you'd need to do something like read a config file per process with the list of symbols you want to watch per process. If you got fancy, you could have a single file with all symbols, one per line, and then start the program by adding command line params that tell it which line to start at in the symbols file and how many of them to read for that particular process.
Anyway -- enough advice based on guessing. If you're already reading multiple symbols per process, then it must be something else.
-Mark D. Valor Edited by mkvalor on Feb 23, 2024 at 02:01 AM Edited by mkvalor on Feb 23, 2024 at 02:03 AM
I found a way to cover most of the edge cases. Recall that one may connect to several different ports on the local IQFeed exe server. One of those is the admin port. So firts, I try to connect to the admin port just as a test to see if IQFeed is already running. If that fails twice (after pausing between attempts), I call a shell script (a batch file on Windows) which runs IQFeed.exe. Then I try to connect 3 more times (after pauses). Since my rust code might looke like gobbledygook, I'll demonstrat this with pseudo code that resembles python (but is not actually python):
admin_conn_attempts = 0 conn_attempt_pause_seconds = 4 while True: admin_conn = tcp_connect("127.0.0.1", 9300) #the admin socket if admin_conn.is_connected(): break else sleep(conn_attempt_pause_seconds) admin_conn_attempts += 1 if admin_conn_attempts == 2: # run external program to start IQFeed.exe run.program(IQFeed.exe) else if admin_conn_attempts > 5: print("ERROR: Unable to connect to IQFeed") exit -1
If i get this far in the code and it didn't exit, that means I'm connected.
I'm aware IQFeed should just start automatically if it's not running but I'm running under Wine on Linux and I have not observed that behavior. I call a shell script to run it (instead of the executable itself) because I set some additional environment things up before running -- such as pinning IQFeed to a certain set of cores on my multi-core processor.
-Mark D. Valor Edited by mkvalor on Feb 10, 2024 at 07:31 PM
Quote: Would it be possible to get an example command for "watching" only the two top depth levels for level 2 SPX option data? Is it as simple as this?: WPL,SPXoptionSymbol,2<CR><CR><LF>
I don't have a subscription for options. However, when I tried your example during off hours for a futures contract, the summary lines returned comprised only two levels of bid and two levels of ask, as expected.
T,20240210 17:58:29 S,CLEAR DEPTH,@ES#,B, S,CLEAR DEPTH,@ES#,A, 7,@ES#,B,5041.50,51,6,2,16:59:02.811840,2024-02-09, 7,@ES#,B,5041.25,22,3,2,16:59:00.820439,2024-02-09, 7,@ES#,A,5042.00,9,3,2,16:59:58.043198,2024-02-09, 7,@ES#,A,5042.25,20,3,2,16:59:03.275683,2024-02-09,
Looks like you might need to subscribe to data or find someone who has options data to be completely sure about those instruments.
-mkvalor
-Mark D. Valor
Quote: If 'Most Recent Trade Conditions' contains HEX 40 (Closing Prints) and also 'Message Contents' contains 'C' or 'O' - field 'Last' contains exactly today's Close price.
This is really great info LeeFox. Thanks for sharing it. Do any of the IQFeed moderators have any comment on using this method?
-Mark D. Valor
Glad your way is working for you, but I'm not sure what problems you encountered using Xvfb. I run Fedora with Wine and Xvfb and this configuration has been rock stable for me for nearly 3 years. So this comment is mostly for the benefit of others who might be dissuaded from trying Xvfb by your message above.
Once Wine and Xvfb are set up, you simply prepend the other commands you want to run in the virtual framebuffer with a couple of wrapper commands and be sure to pass in flags to the program which allow non-interactive gui operation. For example, to run the iqfeed installer:
$ xvfb-run wine $HOME/Downloads/iqfeed_client_6_2_0_25.exe /S
and then to connect to IQFeed such that your separate program passes in the password via the admin port shortly after:
$xvfb-run wine IQConnect.exe -product [PRODUCT_ID] -version [MADEUP_VERSION_STR] -login [LOGIN_ID]
Once you see your program running fine in terminal blocking mode, you can set up cron or systemd timer units to start and stop it like this in the background, so you don't need to keep a terminal window open.
-Mark D. Valor Edited by mkvalor on Oct 14, 2023 at 07:14 PM Edited by mkvalor on Oct 14, 2023 at 07:16 PM Edited by mkvalor on Oct 14, 2023 at 07:18 PM
Great info; thanks Gary. Yes, I was talking about L2 Order ID in Q 3. Hmm, yes, I feel that data type indicates it would not be formatted with leading zeros.
I've definitely seen that, for CME ES, Tick ID typically repeats between trades.
I can certainly test pulling data at different times for CME futures to try to discover the daily(?) volume reset. Just thought I'd ask in case this was well known. If you learn the answer please do post it here.
-Mark D. Valor
Context: the point of my question is to figure out which of these fields is either fixed-length or at least reasonably unlikely to roll over to a higher number of digits or shrink to a lower number of digits across most trading days.
I have my IQFeed client set to start watching symbols shortly before market open, so for futures contract ES (for example) the first lines of my captured L1 feed will show values like this for tick id and total volume (respectively below):
Q,@ES#,271429934,...,276430,
and the L2 MBO feed for ES will show order ids like this (final summary line and first delete, add and update lines):
6,@ES#,6412236089210,..., 5,@ES#,6412236980317,..., 3,@ES#,6412236980495,..., 5,@ES#,6412236980188,..., 5,@ES#,6412236980184,..., 4,@ES#,6412236980185,...,
1) I imagine the L1 total volume resets to zero when ES off-hours trading starts at 6:00 pm ET the night before each trading day. Is that accurate?
1A) When it resets, does the field length shrink to the number of digits needed (such as "1" vs "000001")?
2) The L1 tick id, which is enormous in comparison to the daily total volume (since it includes bid/ask) doesn't look like it contains only a single day's L1 transactions. Does this only increase (starting from some fixed time in the past) or does it reset to zero (for example on the first trading session of each year)?
2A) If it resets, does the field length shrink to the number of digits needed (such as "1" vs "000000001")?
3) L2 field values are similarly orders of magnitude above the L1 tick id values (due to the much broader limit order activity). Does this only increase (starting from some fixed time in the past) or does it reset to zero (for example on the first trading session of each year)?
3A) If it resets, does the field length shrink to the number of digits needed (such as "1" vs "0000000000001")?
Thanks in advance.
-Mark D. Valor Edited by mkvalor on Aug 21, 2023 at 10:51 PM
Well, I hope you will agree with me this is not clear from the docs, since the page for Fundamental Message Format lists the Origin of Data of this field as, "Calculated by DTN".
-Mark D. Valor
Ugh, nevermind -- I was fooled by the trailing comma at the end of the Quote lines to imagine I was missing the Total Volume field (empty value).
I can see now that the final field (which includes a trailing comma) is indeed the Total Volume field and that this value does change on the line with the "v" Message Type.
Thanks for your time.
-Mark D. Valor
EDIT: Nevermind -- see the first reply below. Got fooled by the trailing comma in the Q lines.
--- Original Message -----
In the L1 feed for a futures contract below, notice that the 9th message from the bottom has Message Type 'v'. Yet I don't see a change in anything "volume-like". The bid size is the same, the ask size is the same, the most recent trade size is the same. In what sense did a "volume change" occur which could be detected in sofware consuming an L1 feed for a futures contract?
Separate issue I'll ask in a separate new topic -- no idea why I don't receive Total Volume (final field) in my L1 feed. You can see that I request it in the custom dynamic field response a the top. The documentation for the field says, "Calculated by IQConnect when not sent from servers (or provided by the exchange)". None of my L1 feeds for futures contract receive it (and I'm not currently requesting an L1 feeds for non-futures instruments to see a difference there).
--- begin sample --- S,CURRENT UPDATE FIELDNAMES,Symbol,TickID,Message Contents,Bid Time,Bid,Bid Size,Ask Time,Ask,Ask Size,Most Recent Trade Conditions,Most Recent Trade Time,Most Recent Trade,Most Recent Trade Size,Most Recent Trade Aggressor,Tick,Total Volume ... Q,@MNQ#,234172322,a,09:23:09.529415,15836.50,8,09:23:09.529631,15836.75,4,01,09:23:09.528964,15836.75,1,2,,227513, Q,@MNQ#,234172322,b,09:23:09.529708,15836.50,7,09:23:09.529631,15836.75,4,01,09:23:09.528964,15836.75,1,2,,227513, Q,@MNQ#,234172322,b,09:23:09.529709,15836.50,6,09:23:09.529631,15836.75,4,01,09:23:09.528964,15836.75,1,2,,227513, Q,@MNQ#,234172322,b,09:23:09.529817,15836.50,5,09:23:09.529631,15836.75,4,01,09:23:09.528964,15836.75,1,2,,227513, Q,@MNQ#,234172328,C,09:23:09.529817,15836.50,5,09:23:09.529631,15836.75,4,01,09:23:09.529877,15836.50,1,2,175,227514, Q,@MNQ#,234172328,b,09:23:09.529877,15836.50,4,09:23:09.529631,15836.75,4,01,09:23:09.529877,15836.50,1,2,,227514, Q,@MNQ#,234172329,C,09:23:09.529877,15836.50,4,09:23:09.529631,15836.75,4,01,09:23:09.530059,15836.50,1,2,183,227515, Q,@MNQ#,234172329,b,09:23:09.530059,15836.50,3,09:23:09.529631,15836.75,4,01,09:23:09.530059,15836.50,1,2,,227515, Q,@MNQ#,234172330,C,09:23:09.530059,15836.50,3,09:23:09.529631,15836.75,4,01,09:23:09.530111,15836.50,3,2,183,227518, Q,@MNQ#,234172330,b,09:23:09.530111,15836.25,5,09:23:09.529631,15836.75,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.530111,15836.25,5,09:23:09.530460,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.530111,15836.25,5,09:23:09.531483,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.530111,15836.25,5,09:23:09.531646,15836.50,3,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.530111,15836.25,5,09:23:09.531784,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.531862,15836.25,4,09:23:09.531784,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.531862,15836.25,4,09:23:09.531882,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.531862,15836.25,4,09:23:09.531932,15836.50,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.532203,15836.25,3,09:23:09.531932,15836.50,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532203,15836.25,3,09:23:09.532288,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532203,15836.25,3,09:23:09.532441,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.532722,15836.25,2,09:23:09.532441,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532722,15836.25,2,09:23:09.532722,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532722,15836.25,2,09:23:09.532733,15836.50,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532722,15836.25,2,09:23:09.532735,15836.50,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532722,15836.25,2,09:23:09.532771,15836.50,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.532722,15836.25,2,09:23:09.536253,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.536273,15836.25,3,09:23:09.536253,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.536273,15836.25,3,09:23:09.536360,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.540764,15836.25,2,09:23:09.536360,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.540764,15836.25,2,09:23:09.540764,15836.50,5,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.540764,15836.25,2,09:23:09.586880,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.586880,15836.25,3,09:23:09.586880,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.586997,15836.25,4,09:23:09.586880,15836.50,4,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.586997,15836.25,4,09:23:09.587477,15836.50,3,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587488,15836.25,5,09:23:09.587477,15836.50,3,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587488,15836.25,5,09:23:09.587595,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587599,15836.25,6,09:23:09.587595,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587599,15836.25,6,09:23:09.587609,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587762,15836.25,5,09:23:09.587609,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587762,15836.25,5,09:23:09.587763,15836.75,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587812,15836.25,4,09:23:09.587763,15836.75,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587814,15836.25,3,09:23:09.587763,15836.75,6,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587814,15836.25,3,09:23:09.587816,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587814,15836.25,3,09:23:09.587862,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587814,15836.25,3,09:23:09.587916,15836.50,3,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587998,15836.25,4,09:23:09.587916,15836.50,3,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587998,15836.25,4,09:23:09.587999,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.587999,15836.25,5,09:23:09.587999,15836.50,2,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587999,15836.25,5,09:23:09.588119,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.587999,15836.25,5,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.588161,15836.25,4,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.588162,15836.25,3,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.588335,15836.25,4,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.588336,15836.25,5,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.595870,15836.25,4,09:23:09.588120,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.595870,15836.25,4,09:23:09.595981,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.596018,15836.25,3,09:23:09.595981,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.596158,15836.25,4,09:23:09.595981,15836.50,1,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,a,09:23:09.596158,15836.25,4,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.596285,15836.25,5,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.596620,15836.25,6,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.597092,15836.25,7,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,b,09:23:09.597191,15836.25,8,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227518, Q,@MNQ#,234172330,v,09:23:09.597191,15836.25,8,09:23:09.596285,15836.75,7,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,a,09:23:09.597191,15836.25,8,09:23:09.800264,15836.75,6,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,a,09:23:09.597191,15836.25,8,09:23:09.800280,15836.75,5,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,a,09:23:09.597191,15836.25,8,09:23:09.800281,15836.75,4,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,b,09:23:09.800495,15836.50,1,09:23:09.800281,15836.75,4,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,b,09:23:09.800528,15836.50,2,09:23:09.800281,15836.75,4,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,a,09:23:09.800528,15836.50,2,09:23:09.800528,15836.75,3,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172330,a,09:23:09.800528,15836.50,2,09:23:09.800543,15836.75,2,01,09:23:09.530111,15836.50,3,2,,227519, Q,@MNQ#,234172334,C,09:23:09.800528,15836.50,2,09:23:09.800543,15836.75,2,01,09:23:09.800628,15836.75,1,1,173,227520, --- end sample ---
-Mark D. Valor Edited by mkvalor on Aug 4, 2023 at 09:30 AM
Thanks. I've been issuing a fundamentals query and then cutting off my watches a few seconds after market close. And once I realized I wasn't (yet) seeing the current day's activity reflected, I understood I could just start delaying when I ended my watches. But I figured I'd ask, in case there was specific guidance about it.
As a follow up -- would any particular fields of the L1 F "fundamentals" message get updated at the same time? For example, if the instrument hit a new 52-Week High during that day, would that get updated in the Fundamentals message around the same time that P "Summary" message gets updated? (or if not, when would such an update occur?)
-Mark D. Valor
Here is a real-time L1 'F' fundamental message for the CME ES futures contract. I requested the watch about 15 minutes before open a few days ago using the @ES# symbol which stands in for the currently active contract (in case that makes a difference):
F,@ES#,22,,,4609.25,3683.50,,,,,,,,,,,,E-MINI S&P 500 SEPTEMBER 2023,,,,,,,,,,,,12,2,,12.42,8,43,07/19/2023,10/11/2022,,,,,,09/15/2023,,,ES,,,18:00:00,17:00:00,USD,50,--H--M--U--Z,0.250000000,,,,
Even though volume is recorded through the day in L1 updates and is available in the historical data, I notice that column 5 here for Average Volume is empty. I wondery, why?
-Mark D. Valor
I noticed in this thread: https://forums.iqfeed.net/index.cfm?page=topic&topicID=6953 that Market Summary reports are compiled at about 5:30 pm each trading day. Would that be around the time that L1 "P" summary lines get updated with data for session end (such as close) for that day as well? And is there any difference in the time when this fresh data becomes available for US futures vs equities?
-Mark D. Valor
tl;dr: If you use L2 Market by Order messages for CME/ICE futures, do you feel your systtem greatly benefits from the broader range of updated price points and/or the ability to correlate the OrderId with the L1 messages?
I'm asking because I coded my system under an older protocol so I only have L2 Market by Price. It works fine. But I'm trying to gague whether it's worth it to go back and update my custom software system (a fairly invasive change).
---- Original message below ---- I'm an IQFeed API programmer who mostly analyzes CME/ICE futures and I joined DTN well after the older forum posts which initially requested something like the L2 MBO data. Yet I also joined before protocol 6 was out of beta. As such, I originally coded my apps using protocol 5.2 to pull in the L2 MBP data and 'it wasn't broke so I didn't fix it'. I did upgrade my IQFeed client and protocol once 6.x came out and my apps do use the latest of each (though still only pull MBP data).
Lately, out of curiosity about whether my algorithms might beneifit from more/different data, I've been wondering what the main benefits of MBO data are over MBP. Here's what I believe I understand about the differences:
- MBO data doesn't have the same limitation on number of levels per side (buy, sell)
- As a consequence of the first point, MBO summary messages may be of high volume while the initial state of all(?) prices are transmitted (with a consideration in the documentation to perhaps throttle these requests to avoid saturating the network connection and perhaps stalling other watches).
- There's an OrderId with MBO messages that lets you correlate(?) these messages with the OrderIds of L1 messages
- MBP updates replace the previous message you received at that price with a new aggregated total
- MBO updates are only deltas and you must maintain the structure of the price levels yourself and mathmatically apply the deltas as they come in
- Equity L2 (NASDAQ) comes through as MBO messages and includes MMID but not Order Id
If I made any errors or significant omissions above, please comment to correct them. So , focusing on futures -- I think the main benefits of MBO are the expanded price levels and using the OrderID to correlate with the L1 messages. Yet the cost would be recoding my apps to maintian the structure of the L2 data by applying the updates mathematically as well as potentially ingesting a bunch of price points far(?) from the current price activity. There is a potential fragility if MBO updates are somehow lost.
Please comment with your thoughts about switching from MBP levels to MBO updates (for markets that offer it). Do you make much use of the Order Id correlation and the broader price range (which may include prices far from the current price action)? Does an app analyzing L2 MBO data suddenly start consuming 3X the memory (vs MBP) to hold all the new price levels? If you switched, do you feel it was worth it? etc.
-Mark D. Valor Edited by mkvalor on Jul 20, 2023 at 10:54 PM
A long time ago, someone stated in this Data and Contet Support topic: https://forums.iqfeed.net/index.cfm?page=topic&topicID=2402 that market indexes and stats are updated frequently, perhaps as often as tick-by-tick. I could swear I've read somewhere on the web that many of the famous indexes (such as SPX) have their vlaues updated only as often as every 15 seconds. But anyway...
For those with realtime subscriptions who write their own code with the TCP IQFeed API, what is the best way to consume indexes and market stats, if we want the quickest available updates? Obviously the Market Summary snapshot call wouldn't be ideal if, for instance, the Russell "exchange" summary file for security type Index is updated on DTN servers every 5 minutes.
1. Can indexes and market stats actually be consumed via an L1 Watch API call? (clearly most of the dynamic fields such as those related to bid and ask would not apply, which is why I ask)
1a. If an L1 feed is the way to go, are there any special values for any of the dynamic fields (such as "Message Contents") which woudl have any specific or unusual meanings for indexes or market stats?
2. Do people instead request streaming interval bars for these with an interval set as low as applicable (with a floor of 1 second)? If the old thread I referenced is correct about tick-by-tick frequency, this wouldn't take advantage of the "realtime" subscription.
2a. Many market stats such as "TRIN" lend themselves to line charts rather than bars... In this caseI guess one might simply use "last" from the OHLC values and just discard the other 3 "price" fields? (moot if streaming bars wouldn't be used for live index/market stat requests)
2a. I can grab historical bars off-hours, so I'm not worried about the "8/180" day restriction for second intervals which aren't a multiple of 20; I'd mainly be consuming only the current day's "trades". But, as a good citizen in the DTN ecosystm, is there an upper limit to the number of 1-second streaming bars calls I can make (currrent day only) or is this simply included in the (base) 500-symbol limit? (moot if streaming bars wouldn't be used for live index/market stat requests)
OR
3. If there is a better API call to stream real time data for indexes and/or market stats not covered above, just mention that and ignore the other numberd questions
-Mark D. Valor Edited by mkvalor on Dec 1, 2022 at 12:53 AM Edited by mkvalor on Dec 1, 2022 at 12:54 AM Edited by mkvalor on Dec 1, 2022 at 01:04 AM
Great news! Thanks for taking care of this.
-Mark D. Valor
Ah, I hadn't thought of the wider spread situation since I mostly trade markets where that is rare (during the hours when I am active). Thank you for your explanation.
-Mark D. Valor
Thanks for the reply. User error caused the low count on the web page. I had accidentally typed a single character into the search box which I hadn't noticed.
-Mark D. Valor
|