Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I use IQ Feed, Great stuff as far as data analysis information, storage and retrieval is concerned." - Comment from Public Forum
"I ran your IQFeed DDE vs. my broker vs. a level II window for some slow-moving options. I would see the level II quote change, then your feed update instantaneously. My broker's DDE, however, would take as much as 30 seconds to update. I am not chasing milliseconds, but half a minute is unacceptable." - Comment from Rob
"Thanks for following up with me. You guys do a great job in tech support." - Comment from Phelps
"I was on the phone with a friend who uses CQG and right after the Fed announcement, CQG was as much as 30 seconds behind DTN.IQ. Some quotes were off by as much as 15-18 cents. Your feed never missed a beat." - Comment from Roger
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
"IQ feed is brilliant. The support is mind-bending. What service!" - Comment from Public Forum Post
"I was with ******* for 4 years at $230 a month, this is a huge savings for me, GOD BLESS YOU PEOPLE," - Comment from T.S. via Email
"I just wanted to tell you what a fine job you have been doing. While *******, from what I hear, has been down and out, off and on, IQ feed has held like a champ this week." - Comment from Shirin
"I've been using Neoticker RT with IQFeed for two months, and I'm very happy with both of the products (I've had IQFeed for two years with very few complaints). The service from both companies is exceptional." - Comment from Public Forum
"Can I get another account from you? I am tired of ******* going down so often" - Comment from George
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »NEW IQFEED FORUMS »New IQFeed Forum »Unaligned Bars from BW Command
Author Topic: Unaligned Bars from BW Command (10 messages, Page 1 of 1)

ElliottAnalysis
-Interested User-
Posts: 3
Joined: Mar 29, 2023


Posted: Apr 1, 2023 03:15 PM          Msg. 1 of 10
We made a BW request on 02:52:00 for a 3-minute interval and update interval of 30 seconds. We received BH messages (based on historical data) uptill 02:52:00 and all prior history bars were 3-minutes apart. The problem is that after 02:52:00 marks, the next streaming bar comes at 02:54:00 which is two minutes apart from the last bar. After the bar at 02:54:00, all the streaming bars are once again 3-minutes apart. The stream is given below

B-@ES#-00180-s,BH,@ES#,2023-03-29 02:40:00,4026.75,4026.75,4026.25,4026.75,73084,391,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:43:00,4026.50,4027.50,4026.50,4027.00,73447,363,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:46:00,4027.00,4027.25,4026.25,4026.75,73754,307,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:49:00,4027.00,4027.25,4026.75,4026.75,74208,454,0,

B-@ES#-00180-s,BH,@ES#,2023-03-29 02:52:00,4026.75,4027.25,4026.50,4027.00,74509,301,0,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:54:00,4026.75,4027.25,4026.50,4027.25,74546,338,,

B-@ES#-00180-s,BU,@ES#,2023-03-29 02:54:00,4026.75,4027.50,4026.50,4027.25,74621,413,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 02:54:00,4026.75,4027.50,4026.50,4027.50,74651,443,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4027.25,4027.25,74747,96,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4026.75,74966,315,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4027.00,75039,388,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4027.75,75217,566,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4028.00,4026.50,4028.00,75381,730,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 02:57:00,4027.75,4028.25,4026.50,4028.25,75452,801,,


My question is if there is a way we can make sure that all history, update and closed bars are 3 minutes apart?

In another occurrence, I receive the following stream:


B-@ES#-00180-s,BH,@ES#,2023-03-29 04:26:00,4032.75,4034.50,4032.75,4034.50,113520,792,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:29:00,4034.25,4035.00,4034.25,4034.50,114138,618,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:32:00,4034.50,4036.75,4034.25,4036.75,115551,1413,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:35:00,4036.75,4036.75,4035.50,4035.75,115989,438,0,

B-@ES#-00180-s,BC,@ES#,2023-03-29 04:36:00,4036.75,4036.75,4035.50,4035.75,115989,438,,

B-@ES#-00180-s,BU,@ES#,2023-03-29 04:33:00,4036.00,4036.75,4036.00,4036.50,116292,303,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 04:33:00,4036.00,4037.25,4036.00,4037.25,116458,469,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4037.00,4037.25,116978,520,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.75,4036.75,117217,759,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4036.75,117426,968,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4036.50,117630,1172,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4037.00,117808,1350,,


The request was made on 04:32:00 for a 3-minute interval and update interval of 30 seconds. We received BH messages (based on historical data) uptill 04:35:00 and all prior history bars were 3-minutes apart. There are multiple problems with the stream:

  • How can IQFEED calculate the 04:35:00 history bar at 04:32:00 mark. I understand if it starts sending BU messages for the 04:35:00 bar, but a history bar means that the bar has been closed based on historical data. So, how the 04:35:00 history bar can be closed at 04:32:00 mark.

  • We received a BC close bar message for 04:36:00 (highlighted green) and later received BU (update) messages for the same bar.

  • The previously discussed problem of misaligned interval bars like 04:32, 04:33, 04:35 and 04:36 still remains. The difference between these bars should be 3 minutes.



Can anyone please clarify how can I make sure that all historical and current bars are received in order as well as the same time interval apart?

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 3, 2023 03:20 PM          Msg. 2 of 10
Quote: How can IQFEED calculate the 04:35:00 history bar at 04:32:00 mark.


That looks like an error. I will escalate this to the dev team. BU messages can arrive at any time (if you're using UpdateInterval), but IQFeed should clearly tell you which bar the belong to. If you set a 180-second bar, there should only be messages for every 3 minutes, not 2 minutes apart like that.

Your other two questions are interwoven with that problem, so I can't really answer them until we solve that one. There should never be BU updates after a BC message is sent, because because the BC message can only be sent after the interval is complete. Which in turn only happens when a tick *after* the interval has happened. @ES# trades so frequently this never takes long, but the behind-the-scenes working of BW may be having a timing issue.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 4, 2023 08:48 AM          Msg. 3 of 10
Also, what is the exact BW command you're sending? I can infer some of it from your post, but our investigation would be helped if you can post it in its entirety. Thank you.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

ElliottAnalysis
-Interested User-
Posts: 3
Joined: Mar 29, 2023


Posted: Apr 4, 2023 11:52 AM          Msg. 4 of 10
Thanks Gary for the answer.

I am using the following command (requested at 12:39:xx, where I don't know the seconds for the requested time):

BW,@ES#,180,20230404 093900,,,,,B-@ES#-00180-s,s,'',30

Once again, I got the following output:

B-@ES#-00180-s,BH,@ES#,2023-04-04 12:24:00,4116.25,4120.75,4116.25,4120.75,943647,11233,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:27:00,4120.75,4125.25,4120.25,4124.75,955352,11679,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:30:00,4125.00,4126.00,4124.25,4126.00,961787,6435,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:33:00,4126.00,4128.00,4125.50,4126.25,969806,8019,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:36:00,4126.25,4126.75,4123.75,4124.00,977011,7005,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:39:00,4124.00,4125.00,4121.50,4123.00,986272,8861,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:42:00,4123.00,4124.00,4122.75,4123.50,987705,1433,0,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4125.00,4122.75,4125.00,989230,2958,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4126.50,992407,6135,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4125.00,994452,8180,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4125.75,996636,10364,,
B-@ES#-00180-s,BC,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4126.75,997380,11108,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:45:00,4126.50,4126.75,4125.75,4126.00,998656,1276,,


As you can see, I received a history bar for 2023-04-04 12:42:00 on 2023-04-04 12-39-xx mark and later received BU messages corresponding to the same bar.

The command for the above two examples in my first post are similar with the exception of the request time for the command and BeginDateTime parameter.

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 6, 2023 11:11 AM          Msg. 5 of 10
It is possible for a BH command to be followed by BU commands for the same bar. This is because a BH message doesn't have to be the final message for that bar. This bar behaves in a way analogous to the IncludePartialDatapoint parameter in some historical commands like HDX: it will send a message for an interval even if it's not complete yet.

This is really only likely to happen when the BW command is building the first BC message of the session. Your BW command included some history, which it provided in the form of BH messages up to the current one. Then you received 4 BU update messages for the same bar, because you set the UpdateInterval parameter of the BW command to send an update every 30 seconds. Then you get the BC message for that bar, which is the final message, because BC is only sent when it the bar is complete (with one caveat, see next paragraph). Then you began getting BU messages for the next bar.

Even BC message bars aren't 100% final, because they can still be changed by a subsequent BU message for the same bar. This happens when the exchange issues as a Trade Correction, which changes a tick in a way that the bar must now be recalculated.

Your other example was a BC message followed by BU messages from earlier, which is a different matter still being investigated.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

ElliottAnalysis
-Interested User-
Posts: 3
Joined: Mar 29, 2023


Posted: Apr 8, 2023 01:03 AM          Msg. 6 of 10
Thanks Gary for your insight.

Just to give you some more information about our requirement: we are trying to write data to a csv file which is supposed to be current all the time. Based on your above explanation, I assume that we should take the last bar of the first BW response as 'not closed' and if we receive any subsequent bar with the same timestamp, we should update the last bar. It seems all fine and we can easily implement this logic. However, Example 2 in the original post has some more issues that we cannot resolve with the above logic. The biggest issue is that the bars are not 3 minutes apart uniformly -- some are 2 minutes apart. Furthermore, the update message are not received for the last bar only but comes for a few bars further back as well. Please let us know how the investigation goes.

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 11, 2023 03:53 PM          Msg. 7 of 10
Quote: Based on your above explanation, I assume that we should take the last bar of the first BW response as 'not closed' and if we receive any subsequent bar with the same timestamp, we should update the last bar. It seems all fine and we can easily implement this logic. However, Example 2 in the original post has some more issues that we cannot resolve with the above logic.


Let me make a counter-suggestion: What if you set the bar length to 30 seconds instead of 3 minutes, set UpdateInterval to 0 instead of 30 seconds, then assemble your 3-minute bars from those 30-second bars.

BW,@ES#,30,20230404 093900,,,,,B-@ES#-00180-s,s,',0

This gives you the immediacy of the every-30-seconds update, while also knowing that a BH message represents a complete 30-second bar is complete. The tradeoff is that the BH message will not be sent until a tick has occurred after that 30-second bar. So if something doesn't trade for a few seconds, you might not get the 9:17:30 bar until 9:17:38. If you can live with that, then this approach might work better for you, depending on how your program otherwise works.

I know I still owe you an answer on the 2-minute / 3-minute business, which I will post soon.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Apr 11, 2023 at 03:54 PM

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 14, 2023 09:45 AM          Msg. 8 of 10
I was able to reproduce the issue with the misaligned bar start times. Will post a solution when we have one.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: Apr 24, 2023 11:23 AM          Msg. 9 of 10
I can provide a workaround: if you're using a Start Date/Start Time, make sure it's evenly divisible by the bar length you specified. You didn't give the command for your initial post, but your follow-up was:

BW,@ES#,180,20230404 093900,,,,,B-@ES#-00180-s,s,'',30

This works, because if you start at :39 after the hour, the remainder (21) divides evenly into 3-minute bars. I suspect your command from the OP started at something like 094000, where it doesn't.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

DTN_Gary_Stephen
-DTN Guru-
Posts: 394
Joined: Jul 3, 2019


Posted: May 18, 2023 08:56 AM          Msg. 10 of 10
This issue will be resolved as part of a larger update later on. in the meantime, use the workaround I listed above: if you use a start time, make sure the remainder of the hour is evenly divisible by the bar length you specified.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
 

 

Time: Fri April 19, 2024 3:12 AM CFBB v1.2.0 13 ms.
© AderSoftware 2002-2003