Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"If you want customer service that answers the phone, your best bet is IQFeed. I cannot stop praising them or their technical support. They are always there for you, and they are quick. I have used ****** too but the best value is IQFeed." - Comment from Public Forum
"I just wanted to let u know that your data feed/service is by far the best!!! Your unfiltered tick data is excellent for reading order flow and none of your competitors delivers this quality of data!" - Comment from Peter via Email
"Thanks for following up with me. You guys do a great job in tech support." - Comment from Phelps
"I ran your IQFeed DDE vs. my broker vs. a level II window for some slow-moving options. I would see the level II quote change, then your feed update instantaneously. My broker's DDE, however, would take as much as 30 seconds to update. I am not chasing milliseconds, but half a minute is unacceptable." - Comment from Rob
"Previously I was using *******. IQFeed is WAY more economical, and for my charting needs is just as good, if not better." - Comment from Public Forum Post
"The service is great, I see a noticeable improvement in my volume profiles over [broker]'s data feed" - Comment from Larry
"Can I get another account from you? I am tired of ******* going down so often" - Comment from George
"I cannot believe what a difference it makes trading with ProphetX!" - Comment from Bruce in Los Angeles
"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Latency of Q messages
Author Topic: Latency of Q messages (4 messages, Page 1 of 1)

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 11, 2019 08:00 PM          Msg. 1 of 4
Hi,

I've been trying to optimize my .net socket reader. First, I wrote my own server and pushed 1000 "Q," messages per second to help find any bottlenecks in my code. The mock server writes 1000 dummy Q messages in a loop. I've appended a high resolution time stamp to the messages (using interop GetSystemTimePreciseAsFileTime). I compare this time stamp on the reading side to see how fast I process messages. The writes of 1000 messages only takes a few milliseconds. I noticed that I was unable to keep up with this rate on the read side if I processed the message synchronously. I now push the messages to a BlockingCollection(ConcurrentQueue). Once I did this, my socket mostly reads one or two message per read.

I thought this would help reading the live feed. However, I still see many reads with numerous messages (>50). This is averaging 5, but sometime as many as 100+ messages are pulled in one read. Most of the time, these messages only differ in the Bid/Ask time stamp by a few milliseconds but sometimes are > 50ms (as high as 200 ms) from the first message to the last.

Is IQConnect batching the writes to the socket? Or are you batching the push from your servers? No matter what I do I cannot seem to reduce this.

I should note that I am collecting stats in an array and only dumping one line to the console every 10-15 second. I have timed the stat collection and display and I am confident that it is not skewing the results.

I need to sort this out not because I’m concerned about a few milliseconds but because I can get way behind (1-30 seconds!) in a fast market.

I am running on an AWS EC2 T2.Medium machine (2 cores).

My pings to the IQFeed server are constantly between 45-47 ms.

I am only watching CME products (@ES, @NQ, 5 major currencies)

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2091
Joined: Nov 21, 2005


Posted: Feb 19, 2019 06:38 AM          Msg. 2 of 4
Hello, my apologies for the delay responding to this.

Generally speaking, the markets are naturally very "bursty" in terms of how messages are received from the exchanges. I haven't verified with your specific symbols but you are using some of the most active futures contracts as I'm sure you know. I wouldn't be surprised if your few symbols frequently burst well beyond your 1000 message loop test (unless I'm not understanding your test).

A 30s delay is certainly not normal and figuring out where that data is getting queued will help identify what needs to be fixed. If the data is getting to your machine but queued locally within IQConnect waiting to be delivered to your app, then in the diagnostics app, on the client stats tab, you will see the KBQueued column count up for your app's connection. If this number is zero even when you are seeing delays in your app, then the data isn't able to be delivered to your machine and is being queued on the server. Once we know this information it will be easier to identify what is causing the delay.

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 19, 2019 08:17 AM          Msg. 3 of 4
Thanks for the reply.

I have implemented the latest .net core 3.0 System.IO.Pipelines library. It "queues" and "dequeues" on separate threads while minimizing overallocation. With the help of the SequenceReader, you can access the buffer directly without having to copy to a string and then split. I have yet to test it with my mock server. I'll let you know if it performs better.

I also need to monitor KBQueued in code so I can log these infrequent latency events. Due to some logging changes, I was only able to see these events back in 2017. One of which coincided with an ECB announcement.

I will continue to monitor.

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2091
Joined: Nov 21, 2005


Posted: Feb 19, 2019 08:45 AM          Msg. 4 of 4
The KBQueued value in the diagnostics app is just a display frontend for the clientstats messages on the Admin port from IQFeed via the API.

Turning this on/off and message format is documented here http://www.iqfeed.net/dev/api/docs/AdminviaTCPIP.cfm

Not sure if you were aware of that based on your post so wanted to make sure.
Edited by DTN_Steve_S on Feb 19, 2019 at 08:47 AM
 

 

Time: Sat October 19, 2019 7:31 PM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003