Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"You have an excellent product !!!!!!" - Comment from Arely
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"The people at Nirvana have very nice things to say about your company and I can see why! Price and service is a potent combination." - Comment from Ed
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"I have been using IQFeed now for a few years in MultiCharts and I have zero complaints. Very, very rare to have any data hiccups or anything at all go wrong." - Comment from Public Forum
"As a past ******* customer(and not a happy one), IQ Feed by DTN is a much better and cheaper product with great customer support. I have had no problems at all since switching over." - Comment from Public Forum
"I "bracket trade" all major news releases and I have not found one lag or glitch with DTN.IQ feed. I am very comfortable with their feed under all typical news conditions (Fed releases, employment numbers, etc)." - Comment from Public Forum
"This beats the pants off CQG, I am definitely switching to the ProphetX 3.0!" - Comment from Stephen
"Boy, probably spent a thousand hours trying to get ******* API to work right. And now two hours to have something running with IQFeed. Hmmm, guess I was pretty stupid to fight rather than switch all this time. And have gotten more customer service from you guys already than total from them… in five years." - Comment from Jim
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Latency of Q messages
Author Topic: Latency of Q messages (4 messages, Page 1 of 1)

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 11, 2019 08:00 PM          Msg. 1 of 4
Hi,

I've been trying to optimize my .net socket reader. First, I wrote my own server and pushed 1000 "Q," messages per second to help find any bottlenecks in my code. The mock server writes 1000 dummy Q messages in a loop. I've appended a high resolution time stamp to the messages (using interop GetSystemTimePreciseAsFileTime). I compare this time stamp on the reading side to see how fast I process messages. The writes of 1000 messages only takes a few milliseconds. I noticed that I was unable to keep up with this rate on the read side if I processed the message synchronously. I now push the messages to a BlockingCollection(ConcurrentQueue). Once I did this, my socket mostly reads one or two message per read.

I thought this would help reading the live feed. However, I still see many reads with numerous messages (>50). This is averaging 5, but sometime as many as 100+ messages are pulled in one read. Most of the time, these messages only differ in the Bid/Ask time stamp by a few milliseconds but sometimes are > 50ms (as high as 200 ms) from the first message to the last.

Is IQConnect batching the writes to the socket? Or are you batching the push from your servers? No matter what I do I cannot seem to reduce this.

I should note that I am collecting stats in an array and only dumping one line to the console every 10-15 second. I have timed the stat collection and display and I am confident that it is not skewing the results.

I need to sort this out not because I’m concerned about a few milliseconds but because I can get way behind (1-30 seconds!) in a fast market.

I am running on an AWS EC2 T2.Medium machine (2 cores).

My pings to the IQFeed server are constantly between 45-47 ms.

I am only watching CME products (@ES, @NQ, 5 major currencies)

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2087
Joined: Nov 21, 2005


Posted: Feb 19, 2019 06:38 AM          Msg. 2 of 4
Hello, my apologies for the delay responding to this.

Generally speaking, the markets are naturally very "bursty" in terms of how messages are received from the exchanges. I haven't verified with your specific symbols but you are using some of the most active futures contracts as I'm sure you know. I wouldn't be surprised if your few symbols frequently burst well beyond your 1000 message loop test (unless I'm not understanding your test).

A 30s delay is certainly not normal and figuring out where that data is getting queued will help identify what needs to be fixed. If the data is getting to your machine but queued locally within IQConnect waiting to be delivered to your app, then in the diagnostics app, on the client stats tab, you will see the KBQueued column count up for your app's connection. If this number is zero even when you are seeing delays in your app, then the data isn't able to be delivered to your machine and is being queued on the server. Once we know this information it will be easier to identify what is causing the delay.

mac
-Interested User-
Posts: 17
Joined: Apr 6, 2017


Posted: Feb 19, 2019 08:17 AM          Msg. 3 of 4
Thanks for the reply.

I have implemented the latest .net core 3.0 System.IO.Pipelines library. It "queues" and "dequeues" on separate threads while minimizing overallocation. With the help of the SequenceReader, you can access the buffer directly without having to copy to a string and then split. I have yet to test it with my mock server. I'll let you know if it performs better.

I also need to monitor KBQueued in code so I can log these infrequent latency events. Due to some logging changes, I was only able to see these events back in 2017. One of which coincided with an ECB announcement.

I will continue to monitor.

MAC

DTN_Steve_S
-DTN Guru-
Posts: 2087
Joined: Nov 21, 2005


Posted: Feb 19, 2019 08:45 AM          Msg. 4 of 4
The KBQueued value in the diagnostics app is just a display frontend for the clientstats messages on the Admin port from IQFeed via the API.

Turning this on/off and message format is documented here http://www.iqfeed.net/dev/api/docs/AdminviaTCPIP.cfm

Not sure if you were aware of that based on your post so wanted to make sure.
Edited by DTN_Steve_S on Feb 19, 2019 at 08:47 AM
 

 

Time: Mon May 25, 2020 9:26 PM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003