Effective immediately, IQFeed has made a change to how it handles history requests.

Previously, IQFeed would allow up to 15 simultaneous active history requests. "Active" meant they had to still be returning data. When one request finished returning data, the number of active requests decreased by one, and another request could begin without exceeding this limit.

Going forward, iqfeed will allow up to 25 historical requests in a second, with no regard to whether past requests have completed. If your application makes a lot of small history requests at a time, this may cause you to start receiving the "E, too many requests" error message.

Whenever you make a new history request, your number of active requests is increased by 1. Every 1/25th of a second (40 milliseconds) this number decreases by one. As long as you don't exceed a rolling average of 25 history requests per second, you will avoid the "too many requests" error. In other words: time your requests such that you do not exceed 25 per second.

This change was made in hopes of setting a more consistent, and easier to understand, bar for historical requests. It was difficult to know whether you had exceeded the old "15 simultaneous active requests" standard; a limit of 25 per second regardless of how long each request takes to run is much easier to work with. Also, it sets a consistent standard for history requests of all sizes, that will work with our server.

These changes pertain to request sent to the history server (port 9100) only. Level 1 (port 5009) requests, and other requests, are not affected. If you have any questions about this change, please ask us!

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

UPDATE: The new limit has been raised from 25 historical requests in a second, to 50 historical requests in a second. We are all also sending out emails to users with more information about this change.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Hi, Gary. I'm confused about the purpose and implementation of the new limit.

Each day after the main trading session (at 7PM Eastern) I fetch tick history for all the symbols I'm screening. Currently there are 4090 of them.

My first reaction on hearing about the new rate limit was "That's fine; I'm OK with a lower limit, though I doubt I'm hitting 50 requests per second as it is." I run 8 threads, each of which performs history requests sequentially, so there are at most 8 requests in flight at any given time, and it takes a while for each one to complete and a new one to be issued. (The data has to be received, reformatted, compressed, and written to disk before a new request is issued.)

Yesterday, for example, the history fetch took 339 seconds, for an average of about 12 completed requests per second.

Today the history fetch took 2031 seconds, for an average of 2 completed requests per second. Wow; that's quite a change. It's slower than the days when I had a 6Mbps DSL line.

My best guess is that the majority of history requests finish quickly, so they're hitting the rate limit and are being delayed by as much as a factor of 6. However, a fair number of requests take a long time to finish, which drags down the average way below the nominal rate limit. High peak rate, low average rate.

So I'm curious as to whether your intent is to limit the rate at which requests are initiated, or the bandwidth demand on the servers. If the latter, the current approach may be too drastic; my connection is idle more than 80% of the time.

Hi @taa, can I see your implementation? I mean, 50 requests per second means 3000 requests per minute. How do you get 2 requests per second?

Mathieu

Hi, Mathieu. I don't object to showing you the code (other than embarrassment because some is old and crufty), but it's straightforward. This is a C++ socket-based app that has existed for more than a decade being run on a capable modern desktop machine with an AT&T gigabit fiber connection and no recent changes to the code or to its environment, so an overnight factor-of-six slowdown suggests to me that the new rate-limiting on the feed might be having more effect than intended.

Until the new rate-limiting started, I was averaging 12 completed requests per second, which seems pretty reasonable to me. Some loss of performance is caused by round-trip latency (for simplicity and error recovery, as well as compliance with previous IQFeed policy, each thread keeps only one request in flight at a time). But I suspect the main reason for the low average rate is that for some equities, an entire day worth of ticks takes a significant amount of time to prepare, transmit, and process. It's not just a matter of the maximum rate at which the app is allowed to issue history requests.

Following up with some data from my logs. Minor correction; I see there are 7 active threads, not 8; the code says I chose that to stay under half of IQFeed's suggested limit of 15.

On 2020-12-22, my history fetch showed mean latency (time from sending request to arrival of first data record) of 0.22 seconds. Min was about 0.1, median about 0.24, max about 0.6. Mean data transfer time (from arrival of first data record to arrival of last data record) was 0.09 seconds. Min was 0, median about 0.04, max about 10.2.

So, some equities generate a lot of data. For data transfer alone, the best average completion rate I could ever achieve would be about 80 requests per second. On the other hand, some equities trade very thinly, so their transfer time is essentially zero, and the peak request rate could be extremely high. This is what I mentioned in my earlier postings.

On 2020-12-23, the mean latency was 1.22 seconds (!); min 0.4, median 1.0, max 3.5. Mean transfer time was 1.9; min 0, median 0.04, max 118 (!!!).

Things got really slow on 2020-12-23 for both latency and throughput. Maybe it was just an odd coincidence that this happened the same day we received notice of the new rate limiting algorithm. I'm curious to see what happens tomorrow.

It's to limit the rate at which requests are initiated. If your software works as you describe - by not sending the next request until the last one is complete - then you should never afoul of the limit.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
Edited by DTN_Gary_Stephen on Dec 28, 2020 at 09:50 AM

Hi, Gary. Thanks for the update!

Just to clarify, is the rate limit per-connection, or per-IP-address (or some other key)? Since I have multiple threads issuing requests on separate connections, there are conditions under which more than 50 requests could be issued over the course of a second for all the connections in aggregate, though not for any individual connection.

I implemented a request-rate limiter last week, just in case. Maybe the observed slowdown was unrelated to the changes at your end. We'll see what happens today.

Back to normal performance levels today.

Hello everyone,

Just to inform you that I released IQFeed.CSharpApiClient 2.6.0 with the required change in order to limit 50 requests per second to avoid potential exception.

https://www.nuget.org/packages/IQFeed.CSharpApiClient/2.6.0

Code:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/commit/cc28fa80b009e048065cb9f6cccf1a97a9482132


Please review or comment.
Mathieu

My changes with 2.6.0 works well but you need to limit to 8 concurrent clients. For some reasons, passed this limit, even with the throttler in place, I'm getting ErrorMessage: Too many simultaneous history requests., MessageTrace: E,Too many simultaneous history requests.,

Hope this helps.
Mathieu
Edited by freemind on Jan 16, 2021 at 09:38 AM

For the record, the number of concurrent clients shouldn't be a factor. The server limits you to a total of 50 requests per second, no matter how many concurrent clients the requests are spread across. Let me know if you want me to dig deeper on this.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

That's strange. I believe your limit is a little bit behind 50 requests per second. I mean it should be less than 50 not equal to 50. My code has been well tested and I'm using high precision timers in order to throttle the output and using 50 triggers the exception. Received confirmation from others that adjusting it a little below fixed the issue.

I'll let you know once I have more info.
Thanks,
Mathieu

Just FYI, today I was seeing the response '!ERROR! Unknown Server Error code 0.' during history fetch from servers in the IP range 66.112.156.0/24. This occurred repeatably. I worked around it by adding a small delay between history requests.