Quote: Hi, Gary. I'm confused about the purpose and implementation of the new limit.

Each day after the main trading session (at 7PM Eastern) I fetch tick history for all the symbols I'm screening. Currently there are 4090 of them.

My first reaction on hearing about the new rate limit was "That's fine; I'm OK with a lower limit, though I doubt I'm hitting 50 requests per second as it is." I run 8 threads, each of which performs history requests sequentially, so there are at most 8 requests in flight at any given time, and it takes a while for each one to complete and a new one to be issued. (The data has to be received, reformatted, compressed, and written to disk before a new request is issued.)

Yesterday, for example, the history fetch took 339 seconds, for an average of about 12 completed requests per second.

Today the history fetch took 2031 seconds, for an average of 2 completed requests per second. Wow; that's quite a change. It's slower than the days when I had a 6Mbps DSL line.

My best guess is that the majority of history requests finish quickly, so they're hitting the rate limit and are being delayed by as much as a factor of 6. However, a fair number of requests take a long time to finish, which drags down the average way below the nominal rate limit. High peak rate, low average rate.

So I'm curious as to whether your intent is to limit the rate at which requests are initiated, or the bandwidth demand on the servers. If the latter, the current approach may be too drastic; my connection is idle more than 80% of the time.
--- Original message by taa_dtn on Dec 23, 2020 07:15 PM