Hi,

My objective is to understand for the last 90-180+ days, which stocks had the highest pre-market trading volume. I'm currently using TradingView's screener of 4000+ nasdaq stocks and downloading it each day, building it up over time - which is obviously taking a lot of time...

I was hoping to ask please if there is an efficient way of obtaining this via iqfeed? I thought about using the API to pull the data for all NASDAQ stocks over the last 90-180 days, but wasn't sure if there was a better approach (and if I'd get flagged for pinging so many stock tickers).

Many thanks in advance for any guidance you can provide

ChainsawDR

I still have this use case that I'm trying to address - it looks like I will need to pull the data from the API. Does anyone have any advise on how to structure the iqfeed request? for example, if wishing to pull the MSFT pre-market volume on 6/7/22, is there a way of requesting the volume for 9.29am & 59seconds?

Thanks in advance for any help (wishing to avoid making too many requests to the API)

Depends on what you mean by "avoid making requests." There are two approaches I can suggest:

1. The EDS market summary report. It won't be timed at 9:29:59 exactly, but depending on what data you want to collect, it might meet your needs. It is available on the History/Lookup port, and the syntax is:

Report,Security Type,Group ID,Date

Security Type is the number representing the type of securities you want, such as 1 for Equities and 8 for Futures. The list of codes can be retrieved from the SST command.

Group ID is the group of exchanges you want, like 5 for NASDAQ or 34 for CME. The list of Group IDs can be retrieved from the SLM command. Each exchange contains the group it belongs to: “1,NGM,Nasdaq Global Market,5,NASDAQ” would be part of Group 5.

Date is the date in YYYYMMDD format. Date is omitted for the 5MS command.

This returns a report containing the following fields:

Symbol,Exchange,Type,Last,TradeSize,TradedMarket,TradeDate,TradeTime,Open,High,Low,Close,Bid,BidMarket,BidSize,Ask,AskMarket,AskSize,Volume,PDayVolume,UpVolume,DownVolume,NeutralVolume,TradeCount,UpTrades,DownTrades,NeutralTrades,VWAP,MutualDiv,SevenDayYield,OpenInterest,Settlement,SettlementDate,ExpirationDate,Strike

2. It is possible to perform a tick history request which includes "filter time", meaning it will only return data from a certain time window each day. This is usually done to retrieve ticks data only from 9:30 to 4:00. But you could make a request like this:

HTT,AAPL,20220601 000000,20220609 130000,,092900,093000

This will request all ticks from June 1, midnight through June 9, 1 PM Eastern time. But there is a BeginFilterTime and EndFilterTime, 9:29 and 9:30 respectively. So the response will only include ticks during that time:

2022-06-02 09:30:00.000812,147.8000,26,1515398,147.7900,147.9300,910,O,18,87,0,2,
2022-06-02 09:30:00.000804,147.8400,2,1515372,147.7900,147.9300,909,O,18,87,0,2,
2022-06-02 09:29:59.825505,147.8100,100,1515370,147.7500,147.9300,4682,E,5,17,0,2,
2022-06-02 09:29:59.825502,147.8100,100,1515270,147.7500,147.9300,4681,E,5,17,0,2,

And so on. The highlighted number is the Total Volume. This approach may not be optimal for what you want to collect, but it does work.

By the way: the only hard limits on API requests is 50 individual historical requests per second, and your symbol limit for Level 1 watches (typically 1400). Beyond that it's a matter of how much data your code can process.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Thank you very much Gary. Yes option 2 looks like it will solve the problem. Thanks again for your help!

Hi Gary,

Hoping this may be a quick one... I'm trying to execute in Python the command you shared. During testing I'm trying to download for 4 stock symbols, however requests for GOOG are not returning. Any ideas if I'm doing something wrong here, such as missing specifying the exchange or similar when making the HTT request?

# iqfeed.py

import sys
import socket


def read_historical_data_socket(sock, recv_buffer=4096):
    """
    Read the information from the socket, in a buffered
    fashion, receiving only 4096 bytes at a time.

    Parameters:
    sock - The socket object
    recv_buffer - Amount in bytes to receive per read
    """
    buffer = ""
    data = ""
    while True:
        data = sock.recv(recv_buffer).decode('utf-8')
        buffer += data

        # Check if the end message string arrives
        if "!ENDMSG!" in buffer:
            break

    # Remove the end message string
    buffer = buffer[:-12]
    return buffer

# iqfeed.py

# iqfeed.py

if __name__ == "__main__":
    # Define server host, port and symbols to download
    host = "127.0.0.1"  # Localhost
    port = 9100  # Historical data socket port
    #syms = ["SPY", "AAPL", "GOOG", "AMZN"]
    #syms = ["SPY", "AAPL", "GOOG", "AMZN"]
    syms = ["AMZN"]

    # Download each symbol to disk
    with open("all.csv", mode='a+', encoding="utf8", newline='') as f:
        for sym in syms:
            print("Downloading symbol: %s..." % sym)

            # Construct the message needed by IQFeed to retrieve data
            message = "HTT,%s,20220609 000000,20220610 130000,,092900,092959\n" % sym

            # Open a streaming socket to the IQFeed server locally
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.connect((host, port))

            # Send the historical data request
            # message and buffer the data
            sock.sendall(message.encode())
            data = read_historical_data_socket(sock)
            sock.close

            # Remove all the endlines and line-ending
            # comma delimiter from each record
            #data = "".join(data.split("\r")) note: this is the original that doesn't include the symbol per line.
            #data = data + ","
            data = ("%s" % sym).join(data.split("\r"))
            data = data.replace(",\n","%s \n" % sym)[:-1]

            # Write the data stream to disk
            # f = open("%s.csv" % sym, "w") note: this is the original that saves each in an individual file

            # Move read cursor to the start of file.
            f.seek(0)
            # If file is not empty then append '\n'
            countdata = f.read(100)
            if len(countdata) > 0:
                f.write("\n")
            # Append text at the end of file
            f.write(data) 

or to simplify the question, any ideas why requests for "HTT,GOOG,20220609 000000,20220610 130000,,092900,092959\n" do not work, but requests for others like "HTT,AMZN,20220609 000000,20220610 130000,,092900,092959\n" do work?
Edited by ChainsawDR on Jun 16, 2022 at 03:53 PM

I'm guessing you didn't set the protocol. The S,SET PROTOCOL command should always be set when you first connect. (If you don't do this, IQFeed API assumes you want to use the much older protocol 4.9, which does not always accept the same commands as the current protocol 6.2 does. Your GOOG command fails if I don't SET PROTOCOL 6.2 first. AMZN succeeds. To be honest I'm not 100% sure why they don't both fail, but setting protocol first should render the problem moot.

There are other possible causes I can think, but none would apply to the commands as you list them here:

- Tick data older than 8 days cannot be retrieved during trading hours. (Your GOOG command requests data no older than 7 days)
- You lack the necessary exchange subscriptions. (GOOG is a NASDAQ symbol which is included in the IQFeed core package, so everyone has it)
- If you have delayed data, you can't make history requests that would return that day. In other words, if your NASDAQ data is the standard 10-minute delay as opposed to realtime, you can't make historical requests that would circumvent that. If you try to request ticks from 9:20 to 9:29 at 9:30, you'll get no data back. The same request would work at 9:40, because the 10-minute delay would have passed by then. (You're requesting a day's worth of data so it can't all be unavailable due to delay)
- Make sure the command is being assembled correctly. (The code suggests that it is. And the same command works fine for AMZN)

Send S,SET PROTOCOL 6.2 to port 9100 after you connect and try your code again. If that doesn't sort it, I may want to review your IQConnect.txt log file.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Thanks Gary. So the steps I'm currently performing are:
1) In Windows, open "IQFeed Applications" > Charts (which opens the connection to iqfeed from my computer)
2) In my Python IDE, I run the script shown above.

I have downloaded and installed the latest 6.2 version of the client and tried my existing code again, but no joy for GOOG again. TBH I'm not an experienced developer and I adapted the above script from a tutorial I found online. Do you know please if:
A) there is a way to set the protocol to 6.2 in step #1 above? Or
B) there's a method for adapting the above Python script to achieve this?

I've tried a few methods opening a socket to send that message, but the script stalls. I understand if not able to provide coding support (tried looking for sample files but unfortunately python isn't one of the languages listed).

Many thanks again

ChainsawDR

In case it helps. I tried setting the protocol as shown below, which generated the error further below:

Code:

# Construct the message needed by IQFeed to retrieve data
            message = "S,SET PROTOCOL,6.2\nHTT,%s,20220616 000000,20220616 130000,,092900,092959\n" % sym
#also tried multiple variations such as message = "S,SET PROTOCOL,6.2<CR><LF>\nHTT,%s,20220616 000000,20220616 130000,,092900,092959\n" % sym

            # Open a streaming socket to the IQFeed server locally
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.connect((host, port))

            # Send the historical data request
            # message and buffer the data
            sock.sendall(message.encode())
            data = read_historical_data_socket(sock)
            sock.close
            print("First part of data:" + data[0:100]) 

Error:

Downloading symbol: SPY...
First part of data:E,6.2 is not a valid protocol.,
2022-06-16 09:29:59,370.4600,100,3609350,370.4300,370.4600,80847523 

To answer question A: No. You must explicitly tell IQFeed API which protocol you want to use each time you connect, or it will assume the oldest one, 4.9. All you should need to do is send it a command, like your code does:

message = "S,SET PROTOCOL 6.2"
sock.sendall(message.encode())

or something like:

self._send_cmd("S,SET PROTOCOL,6.2\r\n" % protocol)

You will get a response S,CURRENT PROTOCOL 6.1 confirming this has been successful. Then your HTT command for GOOG should work correctly. I'm pretty sure not doing this is your problem, since i can reproduce the failure of the GOOG command when I don't set it.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Also, I thought of a way to improve the request I suggested. If you're defining "pre market volume" as "the total volume in the last tick before 9:30 am", you can actually do this more efficiently. Make this addition:

HTT,AAPL,20220601 000000,20220609 130000,1,092900,092959

The 1 is MaxDatapoints. This will give you only one datapoint that fits the description and filter. The default order is newest to oldest, so this will give you the very last tick before 9:30. That way you don't have to retrieve all the ticks in that minute.

Note also that the EndFilterTime is 092959. This will include that entire second, up to 9:29:59.999999, which I suspect is what you want. If you make the end time 093000, ticks as late as 9:30:00.999999 will be included in the results.

I should also point out that this approach requires that some trading have occurred between 9:29 and 9:30. For heavily-traded symbols like the ones in your examples, this is a safe assumption. But for lesser-traded symbols, this approach may not always work.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
Edited by DTN_Gary_Stephen on Jun 17, 2022 at 03:24 PM

Firstly, thanks for all of your help with this Gary. Secondly, love your last update - great idea! To confirm, will it always choose the latest tick in the range? Also, am I correct in thinking that for a multi-day request like 06/15 to 06/16, it will only pick the latest across all days, or would it return the latest per day? (if the former I can just use python to cycle through each day).

Coming back to the main problem, here's the summary of what I'm seeing...
If I update my message to send an OLD protocol like 5.1...

message = "S,SET PROTOCOL,5.1\r\nHTT,%s,20220615 000000,20220616 130000,,092900,092959\n" % sym 

... then I get a good response back: S,CURRENT PROTOCOL,5.1 - plus I get all of the info for GOOG (still testing, but specifying an old protocol seems to be working). However if I take this exact same code and the only change I make is to update the protocol from 5.1 to 6.2, then it gets weird. A) I get this error response: E,6.2 is not a valid protocol. B) I now get 1 row of data for GOOG back (before specifying protocol, I used to get 0 rows back). This feels super weird - sending 6.2 is getting an error, but at the same time it is meaning I get 1 row of data for GOOG vs 0 rows of data.

All in all, one request I'd love to make if possible would be to add a basic sample code for Python to the IQ Developer Support section. I know a lot of experienced devs use C++ & Java, but Python is quite a common entry level language so hoping it could be supported to a MVP level like opening a socket, setting protocol, submitting a request (to take off the table situations like this when I'm wondering if its my coding or the system - for this it looks like a problem with the system, both with GOOG data responding differently and with accepting the latest protocol incorrectly).

Will crack on with trying to get it working using the old 5.1 approach, and loop back if no joy (but please shout if you already know there'll be an issue with using the old protocol).

Thanks again for your help Gary, really appreciate it!

ChainsawDR
Edited by ChainsawDR on Jun 17, 2022 at 05:24 PM

To use protocol 6.2, you must of course install the IQFeed 6.2 client. If for example you only have a 6.0 client, then you will not be able to use the 6.1 or 6.2 protocols, only 6.0 or older.
You can download the latest IQFeed client from http://www.iqfeed.net/index.cfm?displayaction=support§ion=download

---

Yair Altman
IQML - IQFeed-MATLAB connector
https://UndocumentedMatlab.com/IQML

I am not a DTN employee; my post reflects my personal opinion

Thanks altmany. Yes understood. I originally had an older client, however updated to 6.2 prior to my recent posts. Unfortunately I'm still getting the issue when setting 6.2 or not setting a protocol at all, however specifying 6.1 seems to be working absolutely fine.

Yes. The MaxDatapoints parameter applies to the entire command. If I ask for 1 datapoint of a two-week period, I get one datapoint total, not one datapoint per day:

HTT,AAPL,20220612 000000,20220620 130000,1,092900,092959
LH,2022-06-17 09:29:59.992136,130.0600,511,1955010,130.0600,130.0700,5783,E,5,17,0,17,
!ENDMSG!,

Omitting the MaxDatapoints feature will return the 9:29-9:30 time period for all days in range. Which may have other benefits: if you want to confirm the volume of the very last tick before trading begins, this may not be the very last tick before 9:30:00.000000. You can collect all ticks in a minute (or any number of seconds) and look to the Basis For Last datapoint (the E in the above example) to identify the final trade that was Extended as opposed to C (qualified) or O (other). It depends on how precisely you're defining "last."

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist