andrewm has contributed to 47 posts out of 20869 total posts
(0.23%) in 2,956 days (0.02 posts per day).
20 Most recent posts:
If IQ Feed would publish the protocol I'm sure it would be fairly easy for the community to whip up a decent open source Linux client.
Happy new year!
It's 2023. It's time to build a Linux command line (no GUI) version of the IQ Feed client.
Here's the others I collect:
cboe_index.py:
from main import start from datetime import datetime
print('starting cboe index at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\cboe_index' TEMP_FILE = 'c:\\temp\\cboe_index.csv' COMPRESSED_FILE = "cboe_index.7z" SECURITY_TYPE = 6 # CBOE MARKET = 13 # index start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
cbot_futures.py
from main import start from datetime import datetime
print('starting cbot futures at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\cbot_futures' TEMP_FILE = 'c:\\temp\\cbot_futures.csv' COMPRESSED_FILE = "cbot_futures.7z" SECURITY_TYPE = 8 # futures MARKET = 30 # CBOT start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
cme_futures.py
from main import start from datetime import datetime
print('starting cme futures at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\cme_futures' TEMP_FILE = 'c:\\temp\\cme_futures.csv' COMPRESSED_FILE = "cme_futures.7z" SECURITY_TYPE = 8 # futures MARKET = 34 # CME start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
cme_options.py
from main import start from datetime import datetime
print('starting cme options at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\cme_options' TEMP_FILE = 'c:\\temp\\cme_options.csv' COMPRESSED_FILE = "cme_options.7z" SECURITY_TYPE = 9 # options MARKET = 34 # CME start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
nasdaq_equities.py
from main import start from datetime import datetime
print('starting nasdaq at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\nasdaq' TEMP_FILE = 'c:\\temp\\nasdaq.csv' COMPRESSED_FILE = "nasdaq.7z" SECURITY_TYPE = 1 # equities MARKET = 5 # NASDAQ start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
nyse_equities.py
from main import start from datetime import datetime
print('starting nyse at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\nyse' TEMP_FILE = 'c:\\temp\\nyse.csv' COMPRESSED_FILE = "nyse.7z" SECURITY_TYPE = 1 # stock MARKET = 7 # NYSE start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
You're welcome. Glad it helps. It's a little rough around the edges. getDirectory should be made functional instead of having the side effect of also creating the dir. Some logging would be nice, etc. But it works...
Here's a couple python files that will collect the OPRA snapshots and compress them to 7z format. I put this in a windows scheduled task. You can replace opra.py values with other markets. I collect several.
opra.py:
from main import start from datetime import datetime
print('starting opra at ' + str(datetime.now())) DATA_PATH = 'c:\\data\\opra' TEMP_FILE = 'c:\\temp\\opra.csv' COMPRESSED_FILE = "opra.7z" SECURITY_TYPE = 2 # equity options MARKET = 14 # OPRA start(DATA_PATH, TEMP_FILE, COMPRESSED_FILE, SECURITY_TYPE, MARKET)
main.py:
import shutil import socket from datetime import datetime from pathlib import Path import subprocess from os import makedirs
ZIP_EXE = 'c:\\util\\7z\\7z.exe' HOST = '127.0.0.1' PORT = 9100
def getDirectory(data_path): today = datetime.today().strftime('%Y-%m-%d') todaysPath = data_path + '\\' + today Path(todaysPath).mkdir(parents=True, exist_ok=True) return todaysPath
def get7zFilename(): now = datetime.now() print("now =", now) today = datetime.today().strftime('%Y-%m-%d') dt_string = today + "_" + now.strftime("%H%M") print("date and time =", dt_string) filename = dt_string + ".7z"; print("filename = "+filename); return filename
def start(data_path, temp_file, compressed_file, security_type, market): print('start sec type ' + str(security_type) + ' market ' + str(market)) with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.connect((HOST, PORT)) s.sendall(b'S,SET PROTOCOL,6.2\n') s.sendall(bytes('5MS,' + str(security_type) + "," + str(market) + '\n', 'ascii')) f = s.makefile() line = f.readline() # for line in f.readline(): print("line="+repr(line)) line = '' outfile = open(temp_file, "w"); while not line.startswith('!ENDMSG!'): line = f.readline() outfile.write(line) #print("line=" + repr(line)) outfile.close() f.close() dir = getDirectory(data_path) fn = dir + "\\" + get7zFilename() subprocess.call([ZIP_EXE, 'a', fn, temp_file]) # shutil.copy2(fn, dir + "\\" + compressed_file) print("done") # print('Received', repr(data))
Any suggestions on this? Can the current bid/ask be added to the 5MS snapshots?
Also the documentation should make it clear that the 5MS snapshots don't contain the current bid/ask since including week-old bid/ask in a 5-min snap is really counterintuitive. Edited by andrewm on Jun 5, 2022 at 08:57 AM
No reply.

I believe it's not supported
if a spread trade executes at a single price on an options exchange's Complex Order Book, how would that be reported on OPRA? How would they decompose the spread into individual legs with an execution price for each leg?
Can we see US options exchanges' complex order books?
The problem with these 5MS market summary snapshots is that they don't give you the current bid/ask. They give you the bid/ask at the time of the most recent trade which may have been hours, days or weeks ago. Right?
Can that be fixed?
A command to get a current snapshot of the full options chain for one expiration date like yahoo shows would be nice:
https://finance.yahoo.com/quote/AAPL/options?p=AAPL
They give: Contract Name Last Trade Date Strike Last Price Bid Ask Change % Change Volume Open Interest implied vol <- probably not needed
Currently with IQ Feed I think you have to query the whole chain, then iterate over it, subscribing and unsubscribing to each symbol. Right?
Keep in mind that the bid/ask in 5MS were at the time of the last trade, which may have been days ago.
The 5MS snapshot is NOT delivering the current market data. The bid/ask is of the time of the last trade which may have been days earlier.
The docs say this:
5MS,[Security Type],[Group ID],[RequestID]<CR><LF> 5 Minute Snapshot Summary - Retrieves a snapshot of the current market data for all symbols in a Security Type and Exchange Group. NOTE: The timing of the snapshot is not guaranteed, but data will be gathered every 5 minutes.
You might find this thread helpful:
http://forums.dtn.com/index.cfm?page=topic&topicID=5679
> You'll find that when you use that mechanism for liquid instruments > the quality of data becomes lacking due to the high frequency aspects of the data.
Agreed that it's not perfect however it's much better than nothing which is effectively what we have now for historical intraday quotes on thinly traded products such as nearly all equity option contracts.
> Collecting data real-time during the day allows you to catch the stream as it happens
Except:
You can only collect as many symbols as your account is permissioned for. If you decide to test a new strategy on 6 months of historical data you must first spend 6 months collecting it. If you want to know how your new strategy would have performed during some historical event when you weren't collecting data on that particular symbol, too bad.
> I believe DTN's choice of quote at trade storage is a reasonable trade-off.
It may work for @ES and SPY but for a product that rarely trades it's mostly useless. The data storage requirements for snapshotting the BBO would be minimal since snapshots would only need to include changed quotes. Even if the symbol's quote changed every second, 23 hours a day and 100 byte records were stored that's only 8MB/day/symbol. Actual sizes would be far smaller.
Being able to see historical bid/ask changes only when a trade happens makes intraday analysis of thinly traded products impossible.
It forces users who need this data to stream subscriptions and capture them in real time locally. That's an unreliable solution over the internet and something users should not be wasting their time building.
IQ Feed could snapshot the BBO once per second and make that historical data available via the API or better as an EOD FTP file for subscribers. It could be a replay of a realtime feed that coalesces ticks into buckets sent at a maximum update frequency of 1 sec. Even 5 second snapshots would be ok.
For now I call a python script every 5 mins to dump the 5MS output for CBOE to a file and compress it. If anyone wants that script, let me know. Edited by andrewm on Apr 7, 2021 at 10:53 PM
It would be great if 5MS, etc would include !ENDMSG! like some other commands do.
edit: disregard. It does have it. Edited by andrewm on Mar 13, 2021 at 02:59 AM
The docs say:
5 Minute Snapshot Summary - Retrieves a snapshot of the current market data for all symbols in a Security Type and Exchange Group. NOTE: The timing of the snapshot is not guaranteed, but data will be gathered every 5 minutes.
What does that mean? If I issue a 5MS command every 5 minutes at times ending :06, :11, :16, etc will I get the 1 minute old snapshot?
Thanks
Bid and ask time would just be the update time of the value. If a bid is updated at 10am and I generate a 5-min summary 2 hours later that file generated at noon would show 10:00 bid time. If there have been no bids or offers that day the bid or ask value and time would show an empty field ",," not a zero value ",0,"
Please let me know if those bid/ask values carry over from previous 5-min periods. Thanks a lot.
Are the fields "Open,High,Low,Close" in these 5 min files the OHLC values for the full day? They're not the 5-min bar OHCL values, right?
Is any bid or ask value shown sure to be from the current trading day because any bids/offers resting in the book are cancelled at EOD?
Could you add bid-time and ask-time fields to these files?
|