Tim,

This message is with reference to your post from:
***
Posted: Aug 2, 2018 07:35 AM Quote Post Reply To Post Msg. 18 of 30
There is some work being done on our website currently and that bumped the automation back, but I did place a file up a couple weeks ago for everything that was expired up to that date if you want to take a look at it.

ftp://www.dtniq.com/beta/IEOPTION.zip
***

Goal:
Get Options Tick Data for Expired Options and Current Options for the 6 months.

I downloaded this file and parsed the file. However the last trade date is 20180713

Q1. I am assuming that ftp://www.dtniq.com/beta/IEOPTION.zip contains only Option Symbols that have expired and the last date they traded? For example, 20180710 has only 24103 symbols as you can see below and
http://www.dtniq.com/product/mktsymbols_v2.zip file contains approx 900k option symbols which are all currently trading symbols.

Can you please verify this assumption is accurate

Q2. Can you kindly update this file ftp://www.dtniq.com/beta/IEOPTION.zip until 20180817 (The last date is 20180713)

After this - i can get latest symbols daily from http://www.dtniq.com/product/mktsymbols_v2.zip and request IQFeed data.

Your help is very appreciated. Thank you.

Please see attached file: (IEOptionsFile_DateSym.xlsx)
Date Symbols
20180713 603
20180711 130
20180710 24159
20180709 112
20180707 385
20180706 403
20180703 27661
20180702 87
20180701 386
...

---

AK786

Good morning,

Yes, your assumption is correct. I will try to get a new file created Monday morning for you when I get back in.

Tim

Thank you very much, your help is very much appreciated. Very satisfied with IQFeed Service.

---

AK786

Tim,

This is WRT the post from:
***
Posted: Aug 7, 2018 08:32 AM Quote Post Reply To Post Msg. 22 of 33
Actually, the expiration date is the blank field, the other date is the last traded date.

I'll correct the email above to be accurate. Expiration can be inferred from the symbol, but we will look into that prior to release as well.

Tim
***

I parsed out Sample Data from the IEOptions File from 20150101 to 20180713

Somehow, for the a Large Number of Options, The Last Traded Date is after the Expiration Date.

Can you kindly explain how that might have happened.

Please see attached. I know you will be generating the latest IEOptions FIle tomm, so I wanted to make sure your efforts do not go to waste.

Thank you

---

AK786

Also another question is how Far Options History do you carry, i tried requesting Options Tick Data between
<OptionsBeginDate value= "20180501"/>
<OptionsEndDate value= "20180713"/>

for all options in (From the above attached file) for Equity Symbol AAPL and i did not get a single data tick data history

---

AK786

@AK786,

Assuming you are paying for the OPRA feed, there is probably something wrong with how you are querying. The typical problem I observed has to do with naming the symbol correctly.

Here is a sample query that worked for me just now. I picked one of the first AAPL symbols from the ftp file mentioned above.

HDT,AAPL1001J210,20090101,,,1,AAPL1001J210_HISTORY_c7gxv4myzxvlt99c8kw6,

(Obviously the query identifier -- the string that has HISTORY in it-- is optional and can be omitted).
The reply I got was one line:

AAPL1001J210 2010-09-28T17:05:54 76.34 68.34 68.34 73.69 20 0

Oh right. You mentioned tick data. Try pulling 1 minute bars to validate the query first, then try for tick.

Thank you J - you are correct,

Tim,

Regarding this - I was making a mistake at my end.
This issue is resolved.

Please let me know about the other situation related to last trade date (after expiration) - its still not a problem, if its a system issue at your end.

Thanks

AK


****
Also another question is how Far Options History do you carry, i tried requesting Options Tick Data between
<OptionsBeginDate value= "20180501"/>
<OptionsEndDate value= "20180713"/>

for all options in (From the above attached file) for Equity Symbol AAPL and i did not get a single data tick data history
***

---

AK786

Hi guys,

Well, evidently there is more work to be done on that file creation process. This file is all I have ready access to give you though, so while the symbols are up to date, these last trade dates do appear to be wrong. We will get that addressed before we go live of course. In the meantime, you could do an HDD request on each to get the last traded date. The expiration is in the symbol itself if you wanted to parse it out for the ones it is blank on, but otherwise when it is there, it appears to be correct.

The current file is on the FTP now.

Tim

Thank you Tim,
I think we should keep the file option.. For me this is a more convenient way to retrieve expired options. Can you expose to us the long term goal get them at the end?

Thank you.
Mathieu

Having the file is part of the long term view, we just need to finish it up and get it QA'd of course, so that when you are receiving it, it is correct.

Tim

Excellent news Tim, I'm in the process of adding support for this file in the IQFeed.CSharpiApiClient wrapper. Will be very easy to use. Ill post back on this thread once available.

Mathieu

We plan on putting in a header at the top as well, so you might want to consider for that as well.

Once I know where the file will live I will let everyone know.

Tim

To all IQFeed Forum Readers,

i just wanted to share that I have been using Matt P. (IQfeed Csharp Client) and its phenomenal. Tick Data download for all US Equities (Approx 8000) can get done within 15 mins / each day. Options data download which can have over a million queries for entire dataset - lightning fast even. I downloaded all US Equity Options for 6 months which has "Expired" + "Recently Traded" Options.

The Live Data Client runs on super low CPU and memory. Check out his code, he has saved me tons of time. I am very grateful to him for responding to my queries almost immediately and resolving minor bugs if any.

Thank you Matt P. and all his IQFeed github contributors.

---

AK786

FYI a new file was posted today if anyone wanted to update their lists.

Tim

Thank you so much Tim.
BTW guys, in the newest version 1.4.2 of IQFeed.CSharpApiClient, I've added a very simple way to retrieve those expired options and market symbols.

Code sample:
var lookupClient = LookupClientFactory.CreateNew();
var expiredOptions = lookupClient.Symbol.GetAllExpiredOptions()
.Where(x => x.EquityOption.Expiration >= DateTime.Now.AddMonths(-6)).ToList();

var marketSymbols = lookupClient.Symbol.GetAllMarketSymbols().ToList();

This will take some time the first time but the code will reuse the cached file and check for new update every time.

Hope this helps.
Mathieu

I added more examples including one with Expired Options:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/tree/master/src/IQFeed.CSharpApiClient.Examples

Thank you,
Mathieu

In using a modification of Mathieu's excellent client (Thanks, Mathieu) I'm seeing symbols that are in IEOPTION but appear to have no data. Could this be verified and, preferably, the data added or the symbols removed from the file.

SPY1018L180 !NO_DATA!
SPY1018L185 !NO_DATA!
SPY1030I71 !NO_DATA!
SPY1228L130 !NO_DATA!
SPX1018L550 !NO_DATA!
SPX1117L525 !NO_DATA!
SPX1118F550 !NO_DATA!
SPXW1531H1795 !NO_DATA!
VXX1228L23 !NO_DATA!
VXX1228X24 !NO_DATA!
AAPL1703K110 !NO_DATA!

Thanks.
Edited by veng1 on Oct 8, 2018 at 09:50 AM

Daily data does exist for the ones I checked. They were likely an implied trade or something along those lines, since we did not create a minute bar, but that is why these are in the files.

I will check into how this is filtered with regards to symbols that receive no trading at all. But, I believe they are filtered.

Tim
Edited by DTN_Tim Walter on Oct 8, 2018 at 10:10 AM

Does the daily data actually include at least one trade?

The ones I checked did, but I need to clarify something with regards to this file. It is not an official offering for us. The file and it's creation are not finalized. It has known bugs and we still have work to do on them, work that does not have a current ETA for completion.

We certainly want to be able to help and provide information for everyone to use. That is why we are here and Mathieu does a fantastic job with his project, staying up to date and in this case, ahead of our release schedule. But, I need to be clear that this is a process that in an alpha phase, not production, so please take everything in them with a few grains of salt.

Tim

I appreciate that this is an alpha product. My code traps an error to a file but does not attempt to save the trade to csv if there is "NO DATA" returned.

My post was intended to be constructive in an effort to help find bugs.

Absolutely, and there are a couple places that I put in research tasks towards that end as well based upon your notes. I just didn't want someone new to the thread coming in and misunderstanding what these files contain. But, at this time the plan is to have symbols be in the file if they have a minute, or daily data available. I did find that I cannot find any data on the AAPL symbol you reported, so I will add that to the task for additional research.

I say files, because eventually we plan to have this for all exchanges and security types, not just IEOPTION, but that is the primary one requested so we are starting with that.

I am particularly looking forward to expired futures contracts. My requirement is to calculate contango and backwardation. In an ideal world, there would be a continuous front month and continuous back months so the the term structure would be simple math.

If there are at least all the past symbols it would be easier.
Edited by veng1 on Oct 8, 2018 at 02:44 PM

I am wondering what your expectations are. Have you followed options on the real time feeds? You'll note that there are continuous quotes available, but there are very infrequent trades happening. And if history only has the trades stored, the trades (if they are available), may not be indicative of the open, the close, the min, the max or anything in between. And that may be the reason why you see NO_DATA, because, well, the option was available, but no one traded anything. Ie, repeating myself, trades are very sparse in the options world.
Edited by stargrazer on Oct 8, 2018 at 03:04 PM

Due to the fact that trades can be sparse on back months on even popular symbols, I've found it of more value to capture and store daily feeds, keep the quotes as well as the trades, and then perform algorithm analysis based upon the quotes captured. I build my own daily summaries from the quotes, and typically use quote mid-points for rough approximations. This allows me to deal with open, close, and any time in between without worrying about the lack of trades.

You will also note that, as various sites say in the footers, past history may not be a good prognosticator of current movements. Which suggests even more value in collecting your own current data, complete with both quote and trade info, to use in your modeling and algorithm development.

After the demise of XIV, I'm looking at option trades for backtesting, primarily on EOD, with a vertical on VXX. I believe that the term structure needs, among other things that are needed, to do a synthetic short, exit or hold cash. The reason for 1 minute data is to look for block trades, not so much as price action.

The term structure for VIX can be calculated on daily basis. Whether there are suitable options trades remains to be seen. I don't much care to simply short VXX.

I agree in the value of collecting one's own data but that can only be done one day at a time. However, it is a fast trip to the poor house to trade based on past history without forward testing.

And yes, I've watched the tape on options, specifically on Livevol for literally years.
Edited by veng1 on Oct 8, 2018 at 03:38 PM

I am curious. I gather you have already pulled down historical option tick data and/or historical option 1 minute bar data for your symbols of interest for what they supply currently. Do you get coverage through out the day, or is it pretty sparse? Do you see ticks or bars falling at EOD regularily?

Picking a random expiration, the most popular strike, by file size, SPXW1708L2570 has 40 EOD prices.

The same strike has 288 1Min quotes starting at 2017-10-12 10:55:00 and ending at 2017-12-08 15:43:00.

Interestingly, SPXW1708L2650_Min has 1866 quotes.

I'm not pulling tick data.