|
|
Home
Search
Register
Login
Recent Posts
| Viewing User Profile for: alexB | |||||||||||||||||||||||
| About | Contact | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Post Statistics | |||||||||||||||||||||||
|
alexB has contributed to 75 posts out of 21185 total posts
(0.35%) in 5,241 days (0.01 posts per day).
20 Most recent posts:
Data and Content Support »
Corrupt data feed Aug 28, 2010 11:38 AM
(Total replies: 23)
Steve hi, I just finished an extensive test. I deleted some of my old historical records going abck to Jan 1, 2010 which were definitely corrupt. I saw the corruption on my graphics. I then re-downloaded the data and lo and behold the corruption disappeared completely. I consider the problem solved. Again, I want to say that I am very greatful to you for such a quick and sharp spotting of those two bugs. Many thanks, - Alex AlexB
Data and Content Support »
Corrupt data feed Aug 27, 2010 07:38 AM
(Total replies: 23)
Steve hi, I downloaded a new set of minute quotes last night and checked a few stocks today, especially the ones that were corrupted before. It appears you've cured the problem. Still I will need a few more days of observation to confirm but as of now I am very encouraged. Also this weekend I will try to review the code myself and try to analyze it with extensive printouts of lines at the juncture of two packets while they are coming from the cocket. Again, thank you very much. This is what a fresh look is capable of doing. P.S. Tonight (Friday August 27, 2010) is the second night of clean download. Everything is working perfectly well. AlexB Edited by alexB on Aug 27, 2010 at 04:42 PM
Data and Content Support »
Corrupt data feed Aug 26, 2010 10:15 AM
(Total replies: 23)
Steve hi, Thank you very much. Inattention is my curse. I fixed the first issue and will try it again tonight. I appreciate your help very much. AlexB
Data and Content Support »
Corrupt data feed Aug 26, 2010 07:57 AM
(Total replies: 23)
Steve hi, I downloaded symbols yesterday after making changes you suggested. I hope I made them all at the right place. Still I beliveve the problem persists. I found an errant value in F (Ford) at 15:30, a typical time when such values occur. The stock preceding it is ERTS and when I looked into it I found for the first time that that bin was missing. These are some F quotes: 11.309 11.32 11.3 11.315 60803 2010-08-25 15:27:00.000 11.31 11.32 11.31 11.32 51536 2010-08-25 15:28:00.000 11.3125 11.32 11.31 11.31 63769 2010-08-25 15:29:00.000 15.18 15.18 15.17 15.18 29753 2010-08-25 15:30:00.000 11.315 11.315 11.3 11.3093 66205 2010-08-25 15:31:00.000 11.3 11.34 11.3 11.33 248474 2010-08-25 15:32:00.000 11.34 11.36 11.33 11.36 257487 2010-08-25 15:33:00.000 11.355 11.36 11.34 11.345 204872 2010-08-25 15:34:00.000 These are ERTS quotes. 15.21 15.22 15.2 15.2 4900 2010-08-25 15:19:00.000 15.2 15.2075 15.19 15.2 4300 2010-08-25 15:20:00.000 23.1 23.105 23.09 23.09 29145 2010-08-25 15:21:00.000 15.19 15.19 15.16 15.18 26927 2010-08-25 15:22:00.000 15.18 15.18 15.16 15.17 15089 2010-08-25 15:23:00.000 15.17 15.17 15.17 15.17 1300 2010-08-25 15:24:00.000 15.17 15.17 15.17 15.17 1000 2010-08-25 15:25:00.000 15.1671 15.18 15.16 15.18 10564 2010-08-25 15:26:00.000 15.18 15.18 15.17 15.1735 1236 2010-08-25 15:27:00.000 15.18 15.18 15.17 15.17 3400 2010-08-25 15:28:00.000 15.18 15.18 15.17 15.18 2000 2010-08-25 15:29:00.000 15.17 15.19 15.17 15.19 48299 2010-08-25 15:31:00.000 15.19 15.23 15.19 15.23 36063 2010-08-25 15:32:00.000 15.23 15.24 15.23 15.24 9738 2010-08-25 15:33:00.000 15.24 15.25 15.24 15.24 37274 2010-08-25 15:34:00.000 You can see that ERTS has one line missing as well as an errant quote prsent at 15:21. Would you take another look into my code? I am sending to you an email with the modified Form1 class. Also tell me if you need other classes. Thanks. AlexB
Data and Content Support »
Corrupt data feed Aug 25, 2010 11:30 AM
(Total replies: 23)
Steve hi, In the previous emails I sent to you I also gave you a briefing as to how to view it. It is my working file which obviously contains a bug somewhere. Also you should keep in mind that this application performs other tasks and it clouds the things you will have to look into. I did not want to edit because I thought it could castrate the code you really need. Some parts may still be outside Form1. If you found something missing, please let me know, I will send those parts as well. I also want to say that I appreciate your help and I hope a fresh eye of an expert may uncover the problem that's been bugging me. Many thanks, - Alex AlexB
Data and Content Support »
Corrupt data feed Aug 25, 2010 07:39 AM
(Total replies: 23)
Steve hi, Yesterday I did what you asked me to. I zipped one of my major files in the project (Form1), changed the extension to .ChangeToZip and emailed it to you. Have you gotten it? Thanks. AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 03:19 PM
(Total replies: 23)
Steve, I don't know if it has any relevance to the subject in question but I also observe the stocks in Quote Tracker very nice pictures/graphics. I have 30-inch screens on my monitors and have all those 40+ stocks displayed. Currently I use IQFeed but the thing I am about to describe is not unique to IQFeed. I observed it also with TD-Ameritrade feed. Once in a while the values of some stocks jump unreasonably high or less frequently drop. The changed value stays for a minute or two and then comes back to baseline. It reminds me what I see in my tables. QT obviously is not connected to my application. AlexB Edited by alexB on Aug 23, 2010 at 03:20 PM
Data and Content Support »
Corrupt data feed Aug 23, 2010 03:12 PM
(Total replies: 23)
It is an amazing dicovery. How could it be? I cannot picture the process that leads to it. Very strange. Thanks. AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 02:49 PM
(Total replies: 23)
Thank you Steve, It gives me hope. Your email address must be -edited to remove address- correct? I will send you my code. This is a complete list of my symbols, the ones that are used to download historical data: AAPL ADP AIG APC ATVI BA BAC BHP BIDU BP BRCM C COP CSCO DELL DLTR EBAY ERTS F GOOG GPS GS HPQ IBM INTC INTU JNJ KLAC MCD MSFT QQQQ RIMM RMBS RSH SBUX T UUP XLB XLE XLF XLI XLP XOM YHOO This is the order they are used in requests. It is strictly alphabetical. You are right, GPS and GOOG are next to each other. I will try to email to you some of my routines, that are conencted with the one I posted. If I missed any, please let me know. I appreciate your help. I hope looking at my mess with a fresh eye will uncover the bug. Thanks. AlexB Edited by DTN_Steve_S on Aug 23, 2010 at 02:51 PM Edited by alexB on Aug 23, 2010 at 03:10 PM
Data and Content Support »
Corrupt data feed Aug 23, 2010 12:00 PM
(Total replies: 23)
OK, I am posting the initial routine that is a starting point for the whole chain. You will see that the names all aroung are Hist_Ameritrade, not IQFeed. It is because I designed the whole system originally for downloading TD-Ameritrade history data but then switched to IQFeed: private void UpdateAllSymbolsHist_OHLC_Ameritrade ( ) { Console.WriteLine ( ); Console.WriteLine ( "Hist_OHLC_Ameritrade Downloads: \r\n" ); Console.WriteLine ( ); string[ ] symbols = FindAllSymbolsInDB.FindAllTableSymbolsInDataBase ( "Hist_OHLC_Ameritrade" ); foreach ( string symbol in symbols ) { Globals.currentSymbol = symbol; Globals.currentDate = DateTime.Now; DateTime latestDate = FindDates.FindEarliestOrLatestDate ( symbol, "Hist_OHLC_Ameritrade", "MAX" ); CreateTableHistAmer ( Globals.currentSymbol ); string cEndDate = DateTime.Now.ToString ( "yyyyMMdd HHmmss" ); string cStartDate = latestDate.ToString ( "yyyyMMdd HHmmss" ); Console.WriteLine ( symbol + " " + latestDate.ToString ( "yyyy-MM-dd HH:mm:ss" ) + " - " + DateTime.Now.ToString ( "yyyy-MM-dd HH:mm:ss" ) ); string sHistRequest = "HIT," + Globals.currentSymbol + ",1," + cStartDate + "," + cEndDate + ",,,,1\r\n"; byte[ ] sendBytes = null; sendBytes = Encoding.ASCII.GetBytes ( sHistRequest ); stream.Write ( sendBytes, 0, sendBytes.GetLength ( 0 ) ); ReceiveData ( ); } } // UpdateAllSymbolsHist_OHLC_Ameritrade I used your code I believe although I am not quite clear on this point. Most of your code must be in ReceiveData, I guess. I want to stress again and again, that for any symbol with values over about $25.00 I never have any problem. Thanks. AlexB Edited by alexB on Aug 23, 2010 at 12:01 PM
Data and Content Support »
Corrupt data feed Aug 23, 2010 11:49 AM
(Total replies: 23)
OK, I definitely use only one socket. I send one request at a time. The request is per symbol. It spans the time frame I need. If I find any data is missing because of my inability, as I mentioned, to arrange for a smooth transition between packets your server sends, then after the data is collected into my internal DataTable I issue another request thru the same socket. That request is for the same symbol and for a narrower time frame (span). There might be a few such requests perhaps per symbol but I have no way of knowing. I can measure them though. I see that I somewhat misunderstood your post at first. Now I am trying to clarify details. AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 11:41 AM
(Total replies: 23)
Yep, it is weird. Your data appears to be all right. Then what is my problem? AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 11:38 AM
(Total replies: 23)
OK, I apologize. I do see the quotes in the attachment. AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 11:36 AM
(Total replies: 23)
Steve hi, Thank you for the reply. I don't see any data you posted though. Answering your question though, this is what I do. I have a list of about 43 stocks (symbols). My program starts with the first one (AAPL). First it checkes the availability of quotes in my Sql Server, by taking the last quote entered. Let say for tonight it will be a quote for last Friday around 7 PM or so. This dateTime is taken as the initial starting date for the IQFeed request. The end dateTime for the same request will be DateTime.Now which will be roughly 7 PM EST tonight. I use HIT command, shape it accordingly and send. I can post the whole routine if you wish or send it to you via email. It is fairly large. As a matter of fact it is not one routine but about five including the one that saves the data in Sql Server. What I found was that I could not make a smooth transition between packets, some Minutes are lost in between. In order to prevent loss of data I have another routine that checks missing bins and issues an extra request or requests to fill gaps which are small. This additional routine slows the execution somewhat, naturally, but it assures complete validity of data in terms of missing bins. Keep it in mind that my system works extremely well for stocks whose values are over $30.00 or perhaps even $25.00. It makes me feel that it is not my problem. I did not mention it in my OP but there are more corrupt data which I ignore. Data that preceeds normal trading hours is so gross that I loath even to look at those values. I have a routine that deletes those once in a while. There are weird quotes for 3:20 AM and so forth. I don't quite get your question about the server. I send all my requests to the only server I know of. I got this information from you. I can post complete details if you wish. The server for me is a proxy server as a matter of fact. It is your API. I use local host and the port number 9100. This is my way: public static class CreateTcpSocket { public static void createTcpSocket ( ) { // ReceiveTimeout 1000 - default is 0; mClient.Blocking Yes; BlockSource - unsupported Socket mClient = new Socket ( AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp ); mClient.ReceiveTimeout = 1000; // default is 0 System.Net.IPAddress ipAdd = System.Net.IPAddress.Parse ( "127.0.0.1" ); int port = Int32.Parse ( "9100" ); IPEndPoint endPoint = new IPEndPoint ( ipAdd, port ); Globals.mClient = mClient; mClient.Connect ( endPoint ); } // createTcpSocket } Thanks, - Alex AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 10:45 AM
(Total replies: 23)
This is a piece of daily data feed for GPS: 19.9 19.9 19.89 19.8925 1529 2010-02-17 14:08:00.000 19.89 19.9 19.89 19.9 1700 2010-02-17 14:09:00.000 19.9 19.92 19.89 19.92 19702 2010-02-17 14:10:00.000 540.3675 540.44 540.21 540.23 2564 2010-02-17 14:11:00.000 19.91 19.92 19.9 19.9 37151 2010-02-17 14:12:00.000 19.9 19.91 19.9 19.9 1475 2010-02-17 14:13:00.000 How did 540.36 come about? AlexB
Data and Content Support »
Corrupt data feed Aug 23, 2010 10:11 AM
(Total replies: 23)
Hi there, I download historical data for 1 min granulars on a regular basis. I do it once a day, at the end of the day, around 6 or 8 PM EST. What I found is that for some stocks whose values (quotes) exceed a certain threshold which I can only sense (perhaps it is about $30.00) all data appears to be correct. The lower the stock value the more likely that a corrput, aberrant value will be thrown in. For instance for C (Citi Group) whose values have been under $4.0 lately such aberrant values appear almost daily. There is some weird regularity in them. I will give a row of 15 dates. The bin where the wrong value appears is shown in parentheses (the timing is approximate because it is taken from a further 5-min granulation I do myself to show in a graph): 8/2/2010 (3:30); 8/3/2010 (3:30); 8/4/2010 (11:30); 8/5/2010 (no corruption); 8/6/2010 (3:30); 8/9/2010 (3:30); 8:10/2010 (3:30); 8:11/2010 (no corruption); 8/12/2010 (4:00); 8/13/2010 (3:30); 8/16/2010 (no corruption); 8/17/2010 (3:30); 8/18/2010 (no corruption); 8/19/2010 (no corruption); 8/20/2010 (12:00 & 3:30). Here is a sample of actual data: 3.7399 3.74 3.73 3.735 1285472 2010-08-20 15:15:00.000 3.74 3.74 3.73 3.74 1623420 2010-08-20 15:16:00.000 3.7305 3.74 3.73 3.74 410455 2010-08-20 15:17:00.000 3.735 3.74 3.73 3.74 2070484 2010-08-20 15:18:00.000 3.74 3.74 3.73 3.7393 782217 2010-08-20 15:19:00.000 3.74 3.74 3.73 3.7305 250245 2010-08-20 15:20:00.000 33.02 33.025 32.99 32.992540927 2010-08-20 15:21:00.000 3.735 3.74 3.73 3.73 1652890 2010-08-20 15:22:00.000 3.73 3.74 3.73 3.73 444935 2010-08-20 15:23:00.000 3.7375 3.74 3.73 3.73 1920830 2010-08-20 15:24:00.000 3.73 3.74 3.73 3.7375 2893528 2010-08-20 15:25:00.000 I do all my programming myself from scratch. When I plot curves for my graphics those aberrant values greatly distort them because I have to squeeze maximum and minimum values into a rectangle and the program has no way of knowing that certain data is wrong. Thus the curve itself becomes diminutive and not helpful. Now I am trying to devise a routine that will through them out and replace them with the average of two bins on each side of the wrong value. It is a painful process because there might be some unpredictable distortions and I've already gotten a taste of some of them. In short IQFeed data IS CORRUPT as far as I can tell. You must do something about it. Thanks, - Alex AlexB
IQFeed Developer Support »
Symbols for DJIA and NAS100? Apr 23, 2010 06:41 AM
(Total replies: 1)
Steve hi, Could you tell me the symbols for DJIA and NAS100 to use for HIT request? Also how can I modify HIT command to get just daily OHLC values? Also I've read about your benchmarking codes. Can I have them? Thanks. AlexB Edited by alexB on Apr 23, 2010 at 06:42 AM
IQFeed Developer Support »
IQConnect 4.7.x beta 90% CPU load Apr 18, 2010 10:18 AM
(Total replies: 14)
I grabbed a couple of our benchmarking apps that we use for IQConnect. These apps simply read the data from the socket, count the messages and throw the data away. Steve hi, I am very interested in this benchmarking. As a matter of fact I started thinking about it two days ago and today got an email notification of the posting, came here and read this. Could you email the codes to me? Now that .NET 4.0 Framework is out I am starting to use parallel extensions and want to dedicate one CPU to running a benchmark of sorts. I was thinking something similar to Performance Counter that will grab packets, count them and present perhaps a running line on the chart or collect statistics and store it in Sql Server for subsequent examiantion. Also I have a fantasy that maybe I should write a program to analyze patterns inside the binary stream and should abnoramlities develop such packets may be blocked. I am now convinced that there are such situations during a typical trqading day but have no proof. Thanks. AlexB
IQFeed Developer Support »
Upgrade and PW confusion Feb 12, 2010 06:52 AM
(Total replies: 3)
OK, I found the reason for the failure. I have my QuotTracker on with IQFeed feeding it. I closed QT and installed the new version. My last wish would be to have downloads automatic. Also I want to change the PW for the forums. Mine is impossible to remember. I cannot find a place where it can be done. I am so busy I cannot spend 10 min searching the site for that. Thanks. AlexB
IQFeed Developer Support »
Upgrade and PW confusion Feb 12, 2010 06:40 AM
(Total replies: 3)
OK, I finally found yet another ID and PW. Found the download site. Downloaded and "installed." This is what I got in the process: IQFeed Client 4.7.0.9 Setup --------------------------- Error opening file for writing: C:\Program Files (x86)\DTN\IQFeed\iqconnect.exe Click Retry to try again, or Cancel to stop the installation. --------------------------- Retry Cancel --------------------------- Naturally Retry does not work and I click Cancel. The installation proceeds like nothing happened. Finish button appears. I click it. Did I install IQFeed new version or not? It is so confusing and messy. It is just disgusting. Sorry. AlexB Edited by alexB on Feb 12, 2010 at 06:43 AM Edited by alexB on Feb 12, 2010 at 06:44 AM |
|||||||||||||||||||||||