|
|
Home
Search
Register
Login
Recent Posts
| Viewing User Profile for: russt | |||||||||||||||||||||||
| About | Contact | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Post Statistics | |||||||||||||||||||||||
|
russt has contributed to 12 posts out of 21251 total posts
(0.06%) in 5,629 days (0.00 posts per day).
20 Most recent posts: This message was posted in a secure forum. Click here to access the topic where message was posted.
IQFeed Developer Support »
bug in iqconnect manager dialog Mar 5, 2010 05:15 PM
(Total replies: 2)
Thanks for the fix. -Russ
IQFeed Developer Support »
bug in iqconnect manager dialog Mar 5, 2010 03:47 PM
(Total replies: 2)
When I press ESC to close the dialog, it shuts down iqconnect.exe completely. This is the dialog that I get by double-clicking on the systray icon, and that has the heading, "IQ Connection Manager" and that gives informaton about the data rates and connection statistics. Clicking the "done" button closes the dialog without terminating the applicaton, as one would expect. I'm using version 4.7.0.9. It is very disturbing (like 3pm est 3/5/10) when trying to debug problems where iqconnect.exe is not serving quotes, and then sending out bad quotes, to have the entire application shutdown unexpectedly. Regards, -Russ
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 14, 2009 04:35 PM
(Total replies: 16)
Sorry, what I meant to say above was that the 4.7 beta version increased performance dramatically on the Phenom, but not on the Xeon. I don't see how the dynamic fieldsets optimization that you referred to could explain that. Also I'm subscribing to bid/ask sizes, which I'm guessing would be the largest part of the update bandwidth. Thanks for all the advice and help. Getting it running smoothly on the Xeon is not looking so hopeful at present, unfortunately. Please let me know if you have further suggestions. As an aside, I'd like to encourage you to pursue the following enhancements for the future: 1) Allow data to be transmitted in binary form rather than ascii. In my experience, this can save a large amount of cpu power by eliminating the need to transform the data back and forth. 2) Have each socket connection running in its own thread. This allows load-splitting across the different processors in a machine.
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 14, 2009 11:07 AM
(Total replies: 16)
cpubenchmark.net is showing the following scores: reported mymachine Dual Xeon 2.8GHz 416 782 Phenom 9550 2486 2156 Yeah, at first glance the Phenom is much faster. But really that's only because it is a quad core and Performance Test launches separate processes that use all the cores. When I set the affinity of the pt.exe process such that it only executes on processor #1, I get: Dual Xeon 2.4Ghz (single processor) 296 Phenom 9550 (single processor) 576 So the real difference for a single threaded app like iqconnect.exe is about a factor of two. What I'm seeing is that on the Phenom, the iqconnect.exe cpu usage (for 4.7 beta) sits at zero and occasionally ticks up to perhaps 10%. On the Xeon 2.4Ghz, it runs at a baseline level of around 5% and occasionally pins at 25%. The processing speed difference doesn't seem to be sufficient to explain how much faster the Phenom is. Processor speed doesn't explain the dramatic improvement with the 4.7 beta version, either.
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 14, 2009 09:21 AM
(Total replies: 16)
Ok, the plot thickens. After upgrading to the iqfeed beta version 4.7.0.2, I'm seeing dramatically different performance on two different machines. One machine, a quad core Athlon Phenom 9550, is showing perfomance similar to what Jay had. The second machine, a dual cpu 2.4 Ghz Xeon, is still showing quite poor performance. Good: Athlon Phenom x4 9550 Windows XP Pro SP2 4Gb ram Bad: Dual cpu Intel Xeon 2.4Ghz Windows 2000 Server SP4 4gb ram Previously, using IQFeed 4.6.1.0, the performance was only a little better on the Phenom. I'm pretty sure that the upgrade worked on both systems, because the connect dialog is displaying a prominent 4.7, and the "REQUEST STATS" command returns 4.7.0.2 on both. Could iqconnect.exe perform better on Windows XP than on Windows 2000? Could it have any processor specific enhancements for more recent cpus? Any reason why the 4.7 beta version is so much better on the Phenom?
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 01:19 PM
(Total replies: 16)
Jay, that's very impressive. Now I have to figure out why I can't achieve anything even close to that performance. Here's another cpu usage graph, this time with a dummy version of my archiving program, which basically does what jimc suggested. It is exactly the same, except that it ignores all 'Q' type records. My program's cpu usage is now very low, but I'm still seeing high peaks on iqconnect.exe. Thanks for all the suggestions, and I'd welcome more.
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 12:45 PM
(Total replies: 16)
Here's an example of what I'm seeing. It's a graph of iqconnect.exe cpu usage versus time, made using Sysinternals Process Explorer (which I highly recommend). As you can see, there are spikes in cpu usage, effectively maxxing out the power of the machine. The graph on the left is my program (python.exe) archiving the data from iqconnect.exe. @Jay: Thanks for the screenshot and the example. I noticed you only show a snapshot, not a graph. What's the maximum cpu usage it ever gets to? @jimc: Thanks for relating your experiences. I'll write a version of my program that just sucks the data down and throws it away, and see if that improves matters.
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 12:42 PM
(Total replies: 16)
Looks like the symbol list didn't post. Here it is as a zipped .csv file.
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 12:40 PM
(Total replies: 16)
Steve, I see what you're saying regarding management of the queue, although in my experience things like buffering data take little cpu power compared to parsing data feeds or maintaining level2 books. I've reduced the number of fields using Dynamic Fieldsets to 8, using just self.send2('S','SELECT UPDATE FIELDS', 'Incremental Volume','Bid','Ask','Bid Size','Ask Size','Last Trade Time', 'Market Center','Extended Trading Last') Previously I had also included 'Total Volume' and 'Last'. Performance does seem better than before, but that may just be due to a slow market. I'll know more by the market close. I've attached my symbol list in .csv format. It's a list of the futures and stocks with the greatest dollar volume. Basically my program just does this: 1) subscribe to Level1 quotes on a set of symbols 2) convert the data stream into my internal format, just a series of quote changes and executions 3) store the data by symbol in a set of chunks in memory (organized by a hash of the symbol) 4) when a chunk is full, write it to a SQL database After hours, I go through and finish the sorting process, leaving me with a price database organized by symbol. Thanks for looking into this further for me. I look forward to seeing the results of your tests. Regards, -Russ
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 11:01 AM
(Total replies: 16)
Steve, thanks for the reply. Regarding your first point, I don't see how optimizing my program will help reduce iqconnect.exe cpu usage. If my program isn't keeping up reading from the socket, that'll just cause iqconnect.exe to block on writing (if the buffers are full). Write blocking won't increase cpu usage. In any case, it's iqconnect.exe that's using all the cpu power, not my program. 2nd point, I'm already using Dynamic fieldsets to reduce the quantity of data. It's possible I might be able to pare it down by another one or two fields. I'll try that and see if it helps. I'm concerned not only with dropped symbols, but also with the latency. Having quotes come in several seconds late because iqconnect.exe can't keep up is not good for trading. Can you give me an example of a test you've run subscribing 500+ active symbols with low cpu usage? I just don't see where the room for improvement is, since the multiprocessing option has effectively been locked out. Regards, -Russ
IQFeed Developer Support »
iqconnect.exe maxxing out cpu on 500 symbols Dec 11, 2009 09:52 AM
(Total replies: 16)
Is there a way of reducing the CPU usage of iqconnect.exe when subscribing to a decent size collection of symbols? I'm subscribing to 500 fairly active symbols and I fairly often see the CPU usage spike to 25% (the maximum) even during the middle of the day. This is on a dual 2.4Ghz Xeon server system. By contrast, my application written in python is able to parse the text data and store it to a database using only about 12-15% cpu max. I'm rather worried about what will happen in a fast market. Even on my local machine, a Phenom 9550, the cpu usage gets close to maxxing out in the middle of the day. I read elsewhere in the forums that all the Level1 socket connections use a single thread, and also that we can't run more than one instance of iqconnect.exe at a time. It feels sad that iqconnect.exe is saturating while there are three other processors on the machine that are not being utilized. Is there any way to fix this? For example: 1) Can't I run two copies iqconnect.exe limiting each to 250 symbols? 2) Can't iqconnect.exe run each socket connection in a separate thread, allowing me to split the symbols between two or more socket connections? What happens when iqconnect.exe maxs out the cpu for a period of time? Does it or the DTN servers start dropping ticks to keep up? Or does the feed just get buffered and delayed? Is anybody out there running more than 500 symbols? How do you do it? Thanks, -Russ |
|||||||||||||||||||||||