Hi.

I'm testing my application which running continously for a long time. It's currently subscribed to 78 symbols (including Level II for few of them) and running IQConnect 4.7.x beta (latest available beta release). IQConnect normally does not load CPU much (about 2% in peaks).

After about 48 hours of uninterrupted functioning IQConnect process started consuming about 90% of CPU time. When I exit IQConnect and restarted it CPU load become normal.

FYI, My application consumes all the traffic coming from IQFeed (including admin status messages). IQConnect does not report that it is out of memory. Unfortunately I forgot to check how much memory IQConnect allocated when it loaded CPU for 90%. In normal mode it occupies 9.2 Mb of RAM. I will wait until next problematic situation with IQConnect and will check how much memory it occupied then.

Thanks for the report.

If you can provide me with your symbol list and dynamic fieldset (if they are in use), that information might be of use as well.

Hi, Steve.

Below is the list of symbols we're currently watching by IQFeed (1 in second column menas that market depth (Level II) collected for this symbol) :

(see private message)
Edited by smt52 on Dec 18, 2009 at 05:55 AM

I got your symbol list.

I don't see anything in that list that would obviously be causing any problems.

It sounds like we might have to wait for this to reoccur to see if we can gather any more information.

-Steve

Hi, Steve

Cat I somehow to force IQConnect to generate the debug log for you?

Steve hi,

I read the thread. What seems to be occuring here is your API has too many instances of classes which are not consumed by GC. You can easily resolve the problem if you replace them with static classess instead. Of course you should check if it is logically possible. If there is no parallelism in some of your flow diagram parts then converting instances to static will completely eliminate the effect the user is seeing.

Than ks.

---

AlexB

Hi, Steve.

The 90% CPU load remains in both 4.7.0.7 and 4.7.0.9 build of IQConnect.
Now this problem becomes vital for me, because IQConnect goes to 90-100% CPU load every several minutes. Same time it stops sending symbols through sockets. The whole picture is following:

1. My application starts IQConnect and then start subscibing to 78 symbols. For each symbol it first reads historical data was missing after last shutdown;
2. IQConnect consumes about 4-5Mb of memory and 3-7% CPU load;
3. When there are about 50 symbols subscribed IQCOnnect starts consuming more and more RAM and when it's about 10Mb it consumes 90-95% of CPU time;
4. IQConnect stops responding on sockets and my application closes it.
5. Go to #1

This happening under Windows Server 2008 R2 (1Gb RAM) and does not happen under Win7 (4Gb RAM). Windows Server 2008 R2 has about 95% RAM load. In normal situation there is no more than 30% total CPU load (including all services and my applications).
Edited by smt52 on Jan 29, 2010 at 09:07 AM

The issue is 100% reproducable on my system.
Edited by smt52 on Jan 29, 2010 at 09:13 AM

Hello, I believe this is going to end up being a combination of low system memory (you mentioned it being 95% in use) as well as inefficient feed processing in your app. I have sent you an email detailing my testing that includes an app you can run for testing.

Hi, Steve.

Thank you for helping me to understand the problem. I'm profiling my application to find a cause of problem from my side. Could you please to send me your variant of C# code working with IQFeed sockets?

Unfortunately I'm not able to run your test applications because APP ID is not suitable for my logins. IQFeed reports an error:

" Sorry, your Product and/or Service for the attempted login are incorrect. If you have recently changed products you may also need to change your type of service. Please contact DTN customer support for assistance."

How can I go over it?

One of the test apps I sent was built using the wrong productID. I have resent both apps (and the C# example) this morning. Please let me know if you do not recieve them or have additional problems.

Hi,
I am getting the exact same problem as the OP, I was wondering if there had been a resolution in this case?

Hello Craig, I don't think I have heard back from sm52 since emailing him the test apps I was using to try and duplicate this. However, here is one of the last emails we exchanged about this prior to me sending the test apps.



==============
Your system specs for a machine running Server 2k8 seem low, specifically the RAM (you didn’t mention the processor speed). The OS itself requires 512MB of ram and the symptoms that you describe are similar to when windows has to frequently swap data to the hard drive (swap file) due to low available memory. Additionally, you mentioned that the System has 95% of RAM in use which would also lend itself to this theory. The reason that IQConnect would be hitting the CPU that hard would be that IQConnect does a lot of memory intensive operations (even though it doesn’t use a lot of memory, the memory it uses is always needed) and when it has to read/write from the hard drive instead of memory, that is going to slow things down and cause it to have problems keeping up with the feed.

Now, with this said, after doing some testing on a RAM starved Vista box (vista having a very similar OS kernel to server2k8 r2) and 4.7.0.9, I believe the root cause of the problem is the efficiency in which your app reads data from IQConnect. Here is what brought me to that conclusion (or at least educated guess at this point).

I started out testing a vista machine with only 256MB of RAM. Just running the OS idle, 70% System memory was in use and approximately 190MB page file usage. I used 2 telnet sessions (one connected to Level 1 port and one to L2 port) to watch the same symbols you sent me in the PM from earlier in the thread. This fairly quickly saturated the CPU with IQConnect using 80-90%. I was expecting the page file usage to spike as well but that didn’t happen to the extent that I expected. It never went above around 210MB. Interestingly enough, IQConnect’s Memory usage (as reported by task manager) never really climbed much either. This lead me to believe that the “swapping to page file” theory was at least partially correct but that IQConnect wasn’t able to read enough data from the servers to start queuing it locally.

At this point, I shut it down and rebooted with 512MB of RAM. Idle OS used ~50% System memory (this is expected Vista usage. More memory available = more memory used by the OS). When I opened my telnet sessions, again IQConnect pegged the CPU quickly and I observed similar results. Page file usage didn’t increase by more than about 20MB and IQConnect didn’t allocate more memory either so it wasn’t being queued there either.

Reboot, 1GB RAM. Idle OS used 33% RAM. Same results once again. At this point I started getting suspicious that my testing was flawed and I noticed that my telnet sessions were using more CPU than I was comfortable with.

I grabbed a couple of our benchmarking apps that we use for IQConnect. These apps simply read the data from the socket, count the messages and throw the data away. I configured them to use your symbol list and started them up. IQConnect runs very smoothly with neither the benchmarking app nor IQConnect using any CPU. No extra pagefile usage, no extra memory usage by IQConnect. Next I took the RAM back to 256MB. IQConnect ran exactly the same in this configuration as with the 1GB configuration.

This leads me to believe that the inefficient processing by the windows telnet client was the major contributor to the high CPU usage in IQConnect.
==========

Hi Steve,
Thank you for the reply, that all makes sense.
After doing some tracing, I think I may have also been saturating the CPU, I will cut back on the CPU and let you know what happens.

I grabbed a couple of our benchmarking apps that we use for IQConnect. These apps simply read the data from the socket, count the messages and throw the data away.

Steve hi,

I am very interested in this benchmarking. As a matter of fact I started thinking about it two days ago and today got an email notification of the posting, came here and read this. Could you email the codes to me? Now that .NET 4.0 Framework is out I am starting to use parallel extensions and want to dedicate one CPU to running a benchmark of sorts. I was thinking something similar to Performance Counter that will grab packets, count them and present perhaps a running line on the chart or collect statistics and store it in Sql Server for subsequent examiantion.

Also I have a fantasy that maybe I should write a program to analyze patterns inside the binary stream and should abnoramlities develop such packets may be blocked. I am now convinced that there are such situations during a typical trqading day but have no proof.

Thanks.

---

AlexB

After some more testing it seems the CPU was being saturated leading to reduced though-put in my applications, all seems to work now.