Hello!We have application, which create multiple asyncronius sockets to level 1 port , one socket for one ticker, about 200.. In iqfeed version 4.9.0.3 we had no problem...Everything worked very well... After updating to last version 5.0.0.11, when application trys to connect to level 1 port via async sockets, iqconnect send such error:

Error: Out of memory.
Box: small information box.
Title in window of box: iqconnect.
Icon: exclamation mark in a yellow triangle (means warning).

After deleting newest version and install older version 4.9.0.3 - everything work without any problems...

After some observe, we found, in older version one socket occupies about 2MB of memory, in the newest version occupies more 8Mb for one sockets....

Please, explain, what can help us?

This was handled via chat but I wanted to post here in case anyone else has noticed the same thing.

We increased the size of the local socket buffers in IQFeed 5.0 accounting for this difference. Most users will not notice a huge difference but since the above poster is using ~200 socket connections to IQFeed it became noticeable very quickly in his case.

In this case the solution is to reduce the number of socket connections to IQFeed.

I'm getting this error too and I'm not making lots of simultaneous socket connections. It seems that over time iqconnect.exe uses more and more memory until it consumes all memory on the machine. If you make a lot of connections for example to pull historical data (NOT simultaneous connections; just sequential) it will use up memory faster.

I believe there is some sort of memory leak in the 5.0.0.11 version of iqconnect.exe.

To add to these observations, I have noticed that if I leave version 5 IQFeed running for longer than a session, it starts to become very slow for history requests. If I start it again, everything returns to normal.

Thanks for the feedback everyone. I'll look into this. The one theme I've seen between all of your scenarios is that you are making multiple connections to the feed (not simply maintaining a single connection and making multiple requests). Is this accurate?

Craig, can you give me a bit more detail about the "very slow" history requests? Do you have any numbers that you can give us to define how much slower the feed becomes?

That is true for me. It seems to leak around 200 bytes per connection. Not so much by itself but over a long enough time the iqconnect.exe process causes the machine to run out of RAM and I get the out of memory error.
Edited by programmer on May 28, 2013 at 01:59 PM

I don't have any hard numbers, to give more detail, I usually download the last days daily bar for ~450 contracts at about 5 est. Usually each request is sub-second. For a couple of days at the end of last week I was waiting another maybe 6 hours before I did the history requests, leaving iqfeed running. When I did so each request was taking in the order of maybe 1-1.5 seconds (remember, this is for one bar). At first I thought that I had a database problem, but no, restarting iqfeed restored the speed. I can take exact numbers if that helps.

Just thought I would note that it's easy to recreate the memory leak simply by connecting to any iqfeed socket then disconnect (no command need to be executed). Do it several thousand times and watch the memory usage grow.
Edited by programmer on May 29, 2013 at 08:07 AM

We have already replicated the problem and have a fix ready for the next version of IQ Feed.

---

Nathan W. Bartsch - Senior Software Engineer - Telvent DTN

Expanding on Nathan's earlier response.

The fix we are releasing in the next build of IQFeed 5.0 will minimize this behavior but we are unable to resolve it completely at this time.

There are 2 sources of the increasing memory footprint that we have been able to identify. The first one (and the most significant) we have direct control over and have limited it to a maximum size in the next build (it was previously unlimited). We will modify this behavior further in a future release of IQFeed. The second source is buried within the socket library we are using and consumes ~60bytes per connection. Unfortunately, due to the extensiveness of the development involved, we are not able to resolve this in a timely manner so it has been slated for being fixed in a future release of IQFeed.

At the current time, the workaround for this would be to either restart the feed periodically or to maintain a persistent connection to the feed for your requests instead of continually creating new connections.

Hi,

I just upgraded to 5.0.0.14 after receiving this same error message from 5.0.0.11 and I am still suffering the same issue as described in this post.

I am only watching 4 symbols and I am not pulling any historical data.

I am watching, YMM13, YMU13, ESM13 and ESU13

I am running on Windows 2008 R2 64bit on a Quadcore XEON CPU with 12GB of RAM running MSSQL 2008 R2

It seems like there is still a memory leak with IQConnect?

Thanks

--
David Tran
MCP, MCSA Windows Server, MCTS Vista, MCITP MS SQL 2008 R2 Database Administrator Pro

David, the fix mentioned in this post was specific to Lookup Port connections so your issue would be unrelated in terms of what is causing it and it wouldn't surprise me that upgrading from 5.0.0.11 to 5.0.0.14 wouldn't have an effect.

With that said, the way you are using the feed (from what you've described) is pretty common amongst thousands of our users who aren't experiencing the same problem (or at least not contacting support if they are) which makes me believe there is something in your setup somewhere that is exposing the problem.

Can you to provide us with a copy of your app that we can run locally to try and duplicate the problem? Alternatively, can you duplicate the issue you duplicate the problem with an apps we distribute with IQFeed? For example, you should be able to open up 4 copies of the IQWatchQuote app simultaneously and watch the 4 symbols or you can use the ConsoleBenchmarkTool example app that is included with the dev version of IQFeed and configure it to watch the exact same fieldset that you are using and watch the symbols using a single connection to IQFeed (if that's what your app is doing).

Being able to duplicate it will make it easiest for us to track down exactly what is going on. Or even knowing that you can't duplicate it with the end user apps we provide would isolate the issue to something with your machine/network setup or something within your app.

Absolutely, please let me know how I should send you my app? It's basically Level1SocketsForm tied to a MSSQL DB using ADO.NET. I've used a try catch for error exception and my app is not throwing any exceptions, just that error popup from IQConnect.

I will wrap up my whole app for you guys so that you can give it a run including my db schema definition.

Kindly

David

If it is less than about 15MB compressed, you can email your app in a zip file that is password protected (to bypass mail filters) to developer support. I will send you a PM with the dev support email address. If it is larger we will have to get another option for transferring it if you don't have a way to host it for us to download.

Email will be fine, thanks

David

Hello,

I have a similar problem like that previously described above. I have an application which connects to IQConnect. I use 3 sockets, one for admin status connection, one for Level1 and one for Level2.
I connect all of them once. I made subscriptions for 500 symbols for Level1 and a single subscription for both levels. Since last week I started to receive a message box with "Out of memory" and I have looked in Task Manager and I noticed that virtual memory is counting up.
At first I thought that there is a bug in my application, slow processing of incoming data but is not true I erased that part of processing, I kept it simple just a counter with number of messages received and the problem persist.

I use 5.0.0.11. Can you provide me a solution for this problem, a workaround for this before a new update ?

Thank you..

mdu, I would recommend updating to IQFeed 5.0.0.14 (the current release version). The 5.0.0.11 release had a bug that caused corrupted data to be received from the level 2 port of IQFeed on occasion (and you mentioned using level 2).

If after updating you still see the same issues, my guess would be that there is something unique in your setup that is exposing the issue in IQConnect and at that point I will need to be able to duplicate the issue locally in order to track this down. Is it possible for you to email developer support with a copy of your app for testing?

Hello Steve, I updated to 5.0.0.14 version and still I have the problem. Also I noticed that I've started to receive more data on Level1 Socket for the same symbol which I did subscription on Level2. Can you provide me an email address from developer support ? What do you mean by sending a copy of my app for testing ? You want the source code or only the application ?

I'm not finding an account based on the information in your forum account. Please send me a PM with your loginID and/or the email address your account is registered under.

Hi,

I'm also experiencing the same "out of memory" error. I have the most updated release version of IQConnect 5.0.0.14 (downloaded it sunday and installed it on a completely fresh server instance).

After the IQConnect error pops up, I lose my data connection. Then when its restarted everything works as expected.

I'm listening to between 350-400 symbols.

Dylan

I have also received the out of memory error after upgrading to .14. However, thus far it has only happened once.