sbRIO & eeePC : Popup "Not enough memory to complete this operation"

[Julien_31]

Hi all,

I'm using an sbRIO-9631 and a eeePC (Config : 1 GB RAM & 1,66 GHz & Windows 7 SP1). I run an executable of my application (developped with LabVIEW 2011).

The architecture is very classic :

- RT code : a while loop performs acquisition (DMA FIFO read), waveform data are sent from this acquisition loop to a TCP while loop (with an RT FIFO), and then the TCP while loop sends data to an eeePC by TCP.

- eeePC code : a TCP while loop receives data from RT target, and then data are sent to a "calculation/log into files" loop (with a QUEUE).

Please find attached some screenshots for RT code and HOST code...As you will see in RT code, acquisition loop has high priority and TCP loop has medium priority : the period for each RT loop is 1000 ms.

The problem is that after less than 1 hour af acquisition, a popup appears : "Not enough memory to complete this operation". When I look at task manager, it seems to be OK : CPU is about 10%, memory usage is stable (700-750 Mo) without swap, and network using is about 2%. When the popup message appears, acquisition sometimes goes on if I click OK button and my application still runs normally ! Sometimes I have 3 or 4 consecutive popup messages, and when I click OK buttun for all of them, acquisition is "crashed", probably on my RT target because I need to reboot the sbRIO...and then it works again)

I've read some posts suggesting solutions or at least explanations, but regarding my application it doesn't really help me (see below if needed).

http://forums.ni.com/t5/LabVIEW/Not-enough-memory-to-complete-this-operation/td-p/1681816

http://forums.ni.com/t5/LabVIEW/TCP-IP-communication-memory-full/td-p/1468316/page/2

Thank you in advance for any help ! 🙂

Julien

PS : at the time I'm writing this post, I've disabled automatic windows standby in "power options", and the "not enough memory" popup has not yet been appearing...do you think it could be enough to solve my problem ??? Since I'm not convinced by this, I prefer posting my message, if you have any other idea...

[rolfk]

How do you send the data over TCP? Binary format?

Most common reason for this error message is that you use a protocol where you first send the number of bytes that follow and then the actual data. Nice and well until you loose a byte somehow. In my experience the loosing of bytes was actually due to oversights on either the sender or receiver side in terms of proper error handling, but there are several reasons possibe. Anyhow becuase the 4 byte length indication is now off by a byte or so, it reads sometimes as a ridiculous large number and when you use that to issue the second read to read the actual data, LabVIEW causes the out of memory error since it tries to allocate the large buffer to read the expected data into it.

Solutions are to make the network protocol more robust, with a header that contains also some identification in addition to the length integer, so that you can do some extra sanity check on the received length. And proper error handling means that anytime you detect an error on the connection, either from the TCP VIs self or when detecting errors in the message, you disconnect the connection and reconnect again, to start with a fresh and clean state.  Your server side (sbrio) must of course support this by detecting dropped connections and allowing new connections to arrive. My network infrastructure does this by a server that allows multiple parallel connections and also disconnects sockets as well, when it detects any errors except timeout errors.

[Julien_31]

Thank you very much for your quick and interesting answer. I'm on holiday at the moment but as soon as I restart working I'll try to follow your suggestions. 🙂

Just a precision, in case I would need to ask others questions : I use the STM library to send data over TCP

http://zone.ni.com/devzone/cda/tut/p/id/4095

[Julien_31]

I just want to add a comment for people who would take a look at this post.

There was an issue with my LabVIEW code : the data were supposed to be sent by RT target every 1 second, and "TCP write" timeout was 1 second as well...and on the other hand, the "TCP read" timeout on the eeePC was also 1 second (default value for STM library). I have increased the timeout for TCP read function (2 seconds) and now it works perfectly.

Here is what BlueCheese told in a post :

TCP streams will never lose data within the middle of a stream. If the receiving side is not receiving fast enough, it will eventually cause the sender's window to go to zero so that no more data can be sent (if the TCP send is set to block, it will block until it can send). The sender's application could choose to drop new data at that point but it would be that application's decision to do so. If you control the TCP socket on both ends then you can be guaranteed (short of memory corruption or bug in the TCP stack) that the data arrives intact and in-sequence if you get data. If you are seeing this behavior I strongly suspect it is because you code on the sending side is dropping data when it hits a timeout.

As far as I am concerned, it seems that I had that kind of issue, even if I don't clearly understand why ! (in each TCP message, the part of the message describing the data size was still the same for several minutes, and suddenly this value suddenly became different as described by rolfk)