Network Stream Max Buffer Size

[Quintin]

Hello,

I recall an AE on here once mentioning that network streams can exhibit problems if you set a buffer size greater than 9MB, but I haven't been able to find any concrete explanation of this.  The reason behind me asking is that, I'm currently running into some memory problems in my application in which I initialize my buffer to 30MB.

Basically, my application starts out running fine, but after 30-45 minutes, all of a sudden, my network stream buffer starts building up until it reaches a max value, even though I am still reading data out of it at seemingly the same rate.  It's as if routine memory allocation/deallocation elsewhere in my application causes issues with the network stream.

Anyone have any insight to this?

thanks in advance,

[The_Right_Choice]

Quintin,

First off, let's see if we can get a little bit more information. 

 - What specific hardware are you communicating with?
 - What OS are you currently running on?
 - What software (and bit size) are you running for this?

 - How do you set up the Network Stream? Screenshots might be helpful in this case.
 - Are you seeing an error and if so, what is the error number and what does it say?

Here is a link to a KnowledgeBase Article that talks about Network Streams. It might be helpful to watch the performance of LabVIEW up until the crash.

The more information you can give us, the more we can help out.

If you can answer these questions, we will be able to see if there is anything else we can do. Thanks!
 

[Quintin]

Thank for the reply Douglas,

Here is the additional info you requested:

• RT hardware is a sbRIO 9606
• PC is LabVIEW 32-bit running on a Windows 7 64-bit machine.
• Below is a screenshot of my block diagram.  The stream datatype is a U32.
• I'm not receiving any errors, the AvailElmntsForRead after running for an extended period of time just starts increasing until it reaches a max of 29999999.  For the first 30-45 minutes of running, this value stays relatively low, jumping around between 0 and maybe ~8000.

[HollyBerry]

Hi Quintin,

It sounds like there is some other buffer that is filling up besides the Network Streams buffer. It does not sound like you are reaching the memory capacity of Network Streams.

I would check memory limits on the buffer where you are storing the read data. LabVIEW32-bit can access up to 4 GB of RAM. You could be reaching this limit after 30-45 minutes and preventing continued reading. Logging the data to disk can help transfer the data from RAM to your hard drive to flush out that memory buffer. 

You can also setup a system where you are inputing the amount of elements remaining on the buffer into the read function so that each time, you are reading everything you can. However, this will not help if you are exceeding the RAM alotment for LabVIEW.

[Quintin]

Thanks for the reply Holly.

Actually, I thought with 32-bit LabVIEW, you can only extend RAM alotment to 3GB, per this KB:

http://digital.ni.com/public.nsf/allkb/AC9AD7E5FD3769C086256B41007685FA

I agree that there could be an issue elsewhere in my application that is causing the RAM alotment for LabVIEW to exceed 2GB over time.  If that was the case, could you comment on why that has adverse affects on my Network Stream buffer, since it should already be allocated from when the my application initial started?

[HollyBerry]

Hi Quintin,

Thanks for posting that KB. What you have said about RAM is correct for Windows 32 bit. However, a little further down on the KB it describes how on Windows 64 bit with LabVIEW 32 bit, you are aloud 4GB. 

What could be happening is that your loop rate is slowing down due to some other process in your loop which is causing the data to be read from the stream more slowly and build up. Are you monitoring your iterations or your loop time? Do you see the loop time increasing?

I would highly recommend inputing the available elements to read into the Network Streams Read VI to just pull down all possible data each time the loop iterates.

[Quintin]

Oh ok that makes sense, and thanks for pointing out that oversight with Win7 64-bit.

No, I am enqueuing the network stream data and it is constantly being dequeued in another part of my application.  From there it gets a little bit involved, as I have an architecture that allows multiple, simultaneous clients to request a live feed of the data.

I'll see if I can put together a diagram or a simplified version of the code (it is a pretty large application >2000 VIs), but you explanation confirms my original speculation, so thank you!  And just to tie up the final loose ends, there is no issue with allocating that large of a buffer for the network stream?

[HollyBerry]

The size of the buffer you set for your network streams will just determine how much memory is set aside within your application. If you make your stream too large, LabVIEW will throw an error such as 'Memory is Full' telling you there is not enough space to create that large of a buffer. 

The important benchmarking data to look at concerning network streams is throughput and latency. The following KB has data in section 6 regarding how quickly data can be passed through a network stream:

http://www.ni.com/white-paper/12267/en/#toc4

Since you are able to function properly for the first 30-45 minutes, it does not seem that you are violating network capabilities with your setup. However, there seems to be something that is building up/slowing down after that period of time.

How did you determine that you need a 30 MB buffer size? If you are originally only storing between 0-8000 items on the buffer, it may make sense to make a buffer smaller than 30000000 to free up resources for other parts of your application.

Here is some more information about how to profile performance and memory within LabVIEW: http://zone.ni.com/reference/en-XX/help/371361J-01/lvdialog/profile/

[Quintin]

I believe a large value was picked just to ensure any connection hiccups wouldn't result in the loss of any data.  I will do some experimentation with lowering that buffer value.

Thanks for your help Holly.