Handling Dynamic String Sizes

[Niatross]

Hi All

I have come across the same 'problem' a couple of times recently...I am not sure whether it is a problem or not.

The best example I have is that I have written some code to receive an MJPEG stream from a UDP connection and extract the full JPEG frames. It took a couple of attempts to get something that worked and I have attached it.

At its core I have a buffer which stores everything received from the stream, when the JPEG start of frame header is found it is assumed that everything in the buffer is the full previous JPEG frame. This works well (Although looking at this code again I can't remember what I am doing with FFD9 check, I think this was meant to be checking for corrupt frames but it certainly doesn't do that. Will have to try and remember and fix.)

My question is....The buffer is constantly being resized as it gets filled up to a size of ~3-4MB. When a full JPEG frame is received it is set back to a blank string. How do I ensure that this doesn't cause me memory problems as in my mind it is effectively the same as using a build array node in a loop.

I cannot find out how LabVIEW is handling the memory management in this example. Am I potentially going to have memory problems if it can't find a contiguous 4MB block. This is part of a much much larger peice of software and I am becoming increasingly concerned that I am going to have memory fragmentation issues after a few hours...especially around these video feeds (I have four of them) and I am starting to think I have created unecessary data copies of them by distributing them via notifiers. That is a problem for another day, for now I am more interested in how LV is handling string memory management.

Ideally I would like to preallocate the memory and earmark 5MB lets say for the buffer, but as there is not a LV primitive of 'Create String of Length x' I am assuming this isn't the best way of doing this. 

Linked to this question is....how are arrays of strings handled. If I initialise an array of 5 strings, each with 5 characters. I then increase the size of the second element to 200 characters I am guessing the entire array has to be reallocated?

[rolfk]

First, 4MB blocks is not usually a big problem to allocate if your system isn't memory constrained (less than 8GB of physical RAM).

Second, strings are not the ideal data structure for this, but I almost always use uint8 byte arrays for that. There you can do Initialize Array with a prefixed size, and also Replace Array Subset to fill in a block of bytes anywhere in the buffer. Some LabVIEW functions like the network Read and Write insist on strings but the Byte Array to String and String to Byte Array Nodes work perfectly for that and are at runtime really simply a No Operation, with neglible runtime performance impact.

If you really want to go all into advanced mode with trying to squezze the last memory performance out of this, you would make the byte array a Data Value Reference and use the Inplace Structure array nodes to work on that array. 

[Niatross]

I agree 4MB isn't massive, hence why I didn't really think it was an issue for the example I sent.

It was more a generic question to find out more about how LV handles strings and arrays of strings.

I did wonder whether converting to U8 array was the way forward.

[Jacobson-ni]

The link below goes over the basics of how LabVIEW stores data in memory.

https://zone.ni.com/reference/en-XX/help/371361R-01/lvconcepts/how_labview_stores_data_in_memory/