(vxWorks cRIO) memory allocation details

[QFang]

Hi everyone, 

A question on memory allocation when dealing with arrays, strings, and array of strings.

-I'm required to construct and produce ASCII Log (LAS 2.0) files on a cRIO.  The cRIO is already doing tons of other operations, and avoiding excessive memory fragmentation and overall memory usage is an important consideration for us, but CPU is also a limited resource.. This new requirement is an addition to "everything else" the cRIO is already doing, and from experience, I know that handling large data and logging as string files is taxing on CPU as well as memory, so suggestions are welcomed.

Worst?) If I construct the whole entiere file by concatenating and building up the file-contents as one long string passing from one shift-register/loop to another, growing as it goes, that strikes me as the worst possible way to do it from a memory management point of view.

Better?) If I construct the file contents line by line and create an array of strings where each element in the array corresponds to one line, will this be better? Since each element is a variable (string) length, I'm not sure how the memory manager handles such arrays? Ideally it is a list of pointers so that each element (of variable string length) is its own (contiguous) segment in memory vs all elements of the array in one single contiguous block? If so, would that mean that an array of strings would require less contigous memory than the same data represented as a 1D array of U8's?

An advantage I can think of with the array (or line) approach is when it is time to write the data to file, I could write each element from the array individually, again, the idea is that it would require less memory overhead to handle lines individually vs one giant stream to disk?  I could then possibly construct/write the file in blocks so that I never have 'the whole file' content in a line-array..

Better still? -anyone have any ideas?

Best? -anyone feel sure they completely tamed this beast? Feel like sharing?

As a parting note, unfortunately, all the data that will eventually go into the file does not exist when the file is first created (additional trace-scans are added as they become available throughout the day), and while the raw data itself in LAS files are in the last section (thus new raw data can be just appended to the string file without loading/parsing the file), there are some fields within the earlier sections that might need updating (such as the STOP tag in the ~W section), I was planning on handling that by tracking a byte-count for the field(s) that need to be updated after creation, but if anyone have any experienced insight they can share on that topic as well, I'm listening (otherwise, I'll likely keep these counters in a variant attribute list for fast and simple retrieval since I'm maintaining individual log files for individual wells and thus need a somewhat more organized way to track them than just an array or similar).

Wish everyone would just learn to consume the TDMS log files we recommend instead of insisting on all these other formats. 😕 

(For a similar project (WITSML files), I ended up finding a rather substantial performance gain by constructing as much of the ASCI file contents as possible from U8 array constants, but the luxury with that project was that so much of the file structure/content was static that this was manageble, not to mention the WITSML requirement was for one scan/trace per file so it was create and forget.  This made a HUGE compute time and CPU usage difference.  Not sure this made any real difference as far as memory usage/allocation went other than constants of course being constant/loaded with the VI and possibly compiler optimizations since a U8 constant is potentially simpler to work with than strings (and string constants?).)

[Akardo14]

Hello,

I wonder if this document could be of your help: http://www.ni.com/white-paper/11198/en/

Please let me know if it helps.

Juan B.

National Instruments

[QFang]

Thank you kindly for the link to that paper.  The word 'memory' is mentioned exactly once in that paper, and unfortunately it did not answer my question on how memory is allocated for array of strings.

[Akardo14]

Hello QFang,

I found some information that might be useful to answer your question

Ok just as an FYI, every time memory is allocated, LabVIEW accesses the memory manager which is a shared resource. The memory manager uses considerable processor time. If the memory manager cannot find a large enough contiguous segment to fill a request, it terminates the program

Therefore, the level of effort required to design your code so that it avoids dynamic memory allocations should correlate to the required uptime of your embedded system. Systems that can handle regular reboots or maintenance can use dynamic memory freely as long as they monitor the memory status. As soon as a system is rebooted, the memory frees up. If the system cannot handle regular reboots, then you should consider the following techniques to reduce dynamic memory allocations. For maximum reliability, move your code to the FPGA or create a redundant system. Also consider that RT targets do not use virtual memory. When programs contain a mixture of dynamic allocations, memory can become fragmented, this can be bad since we can run into a scenario in where we don't have a large enough contiguous segment of memory.

The following are common sources of dynamic memory allocation

     −Queues without fixed size

     −Variable sized arrays (and waveforms)

     −Variable sized strings

     −Variants

As you can see, if you use Variable sized strings, you will have more dynamic memory allocation, which can again lead to segmentation issues. 

I found this document that has some guidelines as to how to get the RE application faster and with fewer resources: http://digital.ni.com/public.nsf/allkb/D58C6375BC58A16586257194004950B8  I hope it helps you. 

As far as memory allocation goes, I think that the first parragraph can give you a good idea, I hope this information helps you.

Juan B. 

National Instruments

[MarkCG · Accepted solution]

I'm logging data locally to disk with my various cRIO projects, because yes, people just can't and won't use TDMS files. What I do is I pre-allocate my string arrays-- for example a the current data point to be logged of the text file is a 1-D string array, 1 element per channel, which I created by doing first creating a fixed size U8 array (INIT 1D array, say 16 bytes), converting to string, and then initializing my string array (INIT 1D array again). 

I keep this allocated 1-D string array shift registered. When it comes time to write a new line to my file, I iterate through the array, using a FOR loop and the In Place Element Structure (not auto-indexing) substituting the new values in to be logged in.

I do everything with either CVT or shared variables so I am just reading tags programmatically and doing a "to fractional string" on them. To write the string to file I keep a last byte count and just go to the file position and write, 

So far I have not had any problems with memory leaks or fragmentation-- you see the largest continuous block solely decrease and then stay constant.  My systems generally run a week to a month between resets. 

It sound like you are already considering doing this-- working with pre-allocated string arrays. I would say this is the way to go. 

[QFang]

Thanks for the additional link(s) and confirmation that pre-initialized string's and string arrays are a robust and workable way to do this in production.