About arrays and memory

[Emmanuel Doit]

Hi Guys,

I'm trying to write 40 channels of data to file in the RT mode. This is being done by putting these 40 channels of data to a build array function, then this array string is passed to a write to binary function. This is in a sequential loop, controlled by a wait until next ms multiple, where a user input defines the rate of data logging.

This data is all double precision, 64 bit. I cannot get good results above 50Hz.

Does anyone know the best way of getting faster results, as ideally I need to be able to log all this faster than 1kHz.

Should I look into improving the array bit, by using initialize array, then replace array element to fill up the array, or should I try to write to memory rather than binary. If this is the case, how is this done?

Many thanks,

Emmanuel

[altenbach]

There is no "write to binary" function in my file I/O, maybe this is specific to RT (which I don't have)?

Make sure to open the file once outside the loop, then append using low level file I/O without ever closing/reopeing the file. You could also accumulate the data a bit longer and write larger chunks at longer intervals.

Could you show a simplified example of your code?

[Emmanuel Doit]

Hi Altenbach,

Yes, I think the function write to binary may be specific to RT. Basically it is similar to the function write to excel, except that it creates a .bin file.

Attatched is a jpeg of the bit of my program I was referring to.

Thanks,

Emmanuel

[unclebump]

Not sure if RT will support working with queues. Send your data to a queue that feeds another loop. Use the other loop to do the actual saving to hard drive while gathering more data. The queue can act like a buffer to hold a backlog of data.

[Kevin_Price]

Emmanuel wrote:

Does anyone know the best way of getting faster results, as ideally I need to be able to log all this faster than 1kHz.  Should I look into improving the array bit, by using initialize array, then replace array element to fill up the array, or should I try to write to memory rather than binary...?

1. Yes, you should definitely use 'Replace Array Subset' rather than 'Build Array'.  In RT, you always want to work hard to avoid any memory allocations inside your main loop.

2. More advanced tip: here's an odd quirk that helped us out on a big RT project.  I don't know if this is a universally good idea or whether it was a quirk of our equipment (PXI controller) and RT version (6.1).  At that time, disk caching wasn't well supported in RT and we had to store all our data in RAM until the end of the experiment.  We were careful to do a one-time 'Initialize Array' before our main loop and used 'Replace Array Subset' inside it.  The problem was that we needed up to 100+ MB of RAM, and after a few consecutive runs we would run out of memory on our PXI controller.

I think it was an issue of memory fragmentation, where we couldn't keep getting the same 100+ MB chunk of RAM.  Eventually, there wasn't a contiguous chunk of that size available, and we'd have to reboot our RT controller.  Here is the solution we found:

We put a big one-time loop around our entire block diagram so we could store our allocated data array in a shift register there.  We made it an unitialized shift register so that it would continue holding onto the same 100+ MB memory space from one run to the next.  Before our real main loop, we checked the 'First Call?' primitive.  If True, we would use 'Reshape Array' instead of 'Initialize Array.'  On the very first call, this would actually allocate fresh memory for us.  On all subsequent calls, since we always requested the exact same dimensions, it would return immediately b/c it didn't have to do anything.  More importantly, we didn't run into the memory leak/fragmentation issue anymore.

I'm trying to write 40 channels of data to file in the RT mode...

This data is all double precision, 64 bit. I cannot get good results above 50Hz.

...ideally I need to be able to log all this faster than 1kHz

So, 40 channels/sample X 8 bytes/channel * 1000+ samples/sec = 320+ kB/sec.  I suspect this may be feasible under RT, at least on PC-like platforms, now that there is better file caching in RT version 7.  However,  if this much datalogging bogs down your RT response, there are ways to help yourself out.

A. Any analog channels can be represented in 2 bytes rather than 8.  Instead of writing a scaled 8-byte DBL, you would write the raw 16-bit A/D conversion value to file.  You'll also need to write the low-level scaling info to a file header so you can recreate the original voltages.  If all 40 channels are analog inputs, you can saved yourself a factor of 4 in storage.

B. Also, others have recommended that you separate your main RT loop from the datalogging via queues or similar.  NI has recommended the RTFIFO in place of queues for better determinism under RT, but they're used very much like a queue.

What you'd do is put your pre-allocated data array in the VI that reads the RTFIFO and writes to file.  Your main RT loop quickly writes its sample(s) to the RTFIFO and moves on.  In parallel, your file loop quickly reads from the RTFIFO and stashes the data into your pre-allocated array.  Then when you've filled up your pre-allocated array, you write it to file.  While busy writing to file, the RTFIFO will buffer up several RT loops worth of samples until you're ready to consume them.  Repeat.

-Kevin P.

[Emmanuel Doit]

Hi Kevin,

Thanks for your reply, most of it sounds like a good idea (and hard work!)

I don't think I can use your suggestion A however, as what is happening is that I am getting measured timing values from an FPGA, then in real time I am calculating the actual values, accurate to 2d.p.

[Kevin_Price]

Re: suggestion "A"

1. After the calculation, can you convert the 8-byte DBL result to a 4-byte SGL representation before writing to file?  Will this preserve the accuracy / precision you need? 

2. Can you store the raw 4-byte U32 to file, and save the calculations for a later post-processing step?

Just some thoughts on ways to shrink the data rate, albeit only 2x rather than 4x.

-Kevin P.