LabVIEW
Best Practices for Real-Time Data Logging in LabVIEW Using NI DAQ Devices
The simplest way is to just configure DAQmx to stream it straight to a TDMS file: https://www.ni.com/en/support/documentation/supplemental/21/ni-daqmx-high-speed-streaming-to-disk.ht...
That'll save all of the data you capture. If you're trying to save subsets of data (e.g., monitoring continuously at 100 kHz for an hour, and need to save 100 ms of interesting content) you'll have to do it manually.
In general though, you want a producer/consumer architecture. And you're allowed to have multiple of these in one program! For example, you can enqueue your data into multiple queues to be removed by multiple loops. You might have one loop for your UI, which takes your 100 kHz data and averages it out, updating your screen every 100 ms. Another loop might write each datapoint to disk. (I'd recommend TDMS files for storage, but you can use CSV's too- they're just a LOT bigger on disk.)
Look up some of the project templates that ship with LabVIEW. IIRC one is "continuous measurement and logging" that might point you in the right direction.
Mostly just reiterating what Bert said...
1. For logging data, the DAQmx Configure Logging is by far the most efficient route to take. It easily beats out a Producer/Consumer for logging data to disk.
2. It is best to have a loop that does nothing but read from the DAQ. Use a queue or notifier to pass the data to other loops that may need the data. There should be no explicit wait in this loop.
3. There is a rule-of-thumb to read 100ms of data each iteration of your loop. Obviously, the number of samples you read will depend on your sample rate.
4. For most setups, do not worry about setting the buffer size. The default is usually larger than you will need, especially if you are following point 3 above.
@crossrulz wrote:
3. There is a rule-of-thumb to read 100ms of data each iteration of your loop. Obviously, the number of samples you read will depend on your sample rate.
Slightly off topic, but this is the first I've heard of this. Is there some history behind it? Granted, I don't save my data to disk (instead I process it in chunks and send it out via TCP), so is it HD write speed or something along those lines that would suggest you should write more frequently instead of in larger chunks? The handshaking alone adds enough delay to justify doing mine in 2-second chunks from a network performance aspect instead of more frequent smaller transfers, but I realize that writing to disk is apples to oranges when compared to sending out network traffic.
@DHerron wrote:
@crossrulz wrote:
3. There is a rule-of-thumb to read 100ms of data each iteration of your loop. Obviously, the number of samples you read will depend on your sample rate.
Slightly off topic, but this is the first I've heard of this. Is there some history behind it? Granted, I don't save my data to disk (instead I process it in chunks and send it out via TCP), so is it HD write speed or something along those lines that would suggest you should write more frequently instead of in larger chunks? The handshaking alone adds enough delay to justify doing mine in 2-second chunks from a network performance aspect instead of more frequent smaller transfers, but I realize that writing to disk is apples to oranges when compared to sending out network traffic.
I can say from experience that it is easy to go down the rabbit hole of buffer size, samples to download, etc. Here are some things that I have found; they are corner cases and really only apply to high stream rates.
- 100 ms of data works 99% of time. (For low sample rates, then a longer interval also works.) However, sometimes using an allowed File Write Size value is better. To use the File Write Size you need to switch temporary to Log Only Mode and see what allowed value is closest to 100ms.
- The sample size, buffer size and file size need to be multiples of each other or evenly divisible, like 1/2 etc. and be a multiple of the disk sector size.
Below are some screen shots of a simulated instrument running at a high rate using the above two points. It has been running for the last 20 minutes without issue in Log and Read mode, but not saving the data.
The values I had were:
Sample Rate 10MSa/s/ch 8 Channels
N Samples to Read 125952 Samples ~ 12ms of data!
Buffer Size 80609280 Samples ~ 8s of data
File Size 20152320 Samples ~ 2s of data
File size is 1/4 of buffer Size
Buffer Size and File Size both multiples of NSamples to Read and disk sector size(512)
You can see that DAQmx can be efficient, as the CPU load is low.
