Hello Keenan, I am glad this is useful to you.
100kS/s/ch on 4 Channels (1.6 MB/s)
50kS/s/ch on 8 Channels (1.6 MB/s)
30kS/s/ch on 16 Channels (1.9 MB/s)
If you haven't seen it already I have benchmarked streaming applications in a whole matrix of scenarios using many different types of cRIO controllers. Those numbers are on the cRIO Wfm Library download page. Your requested throughputs will fit very comfortably to the local disk on chip even if you use the Wfm data type to TDMS (at which point the data is very easy to extract). You will notice however that on that table I don't have any numbers on streaming over USB.
I ran a series of tests last year using several different USB drives. The results were pretty poor and also inconsistent. Not only did the performance vary from drive to drive (expected since not all drives are USB 2.0 high speed for instance) but it seemed the results would vary from test to test even on the same drive. I don't have specific numbers but I clearly recall experiencing both greatly reduced streaming numbers and inconsistent performance.
However, one method that works very nicely for me is to stream to the local disk and then copy the data to the USB drive. This approach limits you to a file size no bigger than the cRIO disk on chip and the time between captures must be greater than the time it takes to copy your file over to the USB drive. As I said before you can do all of this using the waveform datatype and the TDMS file which makes using the data much easier.
Have you considered this approach?
S&V Systems Engineer - Certified Vibration Analyst III
Thanks for the quick reply!
Yes. I have been looking at your benchmark data. Hence my questions regarding streaming to disk using a the 1D Scaled U32 and then writng a binary file. Streaming at higher rates certainly looks possible. I figured that you were writing to the internal disk but figured I would ask about the USB HDD for clarification. I have considered writing the internal disk as a buffering method but I am not sure how to implement it in a way that would minimize to latency between captures (which is a priority). Do you have an example that saves the data locally then moves it?
I presently have a version of code running on my bench which is streaming 100kS/s/ch on 8 channels with 100kS per read which has been pretty stable for the last 30 minutes. There is typically only a couple of seconds of latency between captures as a new file or folder is created. The problem is how to extract the data from the binary file after it is written. I don't have a good understanding of the data type to make the decoding leap. Any help would be appreciated.
Hmm I'm not sure I understand the "low latency" requirement but I can take your word for it.
1. Concerning block sizes I typically operate between 0.1 seconds to 0.5 seconds. You shouldn't need anything bigger than this.
2. The data is saved in the binary file as an interleaved 1D array of U32s. I don't have example code for navigating binary data files however I recommend looking at the 2D SGL version of the rwfm_Read.vi to see how I de-interleave and typecast the data.
3. Moving data files is pretty simple. I have a vibration data logger reference design built on top of the cRIO Wfm Library. In this design I move the files to a "depot" after closing. There may be some other things of use in there as well.
As a quick experiment I just ran the 8 channel 100kS/s/ch code writing to .tdms instead of binary. The binary case remains stable but the tdms case causes the RT Buffer Back log to increase until the buffer overflows. Both use 1D Scaled U32. The only difference is the type of file that is being written.
So it would seem that the limit for writing to a USB HDD using .tdms is about 8 channels at 50kS/s/ch and 100kS/s/ch on 8 channels using binary. Does this make sense?
In terms of latency...
I want to continuously stream the data to disk. However, I want this to run for several days continuously. To keep the file sizes reasonable I write a new file every N minutes. To stay within the bounds of the FAT32 files system I also create a new folder every M files. The latency that I am speaking of is the time between when a file closes and when the next one begins. However, it has just occured to me that because the data in the is being stored in the FIFO the next file contains the sampel that immediately follows the last sample in the previous file. Neat!
No we should be getting better rates. While streaming to the local drive, you should be getting a little over 2 MB/s using Wfm datatype and TDMS. The SGL case to TDMS should get you over 2.5 MB/s. The calculations is sample rate * number of channels * 4 bytes per sample.
Are there any other parallel operations going on which may be stealing time from the CPU? Communication stuff, data processing, anything? If something is reducing your performance to local disk then it is certainly reducing your performance over USB.
The only other process that is being performed is the USB disk writing. See attached.
I can provide the whole project if that would be of use.
Ha I don't have 2011 yet. I misred your original post I thought you were getting those rates to the local disk, not over USB. I honestly don't know what numbers to expect over USB. So where are we at on this project? Is it working how you would like or is there still issues?
No worries. Unless I am doing something lame in terms of writing to the USB disk I think I may have found the limitations of writing to USB via tdms.
Writing directly in binary shows an advantage (as was indicate in the benchmark). However, I can't seem to figure out the following:
1. How to decode the 1D Scaled U32 output from the DMA in order to read the binary file after it is written.
2. How to decode the 1D Scaled U32 output from the DMA in order to extract a channel for other processes.
Could you possibly help dedcode the stream? I have attempted following the 2D example and have only made marginal progress. Apparently there is somethign fundamental that I am missing here.
Maybe I'm misinterpretting your question. But when you read it from binary, you need to read a 1D array of U32 values and the numbers you get will be interleaved together in the same way it is on the DMA channel. The read binary VIs don't have any special smarts to get you just one channel or anything like that (TDMS API does however).
The "1D Encoded U32" version of the Read VI basically returns data straight off of the DMA channel. If you are reading 5 channels worth of data then the 1D array U32s coming off the DMA channel looks like 1,2,3,4,5,1,2,3,4,5,1,2,3,4,5, etc. The binary file has the same format. So if you want to get channel 1 for instance then you need to read every fifth data point and append them together. Another option is to read the block size that you wrote and deinterleave by number of channels and points per channel (as the Read VI SGL does).
Is this what you are asking?