Continuous sampling issue with buffer size

[MohsenJavid]

Hi all, 

I've wrote a program to have multiple readings from a cyclic machine and I would like to have at least 100 samples per second. In addition, I am using NI-cDAQ 9178 chasse and NI-9237, NI-9205 modules. I've set the acquisition mode to "Continuous samples", the sample to read to "10", and the rate (Hz) to "1000". But I receive a buffer size error. Each test will continue for 24-48 hour. So I need to resolve this issue and make sure it will continue acquiring data for the aforementioned time of the test without any issue. 

I 've read some of the previous questions and answers, but I couldn't understand how to resolve this issue. I would appreciate any help in this regard. you can also find the error description in below: 

Thanks

[mcduff]

Can't look at your VI on this computer, but you are updating too frequently. Update every 10 Hz, so increase the number of sample to read 100. (100/1000 = .1 s = 10Hz)

If you change the sampling rate, change the number of samples accordingly.

mcduff

[MohsenJavid]

Hi

Thanks for your response. I increased the Sample to read to "100" and changed the rate to "10 Hz", and I run the code for 10 minutes and it worked without any error. I really appreciate your help. it moved me one step forward. But it keeps record of 16-17 data per second, I mean it records by 16-17 Hz rather than 10 Hz. 

In addition, how can I check that it will perform well during the test which will take 24-48 hours (in terms of buffer size and memory)? With this rate of data logging for more than 24 hours, I have a concern of any error of memory in DAQ system. Do you have any suggestion on that? 

Again, thank you for your help in this regard. 

Mohsen

[mcduff]

Will try to look at you VI when I am on a computer with LabVIEW. The minimum sampling rate for the 9237 1613Hz; your sample rate may be getting coerced. At these low rates, memory should not be an issue.

[Kevin_Price · Accepted solution]

You'll need to learn some new things before your program will be robust and reliable and I don't have the time to try to teach all those things in this post.  My brief recommendations for relevant topics:

1. Use regular DAQmx functions instead of the DAQ Assistant.  They'll give you much more thorough control over your data acquisition tasks.

2. I would also avoid the file writing express vi's.  Right now it appears that you might be generating new files with every loop iteration?  I'm generally suspicious of almost *all* express vi's, as are most of the reasonably experienced folks hanging around here.   Again, using regular file writing functions give you much more thorough control.

3. Learn about Queues or Channel Wires and the Producer/Consumer pattern.  Then use your data acq loop as a producer and a separate file writing loop as a consumer.

-Kevin P

[MohsenJavid]

Thanks for your note. I will try to learn about the items that you kindly mentioned. But for now I have to do few tests before the end of next week and each test will take 24 hours. I will try to change the code in a way that you suggested, but I don't think I can do it really quickly. So, I have to use DAQ assistant for coming 10 days to do the tests. In addition, This loop that I've created writes 3 separate files as outputs. Based on the urgency of the issue, I tried to play with the numbers inside DAQ assistant to see what I can get. I used "continuous sampling", "Sample to read=100", and "rate(Hz)=5000". The code ran for 5.5 hour (19934 sec) and stopped due to the same error as before (see the screen shot). I looked at files and there is 50 sample per second and it works for me for now. I have 10 different transducers and sensors to read. I am wondering if there is any way to extend the time of data logging to have at least 24 hours of data (rather than 5.5 hour)? 

I do appreciate any idea or way to resolve this issue, if it is any. 

Sincerely,

[MohsenJavid]

Thanks for your note. If you see my answer to Kevin, the code ran for 5.5 hours and it stopped with the same error as before. I have about 100k line of data now. I used "continuous sampling", "Sample to read=100", and "rate(Hz)=5000", and this setup was able to record 50 line of data per second. I am really confused. I couldn't understand the relationship of sample to read and rate to the frequency of data logging. I assume the buffer size is somehow connected to "Sample to read" value and if I increase that, it would continue data logging for longer time. But I need to increase the "rate" to be increased relative to "sample to read" to have the same number of data per second (as I told, I have 50 sample per second with aforementioned setup). But there is a limit for the "rate" and it is 10,000 Hz. I really appreciate any thoughts and helps on this case. 

Sincerely, 

[Kevin_Price]

Fundamentally, the error you eventually get is likely caused by inefficiencies in the file writing Express VI's and maybe also the DAQ Assistant.   You'll want to learn to get away from using them when you need higher performance and efficient operations.

In the meantime the main "shortcut" I could suggest you try is to read 1-10 seconds worth of data per loop iteration instead of a fraction of a second worth.  The DAQ Assistant and the file writing Express VI's have a certain amount of overhead, so you're better off iterating less often and dealing with larger chunks of data each iteration. 

    So if you sample at 5 kHz, you might try reading 5k-50k samples at a time.   For starters, let's just try reading 10k samples at a time, which is 2 seconds worth.  Now you'll only incur that overhead penalty once every 2 seconds instead of 50 times a second.

If your averaging needs to be done across smaller time intervals (like 0.01 seconds for example), look into the "Decimate (continuous)" function.  That'll let you wire in your 10k samples as input, averaging across blocks of 50 points (to reduce the effective sample rate from 5 kHz to 100 Hz), producing 200 averages as output.  That reduced output is what you'd write to your files.

-Kevin P

[mcduff]

+1 for what Kevin has told you.

I finally looked at your VI, you need to follow Kevin's suggestions. Do you have a new one to upload?

One other suggestion involves possible post processing of a big data stream. You are saving the mean of the data. In the DAQmx Assistant, enable logging and get rid of your mean and saving data express VIs. After the the DAQ acquisition is complete, parse the file as you see fit. For example rather than a mean, a RMS value would be more appropriate for AC data. You could also change the size of the mean, etc.

mcduff

[MohsenJavid]

Thanks for your help in this case. I've wrote a new application using DAQmx functions and it is working fine.