Writing data to span of file for high sampling rate data acquisition

[Amartansh13]

Here are the tasks which I need to do for taking noise measurements:

1) Take data continuously from USB 6281 board at a 500k sampling rate (50k samples at a time).

2) Save the data continuously for 3-6 hours in any file (any format is okay but I need to save in series of files rather than the single file). I want to start writing to new file after every 2 min.

I have attached my VI and images of my task configuration. I can take the measurement and write data into file continuously for 15 min. After that I am seeing these error:

1) Hardware acquisition cannot keep the pace with the software (something like this, also with a suggestion to increase the buffer size...)

2) Memory is full.

Please help in making my VI efficient and correct. I know you will suggest to remove the "Write in measurement file" from consumer loop because it takes a lot of time to open and close a file in a loop. You may suggest me to open the file outside the loop and write inside the loop. But I want to save my data in new file after every 2 min or after certain samples. If this can be done efficiently without using the Write in measurement file then please let me know.

Thank you in advance.

[mcduff · Accepted solution]

Can't see your code, I only have 2015.

I suggest you look into the functions contained in the snippet. It will do the logging automatically for you, in TDMS format. No need for another loop.

mcduff

[Amartansh13]

Thanks a lot for the reply!

Are you sure that this will work at high sampling rate (500K S/s) for 3 to 4 hours without getting memory full? I mean, can this function work in high speed while loop without lagging or overflowing the memory? 

 Please tell me how to put this function inside a loop.

Earlier when I used tdms streaming, it was using large disk space compared to xlsx file. But this should not happen because we know tdms is better than xlsx streaming in terms of memory usage and speed. Why it is happening? 

Here are the images of my code if you want to see: 

Please reply ASAP

[mcduff]

Post your vi, it will be easier to modify, in 2015 version

[Amartansh13]

Thanks for helping me to this extent. 

I have saved y vi in 2014 and 2015 versions using option: save  n previous version. 

If you still can't open it then please let me know how can I convert my 2016 vi in 2015 or earlier version. 

Here are the attachments: 

[mcduff]

I am having trouble for some unknown reason reading your VI. Attached is an example, you can find it in the example finder that should show you how to do automatic logging. Basically, delete your bottom loop and before the task start configure logging as you see in the example.

mcduff

[Amartansh13]

Thanks for replying

The code which you posted is for saving the data in a single file. Since I want to log data for 6 hours, I need it to be in multiple files. 

Good news is that my above code is now working without any error for 6 hours. I just changed the file format. Earlier in "write in measurement file vi", I was saving data in xlsx format (may be because of which RAM was getting full). Now I changed the format in TDMS. Now everything is working correctly and I am saving around 500k*60sec= 30000k samples per min for continuous 3- 6 hours. 

I just want few more help from you: 

1) Please have a look on my new VI (attached screenshots) and "write to measurement file vi" configuration and let me know that is it optimized and suggest any optimization if needed. Also, please tell me if there is something wrong at small levels. 

2) How can I load this huge data into Matlab (accurately without any data loss) for further processing. There are many codes (Matlab codes) available on Mathworks and NI forums but every code has some bad reviews. Can you please tell me the latest and most optimized way of loading all my data into Matlab? 

Thank you

Amartansh

[mcduff]

Look at the Modified code I am sending, I haven't had a chance to look at your VIs yet.

There is a property node that will break up the files automatically, I added it to the original example.

2) How can I load this huge data into Matlab (accurately without any data loss) for further processing. There are many codes (Matlab codes) available on Mathworks and NI forums but every code has some bad reviews. Can you please tell me the latest and most optimized way of loading all my data into Matlab? 

Batch convert the TDMS files after the data is saved. You can save each channel in a separate file as a flat double binary and tell Matlab to import that.

mcduff

EDIT: Forgot to attach File

[Amartansh13]

Thanks a lot

One last thing and then I am done

Can you please elaborate this:

"Batch convert the TDMS files after the data is saved. You can save each channel in a separate file as a flat double binary and tell Matlab to import that."

I didn't understand this.

Can you give me example or steps to be followed. 

Sorry for the inconvenience

Regards

[mcduff · Accepted solution]

This example here is for one file and one channel, you should be able to loop over this automatically. The bottom comment should be channel name, not group nameby the channel name in control.