Ability to split between multiple TDMS files while logging

[AndrewMc]

It has come up a few times from customers, and I wanted to gauge interest and solicit ideas on how this should work.

Currently, with the built-in TDMS logging support, if you want to change to a new file in the middle of logging, you need to stop the task and start again.  For some use cases, this isn't practical (for example, http://forums.ni.com/t5/LabVIEW/Why-the-TDMS-file-is-larger-than-it-should-be/m-p/1176139#M511099).

The question is: How would you like to specify the "new file" behavior and what are your use cases?

For instance, a couple ideas to get the ball rolling:

1. Add an interval attribute like "Change file after n samples".   We would then auto-increment the file name and change to that file when we have logged "n" samples.
2. Make file path attribute changeable at runtime.  We have a file path attribute for logging.  The idea here would be to support changing the file path "on the fly" without stopping and starting the task.  The problem here is that it would not suit very well a use case where you want a specific file size.  Additionally, it wouldn't be as easy to use as #1; it would be more flexible though.
3. (Any additional ideas/use cases?)

Thank you for your input!

Andy McRorie

NI R&D

It has come up a few times from customers, and I wanted to gauge interest and solicit ideas on how this should work.

Currently, with the built-in TDMS logging support, if you want to change to a new file in the middle of logging, you need to stop the task and start again.  For some use cases, this isn't practical (for example, http://forums.ni.com/t5/LabVIEW/Why-the-TDMS-file-is-larger-than-it-should-be/m-p/1176139#M511099).

The question is: How would you like to specify the "new file" behavior and what are your use cases?

What I'm currently thinking (because it seems the most flexible to different criteria and situations) is to simply allow you to set the file path property while the task is running (on DAQmx Read property node).  The only downside I can think of with this approach is that you wouldn't know exactly when we change to the new file.  We could guarantee within (for example) 1 second, but you wouldn't be able to specify the exact size.

Would this be a good solution for you?  Can you think of a better way to specify this behavior?

[QuantumMonk]

I managed to get this exact thing to work by having the filepath appended with a number corresponding to the number of iterations completed by the data acquisition while loop (and the file writing while loop) divided by 100.  By taking this divided number and rounding it down (toward negative infinity) and appending that number to the end of the filepath (set the filepath as a string, use concatenate strings to add the rounded number, then use a string > filepath conversion module) I was able to make it automatically parse the data into multiple files which are much more reasonably sized than some of the files I was collecting before doing this.

One thing to keep in mind however, the program doesn't seem to close the session with any of the files it writes to until you have stopped the loop from functioning, I'm sure there is a way to code around this but I haven't taken the time to do so because usually when I finish collecting data I can simply stop the program and head out to analyze what I collected in matlab.

Let me know if you have any questions about this method, or any suggestions for improvements.

Jesse

[AndrewMc]

Thanks for the input, Jesse.  I will consider that to be a vote for #1 in adding this feature to the integrated logging support in DAQmx.

[Wayne.C]

I can see the following attributes to control when a new file is automatically started:

1 - Change file after 'n' samples - I just got done writing an app to split large TDMS files and dump to multiple worksheets in Excel workbook.  This is a way to avoid the 65K limit in older versions of Excel.

2 - Change file after time interval - Would be fantastic if I could tell it to start a new file every 'x' hours.  Just what I need for long term life testing.

3 - Change file after max size - Often desirable to optimize size of TDMS files.

[Daklu]

If it's one or the other, I vote for #2, make the path changable at runtime.  I'll pick flexibility every time.  Easy to use options are useless if they don't do what I need.  All the "user friendly" solutions are just different events that trigger a name change.  How about instead of pre-defining what events will trigger a name change you give me the tools to create the solution that fits my needs?

Maybe I want to create a new date stamped file at midnight every night.  Maybe I want to store my wind speed data according to the month on the chinese calendar.  Who knows?  IMO a good api makes the common things easy and the uncommon things possible.  The user friendly solutions proposed certainly make the common things easy, but they completely ignore the uncommon requirements.

[R Sandoval]

Hello There,

There are two ways of logging multiple files using SignalExpress.  Please see the article below:

Logging Continuous Data to Multiple Files in LabVIEW SignalExpress

http://digital.ni.com/public.nsf/allkb/393BA6B1F74DA8AB862572910008661D?OpenDocument

[Dave_Karon]

I like either.

In a recent test program, I ended up having files upwards of 12 GB. Sometimes, although rarely, the files were up to 32GB  That makes it really hard to upload to / download from an ftp server.  If there were soem clean mechanism to switch files in mid-stream, it would really make things quite a bit easier to hadle the data from this kind of test.

[josborne02]

Kudos.  I would like a function like this.  But I don't want it necessarily tied to streaming from DAQmx ... for example:

My custom-built LV application reads data from a serial RS232 connection.  It processes the data and writes to a TDMS file.  The files get big because we are logging continuously, 24 hours a day.  Of course, I could close the file and open a new TDMS every time the current file reaches 1 GB or something.  But it would be great if LV did this automatically. 

BUT, more importantly, is how we read the data.  I would LOVE it if I specify that I want to open the FIRST file and LabVIEW would automatically recognize that this is part of a "spanned" dataset ... eg that is spans over multiple TDMS files and would autocmaitcally jump to the next file when required.  Essentially look at the multiple files as one large file.

[ssomnath]

I vote for option 1. Has this feature already been implented?

[AndrewMc]

This feature has been implemented.  What we ended up with is essentially both #1 and #2, so hopefully your use case will be solved.  See this post here for more details: http://forums.ni.com/t5/Multifunction-DAQ/DAQmx-Logging-New-Features-Split-files-non-buffered-loggin...

[JordanG]

http://forums.ni.com/t5/Multifunction-DAQ/DAQmx-Logging-New-Features-Split-files-non-buffered-loggin...