NI Home > Community > NI Discussion Forums

Data Acquisition Idea Exchange

Showing results for 
Search instead for 
Do you mean 
Announcements
The NI Idea Exchange is a product feedback forum where NI R&D and users work together to submit ideas, collaborate on their development, and vote for the ones they like best. View all of the NI Idea Exchanges to post an idea or add your opinion on an existing one today!
QFang

Supress storing .tdms_index files (useful when generating logs on cRIO and other SSD devices)

Status: New
by Active Participant QFang on ‎02-02-2012 01:10 PM

When using TDMS on cRIO systems, there are a couple of considerations that doesn't normally play in too much when storing data as TDMS files and they are:

 

* The current version of the file system used on cRIO controllers degrades significantly in performance if -any- folder on the cRIO contains more than ~100 files. (work-around> more elaborate folder structures, but a lot of this structuring would be only to work around this shortcoming of the (old) version of the file system 

* The drives are SSD with limited life-length and wear-leveling etc. Writing and re-writing these index files add un-necessary overhead and wear on the disks

* They use up space which is (very) limited on some cRIO's (even if not much). (people may be quick to point out that you can add a thumb-drive, but down-sides to that is the thumbdrives (as far as I know) needs to be FAT. Compared to storing on the cRIO file system which is atomic and fail-safe where you pretty much don't have to worry about sudden power outages and interruptions mid-write.. on a thumb drive you would have all these issues that could worst case corrupt your whole thumb drive.)

 

I propose to add a boolean (default to false) on the TDMS Open called "supress writing tdms index to disk" or some smart name along those lines. What this would do is still allow for the tmds index to be created, but it will remain in memory only and never be written to disk. When the TDMS Close is called, the memory is released and the tdms file is written to disk without the index file. If the same file is opened again, extra time would be needed since the index file would be re-created (again in memory only if boolean indicates this), but I think for the most part this overhead would be more than acceptable.

 

I'm not sure how "simple" modifying the TDMS open and close functions would be, but I do know that there are many cases where this flag would make sense. 

Comments
by Active Participant QFang on ‎02-02-2012 01:30 PM

Something like this TDMS Open (new)

by Active Participant YongqingYe on ‎02-02-2012 07:58 PM

Thank you QFang, for your post of TDMS idea!

 

The purpose of TDMS index file is to help quickly load TDMS file when calling TDMS Open. In quite a lof of use cases, the .tdms file contains raw data and is much larger than the .tdms_index file. So if the index file existing when opening the file, we will just load the inde file, it would be much faster than load the entire .tdms file. 


However, I can understand the siutation on RT, especially when the disk size is very limited. 

 

Thank you again!

by Active Participant QFang on ‎02-03-2012 07:54 AM

Yongqing Ye, yes, I absolutely agree the index file makes perfect sense in almost all use-cases, however, on the RT system, I'd be willing to trade off access speed when opening, for less file clutter on the disk. The index file would still "exist", just as an "in-memory only" instance, so writing to the tdms file itself should still be fast, just the initial open would be a little slower.

 

Thanks for the comments! 

by Member ba_0101 on ‎02-27-2013 04:23 PM

Check the TDMS Advanced Open VI. http://zone.ni.com/reference/en-XX/help/371361J-01/glang/tdms_adv_open/

 

"Opens a .tdms file for reading or writing in the byte order of the host computer. You also can use this function to create a new file or replace an existing file. Unlike the TDMS Open function, the TDMS Advanced Open function does not create a .tdms_index file."

by Active Participant QFang ‎01-28-2014 12:20 PM - edited ‎01-28-2014 12:23 PM

Many TDMS advanced functions are not supported on RT targets.  

nonsupported advanced TDMS functions

by Active Participant deppSu on ‎01-29-2014 07:37 PM

Hi QFang,

The TDMS group has put this feature into implementation in LV2013. Welcome to have a try!

Capture.PNG

by Active Participant QFang on ‎01-30-2014 07:32 AM

Hi deppSu,  indeed they have! Thank you TDMS Group.. and thanks for telling me, I don't know when I would have noticed if you didn't! 

-Perhaps this idea should be marked/moved as implemented?

 

I'll do some testing later, but from the documentation this is perfect!

by Active Participant deppSu on ‎02-07-2014 12:06 AM

Yes, this idea is not "new" anymore and its status should be updated.

About Data Acquisition Idea Exchange

Have an idea for new DAQ hardware or DAQ software features?

  1. Browse by label or search in the Data Acquisition Idea Exchange to see if your idea has previously been submitted. If your idea exists be sure to vote for the idea by giving it kudos to indicate your approval!
  2. If your idea has not been submitted click Post New Idea to submit a product idea. Be sure to submit a separate post for each idea.
  3. Watch as the community gives your idea kudos and adds their input.
  4. As NI R&D considers the idea, they will change the idea status.
  5. Give kudos to other ideas that you would like to see implemented!
Idea Statuses
Top Kudoed Authors