Data Acquisition Idea Exchange

cancel
Showing results for 
Search instead for 
Did you mean: 
QFang

Supress storing .tdms_index files (useful when generating logs on cRIO and other SSD devices)

Status: New

When using TDMS on cRIO systems, there are a couple of considerations that doesn't normally play in too much when storing data as TDMS files and they are:

 

* The current version of the file system used on cRIO controllers degrades significantly in performance if -any- folder on the cRIO contains more than ~100 files. (work-around> more elaborate folder structures, but a lot of this structuring would be only to work around this shortcoming of the (old) version of the file system 

* The drives are SSD with limited life-length and wear-leveling etc. Writing and re-writing these index files add un-necessary overhead and wear on the disks

* They use up space which is (very) limited on some cRIO's (even if not much). (people may be quick to point out that you can add a thumb-drive, but down-sides to that is the thumbdrives (as far as I know) needs to be FAT. Compared to storing on the cRIO file system which is atomic and fail-safe where you pretty much don't have to worry about sudden power outages and interruptions mid-write.. on a thumb drive you would have all these issues that could worst case corrupt your whole thumb drive.)

 

I propose to add a boolean (default to false) on the TDMS Open called "supress writing tdms index to disk" or some smart name along those lines. What this would do is still allow for the tmds index to be created, but it will remain in memory only and never be written to disk. When the TDMS Close is called, the memory is released and the tdms file is written to disk without the index file. If the same file is opened again, extra time would be needed since the index file would be re-created (again in memory only if boolean indicates this), but I think for the most part this overhead would be more than acceptable.

 

I'm not sure how "simple" modifying the TDMS open and close functions would be, but I do know that there are many cases where this flag would make sense. 

QFang
-------------
CLD LabVIEW 7.1 to 2016
8 Comments
QFang
Active Participant

Something like this TDMS Open (new)

QFang
-------------
CLD LabVIEW 7.1 to 2016
YongqingYe
NI Employee (retired)

Thank you QFang, for your post of TDMS idea!

 

The purpose of TDMS index file is to help quickly load TDMS file when calling TDMS Open. In quite a lof of use cases, the .tdms file contains raw data and is much larger than the .tdms_index file. So if the index file existing when opening the file, we will just load the inde file, it would be much faster than load the entire .tdms file. 


However, I can understand the siutation on RT, especially when the disk size is very limited. 

 

Thank you again!

QFang
Active Participant

Yongqing Ye, yes, I absolutely agree the index file makes perfect sense in almost all use-cases, however, on the RT system, I'd be willing to trade off access speed when opening, for less file clutter on the disk. The index file would still "exist", just as an "in-memory only" instance, so writing to the tdms file itself should still be fast, just the initial open would be a little slower.

 

Thanks for the comments! 

QFang
-------------
CLD LabVIEW 7.1 to 2016
ba_0101
Member

Check the TDMS Advanced Open VI. http://zone.ni.com/reference/en-XX/help/371361J-01/glang/tdms_adv_open/

 

"Opens a .tdms file for reading or writing in the byte order of the host computer. You also can use this function to create a new file or replace an existing file. Unlike the TDMS Open function, the TDMS Advanced Open function does not create a .tdms_index file."

QFang
Active Participant

Many TDMS advanced functions are not supported on RT targets.  

nonsupported advanced TDMS functions

QFang
-------------
CLD LabVIEW 7.1 to 2016
deppSu
Active Participant

Hi QFang,

The TDMS group has put this feature into implementation in LV2013. Welcome to have a try!

Capture.PNG

QFang
Active Participant

Hi deppSu,  indeed they have! Thank you TDMS Group.. and thanks for telling me, I don't know when I would have noticed if you didn't! 

-Perhaps this idea should be marked/moved as implemented?

 

I'll do some testing later, but from the documentation this is perfect!

QFang
-------------
CLD LabVIEW 7.1 to 2016
deppSu
Active Participant

Yes, this idea is not "new" anymore and its status should be updated.