Example Code
Products and Environment
This section reflects the products and operating system used to create the example.To download NI software, including the products shown below, visit ni.com/downloads.
- LabVIEW
Software
Code and Documents
Attachment
Description
Read/Write Cluster Description
Attached is a 2015 version and 2018 version. The 2018 version uses VIMs and is a much cleaner way of using the type adaption functionality. In both examples the array of cluster data is flattened into the data, and the type. The type is written as a TDMS property while the data is written as the channel data. Reading it back will read the type, along with the data and then a variant to data can convert it back into the array of clusters that was first written. This solution is much for flexible since unlike other solutions, doesn't rely on unbundling and bundling data based on the specific type of cluster you are reading or writing. To update the data type just update the type def.
Known Issues and Limitations
Because of how data and properties are stored, there is a restriction on the Write function, in that you should not try to write a different cluster to the same TDMS Group Channel Pair. Similarly the Compression option used for a Group and Channel pair should not be changed.
Compatibility/Compression
Starting in version 3, the amount of padding needed in the data is greatly reduced making file sizes smaller. And version 5 adds better compatibility with platform portability. This is when a file is made on one platform, then copied to another. The Read function has the ability to read the older version, and read the newer version automatically. The Write function only writes Version 5, but Version 3 and later comes with three different modes to write. The first is with no compression, the second is compressing each cluster as it is written, and the third compresses the array of elements being written. When using the last mode a second TDMS channel will be used to keep track of the number of elements written with each array.
Version Mutation
Due to how meta data is stored in the TDMS file there are restrictions on appending data to existing files. If a file for instance was made and written with Version 3, and then Version 5 was used and more data was appended to the existing group and channel, then the old data wouldn't be able to be read properly. For this reason there is a check before writing data, that there doesn't exist old data in an old format. If there is the write is aborted, and an error is generated. Included with Version 5 is a Mutate Previous Cluster TDMS Version To Newest.vi. This VI can read versions 2 through 4, and then convert it to Version 5, and re-write it. Developers are only going to need this if they intend on using older files, with the newer version and they want to append data to it. Reading old data works just fine without any extra work.
LabVIEW Version Compatibility
Because this tool uses the LabVIEW primitives for flattening and unflattening strings, it is restricted by the features of those tools. One such restriction is that clusters that are written in a newer version of LabVIEW, might not be able to be opened in older versions of LabVIEW. An attempt to read it may return error 122 (0x7A).
Read/Write Cluster Steps to Implement or Execute Code
Unzip the attached zip, and open and run Main Write Read Example.vi and follow the instructions on the front panel which mention running the VI, changing the cluster, and running it again. Each run will write to a temporary TDMS file, and then read from it.
Get/Set Enum Cluster Property Description
VIMs for Reading/Writing Enums is also included. The Enums will be set as strings, while the Clusters will have their data and type flattened to strings and written to two separate properties. This allows for reading the cluster data as a Variant if the data type of the cluster is unknown.
History
Version 2 Added Compression, and reduced padding needed.
Version 3 Bug fix to address padding type
Version 5 Platform portability was added, allowing for files to be copied between targets and have the data read properly
Other Resources
This source is part of the Tremendous TDMS toolkit posted over on VIPM.IO. That package contains dependency information to other packages, as well as has several useful features not part of this source.
A discussion is over on LAVAG.
Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.
Is that how I did this? When you say it out loud it does sound inefficient. Instead of using the %d then formatting it into a flatten string with tabs, I think a better way would be to flatten it to %02x. This would represent the U8 as hex values. But since each U8 decimal will always be two ASCII bytes, the tab wouldn't be needed as a delimiter. Anyway glad you like it, I use this often but one thing I recently found is that the Excel TDMS add-on version 18.0 doesn't like large amounts of values in a property and often crashes. 17.0 is fine and other viewers are fine.
Hooovahh, I do have a question for you:
is there any reason that you chose to use the "Flatten/unflatten string to variant" functions? Perhaps I'm missing something, but it seems that it would be more advantageous to use the "flatten/unflatten to string" functions since these functions do not require the "Type" input.
I created the following VI:
As expected, string 2 is not equal to the other flattened strings. This is explained by the following article. (Something I just learned.) As such, it does indeed seem better to use the variant to flattened string function:
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019RPPSA2
With respect to unflattening the data though, it seems that I can use either the "unflatten string to variant" or the "unflatten from string" functions. Both seem to work just fine. However, the unflatten to variant solution requires that I store the "Type" property within the tdms file properly - which isn't terribly much work, but it is an extra step. Is there a downside to simply using the "unflatten from string" function?
Follow up to your initial response (apologies for double-posting)
I made the following VI:
I tested your "better way" by flattening to %02x. At first glance, this seems to work like a charm! This eliminates the need for a delimiter. Great suggestion.
This VI also demonstrates how I think that I can get by without having to write to the "Type" property in the tdms file. Is it potentially a bad idea for me to be using the "unflatten from string" function here? It seems to work well for me.
Lastly, thanks for linking your CAR topic. In my use-case here, I don't need to write any tdms properties... only string data. (Assuming that I don't update the "Type" property.) So, it sounds like I'm safe from the issue that you were seeing.
Thoughts? Thanks Hooovahh!
Okay so in making this I discovered two things, one of which you likely have already discovered. When flattening to a string, it needs to be a valid real string. By that I mean it can't be made up of \00 or any other nonprintable characters. For this reason whenever the type or data is being flattened, it needs to become something that contains only printable characters. The TDMS write will truncate a string at the first null and so the data won't be written properly. This is why in my case the type, and data, are both turned into an array of bytes, which then becomes ASCII string. If the flatten, or unflatten to string allows for creating a string that only contains printable characters (which I think it does) then that solution could be used too.
The second thing I wanted to do, is allow for storing the type of the data in the file. Ideally all you need to store is the Data as you've discovered, and Type isn't required in most situations. This is because the Type is provided by the cluster input on the read function. However lets think of a scenario where we write some software, that stores a cluster in a TDMS file. Then years later we update that cluster in our software. Now if we try to perform a read with the updated cluster, on the older file format we will get an error as expected. However if we only are storing the Data, and not the Type, then even LabVIEW can't understand the data format in the file. If you store the Type, and Data in the file together, then you can at least pull the data out of the file as a Variant. Then looking at the Variant you can display the data, or recreate the cluster needed later.
Basically this allows the data to be recovered if the type is unknown, which your logging type won't allow since it relies on the type to be provided, and requires it to be correct. That's why I store the Type as a property of the TDMS file once, and then the data is written into the channel.
I recognize that this thread is 3 years old now, but I just revisited this topic and made a silly discovery. The array to/from spreadsheet primitives don't accept "empty string constant" as a valid delimiter, and will override to using a tab delimiter. I didn't catch this 3 years ago. (And, actually, the documentation for these primitives don't mention this behavior, which is lame.) The code all still works fine, but my string-based tdms files have been getting unnecessary tab delimiters, bloating the files by... 33% or so I would assume.
I've adjusted things with the following code:
Hey no worries added context, and bug discovery is great, even if it is for one person years later. Revisiting this topic to come up with a solution that pads the data in the smallest amount would probably be a great idea. I've tried coming up with other solutions that would make this work better, but one issue I've come up with is if I change the format, then the files I already have won't be able to be opened up, without some kind of mutation preservation code. Maybe I think I can probably come up with some dummy property on a channel that would tell it if it is reading on the old format or a newer one. I think I could maybe add just escape codes to the unprintable characters, but then there is a performance hit for reading and writing. It really is a balance that should probably be up to other developers since there isn't a clear best solution for the read. The write it can probably be just a look up table regardless of the format.
EDIT: Actually now that I'm testing this, only the \00 null character doesn't work, this might simplify things.
oh wow, you might actually be right. This snippet here returns TRUE and NO ERROR... soo it seems like the null character is the only disallowed character:
Yeah thanks, I'm working on some improvements, hopefully I'll post it soon. I've also added a couple of compression options which will help even more with file size.
Ughh, it doesn't look like this forum allows for me to upload files, or else I'd zip this up and upload it. Maybe we should move this to LAVA?
Anyways, here's a bunch of snippets.
After confirming that the null character appears to be the only disallowed character, I wrote some escape/un-escape code.
Disclaimer!: I am not an escape artist. (heh.) So if I'm doing something silly, let me know.
But, basically, here was my strategy:
During escape, replace
\ --> \\
\00 --> \0
During un-escape, replace everything back to the correct characters.
Un-escape was harder code since I had to walk though the string linearly and keep track of offset. A dumb replace-all would not work for un-escape.
To the best of my knowledge and testing, it looks like this code here should work correctly. (And, hopefully, be decently performant.)
Thoughts? Thanks!
Yes everything you are suggesting is basically what I developed yesterday. I did replace the hex 0x00 with the ascii "\\00" adding 3 bytes of padding. Still it is only for when there is a 0x00 byte which isn't all that often. I won't be using the search and replace method because I believe it will be less efficient, but I'll do some speed tests to see what method is best for the pad and unpad functions. So far the fastest pad method is to convert the array of bytes to a string, then using a for loop find all 0x00s, then using a second loop use the Replace Substring adding the padding.
On top of that I have a compression method using zlib inflate and deflate. This works by finding patterns in the bits and replacing them with place holders.
But that got me thinking. If I am writing an array of a cluster, then there are probably lots of patterns in the data for the data types that haven't changed, and compression would work even better. So for that reason my next release will have 3 modes. No compression (just add 0x00 byte padding as needed) compression on each scalar, and compression on each array that is written. This will require another channel to track how many elements are in each array write. This information is a bit redundant because this information can be found in the written array data, but if I want to read the array subset starting at element 100 for instance, it would require reading all the data up to that point and uncompress each write.
Okay you likely already got pinged, but I updated the code to have a version 2 working the say I described. I tested it in several different ways, but please be sure and let me know if you find any bugs. The older no compression mode hasn't changed much from the first version and I have lots of confidence in it. The second compression method had lots of goofy math to get the reading of subsets of data to work and while I believe it is right, I could have made a mistake.
example. This code fails using your escape logic.
Thoughts? Am I misunderstanding something?
My previously attached snippets handle this situation correctly
Are you suggesting that the escape strategy is fine, but the characters I used aren't ideal? I ask because you example also has a similar flaw. What if the literal raw string contained 0x5C 30? Then with your strategy it will also replace it with 0x00 when you go to unescape it. If there is a literal string that contains my escape data of 0x5C 5C 30 30, then it will also contain your escape data of 0x5C 30. I figured the likely hood of the actual data accidentally containing the 4 bytes I look for in that order would be quite low using random data. However it is more likely to come up since I used printable characters.
If that is the case then what 4 characters should never, or rarely be ever seen next to each other? Probably stuff in the Extended codes greater than 128? Or a series of random unprintables at less than 32?
Edit: Oh I see you do two replaces, give me some time to try stuff out, I think your method would be better but does have a larger performance hit.
Okay version 3 posted which uses your escape code method. I have several speed tests, and unit test VIs but I didn't include them in the zip just to reduce all the ransom VIs and optional inputs.
This needs to be a u8 I think.
Looks great!! The optimization with a byte array vs my stringified method is cool
I did some performance testing for my own purposes.
I generated random strings with lengths from 0 to 10000.
Where:
stringified escape:
byte array escape:
stringified un-escape:
byte array un-escape:
cheers
Thanks this mostly confirms what I had seen. I have some test VIs for performance but didn't include them in the last post. I wasn't as through as you, and I just went with what looked the best without spending too much time on it.
Ugh, okay new issue. While the TDMS file can read and write 255 bytes just fine (with null needing special attention), when moving the files between targets, other OSs handle the extended ascii differently. So that means, for compatibility reasons, I may need to have special cases for 0x00, and for 0x80 through 0xFF. I have a pretty crappy but seemingly working padding routine.
Well I made a post here, but forum support from NI hasn't been great lately. I would make a TDMS file on the Windows side, and write to a channel as a string. Then I would FTP the file over to a Linux RT target. Then when I would read those TDMS string channels back, the data wouldn't match the data as it was written on the Windows side. If you viewed the string data as a string it would replace characters with "?" for many in the extended ascii range. I was just using the string data type as a place holder for an array of bytes, but the TDMS API appears to be manipulating it for displaying the string I just bumped the thread again but I expect NI to just say it is working as intended, and in that case more creative escaping is needed.
Okay version 5 is now posted with the help from brentjustice. Because I started using version 4 internally, I made several files with version 4 numbering, but never posted it publicly. So now I posted a new version 5 that is able to read all these older versions, but only writes version 5. I added some mutation code that I hope you don't have to use but I will be so I thought I'd share it anyway.
