08-11-2016 08:34 PM - edited 08-11-2016 08:53 PM
I may make this an idea exchange item later. But for now I'm just curious, are there people who have come up with a solution for reading and writing variant data to a TDMS channel?
My original idea was to turn a variant into an array of bytes, or a string, and then write that data to a channel. Turns out the string write is meant for ASCII string and stops at the first null so that doesn't work.
The solution I came up with is pretty crappy, but uses the MGI Read/Write anything code which turns a variant into an INI string. I then write this INI string to a channel and can read it back later. This usually makes the size on disk much larger than it needs to be.
Just before posting I did a search and came up with a post here, which turns a variant into the type and data, then writes these values as ASCII values which again makes the data on disk take up more space than it does in memory by quite a bit.
So has anyone came up with any other solutions for writing binary data to a TDMS channel? Is there any technical reason that the TDMS read/write doesn't support a raw mode? Is there a work around for reading and writing a binary string to a TDMS channel?
Edit: I just did a quick test and write a scalar I32 with a value of 8. This takes up 4 bytes of memory (obviously) but if I use the code that I linked to to turn a variant into the two ASCII string it takes up 14 bytes on disk.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
08-11-2016 09:27 PM
First, an admission -- I've never tried writing TDMS files. However, I think about it as the "NI Way" to write data, including sampled (binary) data, so was surprised by the notion that you could (only) write strings.
I haven't done the Experiment, but reading the Help file, which pointed to this White Paper, suggested that it was possible to write "raw" files in binary format (there's an example of an I32 file with values 1, 2, 3 written as four-byte binaries). I know your topic says "Variant", but did you really want to write "data of variable data type, unknown until run time, and varying from write-to-write" to the TDMS file? If the data for this particular element will be fixed for the duration of the file, you could pass in the Variant and transform it back to the corresponding Binary, writing "raw" data, couldn't you?
Bob Schor
08-12-2016 07:53 AM
@Bob_Schor wrote:I know your topic says "Variant", but did you really want to write "data of variable data type, unknown until run time, and varying from write-to-write" to the TDMS file?
Yes, that is my exact desire. Probably not what NI thought of when desiging the requirements for TDMS functionality.
Reading through that white paper does help a bit, but I think the real limitation comes from the list of data types that can be written at the end of section 2. The only data type listed that has any variable length is string (maybe DAQmxRawData but I have no idea how that works). And as far as I can tell the string read write, won't write the data as a raw string, but instead writes it as a string that is intended to be human readable, and will stop at the first 0x00, among other restricting rules. I've also tried editing the data in a TDMS file using a hex editor and even if you were able to write a string as some kind of raw data, the read also will do things like stop at the first null, even through the string length is defined.
So if your data type doesn't change from write to write, you can at least put the Type information into the property of the channel, and then the write you can really just write the data. In my situation however my variant will actually change from write to write so I need to write the type. Anyway attached is my attempt to write and read an arbitrary data type modeled after the code mentioned earlier. Still if anyone from NI in the TDMS group wants to chime in I'd be glad to hear other suggestions, I tried some DVR writing stuff but I think that has the same data type limitations.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
08-12-2016 08:14 AM
Where are XNodes when you really need them? [Excellent talk, by the way]
BS
08-12-2016 02:45 PM
I am sorry Brian but why di dyou not want to save the data as a data channel?
I pushed .jpgs and .pngs into a data chanel in the past. The TDMS viewer di dnot like it but the application worked.
Curious,
Ben
08-12-2016 03:14 PM - edited 08-12-2016 03:17 PM
Imagine I have a sequence, and each sample of a channel is a step in that sequence. Each step may have different data to be configured with, since each step could be of a different type.
So idealy I have something like two channels, one called "Step Type" which is a string stating what step to run, and then a channel called "Step Data" which is a variant which is specific to that step type. I understand I can write an array of U8 data to a channel, but that will write each element of the array as a sample, which isn't what I want. In this example I would want to actually write the array of U8 as a single sample, of a single channel.
BTW the TDMS viewer won't be used, it will of course be custom since the viewer needs to be able to display changing data types, either in a single control that can represent any variant as a tree, or as the data type that the specific step uses.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
06-22-2018 09:49 AM
Hello,
I just ran into this post, because I needed exactly this feature recently. As I saw this never got into an Idea, so I did it:
06-22-2018 10:01 AM
After VIMs were official I did end up making a Read/Write Cluster in TDMS example posted here. For completeness I'll be replying to the idea, but I did kudo it too cause something native is always preferred.
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord
06-22-2018 10:11 AM
@Hooovahh wrote:
After VIMs were official I did end up making a Read/Write Cluster in TDMS example posted here. For completeness I'll be replying to the idea, but I did kudo it too cause something native is always preferred.
Super! Thanks, I was not aware of this document! 🙂
06-22-2018 01:10 PM
I copied the two VIM into my project. Works like a charm!
Thanks again! 🙂