TDMS: Marshalling of 3 byte UTF-8 characters

[jonas.d]

When I try to write a 3 byte UTF-8 character (such as '≤', 0xE2 0x89 0xA4) to a TDMS string property or channel I get a System.ArgumentException, Cannot marshal: Encountered unmappable character.

Even when I use our own TDMS writer implementation regarding the TDMS File Format Internal Structure, DIAdem as well as the Measurement Studio TDMS API fails while opening the file.

So I'm very very very afraid that UTF-8 is not supported completely.

Best Regards,

Jonas

[MelanieE]

Hi Jonas,

the link you mentioned above includes the following lines:

All strings in TDMS files, such as object paths, property names, property values, and raw data values, are encoded in UTF-8 Unicode. All of them, except for raw data values, are preceded by a 32-bit unsigned integer that contains the length of the string in bytes, not including the length value itself. Strings in TDMS files can be null-terminated, but since the length information is stored, the null terminator will be ignored when you read from the file.

So the information you try to write to the file is converted to UTF8 format, but an uint32 is expected.

By the way, do you get some error code in Diadem or Measurement Studio?

Best,

Melanie

[jonas.d]

Hi Melanie,

thanks for your quick reply. I'm familiar with the TDMS document and my interpretation is that the byte count of the string - not each character - is coded in 32 bits.

Nevertheless, I made a mistake yesterday regarding DIAdem. It is able to read the TDMS file if we write it using our own implementation.

I have 3 strings, where each consists of one character.

The binary is:

'a'  01 00 00 00 61

'ö'  02 00 00 00 C3 B6

'≤'  03 00 00 00 E2 89 A4

DIAdem works as aspected and displays the strings correctly.

So the remaining problem is that I can't use Measurement Studio to write the third string:

System.ArgumentException : Cannot marshal: Encountered unmappable character.

at System.StubHelpers.MngdNativeArrayMarshaler.ConvertContentsToNative(IntPtr pMarshalState, ref Object pManagedHome, IntPtr pNativeHome)
at NationalInstruments.Tdms.Internal.e.a(i A_0, String A_1, TdmsPropertyDataType A_2, String[] A_3)
at NationalInstruments.Tdms.Internal.g.a(i A_0, String A_1, String A_2)
at NationalInstruments.Tdms.Internal.g.a(i A_0, String A_1, Type A_2, Object A_3)
at NationalInstruments.Tdms.TdmsProperty.a(String A_0, TdmsFile A_1)
at NationalInstruments.Tdms.TdmsFile.AddProperty(String name, TdmsPropertyDataType dataType, Object value)

Language is C#, Assemblies are:

NationalInstruments.Common, Version=15.0.40.49154

NationalInstruments.Tdms, Version=15.0.40.49153

My assumption is that there is a problem with marshalling the string to the underlying TDMS C library.

Thanks for your help!

Best regards,

Jonas

[MelanieE · Accepted solution]

Hi Jonas,

thank you for pointing out the real behavior.

I did some research on the error message you got. As you suggested before, the TDMS .NET class library does not support unicode. If you try to write unicode characters to a TDMS file, you receive the following ArgumentException: Cannot marshal: Encountered unmappable character.  

This information is mentioned under the following link at the bottom of the page:

AddProperty Method (String, TdmsPropertyDataType, Object) - Measurement Studio 2012 for Visual Studio 2010 Help - National Instruments
http://zone.ni.com/reference/en-XX/help/372636F-01/mstudiowebhelp/html/2d61d5b7/

Best,

Melanie

[jonas.d]

Hi Melanie,

reading the docs is always a good choice 🙂

Even if it is not a satisfying answer, I know that I can stop working here. Thanks!

Best regards,

Jonas