Saving to ATFX is very slow

[DIAdemUser2022]

Hi all,

we are using Diadem 2021 in our project load a TDMS file, process the data and then save it to the ATFX format (the output format is a project requirement). Unfortunately, saving to ATFX is very slow (around 15 Minutes). The resulting ATFX file is 54 megabytes in size. When saving the same data to TDMS, saving roughly takes 15 seconds and the resulting file is around 30 Megabytes in size. The behaviour is the same on all of our machines and the customers machines. Files are being saved to a local (SSD) hard drive.

Plugin-Version is 21.1.0f8226.

Unfortunatly, I cannot post the original TDMS file here, as it would reveal project details. The data portal is roughly layed out as follows though:

20 Channel groups with around 800 channels each. About 30% of the channels contain 200 values, the remaining channels only contain up to 10 values.

Does anyone experience the same? Is this a known issue, even?

If you need any more information, please let me know.

Kind regards,

DiademUser2022

[Dia791]

Hey User, we don't use ATFX, but it's an XML structure with additional header elements. Saving a small TDMS (350 kB) as ATFX I receive 600 kB file size - looks similar to your experience. We commonly export the data as ASC (text file) or CSV. The file size is approx. 2x larger, and the export process takes minutes (in comparison to few seconds). Therefore last times we agreed with customers to use TDMS, which they can open with Excel. Summarizing: IMO the issue you describe is not new, and (imo again) is based on the file format and export process. Greetings, Vassili

[DIAdemUser2022]

Hey Vassili,

thanks for your reply! That is kind of unfortunate to hear. Our customer asked for a non-binary format for long term archiving, and ATFX seems suitable for that.

I would be glad to help find/fix this issue, so in case any NI devs need more info from my side, I would be glad to invest some more time.

Kind regards,

User

[Dia791]

Well, what is the actual problem?

The size becomes larger, unless you reduce the scope (remove some channels which are not required). Could you zip the folder? This should reduce the size (for long-time storage); but I'm not sure for ATFX.

The time? Then prepare everything and let the code run via weekend (for all data files in one run).

That's all I can propose by now...

[DIAdemUser2022]

Hi,

Thank you for your help and the suggestions, Vassili!

File size is not an issue here. I mentioned the file size to highlight that saving to TDMS only takes 15 seconds while saving to ATFX takes 15 minutes, with - roughly - the same resulting file size and exactly the same amount of data to process.

The issue is the time is takes to save the files. It is no option to move this job to the weekends, the customer needs the data in time (otherwise, 15 minutes would not be a problem either).

Kind regards