"cannot copy bulk data"

[ramadugu_ravi@yahoo.com]

Hi, 

I am running a script on TDMS files. Basically the script searches for the channel groups (such as Voltage 1, voltage 2. voltage 3...Current 1, Current 2, Current 3.....) and then creates a new TDMS files with combining the channels...such as Voltage..Current...

The scripts works fine for TDMS files upto some 40MB....Now I have a file of 92 MB size....and while running the script Diadem throws an Error "Cannot copy bulk Data".....(screent attached)......

Did any one got the similar problems.....Please help.....

[AndreasK]

How many files do you load?

There are two possible ways to reduce memory usage.

Register the files instead of loading them.

Call DataFileLoad("C:\temp\EXAMPLE.tdms","TDMS","Register")

 Change the point where DIAdem starts swapping to file:

Settings->Options->General : Data Area

Greetings

Andreas

P.S.: Do you use datafinder to find the files to concanate.

P.P.S.: Might it be a good idea to load only the parts needed (selective loading) instead of the whole files.

[ramadugu_ravi@yahoo.com]

Dear Andreas,

We tried registering the TDMS file, instead of Loading...but no luck....we got a similar error message saying that "Cannot register..." (screen shot attached)....

The file is about 22 MB. (twenty two MegaByte only)

We tried opening and reading the channel data from LabVIEW 2011, we are able to read the contents, however it is deadly slow...for one TDMS Read function it takes about 1 sec to read the data.

We have about 80,000 chnnel groups. So it takes about 20 hours to parse all channels of one file which is not at all bearable.

1 )From the tdms file contents we understand that, While logging the tdms files, the software is designed that each group is having one channel only. Is there any way that we can combine multiple groups into single group such that each group becomes a channel in the newly combined group.

2) Is there any better tool (C commands / Matlab  / mathscript commands) to read the tdms file faster?

[AndreasK]

Hmm,

I assume DIAdem has a limit of 65535 channels.

So loading them completely to DIAdem will not work out.

Its possible to selectively load only the channels from the file that you are interested in.

Call DataFileLoadSel("C:\Users\Public\Documents\National Instruments\DIAdem 2011\Data\EXAMPLE.tdms","TDMS","*/Time","") 
Call DataFileLoadSel("C:\Users\Public\Documents\National Instruments\DIAdem 2011\Data\EXAMPLE.tdms","TDMS","*/Res*","")

There is also a programming API in DIAdem, LabVIEW, CVI that would be able to work on the files.

But I am wondering how you end up with such files.

How TDMS creates groups/channels

While measuring to tdms you assign a group and a channel name.

If the group name is identical for all your channels there will be only one group including the channels.

So grouping of channels is done by picking a group name.

In your case it seems to be group_1 ... group_80000

The same with channel name. If group and channel name is identical in a measurment system you append to an existing channel.

Assumtions: You are not measuring 80000 sensors. Are you?

Questions:

• How do you create the files? LabVIEW (tdms VIs, storgae VIs) any other tool?
• Do the files contain 80000 channels. What do want to concanate?
• Is it possible to change the measurment system to end up with more performant files

Greetings

Andreas

[ramadugu_ravi@yahoo.com]

Dear Andreas,

Thank you for the reply.

Let me brief about our process. Our seniors developed a test rig application that tests various parameters (such as voltage, Current..etc) of Automotive ECUs and logs the measured data into TDMS files. 

This application is developed using LabVIEW and develired to test center. 

The application measures & logs the below parameters.

UUT Voltage, UUT StandBy Current, UUT High Current, Chamber Humidity & Chamber Temperature.

While logging the data, the application writes it the following way.

Channel group 1:

Channel 1: UUT Voltage Sample1

Channel group 2:

Channel 1:  UUT StandBy Current sample1

Channel group 3:

Channel 1: UUT High Current sample1

...........

............

............

Channel group 10:

Channel 1: UUT Voltage Sample2

Channel group 11:

Channel 1:  UUT StandBy Current sample2

Channel group 12:

Channel 1: UUT High Current sample2

...........

............

............

What I meant to say is there actually not 80000 (in a typical TDMS file logged) sensors measured, but we have 80000 channel groups, because each time sample is measured, it is logged as new channel in a new channel group.

The application is developed & delivered a while ago and we can't change that.

Now we need to analyze the logged TDMS files.

Basic task is to combine all UUT Voltage samples into one array, Stand by Current Samples into second array...and so on.

The next step is to find the min, max, average and other stastical parameters for each array.

We have written a Diadem script, the script searches such that "UUT Voltage sample x" (simalar to pattern search, i think) and adds all the found channels with this string into a new channel in a brand new file. This same is for other parameters of interest.

The script was working fine for small files such as 5MB...Now we have a long duration tests and hence large files such as 30 MB to 100MB. A 30 MB file typically has about 80,000 channels.

The diadem is not able to load this file. (as mentioned in previous post)

When we try reading this file with LabVIEW, it takes days together (as mentioned in previous post)

We have got some ideas, but dont know how to implement.

i) Splitting the large file into smaller files. May be such as Groups 1 - 10000 as file 1...so on. Reading a smaller is faster, i think.

ii) Attached is the VI we are using to segregate the channel groups. When we measure the times, we observed that "Read TDMS.vi" takes about 1sec to read a channel group. If we have any substitution, to speed up this VI.

iii) If it helps to use Advanced tdms VIs (we have not used them previously)...

Please suggest how can we solve this. 

[deppSu]

I looked at the 22MB tdms file you provided.

1.It has only one segment, thus there is no way to defragment this file or split this file based on segments to improve performance. 

2.Over 90%(19MB) of the file is composed of meta data which makes Labview/TDMS take a lot of time to parse and fetch data out.

3.TDMS Adv API won't help since this file is produced by standard api.

Thus, the bad news is I can figure out no workaounds to improve the performance of your case so far. 

[AndreasK]

While I was thinking how you painted yourself in a corner I tried to think of a little DIAdem Script and a bottle of cool well tasting .... So here is a little sofa script for your file. It works because the world is not that bad as you described it and all channels are named the same. Only the groups are renamed.

Took: 44,305 seconds
Nr of channels: 86516

The Problem is it will not scale.Your 20MB file used 730 MB on my Laptop so I assume your 100MB file will fail.

But short description of your solution.

If I take Excel as an example you created an Excel file with 86516 sheets containing only 

So please  change this for your future products.

Greetings

Andreas

[ramadugu_ravi@yahoo.com]

Dear Andreas,

Thank you very much for your help. It looks like "Selective loading of Channels" is working fine to open the TDMS in parts instaed of whole file at  a time. I am working on this and confirm to you shortly.

[ramadugu_ravi@yahoo.com · Accepted solution]

Dear Andreas,

Last weeks (+ weekends) I could spend time on this and ....

Now I could able to solve my problem with your guidance.....

I did like below..

Basically the problem was to Load the TDMS file into Data portal and second problem is to save the big processed files...

So,

1) First I loaded the TDMS files with "Selective Loading of Channels"...Instead of loading whole large data. (datafileloadsel(...))

2) Now I have all the channels (scattered under so many channel groups) are coming under one channel group.

3) Again there is a problem in saving so many channels under one channel group. I observed that upto 5000 Channels in a file works faster and error free. So I am sainvg the Channels under different files. Eg, if my Voltage has 29000 Channels...File1.tdms has Channel 1 - 5000, File2.tdms Channels 5001 - 10000...so on. (Using DatafileSaveSel(....) function)

4) After making so many files, I am combining them using the command: Copy /b "File1.tdms" + "file2.tdms" + .... /b "Output.tdms"..

These steps looks like requiring manual intervention in between....I am trying to write a Diadem script / LabVIEW code to integrate all these steps towards a unattended processing....

Thank you very much for your guidance....Now it saves my time a lot....

Cheers...

[AndreasK]

You should probably have a deeper look into my DIAdem scripts. I assume your solution is still very slow.

Option Explicit  'Forces the explicit declaration of all the variables in a script.

dim tdmsFile : tdmsFile = "C:\temp\test\script\Test_TDMSLog_channel 1_ECU_SI_No L1(EUT1)_06-07-2012_15-16-10.tdms"

dim targetGroup : set targetgroup = Data.Root.ChannelGroups.Add("TargetGroup", 1)

stopwatchReset(1)
' use wildcard to extract all channels that belong to a given group name. Be aware that % is used as asterix like in SQL
ExtractDataChannel targetGroup, tdmsFile, "MeasureDAQ/USB_Chamber1 Humidity_SM SNo 7_%_Cycle 1_ Zone 1"
ExtractDataChannel targetGroup, tdmsFile, "MeasureDAQ/USB_Chamber1 Temperature_SM SNo 7_%_Cycle 1_ Zone 1"

MsgBox "Took: " & stopwatch(1) & " seconds"

Sub ExtractDataChannel(targetGroup, tdmsFile, groupWildcard)

  dim paramString : paramString = "<filename readonly=""yes"">" & replace(tdmsFile, "&", "&amp;") & "</filename>"
  dim store : set store = navigator.ConnectDataStoreByParameter("TDMS", paramString)
  
  dim channels : set channels = store.GetElementList("tdm_channel", "measurement.name=" & groupWildcard, true)
  
  dim channelTimestampObj : set channelTimestampObj = targetGroup.Channels.Add(groupWildcard & "_timestamp",DataTypeFloat64)
  dim channelObj : set channelObj = targetGroup.Channels.Add(groupWildcard,DataTypeFloat64)
  
  dim channelLength : channelLength = 0
  dim channel : for each channel in channels
    
    channelObj.values(channelLength + 1) = channel.References.Item("local_columns").Elements(1).Properties.Item("values").Value
    channelTimestampObj.values(channelLength + 1) = channel.Properties.Item("TimeStamp").Value
    channelLength = channelLength + 1
  Next
  
End SUb

This easy version of my code should probably do a good job. Because the tdm store is reopened on every real channel there should be no memory problem. Its doing nothing different than you do.

dim channels : set channels = store.GetElementList("tdm_channel", "measurement.name=" & groupWildcard, true)

 does the selective loading based on a wildcard of the group name.

MeasureDAQ/USB_Chamber1 Humidity_SM SNo 7_%_Cycle 1_ Zone 1

Afterwards a simple loop over the retrieved channels is used to attach the single value as alst value of the new channel. The timestamp is extracted from the property and added to another channel. I assume with your approach the timestamp is lost.

Please give this script a deep look. I assume it will make you happy.

Greetings

Andreas

P.S.: Please make sure that this approch is changed for future projects.