Idea to split my 1D-multiple-ramps alternatively?

[Mystogan]

Hi everyone,

I have here a VI  for splitting a .tdms file which may contain multiple ramps. The concept is pretty simple; for example I have 10 appended ramps at the input, each ramp is a simple 1-100 ramp function which looks like a sawtooth function.

The task of this VI is simply to split the ramps (take each element and put them into smaller files, in  .csv or .txt format).

Also, if I set the parameter "File numbers" to 100,  "Word-length" to 1 and running this program, I would get 100 smaller files:

File 1.csv would have 10 elements of "1"

File 2.csv would have 10 elements of "2"

...

File 100.csv would have 10 elements of "100"

Alternatively, if I set the parameter "File numbers" to 10, "Word-length" to 10 and running the program, the results are:

File 1.csv would have 10x smaller ramps from 1-10

File 2.csv would have 10x smaller ramps from 11-20

...

File 10.csv would have 10x smaller ramps from 91-100

This VI runs okay if the source file has total elements less than 10 million elements. But it takes too much time to finish if I have for example 50 million elements.

I guess, there is an alternative way to split the ramps faster, but I cannot find a better way at the moment.

I would be glad if someone has an idea to split the ramps faster 🙂

Any helps are really appreciated,

Yan.

[tst]

Why do you need the source file? Why not just generate the data yourself on the fly? Once you do that, you won't have to deal with large amounts of data (with 50M DBLs you would need 400MB for each copy of the array).

Can you upload the code and a sample file?

[Mystogan]

hi tst,

The main purpose of the vi is to sort the arrays from the .tdms file.

You can test yourself to generate a simple ramp in .tdms file with the included vi.

I need to develop this vi so I can for example extract the ramps; take out 1, 101, 201, and build smaller ramp of 11 elements (2-12, 102-112, ...)

have a look at the vi's

regards,

yan

[tst]

Assuming that you really have to read the data from the TDMS file and that you will have such large pieces of data, I would suggest you try reading the data from the file in chunks. I have absolutely no experience with TDMS, so I can't really comment on this, but I see that the TDMS Read function does have inputs for setting the offset and count and that the advanced palette also has some functions which seems to support reading only parts of a file.

If that works, then you just need to calculate where the data you need will be in the file and I believe the code you have already does that. Basically, you will just need to move the reading part into the for loop and calculate the offset and count for each reading. Assuming that the TDMS functions know how to read only part of a file, this will prevent you from having a large array in memory and will allow you to do what you want.

[crossrulz]

Tst is right about being able to read the TDMS files in chunks.  That is what the offset and count is there for.  I'm done similar things with binary files (try reading and processing a 2+GB file).

[Mystogan]

yes you are right,

I was too focused with complicated ideas, while actually the basic function (read in chunks) is available in the read vi.

thanks, I'll try what I can do with tdms read vi.

Yan.

[Mystogan]

hi,

I've tried to modify my code, but i'm facing another problem.

I'd like to split my ramp which has 1309 elements (1+(109x12)) into 109 new files, and do it like this:

take out element 1,

put element 2 ~ element 13 into first file,

put element 14 ~ element 25 into second... until 109 files done

take out element 1309,

put next 12 elements into first file... until 109 files done

my problem with the recent code, i keep getting in first file: 

2,3,4,5,6,7,8,9,10,11,12,13,2,3,4,5,6,7,8,9,10,11,12,1309,2,3,4,5,6,7,8,9,10,11,1308,1309,...

instead of (what i would have):

2,3,4,5,6,7,8,9,10,11,12,13,2,3,4,5,6,7,8,9,10,11,12,13,... always the same order

I know i'm not programming the "take out" element not in the right way, any help would be really appreciated.

regards,

Yan.

[Mystogan]

simple solution to my own question

I'm changing the input of my (-1) function and seems to work fine