Interleaved data saving and the TDMS Viewer VI.

[Mac671]

I did a Test on a regular XP PC.

It broke too with a too large data piece.

I guess in order to proceed I'm going to have to install Windows7 on my controller, and I should be fine with streaming well fragmented data @360 MB/s.

What do you think?

Is there any quick way you could test the RAID Array with a controller running a Win7 OS?

Maciej

[Mac671]

Hello I got a nother puzzling result which I do not understand.

I streamed some data on my Laptop which doesn't seem to run into the 64MB problem.

Case#1

When I stream "the old way" that means 1000 channels at a time in writes of 20MB each. ( same amount of data as below but saves are 1000 channels each )

The resulting file is 1,56 GB and the index file is 4,5 MB.

Case#2

When I stream 1x 160MB array at one write with 10000 channels.

The resulting file is 1,56GB and the index file is 1,2 MB.

I would understand that this should cause the file to be better fragmented, and reads should be faster.

Loading all data from one channel with the TDMS Read takes 600ms in Case1

Loading all data from one channel with the TDMS Read takes 12s! in Case2

Why would the read be slower in the second case, I have expected the read performance to incerease due to less header information.

Regards,
Maciej

[muks]

Can you post the car#?

[YongqingYe]

Hi muks, the CAR number is #242594.

Hi Maciej, my RAID array also runs on a XP OS while the regular PC runs on a Windows 7 machine. According to your description, it's also possible that related to different OSes.

For the problem of reading performance, do you mean in case #1, you have only 1,000 channels in total in the file while in case #2, you have 10,000 channels? If so, that maybe reasonable.

[muks]

Thanks.

[WibJ]

No in Case 1 I mean I have 10000 channels in total.

But I conduct 10x 1000 channel write ( chans 0-999 , 1000 - 1999 .... etc ). In 16 MB slices.

As opposed to case 2 where I do one big 160 MB write.

Streaming time is comparable.

Case 1 has an index of 4,5 MB

Case 2 has an index of 1.2 MB

Which would suggest that Case 2 file is better fragmented thus reads will be faster.

But unfortunately the opposite is true.

Case1 read of 1 full channel 80000 samples is 600 ms, whilst on Case 2 file is 12 Seconds 😞

I don't know how to explain that.

Maciej

// Sorry for posting from my other account... I was posting from home did not se I wansn't logged in properly.

[YongqingYe]

Well, I have 2 questions of your application:

1) Did you read while writing? Or just read the data after you writing and closing the file?

2) How you calculate the reading time? including opening file's time?

Do you have a VI to reproduce this problem? maybe we can take a look into it.

[Mac671]

In order to reproduce my problem:

You need a windows7 machine ( as it won't work with the 160 MB saves ).

First take my TDMSPerformaceTest.vi from earlier in this thread.

To stream the first Case file use the default settings

8000 - as the number of batches

1000 - as the number of channles per save

100 - as the number of saves to perform.

The second case file:

8000 - as the number of batches

10000 - as the number of channles per save

10 - as the number of saves to perform.

Secondly use the VI attached to load one channel fully and observe the result.

[YongqingYe]

Maciej, sorry. I don't have time now to look into it and can't give you a quick answer. I'll take my vacation next week and I'll have some investigation on this when I getting back. Is that OK for you? Hope this is not an emergency for you.

[Mac671]

The good thing is I'm out on holiday next week as well.

This is quite urgent for me, but at least we're both out of office at the same time 😉

Maciej