ni.com is currently experiencing unexpected issues.

Some services may be unavailable at this time.

LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

TDSM, for handling large data portions - make separate files or one big file?

For a measurement with online feedback I'm working with a host PC and a LVRT target with which generates a large amount of data that I want to store for preprocessing. I have been using LVM files to store the data in one big file for post-processing, but this gave problems in Matlab, as the read matrix was 3x too big for the allocated memory. I then switched to make a separate file for every 'task', which was good for the post-processing, as the separate files are about 600x smaller as when one file was created, but now problem during the measurement show, as the frequent creation and closing of files causes the system to hang for several seconds. 

 

I was wondering if could solve the problem by using TDMS files and how. I could do the same thing of storing separate files, though that would require that generating and closing TDMS files goes very fast (much fast as LVM) or I could go back to storing one file, though then I would need to choose the portions to be read into Matlab or be able to split the file in LV, directly after the measurement. 

 

Which approach should I take? 

0 Kudos
Message 1 of 9
(4,097 Views)

Storing to TDMS files will give you better performance (writing and reading) and a much smaller disk footprint (if you're fine with the accuracy of floating point numbers in LVM, you could use SGL type data in TDMS).

 

The MatLab API for TDMS allows you to specify a starting point and a number of values in order to read a subset of data out of a huge file (TDMS files can be multi-terabyte). The functions you'll need are these:

int __stdcall DDC_GetNumDataValues (DDCChannelHandle channel, unsigned __int64 *numValues);

 

int __stdcall DDC_GetDataValues (DDCChannelHandle channel, unsigned int indexOfFirstValueToGet, unsigned int numValuesToGet, void *values);

 

While the first function will return the total number of values in a given channel, the second function will give you an array of numValuesToGet values, starting at indexOfFirstValueToGet. That way, you can iterate a whole channel from beginning to end. You can also perform random access of any piece of data from anywhere in the channel.

 

Hope that helps,

Herbert

0 Kudos
Message 2 of 9
(4,089 Views)

Hi Herbert,

 

Thank you for your help!

Unfortunately I wasn't able to implement the commands you gave me successfully. Can it be that part of the command is missing? I use the application 'read_TDMdata.m.

 

I put your first line of code into the part where the channel values are observed (maybe you recognize it):

 

        %Get Channel Value if Data Type is 'Double'(10)
        %if ptype.Value==10
            pnumvals=libpointer('uint64Ptr',0);
            calllib('nilibddc','DDC_GetNumDataValues',pchans.Value(j),pnumvals);
            pvals=libpointer('doublePtr',zeros(1,pnumvals.Value));
            calllib('nilibddc','DDC_GetDataValues',pchans.Value(j),0,pnumvals.Value,pvals);
            chanvals(:,j)=(pvals.Value)'; %#ok<AGROW>
            int __stdcall DDC_GetNumDataValues (DDCChannelHandle channel, unsigned __int64 *numValues);
        %end

 

This errors. I think I should also assign the number of values, but I didn't know how to do this with this line of code.

 

Would you know what to do?

 

Thanks

Remco

0 Kudos
Message 3 of 9
(4,067 Views)

What kind of error are you getting and from what line? Note that your second to last line is a copy of the function declaration. That might be a possible source or errors.

Herbert

0 Kudos
Message 4 of 9
(4,064 Views)

This is the error:

 

??? Error using ==> sym.sym>char2sym at 414
Not a valid symbolic expression.

Error in ==> sym.sym at 95
   S = char2sym(x);

Error in ==> sym.int at 49
   b = sym(b);

Error in ==> char.int at 9
y = int(sym(f),varargin{:});

Error in ==> Read_TDMData at 188
            int __stdcall DDC_GetNumDataValues (DDCChannelHandle channel, unsigned __int64 *numValues);

 

 

Without the line the code works, but maybe I put the line a the wrong position?

 

I attached the whole script. Maybe it makes things clearer...

 

 

Thanks

Remco

0 Kudos
Message 5 of 9
(4,061 Views)

You're already calling both of the functions I mentioned through calllib, which is the right way of calling them. If you need another call to DDC_GetNumDataValue, you need to wrap it with calllib again. What you did is copy a function declaration into code that is supposed to be executed, which cannot work.

 

Herbert

0 Kudos
Message 6 of 9
(4,056 Views)

Thanks.

I have been able to retreive the number of values by calling 'pnumvals.Value' 

Though I haven't succeeded in retreiving reduced portions of data. I tried it by altering this library call:

 

calllib('nilibddc','DDC_GetDataValues',pchans.Value(j),0,pnumvals.Value,pvals);

 

by changing the 0 and pnumvals.Value to different values, e.g.

 

calllib('nilibddc','DDC_GetDataValues',pchans.Value(j),1,4,pvals);

 

though this resulted in the same number of values read. The values themselves are different though

 

 

Would you know what I should change?

 

 

 

0 Kudos
Message 7 of 9
(4,034 Views)
Would you also know how I tell Matlab that I'm importing singles? So far, only when I write doubles in Labview, I get sensible data
0 Kudos
Message 8 of 9
(4,027 Views)

I partly solved my question in message 7 in that changing pnumvals.Value to an other number changes the last sample read, but I haven't found how I can choose where I start reading.

 

 

0 Kudos
Message 9 of 9
(4,020 Views)