save data from digitizer multiple file

[Heidar]

Hi,

I acquire data from PXI 5105. 

I would like to save millions data (e.g 100 millions data). Since I can not save in one file, I would like to save it in multiple files and each files has a fixed size (I do not know if it is a good idea because I would like to process it later).

Also, I would like to save it in binary files to use less memory as possible.

Can you please suggest me some advice.

Thank you

Heidar

[crossrulz]

Are you trying to save the data from 1 acquisition or many?  I'm left to assume many since I doubt the digitizer can hold that many samples of data.  Can you elaborate a little better?

[Heidar]

Hi,

Thank you for your answer.

I have a PXI 5105 with 512 MB onboard memory.

For the moment, I am using only two channels, I want to sample at 60MS/s, 1 records and the records length = 120M.

[DFGray]

With a modern operating system that allows files of terabyte size and larger, the major reason to split up a file is usually to allow you to more easily get it from one place to another (e.g. e-mail a piece at a time).  I would recommend staying with one file, unless you have a specific reason not to.  As you mentioned, it will make analysis much easier.

Your data rate is higher than a standard computer can handle (60MS/s, two channels, two bytes per sample => 240MBytes/s). This can not be done with a PXI-5105, since the maximum sustained rate the PXI bus can handle is 133MBytes/s, under ideal conditions.  Normal continuous rates are quite a bit under that, since the bus is handling more than just the digitizer data.

If you really need a data rate that high, you will need a PXIe device, such as the PXIe-5162.  Unfortunately, these devices are rather expensive.

If you get such a device, you will need to ensure your disk drive can handle the data rate.  A single good solid state hard drive would probably work, but I would recommend a RAID 0 configuration of two solid state hard drives for consistent operation.  Make sure you have a SATA 2 controller or better (3GBits/s) for the drives.  This is normal for a modern computer (many have SATA 3, which has twice this bandwidth).

Once you have your hardware configuration, you will need an efficient software architecture to match.  The NI-SCOPE example niScope EX Save to File - HWS Low Level - Single Channel Stream.vi is an example of such an architecture.  It uses a producer loop to acquire the data, then passes the data, using a queue, to a consumer loop to save it to disk.  This keeps the acquire and save in different threads and allows them to work at the same time in an efficient manner.  The example uses NI-HWS files, but could be easily changed to either raw binary or TDMS, if you prefer.  All of these are binary formats.  The TDMS would probably be the fastest, although the differences are not large.

Good luck.  Let us know if you need more information.

P.S.  If you have the 512MByte version of the 5105, you can hold all your data in memory from a single acquisition (480MBytes), then write it to disk slowly, so the speed limits above do not apply.  You will need to read and write to disk in chunks to avoid running out of memory in your application.  If this is your situation, let us know and we can give more info on this specific technique.

[Heidar]

Thank you for your reply.

I will use the example you suggersted.

Also, I would like to know how I can save the data in the onboard memory (and not in the computer). I think it is the difference between Read.vi and Fetch.vi but I do not understand their difference.

Thank you

[DFGray]

You read the data into the scope only by simply taking an acquisition, but not reading it.  So you would do your usual Init, set up the trigger, channels, etc, then run the Initiate Acquisition.  When the scope finishes the acquisition, your data is held in the memory on the scope.  You can use the Fetch properties with the Fetch VI to get any subset of that data you want.  By default, you Fetch all of it.