Continuous acquisition and loss of data

[salomiva]

Hi!

Running a continuous acquisition both with a digitizer (PXI 5124, 32MB per channel) and a digital I/O (PXI6552, 8Mbits per channel) I get a similar manifestation that makes me realize that I don’t understand something. To make myself clear, I have a project with analog acquisition, and another project with digital acquisition. The connection with a laptop is provided via PXI-ExpressCard8360.

Namely, simple examples work fine. When I add some “real time” data displays and/or data processing (for example, storing data in a file) or try to acquire signals from more than one channel, I actually lose data without any warning or error. In just one case I got an error that the on board memory was full and it has been overwritten, and that is clear. What is happening with data in all other cases?

Would Producer/Consumer Design Pattern help me solve the problem? Anyway, before the I get to the solution I would certainly like to understand the problem.

[AECmartin]

Hi salomiva,

Can you please send the code where you see the issues with acquisition rates and more comments? If you acquire data, you have to consider acquisition speed and system capabilities (bandwith, signal processing speed, hard drive performance if you save data). Also displaying the data consumes some resources. Calculation with considerations about you specific HW should be the first you do before creating program, because HW limits are set and you cannot exceed.

This behaviour you describe is not usual, so the code can explain a lot. Using producer/consumer pattern is good to use (it enables you to read the data from buffer fast enough and also works as another buffer so even slower processing can be done but you will be limited due to RAM). You are using NI PXI-ExpressCard8360, there is sustained throughput up to 110MB/s. Consider this, if you will use only digitizer with 1 channel acquisition, you will be able to set maximum speed of 55MS/s (one sample is saved in 2 Bytes). If you will use more channels or even another task (digital I/O), rate will have to be slower. This is for continuous acquisition, you can increase the rate but acquisition time will be limited (depending mainly of buffer size, ...).

So please, send the code, then I can check it and discuss.

Martin

[salomiva]

Hi, Martin!

Here is the concrete exaple with all the details.

My hardware comprises NI PXI-1036 chassis, NI PXI-ExpressCard8360 and NI PXI 5124 (32MB per channel). The input to the system is 1 kHz sine signal. I was thinking if I set the sampling frequency to 10 MHz (that should be ok for the system configuration) and the window size to 10000 I should get one period of the sine signal and the waveform on the graph should be steady.

When I run the Ni example niScope EX Stream to Disk.vi my thinking seems to be right. The recorded signal is ok. But when I run one of my examples, SimpleDataAcqAndWrite_v5.vi,  I see the sine signal moving on the graph all the time and when I turn on the file recording, it slows down even more, and I get strange signal recorded so that I cannot even say that I am loosing frames but random parts of a signal. And there was no error message. I tried to turn off the data graphs but the situation in a recorded file was the similar. I tried to store data to a bin file – the same result.

This is the example that bothers me a lot currently. When I start to change the above mentioned NI example I get errors or loose data. And finally, I have to make a project with two NI PXI 5124 modules and a 4 channel real time continuous acquisition of a signal!

Thanx a lot!

[AECmartin]

Hi salomiva,

I looked at the code and found few parts which are not correct. First of all, your issue can be caused by trigger settings. You should select Software Ref Trigger, both inputs do not need to be wired explicitly in this case (default values are enough). Another thing what is not clear to me is why are you reading data using Multi Fetch WDT and then Multi Fetch Array Measurement Cluster? In this case you will acquire some samples using one VI and another samples using second VI. I think it is better to acquire the data only once and process it in LabVIEW, you can use FFT Spectrum (Mag-Phase) VI or other in the Waveform Measurement Palette since you are using waveform data type (you will get proper x and y-axis values on FFT signal automatically).

Another thing you should avoid is to open or close resources inside the while loop. Usual way is to open all resources (NI-Scope, file for writing) before the loop, then in the loop make all necessary operations (read the data from scope and write it to the file) and after this loop finishes, close all resources used. Even better method is to acquire data in one loop (producer) and in another loop save this data into the file (consumer). This enables you to use multicore processors better and also acquire rate can be different than rate of saving data into the file (you can acquire e.g. 10k samples but save 100k samples long chunk to the file in one step -- saving will run at 10-times lower rate but with bigger chunks of data). Using File > New, list VI > From Template > Frameworks > Design Patterns and select Producer/Consumer Design Pattern (Data). Using this template, you will be able to create the whole code more understandable and flexible.

Also property nodes on the left side of the code should be connected through error cluster to determine dataflow (now are these parts in parallel with the while loop). Also error should cause to stop the while loop as shown in the example niScope EX Stream to Disk saved.vi you attached. The last suggestion would be to think about easier logic in data saving part. You can inspire yourself in State machine design pattern (available at the same place as the Producer/Consumer. State machine is usual way how to create code with many advantages. Please note, that using uninitialized shift registers can cause weird behaviour of the program, in the article http://www.ni.com/gettingstarted/labviewbasics/shiftregisters.htm , Figure 3 explains this issue. It is good practise to initialize all shift registers in this situation.

If you need to measure the signal on 4 channels totally, still you have to consider bus speeds and buffer sizes first. Try to set the lowest sampling rate possible which will meet your needs.

Best regards,
Martin

[salomiva]

Hi, Martin!

Your remarks were of great help to me! Thanx a lot!

Based on your advices I have made two VIs. I definitively have a problem with the capabilities of my system, which still has, as I understand, a remarkable configuration (Intel(R) Core(TM) i7-3612QM CPU @ 2.10 GHz, 6 GB of RAM, 64-bit Windows 7, 32-bit LabVIEW2012 SP1).

Both examples work ok with 1 channel on 4 MHz. With 2 channels on 4 MHz and a buffer size of 10000 after a couple of seconds I get the message that the requested data has been overwritten in memory so it is no longer available for fetching (even in the case when I do not write data to a file). When I turn off the waveform graph the examples work continuously on 4 MHz and 10000 buffer size. With buffer size of 100000 it works ok.

With the sampling rate of 2 MHz and buffer size of 10000 both examples works continuously.

The input to the both channels is the same sine signal. In all cases with 2 channel acquisition I have a strange manifestation: the signals from the 2 channels on the waveform graph are (phase) shifted for a random value with every run of the VI. In the case of ProducerConsumerDataAcqAndWriteToFile_v1.vi the recorded TDMS file exhibits the same manifestation, while in the case of SimpleDataAcqAndWrite_v11.vi in the recorded TDMS file signals are in phase.

What could be the explanation of this shifting of the signals? And what is the cause of the difference in the TDMS files recorded in the two Vis?

Best regards,

Iva

[AECmartin]

Hi salomiva,

I am glad that this information helped. I think that issue is caused mainly by buffer size buffer. 10k is for this speed of acquisition very small size. If you calculate the time when it the buffer will be full, it is 10k/2M=5ms. So the program has to read the buffer at least once in this time interval during whole acquisition. Because you use non-deterministic operating system, the program can be interrupted by another one with higher priority and this delay will cause that buffer will be overwritten. The bigger the buffer size and faster the procesor are the lower possibility of this issue.

Another issue is plotting the data into waveform graph. In 2MSa/s acquisition and 10k buffer size, the graph must be redrawed every 5ms. The whole redraw process takes some time and resources and can significantly decrease speed. The advice is the same -- increase buffer size to e.g. 100k, it will mean redrawing every 50ms and increase speed. Another approach is to acquire even more samples, decimate them and plot this decimated data (graph width is usually not bigger than 1000 pixels, so plotting e.g. 100k samples in one moment does not make a sense, you will see only few of them).

Now about the random phase shift issue. If you plot the data, it is shifted in both programs and in saved data, in SimpleDataAcqAndWrite_v11.vi everything is OK and the other is shifted as well? Is this phase shift the same in file and in graph? I would suggest to try set Chan Characteristic VI for both channels (use one instead two). Also would be good to investigate the behaviour when you remove niScope property node and/or remove the trigger settings.

You mentioned that you use 64-bit Windows and 32-bit LabVIEW. In case of problems with memory (mainly with Producer/Consumer pattern because it works as buffer), 64-bit LabVIEW can help to use more memory as mentioned in http://digital.ni.com/public.nsf/allkb/AC9AD7E5FD3769C086256B41007685FA .

Best regards,

Martin

[salomiva]

Hi, Martin!

In ProducerConsumerDataAcqAndWriteToFile_v1.vi the phase shift is the same in the file and in the graph. I tried to set one Chan Characteristic VI for both channels as you suggested and nothing changed. When I removed the trigger settings the channels are in phase but it actually acquires only one buffer of samples (when I change the frequency of the sine signal online nothing changes until I stop the program and run it again).

Best regards,

Iva