real time data logging is slow

[abdel2]

I am using a real time Desktop PC which is acquiring data from 4 different channels simultaneously. The real time system has a deterministic loop that uses a timed loop (connected to the DAQ sample clock). It acquires 1 sample per channel per iteration and puts the scalar value from each channel into a separate network-shared variable. So the ideal sampling rate is 1Ms/4=250kS/s. In other words the timed loop is running at a frequency of 250kHz.

On the host computer, I'm reading the data from the shared variables and writing it to a text file. The problem is that the data is not completely correct. It seems like it is either mixing different scalars or skipping samples. Because the host is a Windows OS, the timed loop on it is running at 1 kHz.

My test procedure:

-Generate a sine wave with a sampling rate of 10 kHz on ao1 which is connected to ai3 using a BNC cable.

 -Read the data from ai3 and log it on the host computer.

If I'm generating the sin wave with a sampling rate of 1 kHz or lower, the logging is ok. However, I would like to log data as fast as possible (~200 kHz). I understand that I am skipping samples because the host computer is not reading data from the shared variables fast enough but I have tried creating a network or FIFO buffer of 100-1000. Still doesn't make a difference. The problem is definitely with the host computer because if I display a graph of the acquired data inside the deterministic loop on the host computer, I can see a non-aliased sin wave even when the generation had a sampling rate of 250 kHz.

What are some solutions to this problems?

Thanks. 

[RavensFan]

I doubt that your network would be able to handle that kind of data rate.  You are trying to send 250 kS/second x 4 channels.  I am going to assume they are doubles at 8 bytes/sample.  So that is about 8 megabytes/second.  x 8 bits/byte is 64 Megabits /second.  That is quite a bit of bandwidth.  And for every update, you are not just sending 8 bytes, but all of the header and addressing information in each packet.  If a packet was 512 bytes,  now you are up to 4 gigabits/second.  Impossible!  To have a chance, you will have to combine your channels and multiple samples into a single piece of information that you can transfer.  Perhaps setting up a TCP/IP transfer between the two will help eliminate some of the overhead of using shared variables.

I'm not even sure how you are able to get smooth acquisition if you are only acquiring 1 sample per channel per interation on a 250 kHz clock.  I think you would have to be reading larger chunks of data even on a realtime P

Also, if your realtime PC is putting data into the shared variable at 10 kHz  (I'm not sure why you are talking about 10 kHz in one paragraph and 25 kHz in another  ???) and your host PC is running at 1 kHz reading the data out, you are filling up the buffer 10 times faster than you are reading it  Of course you will lose data, or wind up crashing the shared variable engine or PC when it fills up with data.  That is why if you are acquiring at 1 kHz the logging is okay.  They are writing and reading at the same rate, so the host PC has a chance to keep up with the realtime PC.

Message Edited by Ravens Fan on 09-11-2008 12:11 AM

[abdel2]

Thanks. I guess I have to find another method of logging the data. TCP/IP is a little too much but maybe I'll log the data in a binary file on the real-time system.

Before all that, I have run into another problem. It looks like the code before was set to sample at only 1kHz (from DAQmx Timing). Since I was doing hardware single-point sampling, I thought that property wasn't really used. Anyhow, I have two questions:

1) If I'm doing single-point hardware sampling, why do I have to read the data very fast? Since there is no buffer, there shouldn't be any overflow. I was thinking I could ideally run a timed loop at 500kHz/s using a hardware clock and read one sample with each interation. For some reason, it locks up the whole system (100% CPU usage). Does the card still use a buffer of 1 and does it read data from the sensor at Start Task or Analog Read?

2) Why can't I run a Timed Loop faster than the sampling rate if I'm using a hardware clock? If I'm running the loop faster, there is a higher chance that I'll read the data faster than it's being written (using continuous data acquisition) and there won't be a buffer overflow.

[Pie56694]

Hi Abdel2,

It sounds like you're trying to do a hardware-timed acquisition in software.  It may be better to do hardware-timed acquisition with a continuous read, then use the array or waveform data type with a real-time FIFO to move data from your high-priority loop to a low priority loop on the real-time system.  The low priority loop can then process, transfer, or store the data without affecting the acquisition.  Test 3 here shows that TCP/IP data transfer can outperform a Network Published Shared Variable, but the performance increase is not a function of payload size.

The system is probably locking up because of "thread starvation".  Most DAQmx VIs are blocking nodes because they use the Call Library Function Node.  Keep in mind that there is more than just the Timed Loop executing.  Depending on the application, there may be timer threads, Ethernet communication threads, background worker threads, etc.  If the application does not allow time for background system threads (more likely if using blocking nodes), you can lock up the host with code in a time-critical loop.

LabVIEW loops will not iterate until each node in the loop structure has data on all of its output terminals.  If you attempt to iterate a Timed Loop faster than the acquisition is returned, the loop will finish late.  The default behavior in this case (see here) is to discard missed periods and maintain the original phase.  So, running the loop faster than the acquisition will cause you to lose data unless you configure the Timed Loop properly.

All of the examples in the LabVIEW Example Finder with the clock icon are approved for real-time targets.  You can look at examples for continuous analog acquisition, and use them with the LabVIEW Real-Time Continuous Communication Architecture template.

Benchmarking a Typical Control Application using LabVIEW Real-Time & NI-DAQmx 

I hope this helps.

Message Edited by Pie56694 on 09-12-2008 03:35 PM

[abdel2]

Pie56694 , thanks for the insight.

I am now using a continuous data acquisition with maximum buffer size (1000000) and I'm reading 100 samples/channel/iteration. It seems to be working great.

The problem:

I'm logging data in the acquisition (deterministic) loop to a tdms file. This obviously adds more time to the loop iteration. I would not mind logging data in a low-priority loop but I don't know how to transfer the 2D arrays for DAQ Read VI to TDMS Write VI without using a direct wire.

My first application is going to log data from 6 channels, depending on a GO bit (which is also monitored using data acquisition). Every minute or so, the GO bit goes false for 2-3 seconds and I have time to complete my data logging. If I understand correctly, there will always be some data left in the buffer because the data logging loop is running slower than hardware acquisition. Is there any way to check how much data there is left so I empty out the buffer within those 2-3 seconds?

[Pie56694]

Hi Abdel2,

Much of the work of passing data from a deterministic loop to a non-critical loop and logging that data to a file has been done for you.  From LabVIEW » New... » Project » Project from Wizard » Real-Time Project » Continuous Communication Architecture » Application Includes deterministic components » Two Loops » Include File I/O.  This creates two Timed Loop structures and a single-process shared variable that moves data from the deterministic loop to the non-critical loop. If you are reading data from multiple channels, you can create a single-process or a network published shared variable of type Array of Double Waveform.  This works very nicely with a DAQmx Read: 1D Waveform NChan NSamp.

If you create two Timed Loops, such as in the Continuous Communication Architecture template, there is no hard reason to run them at different rates.  You can run them at the same rate, and the LabVIEW Execution System will interrupt the low priority thread assigned to the non-critical loop to do another acquisition if necessary.

If you are using the RT FIFO VIs, there is an empty? terminal on the RT FIFO Read that can be used to check if the FIFO is empty.  If you are using a shared variable, you can Enable Timestamping in the variable configuration screen.  A shift register may then be used in the non-critical loop to check the timestamp of the shared variable and only write it to the file if it is new data.

It sounds to me like your best bet is to use a Producer/Consumer template for the acquisition.  This ensures that the consumer loop will only run (perform file I/O) when a new data set has been pushed into the queue.  From LabVIEW » New... » VI » From Template » Frameworks » Design Patterns » Producer/Consumer Design Pattern (Data).  I hope this helps.

[abdel2]

Thanks Pie56694.

I have played around with producer/consumer and other type of host-target communication VIs but I still need to do some modifications for my own application.

You mentioned I can use an Array of Waveforms is simply a waveform to transfer data but it is not working properly.

I have a loop that runs at 50K. Each iteration acquires 1-1000 samples (reads all the samples available) into a waveform for 8 channels. I am simply putting the waveform of one channel into a shared variable (Double Waveform) and reading it in a loop that runs at 1K on the host computer. For some reason it only reads the first sample of each waveform and obviously also skips a few waveforms because there is no buffer. The channel I am reading has an input sine wave at a certain frequency. It is aliased on the host when the frequency is above approximately 5 Hz.

I even tried reading the shared variable (converted to Single Process now for better performance) on the target right after I right to it in the same 50K Time Loop. In that case there is aliasing if the frequency of the input is approximately 500 Hz. There seems to be an issue with the shared variable system. I have trying Multiple Elements buffer with different parameters but it doesn't seem to work. 

Message Edited by abdel2 on 09-26-2008 11:07 AM

[Pie56694]

Hi Abdel2,

What version of LabVIEW are you using?

[abdel2]

Labview 8.6. I just bought the package about 2 months ago.

[abdel2]

PS: with shared variables you have to read data either at the same rate or faster than writing the data. Otherwise there is a buffer overflow eventually. Is there any way to prevent this while using the same fast and slow loops for writing and reading data respectively? For example can I empty the whole shared variable buffer with just one call rather than reading one value at a time in a loop? I would think it would be faster unless the "emptying out" command used a loop internally.

Thanks.