NIDaqmxbase Read VI till buffer empty challenge

[sth]

Ok, I have a loop that I want to execute 1/second.  I am timing it with a DAQ and reading 1 second of data to regulate the loop.  So far, so good.

But every once in awhile, other software, hardware, user interaction, equipment problems occurs and the loop misses a beat or is delayed.   Then I would like to read the entire buffer.  At the moment, I have a loop that reads 1 second of data (say 32 channels @ 2500 Hz) with a timeout of just over 1 second.  Then I go back and read data in more chuncks with a 0 second timeout.  That call returns all the data in the buffer since it loops until the timeout of zero is reached.

So what is the problem?  Well that second loop still takes finite time so I always get about 2 extra samples.  So my timing is off by about 1 msec every loop.  This is not a big deal but annoying since this software is designed to run for months at a time.

So how do I read exactly 1 second of data unless there is a large number (ie at least 2 seconds of data) in the buffer? 

This is on Mac OS X, so no timed loops

LabVIEW 8.5.1

VISA 4.5.0

NIDAQmxBase 3.2.0 

Here is the read until empty Code. 

[ColeTrain]

Hello sth,

A few things I have noticed with your code.  First, where you have minimum scans, that input is actually called 'number of samples to read per channel'.  The number you put in will cause the read VI to block until it sees that many samples in the buffer.  My guess is, is that you have -1 set there right now which defines the read VI to just return all samples in the buffer.  If that isn't the case, let me know so I can verify this.  

You said you wanted to run this VI for a month at a time, but you are constantly passing an array through a shift register.  If we look at one channel @ 2500 Hz and run it for one month (30 days) we will have almost 6.5 billion samples in the array for that one channel.  Your computer will start running terribly slow if not crash completely when trying to do this.  A better approach would be to build a master/slave loop architecture and write your data to a file.  

The final thing is with your errors.  You really do not need to use the select in your architecture, as you're stopping the loop on an error.  Because the read VI is the only VI that will throw an error, you will not need to pass an empty error cluster through shift registers as this is just more overhead.  Even with the shift registers you have now, you will always pass out an empty error cluster unless you see an error on your first iteration as your select will always give the false case.  Hope this helps!

[sth]

---

ColeTrain wrote:

First, where you have minimum scans, that input is actually called 'number of samples to read per channel'.  The number you put in will cause the read VI to block until it sees that many samples in the buffer.  My guess is, is that you have -1 set there right now which defines the read VI to just return all samples in the buffer.  If that isn't the case, let me know so I can verify this.  

---

Aha!  I had that, it was left over from using regular NIDAQ a few years back.  But under NIDAQmxBase (note the Base), -1 is not a valid input to the number of samples.  If you use -1 it does NOT return all the samples in the buffer.  It may in NIDAQmx but not in the Base version.  In this case I am calling it with an arguement of 2500 scans.

This VI will only execute until the buffer is empty.  Thus the shift register is reinitialized constantly.  This VI is called once/second and theoretically will not have more than 2500 pts in the array.  If the parent VI is hung up for some external OS, networking or hardware reason then it might get 2 seconds of data or 5000 scans.  Worse case it will read an entire buffer which I think I set to 10 seconds before overflowing and stopping the acquisition.  So yes it does run for weeks at a time, but has long term stability when it glitches.  If I just read 2500 scans of data then once it misses a beat, there will be a constant 2500 scans in the buffer for ever.

This is just a sub-vi and not the whole system.  There are many many parallel loops in the top level, reporting, saving to file periodically, PID Loops, calling operators pagers, etc.   This is just the low level replacement for the DAQ read function.  Actually I should build in a trick to restart the DAQ if there is a buffer overflow to make it even more robust.

 Third, I do need the select with errors.  My intent is to return the error from the FIRST call to read.  The final call will have a timeout, but since I force that error I don't want to report it.  If the first call errors out that is the significant error.  The last call is uninteresting since it will always be an error.  The shift register will always lock in the error from the first call to the read.  That is the one I want to return.  Yes that should be no error most of the time since the system will be behaving correctly.  That no error should be returned to the calling VI.

Yes the selection is a bit kludgy but that is how to do correct error reporting.

Thanks for the comments.  You have noticed much of the Non-standard tricks in this VI but most of them are intentional and have a purpose.

Cheers,

-sth 

[ColeTrain]

Hello sth,

I am glad that all of this has a purpose and I can see what you are going for with your code and I admire your ingenuity.  However, if you are passing 2500 as the input to the sample per channel, then you should be receiving 2500 samples back everytime you call this read.  So... I'm not saying I don't believe you... but... I don't believe you. 😉 

What card are you specifically using?  I would like to run your code with the same hardware to see if I can figure out why you are getting the extra samples.  Thanks sth.

[sth]

Cole,

This is to handle the non-ideal case of a non RT OS. 

This VI will never return exactly 2500 samples.  I have many loops in the application this one is supposed to be called 1/sec and look for 2500 sample each time at 2500 scans/sec.  Sometimes an external problem (such as a dropped ethernet connection from Timbuktu, loading a GB data file) will cause the system to have more than 2500 samples available.  I have set the internal buffer size to 10 seconds to make over-running the buffer very rare.  But if 3 seconds pass before the loop executes, then there will be 7500 samples in the buffer.  (Which doesn't have to be an even multiple of 2500).  So now the loop is running stale data, I want to clear that out and empty the buffer, not continue to read 2500 samples.

The VI I showed above  will read the first 2500 samples, then it will read twice more getting another 2500 samples each time to empty the buffer.  Then it will read a 4th time and return about 5 or 10 samples that have arrived while the first 3 reads occurred.  This last one will throw a timeout error and the loop will terminate returning 7512 samples for instance.  Even when running at 1 second intervals, it will end up returning 2502 or something like that not exactly 2500 samples.

The real problem is that I can't peer into the buffer and see how many samples are in there. 

If you want to test this, you need to put in the read loop a wait that is triggered by the push of a FP button.  Every so often add a 3 or 4 second wait.

The card is a 6033 (or 6034?) with 32 channels  in full differential mode.

Mac OS X 10.5.7

VISA 4.5.0

NIDAQmx Base 3.2 (and 3.3 beta) 

[ColeTrain]

Hello sth,

I think this all comes down to separating hardware verse software timing.  If we only want to read in one second's worth of data at a time, then we can wire the rate to the "number of samples to read per channel", in the example we've been working with here this would be 2500.  The hardware will be building these samples up in its buffer and will return only the number of samples we want.  If they are not ready, the read will block until the requested number of samples have been obtained.  

So we'll use your example of a file being read from timbucktu.  If this then takes us 3.13 seconds to get to our loop in software there will be 7825 samples in the hardware buffer.  *Hypothetically* let's say each time our loop executes 7 more samples build up in the buffer. With 7825 samples already present and 7 more coming each time and we're requesting 2500 samples, I'll walk through how many samples we pull into software and how many are currently in the buffer.

Enter loop: #SW: 0       #HW: 7825

1 Iteration: #SW: 2500 #HW: 5332

2 Iteration: #SW: 5000 #HW: 2839

3 Iteration: #SW: 7500 #HW: 346

4 Iteration: #SW: 7500 #HW: 353 (SW didn't go up, requested samples are not there, loop errors out)

By requesting 2500 samples with no timeout, we allow ourselves to pull out exactly one second's worth of data whenever we come to the software call.  It will not matter how many samples are left in the buffer or how quickly we get to the samples as your DAQ card will keep on chugging along.  Please let me know if this does not clarify things or if this is not what you are seeing when you pass 2500 as your input.  If you are not this is a bug and will need to be reported to get it fixed. Thanks and have a great day. 

[sth]

---

ColeTrain wrote:

So we'll use your example of a file being read from timbucktu.  If this then takes us 3.13 seconds to get to our loop in software there will be 7825 samples in the hardware buffer.  *Hypothetically* let's say each time our loop executes 7 more samples build up in the buffer. With 7825 samples already present and 7 more coming each time and we're requesting 2500 samples, I'll walk through how many samples we pull into software and how many are currently in the buffer.

Enter loop: #SW: 0       #HW: 7825

1 Iteration: #SW: 2500 #HW: 5332

2 Iteration: #SW: 5000 #HW: 2839

3 Iteration: #SW: 7500 #HW: 346

4 Iteration: #SW: 7500 #HW: 353 (SW didn't go up, requested samples are not there, loop errors out)

---

In the above example I am not sure what you are using as a timeout. Even the 4th iteration should just wait for the 2500 samples to be available 1 second after the previous read. 
Because of how the system responds to hardware instrumentation issues outside of the LabVIEW system I would like it to respond the following way.  This way the system will never be using stale data and I can discard any outdated information in the buffer.  I would like it to empty the loop or 2500 samples.
 Enter loop: #SW: 0       #HW: 7825
1 Iteration: #SW: 7825 #HW: 0007
2 Iteration: #SW: 2500 #HW: 0007
etc.
 
But with the example I posted I get and a 1.4 second timeout I get.

 Enter loop: #SW: 0       #HW: 7825
1 Iteration: #SW: 7832 #HW: 0000
2 Iteration: #SW: 2507 #HW: 0000

3 Iteration: #SW: 2507 #HW: 0000

 etc.

I don't see a way to do this without that extra 7 samples.   If there are more than 2500 samples going into he loop I want to read them all.  If there are less than 2500 samples I want to wait until there are 2500 and return that.

[ColeTrain]

sth,

The example I gave was based off the example code you posted above where there is a timeout for the first call and a '0' timeout for all subsequent calls.  I was never clear what you used for the number of samples to read, but it sounds like you have '-1' for your input.  Using the method I outlined above, you can just check the number of rows in your resulting array and always just pull the final row that is there.  This will assume that you always read data in 1 second chunks and you can discard any extra data that had accumulated.  Again, by wiring the number of samples we want to receive, we do not have to worry about SW execution speed as every chunk of data that is read will always represent exactly 1 second's worth of data.

If you want to clear all samples and only take the most recent 2500 samples, then you're right that there is no way to do this with DAQmx Base.  You will have to just use array subset functions to pull out only the most recent number of samples that you have aquired.  

[sth]

Cole,

I use 2500 (at 2500 hz) for the number of samples.

I have been informed by NI R&D that -1 is NOT a valid input for number of samples for NIDAQmx Base please confirm if this is true or not.

I cannot find a use of -1 in the documentation for the DAQ read function.  Can you show me where that is documented and what the behavior is?

In looking at the internal code for NIDAQmx Base I don't believe that -1 is treated specially in terms of amount of data read. 

I have no problem using the array subset to work with the resulting data when it is more than 1 second of data.

But when there are less than 2500 samples in the buffer upon entrance to the VI, I want to read EXACTLY  2500 samples.  Not 2507.

If there are more than 2500 samples in the buffer upon entrance to the VI, I want to read all those samples.  (and not those samples + 7 but that is a minor irritant).

I will see if I can get around to making a full test harness VI that demonstrates this problem.

[jschwartz]

Hi sth,

You are correct - using -1 as the Number of Samples with continuout sampling is not officially supported.  It is supported for finite samples.  I agree that the documentation in the DAQmx read help is a little vague.  In order to get exactly 2500 samples, you will have to use hardware timing - right now you are using software sampling.