Error -89131 only when VI is run for the first time of the day; Error -200279; Synchronisation of several AI Voltage and Encoder Channels

[MoritzA]

Hey guys, 

Sorry for putting this much into one thread but as one error may be caused by the other I thought I'd try to put everything together into one post. 

Hard- and software used, full error messages and the VI itself (incl. Sub-VIs) are attached below. Thanks in advance! 

I'm currently trying to program a VI which records Voltages and the Encoder information of several sensor with several NIs. It's important for my application to have all the channels synchronised so I've tried to combine some of the templates/examples that I found online to achieve that. 

I've googled through a lot of them but I think these are the main ones that I tried to adapt: 

https://forums.ni.com/t5/Example-Code/Synchronize-Encoder-Counter-Input-and-Analog-Input-with-DAQmx/...

https://www.ni.com/en-gb/support/documentation/supplemental/21/signal-based-synchronization-of-analo...

1.) As I'm using different kind of NIs and different kind of signals I'm worried about my wiring - did I do it in the correct way? So are the channels really synched now or did I miss something / mess something up? 

All in all it seems like the program is running fine but there are two/three errors that pop up every now and then. 

2.) First of all there's Error -89131, which only pops up the very first time I try to run my VI (after a restart of my laptop) and leads to Error -200284. After dismissing the errors and trying to run the program again, it's working fine (until the next restart of my laptop - it doesn't matter if I disconnect and reconnect myself from the Device in the mean time)! As far as I understand the second error is the result of the first one, so by solving the first one the second one should also be gone. 

As it only pops up the very first time running the program I'm not too bothered by it but suggestions for a solution would still be welcome as I'm hoping it doesn't interfere with a potential synchronisation issue or the following error. 

3.) The second and more annoying error is Error -200279 which I'm not really able to reproduce and therefore it's complicated for me to fix. Sometimes I'm able to run my program for about 2 hours without any issue (thinking I've fixed it) and other times I'm not even able to run it for 10-20 minutes before it crashes cause of Error -200279. I've tried to monitor the available samples per Channel but they seem to stay between 0 and 4500 until the program stops and they're suddenly at higher levels (higher than Sample rate = Samples per Channel = 60.015). As the problem only pops up every now and then it's difficult to monitor the AvailSCh rising but they don't seem to rise in the first few minutes. 

Thanks in advance for any help provided! 🙂

Best regards, 

Moritz

PS: I've tried to tidy up the program and put everything into english so I hope it's easier for you to get through the program. If you have any questions please let me know. 

Hardware: 

NI CDAQ-9188, 2x NI 9215, 3x NI 9423, 1x NI 9205

Software: 

LabView Version 20.0.1f1 (64-bit)

Error -89131 occurred at DAQmx Start Task.vi:7220004

Possible reason(s):

An attempt has been made to perform a route when the source and the destination are the same terminal.
In many cases, such as when configuring an external clock or a counter source, you must select a PFI, PXI Trigger, or RTSI line as the source terminal.

Property: SampClk.Src
Property: SampClk.ActiveEdge
Source Device: NI-cDAQ-9188
Source Terminal: ai/SampleClock

Task Name: _unnamedTask<4>

Error -200284 occurred at Final Program_v3 - english version - bug fix.vi

Possible reason(s):

Some or all of the samples requested have not yet been acquired.
To wait for the samples to become available use a longer read timeout or read later in your program. To make the samples available sooner, increase the sample rate. If your task uses a start trigger, make sure that your start trigger is configured correctly. It is also possible that you configured the task for external timing, and no clock was supplied. If this is the case, supply an external clock.

Property: RelativeTo
Corresponding Value: Current Read Position
Property: Offset
Corresponding Value: 0

Task Name: _unnamedTask<1>

Error -200279 occured at Final Program_v3 - english version - bug fix.vi

Possible reason(s):

The application is not able to keep up with the hardware acquisition.
Increasing the buffer size, reading the data more frequently, or specifying a fixed number of samples to read instead of reading all available samples might correct the problem.

Property: RelativeTo
Corresponding Value: Current Read Position
Property: Offset
Corresponding Value: 0

Task Name: _unnamedTask<5>

[Kevin_Price]

No apology needed (or even wanted).  Plenty of detail is good!  We often have to beg for *enough* info.

While there are many opportunities to neaten and condense the code, it also shows a substantial investment in learning about DAQ, sync, and the Producer/Consumer pattern.  (Notable exception: the encoder tasks should tunnel through the 1-frame sequence like the AI tasks instead of going under or around it.  This will make sure they're started before the first AI sample clock arrives.)

I think your errors are mostly due to some pretty subtle stuff.

-89131:  One of your AI tasks is going to be the "master".  It will produce the '.../ai/SampleClock' signal and bring it into existence.   Then all the other tasks can use it as their own sample clock.  Trying to configure the master task to use that signal as a sample clock *before* it has a chance to create it is the problem.   The solution gets into other subtleties of cDAQ.

    Your cDAQ chassis probably has multiple "timing engines" that can be used by AI tasks, 3 is a number I've run across a few times.  They are known as 'ai', 'te0', and 'te1'.  Whichever task is first to reserve resources will get the dedicated 'ai' engine, the ones that come after will be assigned 'te0' and 'te1' if they're still available.  Your 3 tasks execute in parallel up until after the point where one does a "reserve" and the other two are started.  Starting has to first pass through a reservation stage, so there's no telling which of the 3 arrive at the reservation point first and get to claim the 'ai' timing engine.

    The solution: just a little bit of re-ordering.  By reserving the uppermost AI task early, one can then *query* the task for its sample clock terminal and get a valid result  Then no matter whether it's '/ai/SampleClock' or '/te0/SampleClock' or '/te1/SampleClock", it can be wired to all the other tasks' calls to DAQmx Timing.

-200284: I tend to agree that this may well stop happening once you solve the one above.  Let's assume so for now unless proven otherwise.

-200279: This is a classic error for continuous tasks where the app doesn't retrieve data from the task buffer as fast as the device is filling it.  Your code is constructed predominantly correctly though, no sign of the most common reasons new posters run into this error.

   My advice: increase your buffer size.  DAQmx automatically picks a size based on your sample rate.  It appears your rate is ~60 kHz and your requested buffer size is similarly 60k samp/channel, or 1 second worth.  DAQmx is going to bump that up to 100k samp/channel.

   I usually like to oversize my buffers to be 5-10 seconds worth.  Memory is plentiful and cheap these days, no need to skimp.

Attached is your top-level code with some minimal changes I made to implement my suggestions.  I added comments in teal to highlight the changes.

-Kevin P

[MoritzA]

Hi Kevin, 

thanks a lot for your help! 

I had put the encoders in the first part of the flat sequence. It seems when I deleted that part (and rewired them outside of the sequence) I managed to wire them on the outside. 

Error -89131 is definitely gone (and -200284 - as expected - with it)! 

I guess only time will tell if Error -200279 pops up again or not but all in all my questions are answered and solved for now so thanks again to you! 🙂

Best regards, 

Moritz

[MoritzA]

Hi Kevin (and all other community members),

thanks again for your help. Really appreciated!

Quick update regarding the error -200279: I found out that it only pops up (sometimes) when I'm using LabView after using other apps on my laptop (e.g. Chrome - it doesn't make a difference if I close them after using them and before starting LabView though). It seems like my laptop focuses other tasks ("even closed ones") over LabView. The problem doesn't seem to pop up when I restart my laptop and immediately open up LabView (and NI Max) only so I don't mind too much and I guess it's rather a hardware problem with my lapop than a programming issue with LabView. 

Anyways: I have another question regarding the synchronisation of my tasks (and regarding your improvement suggestions) - as a lot of this still relates to the old program I've decided to post it in here... If opening up a new post is the better/correct way please let me know and I'll do so.

1.) I've come to realise that you didn't put a 10 second buffer on the master task (and I've left it like that for now) - did you do that on purpose (and if yes why) or was it just by mistake? So is the purpose of the buffer you introduced to buffer the slave tasks over the master task or did you intend to implement a buffer in general (and therefore it should also be implemented for the master task)? 

2.) Also I've tried to implement another task in my program: a GRAS 40PH-10 CCP Free-field Array Microphone and therefore a sound pressure measurement. We're using an NI 9234 to record the sound and therefore are only able to sample at a max frequency of 51.200 S/sec while the other tasks need to be sampled at (at least) 100 kHz. I've tried to implement the microphone and synchronise the tasks by combining my former program (with your improvement suggestions) with one of the examples I found online (Figure 10 in https://www.ni.com/en-gb/support/documentation/supplemental/21/signal-based-synchronization-of-analo... - I hope it's correct to use the trigger this time. According to my understanding it should be fine to simply synchronise the NI 9234 (master task) with my NI 9215 (which is synched to the other NI Cards in the way of the former program). I had some errors popping up while changing up the program (exceeding the maximum number of tasks) so I tried to save up tasks by using Channel Expansion.
Also I've decided to save the data of the microphone into a seperate file to avoid any problems with the different amount of samples per second. The program seems to be working fine without any errors popping up but I would like to get an/the opinion(s) of more experienced LabView users - are all of the channels in synch or did I mess something up and are you happy with the way I did it or do you have any suggestions for improvement?
You may wonder why I'm using two NI9205 but only one of the inputs on each of the cards... By looking at the datasheet I found that you aren't able to measure two of the inputs on one card simultaneously and we had a second NI9205 laying around so by using two and putting them in the same task they should be in synch (also with the other cards) - correct?

3.) When setting up the sample frequency of the second "chunk" of tasks (NI 9215 and its slaves) I've found that by picking 100.000 as the desired sample rate the actual rate will always be at 100.055. How come it's exactly that amount of samples - what is the formula/reason behind it? I guess it's in relation to the Timebase of the Master but I'd like to understand the things going on in the background.
Also a frequency of 100.055 is higher than the 100.000 S/sec/Ch of the NI9215 (specified in the datasheet) - what happens when I'm sampling at a too high rate (and are the 55 S/sec over the limit relevant at all)? Will the channels get out of synch/not sample simultaneously or will I simply get duplicate values (I've already heard about both) or does it depend on the NI cards that I'm using (only relevant for my NI9215 atm, not really for NI9423, NI9205 and NI9234 as I'll stick to rates lower than 250 kS). It'd be good (for my encoder tasks) to sample with a frequency of 144 kHz but for now I've sticked to the 100 kHz to avoid any problems.

Thanks in advance for all help provided!

Best regards,
Moritz

EDIT:
While setting up the VI for the forum (tidying up branches a little bit) I stumbled upon Error -200279 again (even after restarting the laptop). I'm currently accessing my NI-Hardware through Wifi and a VPN - could that be the source of the problem (e.g. caused by a too instable internet connection)? In the screenshot attached you can see the error incl. the samples available at that exact moment - according to my understanding the buffer shouldn't be overfilled so do any of you have an idea on that issue? 

 

Thanks again for everything! 

General overview on the hardware in use:
NI cDAQ-9188: 2x NI9215 (Currents + Voltages), 3x NI9423 (Encoders), 2x NI9205 (Voltage), 1x NI9234 (Sound pressure)

[rolfk]

Wifi is always a potential problem. Your DAQ system requires a very good and sustained throughput to operate properly. While it may theoretically have that throughput on average it may have latencies caused by interferences with other Wifi users and Wifi networks (or even simple bugs in the Wifi hardware, they still happen) and are for the typical residence usage of Wifi often not a big problem but DAQmx requires a stable and reliable connection.

So I would definitely try a wired solution as comparison!

[MoritzA]

Hi Rolf, 

thanks for your reply. Normally we're using a crossover ethernet cable to connect the laptop directly to the devices but in times of home office it's getting more common to test the programs via VPN but I'll give it a new try asap. 

Thanks again for the input on that issue and looking forward to getting suggestions on the other ones! 

Best regards, 

Moritz

[rolfk]

VPN is almost certainly even worse than Wifi. Unless you have a reliable Gb connection in your home office to a big fat internet hub (and of course an at least as good connection of your work office, although if it is a reasonable sized company, that is usually a given nowadays), it is likely the real culprit here.

In fact I’m impressed that DAQmx could handle these types of sample rates for multiple channels over VPN.

[Kevin_Price]

[Edit: I skipped right over the WiFi and VPN part, but fortunately rolfk addressed it.  And I agree that a -200279 error is a likely outcome from those operating conditions, and probably sooner rather than later.  But the stuff I say below is *also* a distinct and independent possible cause for the error.  Both aspects need to be addressed.]

Thanks for your efforts in the detailed write-up, but please realize, there's quite an awful lot to look at, digest, and cross reference to your description.  I've only got time for a few comments.  First your specific questions:

1. Pretty sure it was just an oversight that I didn't make a bigger buffer for the master task before.  No reason all the buffers couldn't be pretty big (like ~10 sec worth).

2. The 9234 uses a Delta-Sigma converter with its own internal timebase which (AFAIK) won't automatically stay in sync with the timebases in the chassis (and thus with all the other tasks).  My rule of thumb for the many devices spec'ed at 50 ppm timing accuracy is a possible timing error up to 3 msec per minute of acquisition.   That's an apparent shift of 300 samples at 100 kHz.   I'm not sure the method you tried so far using the 9234's SampClk.Timebase.Src property is going to accomplish the sync you're aiming for.

   The Delta-Sigma converter in the 9234 *also* brings with it an input delay due to its built in filters.  That will require some compensation too.

    I would stick with just 1 TDMS file.  TDMS is really good at handling multi-rate data for different channels.

    I'm not sufficiently familiar with the 9205 module to comment about using two of them instead of one.

3.  While 100.055 kHz is technically out of spec, my "gut feel" is that it's quite unlikely to be a true problem.

There are two other fairly fundamental things to comment on:

4. In a multi-rate acquisition, you kinda need to think about a master task for timing your reading loop as well.  You can only read integer #'s of samples at a time, but the rate at which samples come into the respective buffers will not generally be a perfect integer ratio.  So when you specify fixed integer #'s of samples to read for all the tasks, this will *always* put you at risk for the -200279 buffer overflow error.

    Read a fixed # samples from 1 task first, then tell all the other tasks to read "all available samples" using the special (and default) value (-1).

5. After that, I would very strongly recommend you create a typedef'ed cluster for all your acquisition data.  Each task gets its own field, and I further recommend that the AI data should be held in waveforms rather than mere arrays.  The waveforms retain timing info and other properties that TDMS will embed in the file for you.

    Waveforms aren't inherently available for your encoders.  There are possible workarounds, but too much to get into in my limited time now.  You can get by leaving them as arrays if you can be sure all your hardware sync keeps their data in lockstep to one of the AI tasks.

    Once you do this, you only need 1 queue for your producer/consumer work.  The queue type will be your cluster typedef, then you'll use "bundle by name" in your producer loop to package all your data into a single cluster wire.  Enqueue that cluster, and then your consumer loop can use "unbundle by name" to deal with data fields individually and write them to TDMS. 

Past my intended time already and have to go now.

-Kevin P

[MoritzA]

Hi Kevin, 

thanks again for your help! Very appreciated! 

I do understand that it's quite a lot to read through and tbh: I really admire guys like you not getting lost in the coding of others - especially as the programs get bigger and bigger...

Regarding the issues: 

1.) Ok thanks! 

2.) My main concern for the program will be the tasks mentioned in my first post - therefore the synchronisation of the 2x NI9215, 2x NI9205 and the 2-3xNI9423s. 

My ideal goal was to synch the Microphone to the other channels as well but if it's not perfectly synched then that (has to be) fine for me (we'll take the average of that channel for the analysis) - again: all the other ones are more important. Do I also mess up the synchronisation among the important ones by adding the microphone in the way above or did you mean that I'm just messing up the synchro between the micro and the batch of other tasks? (I hope you know what I mean with that sentence) 

The alternative would be to not link the microphone to the other tasks at all - as long as the important ones are in synch that's also fine for me. (I think you get my point)

Also the two NI9205 channels are more important than one of the encoders - I'll probably get rid of one of the encoders and therefore have a task for the second NI9205 left. This way I shouldn't have any problems with these. 

4.) that does make sense - thanks for the suggestion. 

5.) see 2.)

Thanks again for the time you've taken and in general! 

Best regards, 

Moritz

[Kevin_Price]

2. This is an important decision point.  The shared sample clock method used to sync everything *except* the microphone and 9234 cannot extend to the microphone task.  Full sync would require a significant overhaul of the whole timing scheme.

    Here are 3 sources of "anti-sync":

- inherent input delay of the 9234 due to filtering, a non-cumulative offset.

- a non-cumulative offset in start time if not triggered together

- a (possible) cumulative timing skew as mentioned earlier, accounting for as much as ~3 msec per minute of capture

    You're going to need to decide how important these things are for your use case.   Especially the third one.  That's the one that could make you rework your whole timing scheme because its effects are cumulative.  The first two can be handled without major rework.

5. I'd give the same advice with or without the microphone.

-Kevin P