Producer/customer failure

[GiovanniV]

Hi,

I have been working on a large project for several months and I am at the final stage of the project. However, when I want to use my program it gives me several problems. Let me first elaborate what the main progam does:

• The machine it operates has 5 sensors, and 4 motors
• The main program consists of 8 while loops:

1. [DAQ loop] acquires 5 sensor voltages -> filters these (sample compression - mean) -> puts them in a queue that is accessed by the second loop
2. [Visualization loop] dequeues measurement data from the first loop -> does some calculations and shows them in different sets of graphs (located in different tabs of the front panel) -> puts the original measurement data in a second queue for the third loop.
3. [Calculation/storage loop] dequeues data from the second loop and stores the desired calculated values in a txt file
4. [Motor Control loop] controls one motor. It listens to the 'Communication channel' (notifier between two motor control loops). If it receives 'Start motor', it starts (or continues) the internal case structures. If it receives the 'Stop motor' command, then the motor ceases all activities. It listens to the 'Mean gap' channel (notification from the DAQ loop with a mean value) to get the latest mean gap. If it matches the desired gap height, the loop continous with an iteration through the different shear rates. If it doesn't match, it sends a command to the second motor loop to 'Go to gap = xx.xx' (over the 'Communication channel'). In the latter case it subsequently waits again for a new 'Start motor' command.
5. [Picomotor Control loop] controls the second motor. It listens to the 'Communication channel'. If it receives 'Go to gap = xx.xx' (with x a float), it reads out that floating number as a new target gap and puts it into a shift register. If it receives the 'Stop motor' command, then the picomotor ceases all activities. It listens to the 'Mean gap' channel to get the latest mean gap. As soon as this value is stabilized the picomotor loop will enter into the case structures. The difference between the target gap and the mean gap is used to calculate a step size for the picomotor. As soon as this step is zero, the picomotor loop sends a notification to the 'Communication channel' to 'Start motor'. It also listens to the 'Stop command' channel for a stop command.
6. Loops 6/7/8 are event listener loops

Now in short what do I do: I start a measurement, the gap height is adjusted to be the desired gap, then the first motor moves over a certain trajectory and then everything stops. The whole time, the DAQ loop is acquiring data and that data is visualized and calculated accordingly.

Problem:

I start the measurement, the gap is adjusted to the desired gap and then the first motor starts. It starts to move and then suddenly the program pauses for a few seconds (the motor can in the meanwhile continue to it's latest destination). After this pause, the program continues normally until a certain moment where it does the same thing. Just halts for a few seconds and then continues. This is really feaking me out because I cannot seem to figure out where the problem is located.

Discarded explanation:

• queues getting full and the program having to wait for a dequeue operation: this was my first guess, but I included a check before each enqueue action to check whether the queue is full or not. If it is full then the oldest element will be discarded

It would be super if someone has experienced anything like this before and could shed some light on this problem. If anything is unclear, just ask!

I am using LabVIEW 8.5 on Windows XP.

Thx,

Giovanni

[WayneS1324]

Nearly impossible to help without the code.

Does it seem like the longer it runs, the more it stops (or even slows down)?

[GiovanniV]

Hi,

Yeah I know, but I think that supplying my code would be a bit difficult. I can if you want, I tried to document it as good as possible.

To answer your comment: no it slows down at kind of arbitrary moments. It's not like when your using a XY-graph that performance slowly drops.

Additional information:

• I included indicators on my front panel that show the iteration count of each loop, to see what happens whenever the program stops.
• When it pauses now I can see that actually only my DAQ loop is halted. The rest just continues.
• I also get a warning in the error wire after the DAQ: 

  Warning 200011 occurred at DAQmx Control Task.vi:8
  
  Possible reason(s):
  
  Measurements: Clock rate specified is so high that it violates the settling time requirements for the acquisition.
  
  Reduce the clock rate, or the accuracy of the measurement might be compromised.

  .

[TCPlomp]

Hello Giovanni,

I would try to debug this step by step.

It's not clear to me what you mean by 'paused'.

Some easy things to try is showing an iteration counter for each of the loops in a 'debug'  page.

Second thing I would try is disable parts:

-Don't save to file (third loop) or use a TDMS (binary file) that should be faster than text, do you open close the file on each iteration?

-Don't show data in graphs (second loop)

You could shorten all the timeouts on queues, notifiers.

Why would you relay the queue in the second loop? You could sent the data into two queues from the first loop. If I understood it correctly currently the third loop is one iteration late.

How heavy is the sample compression? Is there a specific reason to apply a digital filter? Your warning state mentions that you are pushing to the limits of the DAQ-device. Perhaps a lower sample rate will result in less calculations for compression.

Run the VI analyzer toolkit and specifically focus on performance tests.

Ton

[carlos_camargo]

Hello,

How about this, since you are having problems with your daq loop only, make a new version of the code, strip out everything but the daq loop, see what happens.  Maybe keep the second loop (Visualization).  If you are still getting the problem then clearly the other parts are all irrelevant, deal with the daq problem, everything else might fall into place.

 C

[GiovanniV]

---

TCPlomp wrote:

It's not clear to me what you mean by 'paused'.

Some easy things to try is showing an iteration counter for each of the loops in a 'debug'  page.

---

Well I tried yesterday to show the iteration counters, and it's clear that only my DAQ loop is halted for 15 seconds and then continues. The other loops still run smoothly.

---

Second thing I would try is disable parts:

-Don't save to file (third loop) or use a TDMS (binary file) that should be faster than text, do you open close the file on each iteration?

-Don't show data in graphs (second loop)

---

I will try to do this today. I use a txt file to write to and I open the file before the while loops start and I close it after the loops have stopped.

---

You could shorten all the timeouts on queues, notifiers.

---

I already tried changing this, but no effect.

---

Why would you relay the queue in the second loop? You could sent the data into two queues from the first loop. If I understood it correctly currently

the third loop is one iteration late.

---

I did this to make sure the first loop has not too much to worry about but that first queue.

---

How heavy is the sample compression? Is there a specific reason to apply a digital filter? Your warning state mentions that you are pushing to the limits of the DAQ-device. Perhaps a lower sample rate will result in less calculations for compression.

---

Now I have set the DAQ to a rate of 100kHz and the number of samples to 1k. The sample compression then compresses each set of 1k samples by a factor 500. The reason why this filter is there is to smoothe the signal. Otherwise I tend to get a lot of noise. The problem I had in the past with decreasing the sample rate is that I got weird signals, mainly because of the filters I think. My signal is sometimes attenuated a lot.

---

Run the VI analyzer toolkit and specifically focus on performance tests.

---

Where can I find this toolkit? I haven't heard from it yet.

Anyways, thank you already for your insights. As soon as I get some new information I will post it here. I will also try to use only the DAQ and visualisation loop to check whether it persists.

[TCPlomp]

The VI analyzer toolkit is part of LabVIEW, you should be able to start it via tools\VI Analyzer (it can be that you don't have the right license).

I would advise to implement a hardware signal conditioning and use a lower DAQ rate.

Ton

[GiovanniV]

Hello again,

I managed to figure out where the problem was. The main motor is controlled over a GPIB interface and the motor control loop actually sends a command (move to position x) and then a small embedded while loop checks whether the movement has finished (every iteration a query is sent to the controller to check whether it is still moving or not.

On the other hand, in the DAQ loop I am querying the motor controller for the actual speed it is moving at. Now since the pauses are spaced in a random fashion, I imagined that the problem would be that whenever the motor control loop is waiting for the motion to finish, the DAQ loop is waiting to send it's speed request. To check whether I was right, I disabled the query for the actual motor speed in the DAQ loop, and now the problem is solved.

This poses a new problem:

How can I probe the motor speed (each set of datapoints from the DAQ loop should contain the motor speed value of that moment) while still being able to control the motor. Is there a way to have like a multiplexer (something that accepts all commands to the motor controller and then prioritizes them)?

If anyone should have a suggestions, you are most welcome to make it!

[johnsold]

Giovanni,

One way to deal with this situation is to use just one subVI to do the GPIB communications.  It can be set up as an Action Engine or a parallel loop which communicates with the main program via queues.  This subVI would know when a request expects a response and would not process the next request until the first response had been received and passed to the appropriate place.

By doing all the communication in one place the possibility of getting responses mixed up is minimized or eliminated.  It can also allow clean initialization and shutdown of the process and handling of communication errors.

Lynn

[Kevin_Price]

Giovanni,

I have done some fairly similar architectures before and I think I recognize the issue.  My first encounter with a similar (I think) problem came when a single external comm device wanted to be controlled by one loop while also being regularly queried by another parallel loop.  I solved it, in part, through the use of semaphores.

Another part of the solution followed the suggestion from johnsold.

Let me boil down how I see your problem.

1. When the DAQ loop wants to query for info from the GPIB motor controller, it must receive its answer without excess delay time.  The answer should represent the most current info possible.

2. When the motor control loop commands a move to the GPIB motor controller, it needs to avoid hogging the GPIB communication channel which may prevent the DAQ loop from having timely access.

Once you set up your GPIB comm functionality as a single distinct module, maybe all you need to do is add some delay time in your motor controller's "done moving yet?" query loop.  Or maybe your controller can use knowledge about the expected move time to delay the start of its queries.  Use of queued comm requests or semaphores can help make sure the DAQ loop doesn't get starved out of access to the shared GPIB resource.

If you follow johnsold's thought and make a separate GPIB comm loop, another option would be that the comm loop waits for queued requests, but in the absence of any request it simply queries the info needed by the DAQ loop and writes it to a global-like object (such as a named notifier or functional global, etc.)  Then the DAQ loop can simply read the most recent info from this global-like object and never directly need to access the GPIB comms.  The tradeoff is that the DAQ loop gets slightly stale info but gets it instantly.  The other way gives you slightly less stale info after an unknown delay.

-Kevin P