Multilpe timed-while loops, buffer overflow (error 200279), fast acquisition time

[E1]

<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta name="ProgId" content="OneNote.File"><meta name="Generator" content="Microsoft OneNote 11">

Keywords: buffer overwritten, 200279, -200279, timed while loops, input channels, ReadMX, PCI 6229 DAQ, Windows XP, fast acquisition

Hello everyone,

My problem is related to a buffer overwritten error (200279). I have 8 concurrent tasks running (i.e. 8 while loops):

1. Input - read data from DAQ  (runs in a separate .vi using a timed while loop - running @ 2ms). The .vi that's running the reading is set to high priority (critical).
2. Output - output data to DAQ (runs in a separate .vi using also a timed while loop - running @ 2ms). The .vi that's running the writing is set to high priority (critical).
3. 3 other timed while loops dealing with refreshing User Interface, error handling and data processing (running @ 200ms, 200ms and 2 ms respectively).
4. Two non-timed while loops (one uses a wait - 500 ms, the other runs only when there are elements in a queue).
5. A non while loop containing an event structure diagram (deals with taking user input from the front panel).

The data acquisition is set on: continuous acquisition mode, acquire 20 samples, at 9kHz and there are 13 channels (analog inputs) that are being read. For the data generation there are 4 voltages being output every 2 ms.

The symptoms are: the program runs for a while normally but then it returns a 200279 error (there's no consistent time for when the error shows up) and stops. Just before (or at the same time) as the 200279 error is created the timed while loops appear to stop for 5 seconds (+/- 0.5 sec). I've found this out by time-stamping every iteration of all the loops in my program. Interestingly enough the non-timed while loops seem to be running (this includes the while loop that contains the wait which keeps on running) normally during this 5 second stop. I changed the while loop that contains the wait to a timed loop and that results in it also not running during the 5 second stop. Moreover if I change the timed-while loop that performs the data acquisition into a non timed-while loop (with a delay) I still get the 200279 error but this time LabView crashes!!!

My problem is that I don't know what causes the buffer to be overwritten. I am reading 20 samples every 2ms (with very few loops running late (i.e. 1 or 2)) which translates to reading 10kSamples every second. My point here is that if my acquisition is running at 9kHz I am reading 10kSamples then I'm reading more samples than I'm acquiring--- presumably I need to wait to acquire those samples --- and that should mean I can't be filling up the buffer. Note: I also tried reading all the samples in the buffer (by setting number of samples read to -1) but this doesn't help either.

From the debugging I've done I have a feeling that having 5 timed-while loops (of which 3 are running at 2 ms) might be creating problems. Additionally I'm using a queue to store the samples I read (this might lead to some priority-inversion problems … but I don't think it should be resulting in waiting times beyond a couple of hundred ms).

The next thing I've done was to go for N-samples N-channels and this option works without returning any error (as expected). However every now and then the program hangs for about 5 seconds (similar behavior that was causing the buffer overflow error during acquisition mode).

More information:

Running LabView 8.0, DAQ: M-series 6229 on PCI slot, Windows XP SP3, Intel Core2Duo @ 2.33 GHz, 2 GB of RAM.

My questions for you guys are:

1. Have you encountered the 200279 error (buffer overwritten) when running multiple timed-while loops?

Or any pausing of the timed-while loops?

2. How did you debug it (identify which task was the one that was hanging?
3. I tried to use the buffer overwrite flag on the task I'm reading from… but it doesn't seem to work. Are there any other ways to flush the buffer before writing?
4. How do you guys do fast data acquisition? I looked at the examples … and they are a little bit too slow for what I need. The overhead of setting up and taking down the channel is too much for my application. I need continuous acquisition without the nagging buffer overload error.
5. As I am using a dual core Intel processor I'm wondering if I can force only a small numbers of threads to always be on one of the CPUs (i.e. the thread that's reading from the input buffer and another thread that needs to access the data that has been read) - sounds kind of dicey with the OS thread management.

Thank you for your input!

Regards,

E1

[Brooks_C]

Hello E1,

I believe that the source of this error is that you're trying to run you're loops at 500Hz.  On a Windows operating system the fastest you can comfortably run an empty loop is several kHz, but when you add in other processing into this loop this rate decreases dramatically.  In order to avoid buffer overflow errors it is recommended that you set your samples to read to be at least 10% of your sample rate in Hz.  When you call a DAQmx Read LabVIEW asks Windows to move data from the DAQ card's first in first out (FIFO) queue in RAM into LabVIEW's memory space in RAM.  The largest source of delay in this process is waiting for Windows to perform this task--in general the response is on the order of a few ms.  You can reduce the effect of this delay by reading more samples each time you call a DAQmx Read.  For instance, on an average computer you can comfortably acquire at 1 MHz if your DAQmx Read is reading at least 100,000 Samples/sec.  However if you run the same program acquiring at 1 kHz, but only reading 1 sample at a time the same machine would likely encounter a buffer overflow error.

I would recommend modifying your code so that all the loops run at about 10 Hz by increasing the number of samples to read to be 10% of your sample rate.  If you have multiple input and output tasks in your program it’s entirely possible that it isn't the same buffer overflowing each time so tracking down exactly which task is causing the problem is a moot point.  The real issue is that the CPU is being worked too hard to read from all the buffers fast enough.  Also, in general, timed while loops are more useful if you're using a deterministic or real time operating system.  In these operating systems software execution always takes the same amount of time where as in a non-real time operating system (like Windows) other processes, including background processes, can change the amount of time it takes for code or a VI to execute.  For these applications I would usually just use a while loop with a Wait (ms) VI in it.   It is important to put the wait in the loop so that the CPU doesn't continually run the loop as fast as it can (this is another tactic for reducing the CPU's work load).

I hope this helps get you started, and have a great day!

Cheers,

[E1]

A follow up to my problem.

The problems seems to be linked to dual core units. I tried the same program on a single core (much slower than the dual cores I have) and it NEVER displayed a 5 seconds stall. After setting LabView.exe process to run only on one of the CPUs I noticed that the stall was occurring much more infrequently (about 100 times less infrequent). I believe that having so many concurrent while loops creates a pretty big mess when passed through the scheduling algorithms.

My next attempt will be to disable hyperthreading and observe the results.

E1

PS Thanks Brooks_C for the info. It appears that making all the timed loops non-timed with a delay does prevent the 5 second stall. However I have very little control over the period of the loops. I looked into RT systems but they do not fit into my set of specifications. So for now I will  keep running on Windows XP only on a single core.


Message Edited by E1 on 05-28-2008 07:19 PM

[Brooks_C]

Hello E1,

This is interesting behavior that you're seeing.  I think that it may be caused by the loop priority settings--while this feature is enabled for Windows, Windows is not designed to handle these priorities so in general it is not recommended.  I would suggest removing the priorities and trying it that way.

Also, there are a couple benefits to using timed loops over while loops with a wait VI including the ability to set which processor each loop executes on, however, as far as timing goes both of these loops will be identical.  On a Real-Time system these loops operate deterministically, but on a Windows systems processes, including timed loops can be interrupted anytime so the timing between these two types of loops is equivalent.

Just to make sure we're on the same page, when you use a while loop you should put a Wait (ms) VI in the loop to control the loop timing.  If you specify 100 ms on the input of this VI then each loop iteration will take either 100 ms or the length of time it takes to execute the other code in the loop, whichever is longer.  As long as the rest of your code can execute in 100 ms each iteration will take 100 ms and you can control the timing in this way.

I hope this helps, and I'd be interested to hear what you find out.  Have a great night!

Cheers,

[mattias@hv]

Hi!

I thought I might give you some of my experience regarding having two timed-while-loops.

I had the problem with stalling application (~5-6 secs.) due to having two timed while loops.
I've been trying for some days to find what could be the problem, changing priority, cpu#,
timings etc. However, today I discovered that when I replaced one of the timed-loops with
"a standard while with a wait-statement inside", everything worked out really good. No more
stalling, and everything is working as expected.

I use a Core2Duo @ 2.4Ghz, 2GB ram, Labview 8.2.1 and Labview 8.5.1 on WinXP Pro SP3.
As already stated in the comments above, I also think windows is not as capable of handling
these time-loops as one can think (anyway, as I thought 🙂 ).

Attached is a picture showing my current solution (one timed loop and one standard while loop).

Regards,
Mattias

[E1]

Hi Mattias,

Thank you for adding your fix to this issue.

After looking into how Intel does it's scheduling on the core 2 duo processors and trying to figure out how the OS (i.e. Windows XP) sets its scheduling of processes and then also throwing in the scheduling that LabView tries to impose (that's three layers of scheduling) I decided that there's too much complexity for me to handle. One of these schedulers needed to be disabled! So I disabled my hyperthreading scheduler (that's called Core Multi Processing (CMP)... if you're running Intel ... I'm not sure how AMD calls it).

After disabling the CMP all my timed while loops behaved (so far) really well. No more 5 second stalls even when I really get the CPU usage close to max. I have to say that I still observe some lates, for example if I zoom in and out of images acquired in real-time while my control software is also running. However these are on the order of 100 msec and I can live with that (as long as it's not frequent).

So if it's not too much trouble you could try disabling the Core Multi Processing and check if there is an improvement in the reliability of your software.

Note: To disable CMP you have to enter the BIOS of your machine (i.e. Pressing Delete or F2 during the boot up screen) Go to Advanced Settings (or something like that) then CPU Settings and then there's a Core Multi Processing value that you can set to disable.

Let me know if this worked for you!

Thanks again for adding your solution!

E1

[E1]

Additional information: I've ran my programs on a single core CPU with hyperthreading enabled running on WinXP and observed multiple 5 second stalls. Disabling hyperthreading results in no 5 second stalls.

CONCLUSION: Using multiple timed-while loops can cause 5 second stalls on Intel processors that use hyperthreading or equivalent technology. Disable hyperthreading to solve the problem.

[Brooks_C]

Hello E1,

As far as I know there are no known issue with LabVIEW and/or DAQmx having to do with hyper threading.  Would it be possible for you to post your VI so that I can look into the issue that you're seeing?  If you're not comfortable posting your code on the public forum would it be alright for me to look up your email and contact you directly?

Thanks,