Program shutdown after ~12 hours

[djs_at_eaton]

On a PXI system running a 6071 DAQ at 50K samples/s on 16 channels, W2000 shuts the LabVIEW app off. Sometimes it shuts everything else off too, and the OS crashes. Sometimes the OS stays alive enough to ask for a reboot. Sometimes a few items remain on the screen, but mouse and keyboard are dead.

[johnsold]

You have generated 138 GB of data after 12 hours, but your chart history is 262144 points deep. I suspect a memory leak somewhere.

Suggestions: 1. Put a shift register on the error signal in the read loop. If an error occurs it will prevent repeated attempts to read.
2. Put the second read inside a case structure which only executes if Scan backlog > 0.
3. Use the Profiler to check for changes in speed or memory usage over time. Also watch the Scan Backlog indicator. Does it build up before the crash?
4. I do not use Windows, so I cannot offer specific suggestions, but OSes sometimes decide to do their own things, like indexing disks or checking the network for software updates. Try to find out if something like this could be interferring with y
our program.

Lynn

[djs_at_eaton]

Shouldn't the unused data simply be dumped as it falls out of the indicator's range? I'm assuming the chart history is simply a circular buffer that overwrites old data.

I tried the profiler, but it showed nothing unusual. I've activated W2000 task manager to track memory usage. Nothing obvious after a few minutes. Meomory use fluctuates around 47Kb, but doesn't seem to increase. I should know more in a couple of hours.

I've used an alomost identical program on this computer with this DAQ card before and ran it for days at a time.

[Ben]

I think Lynn was correct in suspecting a memory issue.

I will venture a guess that after 12 hours you getting the trigger you are looking for.

After the trigger the history is read and formatted for file.

Now it is time to math.

After 12 hours of 50,000 samples/sec on 16 channels you have alot of data.

I wipped a quick test VI to help figure out how much memory is being used.

You are collecting 800,000 samples/second.

Formatted as text, the string length is 4G (that is for one second).

For 12 hours you need 2.1 tera bytes of memory to buffer the string!


So....
I would suggest that you truncate the data to a couple of seconds prior to the trigger event and save the data as binary.

Do post processing to convert it to text
latter.

I hope this helps,

Ben

Demo attached

[Ben]

Ignore those numbers!

Repeat proceedure using history buffer length.

I still believe this is a memory issue.

Ben

[djs_at_eaton]

I'm not arguing that it's not a memory issue. I just don't think it has anything directly to do with the amount of data.

I'm not saving the data as a string until the write to disk after the DAQ trigger event. The data is "stored" in the chart history, which I assume is a fixed quantity, allocated once, and treated as a circular buffer in which new data overwrites old. That is, in fact, what the W2K task manager shows. I'm not "leaking" or otherwise abusing memory.

I revised the program to clear the chart history every 10 hours. So far the program has run about 28 hours. The call to the clock function takes extra time, so it's also running fewer iterations of the while loop. Previous crashes happened somewhere around iteration 16
x 256 x 256. That number should happen around 36 hours.

If the program operates for 48 hours, I think it will be OK "forever". My theory is that at some count on the graph X-axis, the pointer into the circular buffer fails to index properly and writes data outside the buffer. At that point the OS gets upset or program memory is overwritten with data. Clearing the chart history sets the X-axis values back to zero.