08-14-2012 10:05 AM
As suggested, I added error handling/logging but it didn't catch any error code when it crashed after 3 hours of operation yesterday. Here are some more diagrams.
Part of the main VI
W3 all files subVI
W3 all file subVI (there are six of them in the upper level VI as shown above)
The newly added error indicator is grey unlike the other ones. What's the difference?
deppSu wrote:
Looks like the logging for 4 TDMS files is concurrent, have you tried wire the logging VIs sequentially?
I haven't tried it. In fact, as shown in the diagrams above, 6 * 4 = 24 TDMS files are being written at the same time. (This is for this partiular function W3 alone. If I run full calculations, it will be writing to nearly a hundred TDMS files each second.) It will be hard to make the 24 TDMS Writes completely sequential but I guess I can try at the lowest level like this.
08-15-2012 02:42 AM
MemoryGrowth,
You mentioned that LabVIEW 2011 crashed. Could you provide a mini-dump file for a LabVIEW crash? Refer to http://digital.ni.com/public.nsf/allkb/2FF01AD2F6A02ABD862578D40053FCC7, NI Error Reporting could generate a mini-dump when LabVIEW crashs. Mini-dump file could help us analyze the root cause of your LabVIEW crash, especially when you could not provide a reproduce VI.
For your sentences "There is a known memory growth issue with TDMS. Our application basically writes to TDMS files 24/7. Memory growth caused by TDMS is very critical for us", did you have any post in this forum on your memory growth issue using "TDMS Write" (not "TDMS Adv Sync Write")? I search this forum with the username "MemoryGrowth" but I could not find it.
If possible, could you provide a very simple VI to reproduce your memory growth issue using "TDMS Write" (not "TDMS Adv Sync Write")? A simple workaround is to write multiple tdms files, and each tdms file only contains one channel. I don't know whether this workaround works for you or not.
Best Regards,
Bo Xie
08-15-2012 08:40 AM - edited 08-15-2012 08:42 AM
Re-wiring the error handling line as in the above diagram didn't help.
So far, I have identified three cases when the program does NOT crash.
1. When there is no TDMS writing part. It doesn't crash even with the short save interval and high sampling frequency as long as TDMS writing is disabled.
2. Longer save interval. (ex. 1 sec vs. 0.2 sec) I described what "save interval" here means in one of my previous posts. Less data to write per second.
3. Lower sampling frequency. (ex. 10240 S/s vs. 1920 S/s) This doesn't change the final number of data points written to TDMS but calculations are less intensive since it gets less data to deal with from the DMA FIFO.
Therefore, it may not be that there is an inherent problem with TDMS writing. It just seems the program didn't crash because it used less system resources without TDMS writing. System resources are not infinite so some heavy load must hit the ceiling at some point. What I want to understand is the random nature of the crashing time. Would it be that something gets gradually accumulated until it reaches some threshold or that just one probability-based random trigger crashes the program? I haven't been able to find an effective method to monitor any signs to predict when it will crash.
@Bo Xie wrote:
MemoryGrowth,
You mentioned that LabVIEW 2011 crashed. Could you provide a mini-dump file for a LabVIEW crash? Refer to http://digital.ni.com/public.nsf/allkb/2FF01AD2F6A02ABD862578D40053FCC7, NI Error Reporting could generate a mini-dump when LabVIEW crashs. Mini-dump file could help us analyze the root cause of your LabVIEW crash, especially when you could not provide a reproduce VI.
For your sentences "There is a known memory growth issue with TDMS. Our application basically writes to TDMS files 24/7. Memory growth caused by TDMS is very critical for us", did you have any post in this forum on your memory growth issue using "TDMS Write" (not "TDMS Adv Sync Write")? I search this forum with the username "MemoryGrowth" but I could not find it.
If possible, could you provide a very simple VI to reproduce your memory growth issue using "TDMS Write" (not "TDMS Adv Sync Write")? A simple workaround is to write multiple tdms files, and each tdms file only contains one channel. I don't know whether this workaround works for you or not.
Best Regards,
Bo Xie
Hello, Bo Xie. This was my very first thread on this forum.
The program runs on an RT controller (PXI-8108) so when it crashes (all running VIs on the controller are abrubtly closed), I don't think I can generate a mini-dump file.
When I was using the old TDMS Write, memory leak/growth was very visible and consistent. RAM usage went up almost perfectly linearly with time. In my OP, there is an official link for this problem so it must be an issue well-recognized by NI. After using TDMS Advanced, the memory growth problem is completely gone. RAM usage is totally flat. It also doesn't seem like the use of TDMS Advanced newly introduced the random crashing problem. It may just be that it was overshadowed by the consistent crash caused by obvious RAM overflow with old TDMS Write.