Out of Memory Error when Saving SequenceFile generated using API

[Mikewaldon]

I am currently developing an application that creates a SequenceFile using .NET 4.0, C# (2010), Windows 7 Pro x64, Dual Core 2.5 GHz, 3.0Gb Memory.  Using the API EngineClass, I create a complete SequenceFile based on user/file inputs and then save it to disk.  When I'm done, I save using SequenceFile.Save() and then release using ReleaseSequenceFileEx().

For smaller files, there is no problem.  For larger files, I sporadically get the following error:

System.Runtime.InteropServices.COMException (0xFFFFBD98): Out of memory.

When the file does save, it is ~2.8Mb.  Breaking the file up is not an option.  I am not looping, this is one shot.  I've tried shutting down VS2010 and restarting the computer.  It is still inconsistent on saving.

I also had to set the Embed Interop Types to False per guidance from NI.  They said that it is unstable in 4.0.

Questions

1. Is there a file limit size to the Save() API function?

2. Is there a workaround to using the Save()?

Best Regards

[asbo]

I believe I've run into this problem before, but it was with extremely long sequences within a file, rather than a large sequence file itself. The solution to my problem was to break it up into more subsequences but I don't think that's relevant to your issue.

What file format are you using?

Have you tried manually reproducing the failure in the Sequence Editor?

[Mikewaldon]

There is a single sequence file (.seq) that contains mulitple sequences and variables.  It is the large versions that are causing problems.  Below is an example.

Sequences: 346

FileGlobals: 2207

Sequence Calls: 8252

Total Parametes (in all sequences): 255

Total Locals (in all steps): 56

Total Steps (in all sequences) : 52706

I can open/close/save the file at will in the editor.  The steps are generated by my test software, so generating them manually isn't an option (see # steps above).

I can build the sequencefile with no problem/error.  It only occurs if I try to save it using the Save() method.  Percentage with, it save successfully only about 25% of the time.  I've tried looping the save and if it fails on the first attempt, it will always fail.  I need to dump out and rebuild before I can attempt to save again.

[asbo]

What does the memory of your test software look like as the generation progresses? 

Can you save in stages? This may help if TestStand is smarter enough to only manage the changes to the working set.

Beyond this, you should probably open an issue with NI support, R&D will be able to help you much more than the forums can.

[Mikewaldon]

Memory under Task Manager varies between ~780Mb to 1.2Gb.

I'm currently trying the incremental saves, but not much success.  It slows the generation process do substantially.

[dug9000 · Accepted solution]

Before you save the file try calling:

GC.Collect();

GC.WaitForPendingFinalizers();

This will force the .NET garbage collector to recover whatever memory it can to make room for the unmanaged code in the TestStand engine to be able to save the file. Large blocks of contiguous space are needed when saving files. This will likely fix your problem, however there are some other things you might want to consider that would also help.

1) What version of TestStand are you using?

1b) What file format are you using? Binary (binary is the default for new files created in TestStand 4.0 and higher), XML, or INI?

2) Do you have any large numeric or boolean arrays in your sequence? If so you might want to consider making them initially empty (if there isn't really any data you care about them saved at edit time), then in your sequence you can programmatically resize them to the size you need at runtime.

Hope this helps,

-Doug

[Mikewaldon]

You rock!

5 times in a row makes me happy.  I'll try to break it, but I think this will do it.

BTW:

1) TestStand 2010

2) Binary

3) Not many arrays.  Some are 80 element, but only a couple.  I can't determine what they'll be because the generation is determined on end user inputs and/or files.

[dug9000]

Glad it worked. Garbage collection doesn't necessarily always make things simpler. It has its own separate set of problems.

1) 80 elements isn't that many so that's probably not worth optimizing. Also newer versions of TestStand (either 2010 or 2010sp1, I forget which), write arrays as a separate chunk in the binary format so it's not as likely to cause an out of memory error unless the array is so big that the array itself doesn't fit in the 32-bit address space when you read the file back in.

2) Binary format is the way to go.

3) I think the GC.Collect(), GC.WaitForPendingFinalizers() solution is the right answer for your problem. You might even want to call it periodically while you are building the file if memory on the machine seems to be getting low. For example, maybe every 1000 steps you create or so. It might improve performance if you are running low on memory while building the file.

-Doug

[uoi]

Hi,

May I know how to call the followings in teststand?

GC.Collect();

GC.WaitForPendingFinalizers();

Im having the same problem.

Thanks in advance.

[Mikewaldon]

I am using C#.NET.  I called them in my save method immediately prior to the actual test save sequence.

GC.Collect();

GC.WaitForPendingFinalizers();

GC.Collect();

GC.WaitForPendingFinalizers();

try

{

       CurrentSequenceFile.Save(outputPath);

}

       catch (Exceptionerror)

{

       FireLogEvent(ParserLogType.Parsing_Error, string.Format("{0}", error));

}

Factory.TSEngine.ReleaseSequenceFileEx(CurrentSequenceFile, ReleaseSeqFileOptions.ReleaseSeqFile_UnloadFile);

Hope this helps