There are a lot of possiblities for running out of memory. Check for references opened but not closed, ineffecient array operations, overuse of local and global variables, attribute nodes in subVIs. You might want to try turning on the Performance Profiler to see if you can isolate where all of the memory is being used. The application note "LabVIEW Performance and Memory Management" is a good place to start for general information.