The starting place for managing memory in LabVIEW is to read
Application Note 168. Bring along some coffee. It's not the most exciting document in the world, but it is an excellent description of LabVIEW memory usage.
As to your particular problem, I have never seen the ability to explicitly define a memory limit in LabVIEW. Most of that seems to be handled by the operating system that you use. Personally, I've loaded a gigabyte of string data into memory (on a machine with 256M of ram!) without crashes under win2k.
Good luck!