system resources usage (questions) algorithm (questions)

[TaurusTiger]

Preface:
We have a producer consumer design pattern using queues for data buffering and communication. The data must be consumed in order. The queues have a max size. When set to -1 the queue can hold an unlimited number of elements. While we all wish memory was unlimited, it is not. Using memory beyond a certain percentage of a system's total memory causes 'bad things' to happen.
Programmatically with LabView how does a VI...

Questions:

1. Get a system's total memory size ?

2. Get a system's virtual memory or is this included in the information received by asking for system's total memory size?

3. Poll for a system's current memory usage?

4. Poll for a system's current virtual memory usage?

5. Is there a way to determine the memory requirements for a single queue element (Note the element does not consist of simple data structure(s) with known sizes in memory and besides the element size is dynamic)?

6. System's long term storage size?

7. System's long term storage current usage or free space?

8. Processor/core usage?

Preface:

I am looking for ways to protect against the VI consuming the entire system's resources, VI crash, system crash, 'bad things', unpredictable behavior, Armageddon, etc.

My answers thus far include:

Questions:

1. Don't do anything which includes the possibilities listed above.

2. VI(s) is the sole system's task.

3. Maximize virtual memory size and cross fingers.

4. Use a system with a ridiculous amount of memory, or run the VI in such a way as to use memory from multiple systems. Essentially use VI on a cluster. While a fascinating concept, more hardware is more expensive.

5. Use parallel analysis inside consumer to increase consumption rate without the continuous use of core(s)/processor.

6. Sequence the VI. Producer generates data for queue one. Consumer one writes/streams a specific number of elements to long term storage from queue one. Producer two streams data from long term storage to queue two. Alter producer two's rate via timing control (timing control is determined based on calculation using average element size, size of queue, current memory usage, max memory usage, max storage, current storage usage). Consumer two processes data in order from queue two. Analysis is presented continuously to user(s).

7. Some combination of 5 and 6

8. Remove Queues. Producer streams data to long term storage and the consumer analyzes the data.

7. Any other ideas?


Thoughts and answers are fully appreciated!

Respectfully,
Taurus Tiger

[pallen]

Hi Taurus,

Just heading out the door to the office, but I did find something that may be of some use to you:

http://decibel.ni.com/content/docs/DOC-9292

This is a Labview implementation of a task manager like application. View the processes, select one to view CPU load and memory consumption. A new feature is the memory deallocation control, just in case the process start consuming to much memory.

Might not be exactly what you need.  But could be of some use. 

[TaurusTiger]

Patrick,

proc_manager.lvproj and the associated VIs do contain much of the functionality I am looking for. The user interface in TaskManager.vi does show CPU load, memory usage, and a lot of process specific information. I might be able to derive the system CPU load and Memory usage from the data. This is definitely a useful project. Someone put some time in to make the project work. The other three VIs each make use of a library call from a DLL (Dynamic-Link Library). The required Dynamic-Link Libraries are included in the project. This really is a great solution.

wtsapi32.dll, and psapi.dll are Microsoft Libraries. wtsapi32.dll and psapi.dll are proprietary to Microsoft. You can definitely find them in C:\WINDOWS\system32\. Sadly, this limits the VI to systems running a Microsoft operating system. I would not be surprised if the specifics of the DLLs changed over operating system versions.

I would prefer to find something a little closer to platform independent, but the idea of platform independent code is wishful thinking.

A close example might be getting the complete number of processing cores on a system by packages times, cores per package, times logical processors per core. This information can be obtained from the CPU Information Function: http://zone.ni.com/reference/en-XX/help/371361F-01/glang/cpu_information/

I say might be because according the site the function requires the base package. I am not sure if the base package is applicable to Windows, Mac, and Linux. Is it applicable to Windows, Mac, and Linuz

In my searching around online today, I did find something which has more information. sysinfo.zip has all of the information regarding short and long term storage. The user interface shows Operating system, Windows Directory, System Directory, Total Space on Drive C (MB), Free Space on Drive C (MB), Total Physical Memory (MB), Free Physical Memory (MB), Total Virtual Memory (MB), and Free Virtual Memory (MB). Investigating the C code a little shows it too is for Windows.

I can pull information from both to get all the information needed. It will work on Windows at least 😉

Thank you for your response. It has been most helpful.

Larga Vida y Prosperidad,

Bill

[Oskar_Bosch]

I'm surprised as well that there are not more functions available to access system functions.

sort of defeats the system independent G programming when you have to manually interface to windows DLL's.

Had to resort this several times as well (for example, get the user directory).

Would be nice if labview would provide more of these functions build in.