Very large scattergraph - out of memory exception

[bnils]

Hi all!

When loading and displaying a very large data into a scattergraph (displaying all), e.g. ten plots with more than a million entries each (!) I have received out of memory exceptions and/or the red envelope/cross images displayed on the scatter graph.

1. Is there a way to detect this bad state in the code?

2. Are there any recommendations for how much data I can load and display in a scatter graph? Of course depends on the physical memory on the target machine.

3. (C#) Is there a way to catch System.OutOfMemory and if so cancel the loading of all this data?

Using Measurement Studio v8.9.0.246.

Best regards, Björn Nilsson

[bnils]

Exception details:

While trying to pan the gigantic graph, I get an exception with the following call stack:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.   

at NationalInstruments.UI.Internal.XYDeviceDataManager.Clip(Double xmin, Double ymin, Double xmax, Double ymax, Double[] xData, Double[] yData, Int32 dataLength, Double[]& clippedXData, Double[]& clippedYData, Int32[]& clippedDataIndices)   

at NationalInstruments.UI.Internal.XYDeviceDataManager.Clip(Double xmin, Double ymin, Double xmax, Double ymax, Double[]& clippedXData, Double[]& clippedYData)   

at NationalInstruments.UI.Internal.XYDeviceDataManager.a(Domain A_0, Domain A_1, Double[]& A_2, Double[]& A_3)   

at NationalInstruments.UI.Internal.XYDeviceDataManager.Map(Domain fromXDomain, Domain fromYDomain, Rectangle bounds, Boolean invertedX, Boolean invertedY, PointF[]& mappedData)   

at NationalInstruments.UI.Internal.CartesianPlotElement.Map(Rectangle bounds, PointF[]& mappedData)   

at NationalInstruments.UI.Internal.CartesianPlotElement.DrawForeground(ComponentDrawArgsInternal args)   

at NationalInstruments.Restricted.ControlElement.a(ComponentDrawArgsInternal A_0, Rectangle A_1, Boolean A_2)   

...

at NationalInstruments.Restricted.ControlElement.Paint(PaintEventArgs e)   

at NationalInstruments.UI.WindowsForms.ControlBase.OnPaint(PaintEventArgs e)   

at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer, Boolean disposeEventArgs)   

at System.Windows.Forms.Control.WmPaint(Message& m)   

at System.Windows.Forms.Control.WndProc(Message& m)   

at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)   

at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)   

at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Other cases when recalculating graphs, the following fails:

at NationalInstruments.UI.Internal.XYDataManager.GetYData()   

at NationalInstruments.UI.XYPlot.GetYData()

So just feeding the scatterplot may succeed but once you start interacting to it, it fails.

I could of course limit the feeding of the scatterplot but that limit is likely to be depending on the target machine capacity so its hard to specify a general number, suggestions?  

[D_Biel]

I was able to reproduce the red X you posted. It appears that there are two scenarios where your graph will display a red X.

1. There is an out of memory exception and the exception is passed to your code. This is good because you can then handle the System.OutOfMemoryException in a catch block. Graph.ClearData( ) will return the graph to a normal state.

2. There is an out of memory exception and the exception is handled internally by our control. This is bad and I am reporting this to our development group as a bug. The problem with this is that at this time I do not know a way to programmatically determine that you are in a bad state.

I would recommend avoiding these scenarios to begin with by reducing the memory your app uses. The graph does not really have any limitations on how much memory it can display, this is limited purely by the free memory of your system. So yes, it does depend completely on the system. If you need to push the limits of the system, what you could do is check how much memory is available on the system and then you can estimate about how many data points you plot. You can use either GC.GetTotalMemory or System.Diagnostics.Process.GetCurrentProcess().PrivateMemorySize64 to check how much memory your application is currently using.

You can track this issue with bug report #332295.

[bnils]

Thank you for your answer!

I have both cases (1 and 2) were exception is thrown. In the first case i can catch this in a single function of my own, but in other cases e.g. panning the plot etc, the exception is thrown by the GUI/National Instrument library I really cannot catching the exception other than at a very high level (e.g. main.function), so no possibility to clean up and return.

Calling Graph.ClearData() did not make any difference for me, but can of course be used for clean up!?

As you say I intend to reduce the risk of getting the exception. In my case I will try to split large data sets into several smaller. This wont eliminate the problem (if you choose to show all data sets in my application you still end up in an exception) but i may be more user friendly.

Best would be to perform some kind of memory usage analysis before loading the data, as you indicate at the end.

I'll follow the bug you mentioned.

Regads Björn

[D_Biel]

We came up with a way to check if the graph is in a bad state. Although it is a bit hacky. We can get the bitmap of the graph and check the top left pixel to see if it is Red. This should work so long as your graph control background color is not red. The point we are checking is actually in the axis label part of the control under default settings.

scatterGraph1.Refresh();
Bitmap myBitmap = new Bitmap(scatterGraph1.Width, scatterGraph1.Height);
scatterGraph1.DrawToBitmap(myBitmap, scatterGraph1.Bounds);
//myBitmap.Save("C:\\1.jpg");

if (myBitmap.GetPixel(12, 12).R == 255)
{
    //Handle bad state
}

 Note that I am checking the pixel at location (12, 12) this is because there is an alpha region on the top and left of the bitmap. You can see what I am talking about if you uncomment the myBitmap.save( ) line and look at the resulting jpg in a image editor that can handle alpha (MS paint won't work).

[jdl]

This post references bug report #332295 - what is the status of this bug?  Was the bug resolved? Is there a patch fix available for this?

Thanks,

Jon

[D_Biel · Accepted solution]

Hi Jon,

It was determined that this issue is not a bug. It is actually an issue inherent in the .NET control drawing architecture. When you programmatically change the appearance of a control, like plotting to a graph, the drawing of the modified control does not take place immediately. Instead, the redraw happens at a more convenient time for the UI thread. If you run out of memory during the time between when you plot to the graph and the redraw actually occurs, then you will receive an out of memory exception during the redraw. The difficult part here is that you can't handle the OOM exception.

To avoid this, you can force the control to redraw immediately using the refresh method as shown below.If you place the plot and refresh in a try-catch, then you have the opportunity to handle the exception.

try
{
    scatterPlot.PlotXY(...);
    scatterGraph.Refresh();
}
    catch(Exception e)
{
    //catch OOM here!
} 

[jdl]

Thanks for replying.

The problem I have is the same as bnils in that the OOM exception is thrown by the NI libraries when panning the display.  Is there anyway of catching the exception from here, or is there another workaround other than checking the colour of the pixel in the top-left corner?

Regards,

Jon