How can I display more graph data than will fit into memory?

[bmihura]

I'd like to graph more data than will fit into my computer's memory on a LabVIEW graph or some other LabVIEW control. In the past I've hacked this with the 2D Picture Control. LabVIEW would read through the huge file one chunk of manageable memory at a time and fill in the appropriate pixels in the 2D Picture Control as it read through the file.

Is there a better way?

[jcarmody]

Decimate?

[altenbach]

---

@bmihura wrote:

I'd like to graph more data than will fit into my computer's memory on a LabVIEW graph or some other LabVIEW control. In the past I've hacked this with the 2D Picture Control. LabVIEW would read through the huge file one chunk of manageable memory at a time and fill in the appropriate pixels in the 2D Picture Control as it read through the file.

---

What kind of data and how should the final display look like? We definitely need more detail!

If you just want to create e.g. a 2D histogram of ALL data, an intensity graph would be more suitable than a picture control (e.g. as seen here).

(link to example)

[bmihura]

altenbach,

I'm collecting oscilloscope-style traces at 250 kHz from eight channels during a one-hour test....

This results in a 28.8G of data in memory when I want to show all the data on the graph but I'm really interested in spikes that may happen on any of these eight channels.

Yes a 2D Historgram would be more suitable than a 2D Picture Control. Either way I'd have to write the code to read in chunks of my too-big-for memory data file and map them to pixels on the 2D indicator. I'm hunting for existing code that does this so I don't re-invent the wheel again; I've lost track of my LabVIEW 4 code that does this.

[Stuart.Parkinson]

So you have a massive stream of data and you want to graph it all at once. But be it a graph, a picture control, or something else I'm gussing that it can only be a maximum of around 2000 pixels across (most monitors are 1920). If you wanted to fit say 200000 samples into these 2000 pixels, then each pixel will have to represent 100 data points (just by way of example)

Do you want the pixel to display the min, max, or average or those 100 data points? Or just the first one?

Is it a periodic signal that you want to overlay (like altenbach's image which resembles a persistent scope)?

Or are you looking for a 'live' data plot that allows you to zoom and scroll on the data?

It should be fairly trivial to read in say 100 points at a time, (min/max/avarage/etc.) them and add the single resulting point to a chart. If it is one of the other two you are after then that will be a little more challenging.

[mikeporter]

Here is some additional information that you might find helpful:

http://www.notatamelion.com/2015/07/31/plotting-large-datasets-without-waiting-forever/

Mike...

[bmihura]

Stuart,

I'm looking for a live data chart that allows me to zoom and scroll. Since I'm looking for peaks I'd want the min and max per pixel just like LabVIEW's graphs do.

Effectively, I want a LabVIEW graph that handles data from files rather than in memory.

[Stuart.Parkinson]

Looks like mikeporter has done most of the heavy lifting for you in his excellent article.

If the data will not fit in memory then you have little choice other than reading chunks at a time from file. I would probably so something like:

Decide how much you want to work with at a time (say 1 MSamples). If your selected 'dataset' zoom level is 'all data' then you will have to read in all the file piecewise, decimate the data, and build up your 1MSamples array. You can then use the LabVIEW graph to zoom in and out and scroll that set of samples. This gives you some limited 'live' functionality on the decimated dataset, and means you don't have to read from the file every time you scroll or zoom.

Then if you want to examine part of the data more closely, you need to change the 'dataset' zoom level to 'just these 100Msamples' (say). Re-read just that section of the file, decimate as needed, and display. As your 'dataset' of interest becomes smaller you will need to decimate less until you can just read sample for sample.

These two levels of 'zoom' - a macro 'dataset' selected from the overall data, and the native graph zoom and scroll, should give a balance of usability without having to constantly read from file as you move around small sections of the data.

Sorry I don't know of any existing code (though I'm sure someone has done it before!) - if you do implement something like this then maybe you could share 😉

[altenbach]

I assume that the data is stored flat inside a binary file. In that case it is easy to calculate file position for any desired chunk of data and read the subset.