XY graphing with large amount of data

[Kevintb]

I was looking into graphing a fairly substantial amount of data.  It is bursted across serial.

What I have is 30 values corresponding to remote data sensors.  The data for each comes across together, so I have no problem having the data grouped.  It is, effectively, in an array of size 30.  I've been wanting to place this data in a graph.

The period varies between 1/5 sec and 2 minutes between receptions (its wireless and mobile, so signal strength varies).  This eliminates waveform graph as the time isn't constant and there's alot of data(So no random NaN insertion).  This leaves me with an XY graph.

Primary interest is with the last 10 minutes or so, with a desire to see up to an hour or two back.

The data is farily similar, and I'm tring to possibly split it into groups of 4 or 5 sets of ordered pairs per graph.


Problems:
1.  If data comes in slow enough, everything is ok, but the time needed to synchrounously update the graph(s) often can exceed the time it would take to fully recieve the chunk of data which contains these data points.  Thinking asynchrounously is useless, as the graphs need to be reasonably in tune with the most recent data recieved.  I can't have the an exponential growth in the delta of time represented on the graph and the time the last bit of data was recieved.
2.  I could use some advice on making older data points more sparse to allow for older data to be viewed, but with a sort of 'decay' of old data I don't value that 1/5 second resolution at all.

I'm most concerned with solving problem 1, but random suggestions on 2 are most welcome.

[chilly_charly]

I don't think the display should be your main problem.

On my PC, displaying 30 channels, with 35000 points each (corresponding to 2 hours at 5 pts/sec), takes 0.35 s. 3 frames per second. Even with a slower computer, display time should remain acceptable, since you will end with a much lower number of points.

Memory doesn't seem to be a big issue either, since everything should fit into 8 Mbytes (as dble !!!)

Either your video card is deadly slow, or the timing problem is somewhere else. May be in the way you store the incoming data ?

[Tomi_Maila]

I didn't quite get the first question. Could you try to find out where exactly the time is consumed in your program and then refine your question.

To the second question (which may also solve the first question). You can store all the data to a file as it arrives. Keep the most recent data in a let's say shift register of the update loop. Add a data point corresponding the start of the mesurement to the data and wire it to a XY graph. Make X Scrollbar visible. Handle the event for XY graph:X scale change. When the scale is changed i.e. the user scrolls the scroll bar, load data from the file and display it on the XY graph. This was a little simplified, it's a little more complicated.

In my project, I am writing an XY graph X Control to handle a bit similar issue. I have huge data sets i.e. multichannel recordings  with millisecond resolution over many hours. One data set may be several gigabytes, so the data won't fit on the main memory of the computer at once. I wrote an X control which contains a XY graph. Each time the time scale is changed i.e. the scroll bar is scrolled, the X control generates a user event if it doesn't have enough data to display the time slice requested. The user event is handled outside the X Control by loading the appropriate set of data from the disk and displaying it on the X Control. The user event can be generated already a while before the X Control is out of data. This way the data is loaded a bit in advance, which allows seamles scrolling on the XY graph. One must notice that the front panel updates must be turned off in the X Control when the data is updated and back on after the update has finnished. Otherwise the XY graph will flicker annoingly.