11-03-2021 03:37 PM
I am streaming data from multiple particle sensors (currently 3, but would like to use more) and I have little control over how the data is sent from these devices. The input I receive is a string and looks like this:
1 789 246 12 0 0
First number indicates the sensor, the next five indicate particle counts in a particular size bin (0.3um, 0.5um, 1.0um, 2.5um, 5.0um), each separated by a space. The stream of data looks like this and adds a new row every second.....
1 789 246 12 0 0
2 876 279 18 0 0
3 756 239 14 0 0
1 792 240 2 2 2
2 888 283 18 0 0
3 756 239 14 0 0
I'd like to be able to visualize the data from each individual sensor in its own chart in real time, but I cant figure out how to separate this data stream in a way that makes the most sense. A single chart (say from sensor 1) would have 5 different lines corresponding to each particle size bin. I tried parsing each new line based on the first number (the sensor value), and this leads to 3D arrays with each page as a new sensor, but the 3D array much have uniform 2D array sizes per page and sometimes a sensor has a hiccup and doesn't send data.
Any thoughts on an efficient way to sort this data stream and display on multiple charts? Ideally, this would be scaled up to multiple sensors, more than the three I'm showing here.
Solved! Go to Solution.
11-03-2021 04:58 PM
Is it reasonable to assume that all of your particle sensors "do their best" to send data at the same rate, with the possibility that occasionally a point will be "missed" (but easily detected by you)? If so, then saving the data in a 3D Array, Sensor by Channel by Sample makes sense. You can handle the "missing point" by writing "XXX" (or some other easily-recognized code that says "Skip this sample for this Sensor (or Channel)" (and program your graphing routine to not plot that point).
Bob Schor
11-04-2021 11:36 AM
A 3D array makes no sense at all for visualization and charts have their own history, so all you need is add the new data to the chart (And stream to a file if you want to keep a record of all data).
Split the incoming data into the various detectors and update each chart once per second and if one particular set is missing, update either with the most recent valid data (kept in a shift register) or NaN (to create a gap), whatever is more appropriate. For example if the data is typically slowly changing, repeating the last good dataset is a good choice.
Do you have a simple code skeleton example so we can get a better idea on how it all looks?
11-04-2021 01:45 PM
Thanks for the input! I'm making progress, and I've attached two VIs as examples, both generate dummy data that would stream from these little particle sensors via VISA. The dummy data here looks similar to the real thing. One example shows streaming data that has a numerical header which indicates the sensor number, followed by a space and then additional numbers representing particle counts per bin (each separated by a space, 6 total bins).
The other example VI shows a different approach where I could add alphabet letters with the streaming data as sensor headers instead of numerical values, i.e. A B C, etc for sensor 1,2,3...
example: A 344 546 223 123 432 546 would be from sensor A.
Is there any advantage to having letters as sensor references? I can't figure out how to sort via letter so any advice would be super helpful.
I can plot data from a single sensor as it populates with data. Is there a way to scale this programmatically depending on the number of sensor I'm connected to? So four sensors would automatically populate four plots on the front panel?
More general question: Am I going about this the correct way? I tried 3D arrays but that didn't seem to work well, especially when I tried to plot the data. Is there a better/more efficient way of sorting individual 1D arrays with specific headers and then displaying them as they get updated?
Many thanks for the help!
-Matt
11-04-2021 01:58 PM - edited 11-04-2021 02:09 PM
Numbers are easier as references, because you can parse the entire string as numeric using "spreadsheet string to array" with space as delimiter and a 1D I32 array as type. (Not sure why your array is orange, because channels and counts are of course integers).
You don't have a 3D array, but a ragged array (1D array of clusters containing 2D arrays.
You also don't have charts, but graphs. Big difference!
I'll have a more detailed look. I am sure you can do the same with 20% of the current code. 🙂
11-04-2021 02:26 PM
Here's a literal rewrite of your code to give you some ideas (zero change in functionality!).
I really don't like these ever-growing arrays inside convoluted data structures and ever-growing data in the graphs. Ultimately, you'll run into memory problems. You really should use charts with a reasonable history length and get rid of all data in the shift register. If you want to record all data, stream it to disk using low-level file IO.
11-04-2021 03:00 PM
Thanks so much! This is really helpful!
11-04-2021 03:04 PM - edited 11-04-2021 03:06 PM
@mes291 wrote:
I can plot data from a single sensor as it populates with data. Is there a way to scale this programmatically depending on the number of sensor I'm connected to? So four sensors would automatically populate four plots on the front panel?
While you cannot generate charts on the fly at runtime, you can create an reasonable upper limit of charts and hide the one's you don't need. Here's a quick example how to update and display up to three charts and charting previous data if no new data is available.
As I said already, if you want to keep a record of all data, you should stream it to disk (not shown).