LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Better performance in mathscript - multiprocessing?

I have a vi that reads log data from a file to an array, one row for each record. A for loop then takes one row at a time, makes some simple mathscript calculations on three of the columns, builds a new array which is then inserted into the original. When the log file is large, this takes very long time, so I need some optimizations.

My first idea is to use matrix calcultations, so that column C=A+B in one go, instead of looping through every row with a=b+c. I found the functions c=plus(a,b), c=ldivide(a,b) but didn't now how to use them. Nor do I understand how to enter a whole array into mathscript, when I "add input variable" it is only scalar?

Another idea was to make the code muliprocessor friendly, to reduce the time by half. or so. Also interesting to learn more about this, even if there are better things to do to optimize this particular task. As a test I made two loops that calulate the sum of 1...n, and as expected the vi uses full CPU  (both). When I try the same with my problem, dividing the input array into two, and giving two calculating loops half of the rows each, only one CPU (50%) is used?

Take a look at the pictures in attached word document.

Ola A.
0 Kudos
Message 1 of 13
(3,190 Views)
Why use mathscript at all?? All of the math functions that you are using can have arrays wired to them as inputs and outputs.
0 Kudos
Message 2 of 13
(3,176 Views)
I've solved a problem like this by reading my data one record at a time in one loop and passing that to a separate loop where I evaluate it. I pass the data via a queue.

When the file read function runs out of data, it loops and checks the size of the queue. If the queue is empty, then the processing loop is finished and I release the queue from the file reading loop. This causes an error in the processing loop that I use a a signal to exit and return the results.

I saved additional time because my data was binary and I could determine the number of records by dividing the file size by the record size. I calculated the output array size in advance and used replace element instead of build array.

I went from several minutes to process a data file down to 10 seconds using this technique. Here's a picture of what I did.





Message Edited by Phillip Brooks on 02-19-2008 11:13 AM

Now is the right time to use %^<%Y-%m-%dT%H:%M:%S%3uZ>T
If you don't hate time zones, you're not a real programmer.

"You are what you don't automate"
Inplaceness is synonymous with insidiousness

0 Kudos
Message 3 of 13
(3,175 Views)
Why use mathscript? The only reason is that we used to write the formula this way, and to me it is not yet so easy to understand a formula in dataflow representation. But you're right, I should use this as a lesson in doing math the labview way! 😉

0 Kudos
Message 4 of 13
(3,147 Views)

Unclebump, I rewrited the code in simple math operations, on whole columns. I don't know if it is faster because I found that the calculations are not the problem at all! They are fast enough in the original version. But we have a loop afterwards (se attachment) that loops through all rows, and if the status column is anything else than zero it adds the row to an resulting array. This takes time! What we want to accomplish is to remove all rows where status=0.

Suggestions on how to do this faster?

PS. I see all those nice .png attachments, how do I make them?

0 Kudos
Message 5 of 13
(3,134 Views)
There are several ways to do this. You could try the delete from array function. You could write only the good values into a queue. For each row, you could make a cluster with the status value as the first item and all of the original data as the second item. Sort the cluster array based on the status value and split this sorted array where status=0. Kind of depends on how your data is arranged. I haven't used 8.5 enough yet to know if it has any better ways of doing this now.
 
Not sure about putting images inside the note, I usually just post them as an attachment.
0 Kudos
Message 6 of 13
(3,120 Views)

Thanks, many good ideas there! This should be fun!

I this project I start with an array, around 50 columns and maybe 20000 rows. I want to end with an array also, with maybe 90% as many rows.

I first made queue for the good data, for the fun of it. It's ok to fill the data in the queue, but the consumer loop never stops. In one of the LV examples they destroy the queue when all data is put in, but that gives immediate errors in the consumer loop. I try to look at "queue status" to find the number of elements in the queue, if it is zero and my flag "all_data_put_in" is set it should terminate, maybe I've done some simple error.

To go from queue to array also needs to be done in a good way, not build array again...

The approach to make 1D array of cluster with [status; all_data] as elements and then sort them seems good, but how do I make this array from the original? I think looping through and manually building it should be inefficient.

Ola

 

 

0 Kudos
Message 7 of 13
(3,101 Views)
Post your data file and someone will come up with a solution.
0 Kudos
Message 8 of 13
(3,093 Views)
I found a simple and fast way to remove rows from an array, where status column = 0. Instead of 'building' a new array, I start with an array filled with zeros and put in the new rows by 'replace array'.  In the end i have to remove the extra rows.
The time when down from 2 minutes to about a second!
See attachment.
 
Ola
0 Kudos
Message 9 of 13
(3,069 Views)
Hi Ola,

so you learned the common way to work  with arrays (in a fast way) by avoiding data copies/memory allocation in each loop iteration! Smiley Happy
Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
0 Kudos
Message 10 of 13
(3,059 Views)