formatting cluster into string (performance issue with open G varaint library)

[Olaf_Stetzer]

Hi all,

I have a labview program (running either on compact fieldpoint or cRIO) which samples data. This data is stored in a large cluster (around 30-40 elements) internally but
for communication with the user interface and for data logging the cluster needs to be formatted into strings at some point. For some reason this has to be done in the
time critical part of the software that runs on the real-time target, so performance is an issue.

I am currently using the open G variant library to do the formatting but this seems to be a very ressource hungry solution. For this reason I am looking for a way to improve - performance-wise -
this sub-routine.
One hint: The cluster may change with new versions of the sorftware, but it is only one cluster which obviously does not change within one runtime of the software - so a possibly time expensive
analysis of the cluster needs theoretically be done only once, but I am afraid it is done once per second - each time the data is logged.

Among the longest (VI time in performance monitor) are the following routines:
Parse String with TDs__ogtk.vi
Variant to Header Info___ogtk.vi
Get TDEnum from Data__ogtk.vi
Split Cluster TD__ogtk.vi

The vi and an image of the block diagram are attached. The data cluster in the outer program is wired directly to the varaint cluster and therefore automatically typecast (red point at connection).

Cheers,

Olaf

[tst]

Since you're using 8.2, you might wish to look at MGI's read\write anything VIs. They work like the OpenG variant config VIs, but use functionality which was added in LV 8.0 and which makes this much faster and much less resource intensive. I'm not sure if the code works on RT targets, but it might.

[Olaf_Stetzer]

---

@tst wrote:
Since you're using 8.2, you might wish to look at MGI's read\write anything VIs. They work like the OpenG variant config VIs, but use functionality which was added in LV 8.0 and which makes this much faster and much less resource intensive. I'm not sure if the code works on RT targets, but it might.

---

Hi,

thanks for the suggestion. I have downloaded the MGI library and looked at the vi's you suggested. Unfortunately, I need the steps for formatting and writing to a file seperated, and I alos need one tab-separated line per cluster (without element names, that would be too much oberhead for 30-40 values per second to log!)

I also forgot to mention that some cluster elements are floating point values with some without physical units and there are also binary values. I do not have subclusters
though.

Cheers,

Olaf

[tst]

Since both the OpenG and the MGI tools are open source, you can just open the VIs and try to figure out how to change them to do what you want.

---

Olaf Stetzer wrote:

For some reason this has to be done in the time critical part of the software

---

What's the reason? This is definitely the source of your problems, because TC code should not format data for display and for logging. That should be done in lower priority threads and less frequent. You can transfer the data using a queue.

[Olaf_Stetzer]

---

@tst wrote:

Since both the OpenG and the MGI tools are open source, you can just open the VIs and try to figure out how to change them to do what you want.

---

@Olaf Stetzer wrote:

For some reason this has to be done in the time critical part of the software

---

What's the reason? This is definitely the source of your problems, because TC code should not format data for display and for logging. That should be done in lower priority threads and less frequent. You can transfer the data using a queue.

---

For some part, the reason is historical. We have two TC loops and one non TC loop. The latter corresponds with the TC loops via RT FIFO's. Since there is data and mesages to be sent, it was decided to communicate via strings and use a custom RT FIFO that uses strings. Maybe not the best solution but it would be a lot of work to change this. If I go with your suggestion we would need even more FIFO's or QUEUE's to communicate between the different loops (we already have 4 currently) because we would need one per datatype.

I am not happy with the current solution but I have only littel time before we go on an expedition where everything has to work. And it currently works, just a bit too slow....

Olaf

[tst]

How about flattening the cluster to a string and unflattening it on the other side and then converting it for display?

Flattening isn't the best method either when doing TC code, because it also has to play with memory allocation, but it's MUCH faster than the other options.

[Olaf_Stetzer]

---

@Olaf Stetzer wrote:

---

@tst wrote:
Since you're using 8.2, you might wish to look at MGI's read\write anything VIs. They work like the OpenG variant config VIs, but use functionality which was added in LV 8.0 and which makes this much faster and much less resource intensive. I'm not sure if the code works on RT targets, but it might.

---

Hi,

thanks for the suggestion. I have downloaded the MGI library and looked at the vi's you suggested. Unfortunately, I need the steps for formatting and writing to a file seperated, and I alos need one tab-separated line per cluster (without element names, that would be too much oberhead for 30-40 values per second to log!)

I also forgot to mention that some cluster elements are floating point values with some without physical units and there are also binary values. I do not have subclusters
though.

Cheers,

Olaf

---

So I used the MGI write anything as a template for a new version of my formatting routine (I removed the QUEUE and therefore also the nested cluster/array functionality). When using the same format string (this makes a big difference in performance I realized), the old opne G routine needed 2.6 seconds for 1000 cycles on my host PC and the "new" one 3.3, so the old version is still faster. This is for the host PC I did not yet check the performance on the RT target but I guess the tendency will be the same.

Olaf

[tst]

I can't comment on what you've experienced, but I can say that I recently ran into trouble with the OpenG VIs taking up a lot of CPU and taking a long time to run with a complex cluster with nested arrays and clusters (in LabVIEW 7.0).

Testing it on the same computer in 8.5.1 gave the following results:

OpenG 7.0 - ~17 secs, lots of CPU.

OpenG 8.5.1 - ~9 secs, lots of CPU.

MGI 8.5.1 - ~0.5 secs, almost no CPU.

This includes writing to the file, which in the OpenG code is done by the inefficient NI VIs. I suppose it's possible that part took a lot of the processor time, so I can try to change it just to see how it would work. Thanks for the idea.

[tst]

---

tst wrote:

This includes writing to the file, which in the OpenG code is done by the inefficient NI VIs. I suppose it's possible that part took a lot of the processor time

---

I did a quick test. I went into NI's VIs and disabled the piece of code which actually does all the stuff. This caused the OpenG code to run in ~0.5 sec, showing that it is indeed NI's VIs which are the problem.

After looking into it some more, I realized that while the MGI VIs use functionality which was added in 8.0, that functionality already exists in the OpenG code. I did a quick test of opening up the MGI VIs and replacing the built-in VIs with the OpenG VIs. This basically worked, although the OpenG VIs are not as efficient. It also took about 0.5 sec, but it took more CPU. Obviously, to do it properly, the code needs to be written more carefully.