Why table control is working so slowly with array of 70000 rows

[gigi85]

transform your array to string at the beggining, manipulate your data in a string form and then re-transform your string to array 🙂

[RavensFan]

---

@gigi85 wrote:

PS: don't use array, only string...and then before the output transform the string in array (spreadshet to array block)...let me know if it will be better 🙂

---

Why would you think this would help anything? If you have a lot of data, it is going to take up pretty much the same amount of space whether you have it defined in an array or defined in a string.  Bytes are bytes.  Doing the transformation from an array to a string and back again is going to be worse because now you are making multiple copies of a large dataset that are just organized differently.

To the original poster:

You may want to read this

Managing Large Data Sets in LabVIEW

I'm confused as to why you think the code you posted does not include local variables when it is quite obvious just by opening it there are local variables all over the place, just as Gerd says.

[Current_93]

---

@GerdW wrote:

Hi current,

sorry to say, but your VIs use way too much sequences and duplicate code.

And I do find several locals, not all are needed.

- Do you really need SharedVariables?

- When providing a ZIP with a handful of VIs you should give the name of the main VI instead of let us search for it...

- clean up your code (like put duplicate code sequences in subVIs, merge property nodes of the same control instead of using several of them, merge IndexArray nodes when accessing the same array, a lot more) and we will have a look at it again...

---

Dear GerdW

I like SharedVariables really, because they are very easy to work for me and I need them. Good for you I clean up my code as you suggest. But but really can change the order of the 800 MB number ?  I think not.

At the present time I care about how deallocating these 700-800 MB after closing the form with my code.  How to do the deallocating in this case??

 

 

 

[GerdW]

Hi current,

using SharedVariables is kind of overkill when you only need them inside a simple VI hierarchy in one project on one computer. "Regular" global variables will work then too...

Deallocation of memory is done as soon as you remove that VI from memory. You may request a deallocation by calling that function...

More hints:

- Also FP indicators use their own buffers for the data. Keeping your big array in an indicator may create another copy of all the data in memory...

- Stay with shift registers to keep the data in one place (as good as possible)...

- Clean up your VI...

[Ben]

---

@GerdW wrote:

...Deallocation of memory is done as soon as you remove that VI from memory. You may request a deallocation by calling that function...

...

---

Request is the operative word in that sentance.

You can requst it but LV will only release it after the VI is removed from memory.

DIsplaying large data sets is a bit of a challenge in LV becuase it will make some assumptions that will back-fire unless you approach this challenge carefully.

There are two (actually more) ways to minimize how much memory is used but we have to be aware of where and when LV copies memory buffers. In this case you have to be aware of almost all of the rules to keep the memory useage low.

Since DVR's require additional thought to work with I will suggest using an Action Engine to provide any storage you need and it will also let you focus on the perfromance of a single VI rahter than playing "Wack-a-mole" trying to kill buffer copies in multiple VIs.

Challenges

1) LV will not release memory for a buffer once allcated unless you request deallocate and the VI is removed from memory.

You can use VI server to present the screen the user selects from. When the user is done the screen can be closed and closed so the memory will be freed up.

2) Locals and globals reuire duplicates of the buffer. This allows for LV multithreading to to schedule any computable code at anytime. In this situation avaoid locals and globals like they are the plague.

3) Wack-a-mole is my code speak for the work eliminating buffer allocations dots you see when doing Porfile >>> Show buffer allocations". Use "inplaceness" operators to elelimante data copies.

4) "Clear as mud" therad talks about the proper way to wire data into and out of a sub-VI. Put the terminals on the root of the sub-VI diagram and avoid functions that do not work in-place.

5) Put all file related functions in an Action Engine so the data goes from file to SR and then to the terminal used to display the data. Important, use a terminal not a local.

6) Gird up your loins - All of that work may get it inot the display with minimal memory but having that many entries in the display could end up being very slow or sluggish when scrolling. For this part of the game, I have resorted to "lying" to the user and writing my own table that LOOKS like it had more data but I kept the data elsewhere and only updated the part of the data they where viewing at the moment.

So ...

If you stick to only the easy LV functions you will abuse memory.

In the same way you have to learn how C++ fucntions work we have to learn how LV works to be efficient.

Ben

PS: Serach this site for any terms I used above that you may not understand. I have writien on all of them extensively in other posts. 

[Mark_Yedinak]

---

@Current_93 wrote:

---

I like SharedVariables really, because they are very easy to work for me and I need them. Good for you I clean up my code as you suggest. But but really can change the order of the 800 MB number ?  I think not.

At the present time I care about how deallocating these 700-800 MB after closing the form with my code.  How to do the deallocating in this case?

---

This will eventually cause you problems. Over use and inapprorpriate use of variables, be they shared, local or global, break data flow and will lead to race conditions as well as excessive copies of your data. You would do very well for yourself to learn how to use dataflow properly and avoid the use of variables in your code.

[Current_93]

Thank you Mark!

Then why none of the NI team are not in the manual says about this horrible properties of local variables? Would write about it immediately that for a 24/7 mode can not use local variables. 

I will accept your suggestions on the continuous dataflows and in the following projects will be attentive to this.

[Current_93]

Dear Ben thank you for your detailed answer

I do not quite understand about "Action Engine" and "therad talks" 

[P@Anand]

You can take a look at here for Action Engine