Capture and view large size data

[sumsu]

Hi all,

I'm using LabView 2012 version and i need help about large size data storage and viewing by Table. According to my application i'm collecting 11 different data (32 Bit int) in each sample with 20 Hz by RS232 protocol. I mean that i have 20x11 matrix in a second. While i'm just listening the RS232 port i can capture all data without any lossless. But if i want to write these data to Shift register and show all data by Table in each loop i've a lot of data losses.

According to my application i need to collect these data during 45 minutes and it means that i must capture 54000x11 data. Do you recommend me some solution to use instead of Shift register?

[GerdW]

Hi sumsu,

you should use shift registers as they are very efficient!

But DON'T use BuildArray within your loop.

Instead you need to initialize the array to the needed size before your loop and use ReplaceArraySubset inside the loop!

Other simple solution: save the data to a file instead of collecting it in a big (and growing) array!

[Bob_Schor]

Two other possibilities:

• If you don't need to view the data coming in, simply bring it out of your loop in an Indexing Tunnel (or Tunnels, since you seem to have 2D data).
• Use a Producer/Consumer architecture and "let the Queue be your Buffer".

Bob Schor

[Mark_Yedinak]

If you do need to display the data I would definitely go with a producer/consumer architecture. I very rarely have my data acquisition loops display or process the data. I generally post the data to a queue and let another loop actually process the and display the data. 

[sumsu]

Thank you my friends,

I'll try "ReplaceArraySubset" and "Queue" solutions that you recommended me. After that i'll share my results and observations with you.

[sumsu]

Hi,

Before apply to my test bench i used "ReplaceArraySubset" blocks in trial versions. I generated 10 different random numbers instead of data from RS232 protocol. 11st data indicates index on the left of table in Front Panels.

As you see from Front Panels if i place Table in the loop i could capture 34811 data in 2700 seconds. On the other hand if i place Table out of the loop i could capture 41004 data in 1350 seconds (half of 2700 seconds). 

I think the main problem is, to indicate all data on Table. If i don't want to indicate data everything is good, but when i want to see all data i cannot capture all of them. 

Could you recommend me something about this? 

[GerdW]

Hi sumsu,

why do yu think you need to build so many (sub)arrays when all you need is the one big predefined array?

Like this:

Defining an array of 50k rows and 11 columns before the loop.

Creating an array of 11 "random" measurement values.

Replacing a row in the big array.

[sumsu]

Hi GerdW,

I'm generating 10 different data since seperated data coming from RS232 protocol. I recommend you that place Table in and out of your loop. Then you'll see difference between both of them. I found the main problem is show all data on the Table.

Best regards,

sumsu

[GerdW]

Hi sumsu,

I updated my snippet to reflect your comments. I don't see any "main problem" here…

When this doesn't solve your problem you need to explain it better!

[Ben]

---

@sumsu wrote:

Hi GerdW,

I'm generating 10 different data since seperated data coming from RS232 protocol. I recommend you that place Table in and out of your loop. Then you'll see difference between both of them. I found the main problem is show all data on the Table.

Best regards,

sumsu

---

Of course it will take longer if you are updating the table every time you get more data that if you collected the data and THEN posted it to the table.

If you want to display massive amounts of data as it comes in then switch over to a producer/Consumer architecture as was mentioned early in this thread.

Beyond that

You may get a performance improvement if you Defer Front Panel Updates prior to the GUI update

AND

Reduce how often you are updating the table since the user can only see about 30 updates a second anyway.

Ben