Storage and processing of large volumes of data

[daisy_11]

Dear all

I currently need to store the data in FGV for subsequent use. The data per array in FGV is 900,000, and the reading and writing speed of FGV is too slow. May I ask how to optimize or do you have any better Suggestions?

[RavensFan]

A picture of a VI doesn't  help us help you.  Certainly not one that doesn't even show the file writing part of the code.

Please attach an actual VI.  Perhaps even a data file.

When you say "too slow", how slow is it?    How fast do you want it to be?

[altenbach]

It is inefficient to place connected terminals inside structures. they belong on the toplevel diagram (details). You could also use feedback nodes, eliminating the FOR loop.

Is debugging disabled and front panel of the FGV closed when you test?

What else is going on? What other parts of the code access the FGV and how often? How is it used? How do you determine performance? What's in the other cases? We can't really debug pictures!

[daisy_11]

Each array has a data size of 900,000, and the time it takes to write and read is between the 30-50s, so hopefully it can read and write faster

[daisy_11]

Just call this FGV in another VI and turn off the debug and FGV front panel while using. Since each array has data of 90000000, FGV reads and writes are slow

[altenbach]

---

@daisy_11 wrote:

 Since each array has data of 90000000, FGV reads and writes are slow

---

Did you just add two more zeroes to that number??? (that would require 64bit LabVIEW!)

[altenbach]

In my testing, reading 3x 900k data from the FGV takes less than 10ms. 

(Why do you even need the init state?)

[daisy_11]

Init is to make sure that the data is up to date The data should be 3X90000000. I typed it wrong the first time

[daisy_11]

The data should be 90000000. I typed it wrong the first time

[altenbach]

That's a lot of data!!! (>2GB per data copy!) Do you have LabVIEW 64bit? How much RAM do you have? What speed would seem reasonable to you?

This is only the "storage". What kind of "processing" are you planning to do?

What does the data represent and why do you need to handle it all at once?

Maybe a FGV is not the ideal tool here.