Extremely large files

[Camu]

Hello, guys, how do you save and read extremely large files (e.g. >10 million data points), without overusing the physical memory and slowing down the program? Currently I'm using "write to/read binary file" but only below 5 million data points it works well.

Thank you very much.

best

Camu

[altenbach]

What happens if it "doesn't work well"?

10 million data points are not that much....

What is the datatype of your data (U8, extended precision complex, etc.).

What is your OS? Can you show us some code?

[Camu]

altenbach

What happens if it "doesn't work well"?  10 million data points are not that much.... What is the datatype of your data (U8, extended precision complex, etc.). What is your OS? Can you show us some code? Hi, Altenbach, the memory will corrupt when i'm trying to read a binary file containing more than about 5 million DB data points (time information in ps) to build a histogram. I'm using Labview 8.2 on a computer with Xp and a 2GB memory. thank you for answering. best Camu

[TCPlomp]

This is not a reliable way to store binary data, use the LabVIEW binary VIs instead.

Or even better go for TDMS.
As Altenbach has mentioned 10 Million points isn't that much.

Ton

[altenbach]

Please provide more information on how you do things (saving, histogram, etc). Do you use express VIs?

The attached (downconverted from 8.5.1, I hope it still works ;)) generates 10M random DBL date in the range of 0..100 and then calculates a simple hinstogram.

1. Directly
2. After saving to a binary file and reading it back.

I always get matching results.

[altenbach]

---

TonP wrote:
This is not a reliable way to store binary data...

---

Did you actually figure out what "this" is here? 😄

[TCPlomp]

---

@altenbach wrote:

Did you actually figure out what "this" is here? 😄

---

Me or Camu?
Anyway, using the Get characters from string for this function isn't right I believe honestly.

Ton

[Camu]

HI, Altenbach, Thanks very much for providing this example.

It works really well. However, when I want to generate more DBL data (20M), then the computer slows quite a bit down and the memory almost corrupt (the computer I'm running the example vi has only 1GB memory). I think there is a limit for indexing at a for loop.

I will attach the details of my programs quite soon.

best

camu

[altenbach]

---

@TonP wrote:

---

@altenbach wrote:

Did you actually figure out what "this" is here? 😄

---

Me or Camu?
Anyway, using the Get characters from string for this function isn't right I believe honestly.

---

LOL, I meant you. I must be blind then, because all I could see in the question was "Currently I'm using "write to/read binary file" ".

Where did you see the use of "Get characters from string "?

[TCPlomp]

I openend up the Read routine  and it uses the 'Read Characters from file' routine which get typecasted. This is IMHO a very unreliable way to read binary data.

Ton