LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Can i optimize write of large data array into text file ?

Highlighted

Thanks all for your answers and your explanations

I've modified my VI using Mythilt and mcduff examples.

By writting block into my file, i divided my time about 4-5. That's very helpfull for my application...and the user who will to use it 🙂

 

I've also tried the example of Bob_Schor

 

0 Kudos
Message 11 of 23
(821 Views)
Highlighted

I am just courious: what is the reason you need to use ASCII file? At such sizes a binary (TDMS very useful) file is much simpler to handle, and also to read the data out of it later on... Of course if you have a special reason, like the file needs to be read by another software which you cannot modify...?

0 Kudos
Message 12 of 23
(818 Views)
Highlighted

The point of my "example" was to test an algorithm using a specific example.  If it flat-out failed, you stop there.  If it looks promising ("it is also very fast"), then you spend a little bit of time making it "generalizable", i.e. adding the bells and whistles.

 

Bob Schor

Message 13 of 23
(814 Views)
Highlighted

Hi,

 

@Bob_Schor

Don't misunderstood me, i didn't say your example is not good. I've tried it with a few tests.

However, i can't have so much time to modify it to be more used with various size of data array. Well, by 'no much time', i mean that with my skills in LV it will take me some time that i don't really have. 

But i don't forget you help me, and i really appreciate it, and i don't forget your example, because maybe i will come back later on it, if i need.

 

@blokk

You're right. In other applications i've made, i used TDMS file, it is much quicker and easier to handle. However, in my current application, the file is wrote by LV but it used (read) by another application (in fact a Perl script).

 

0 Kudos
Message 14 of 23
(811 Views)
Highlighted

Any computer language should use binary file ops at such file sizes, much faster. You can read tdms from PERL too...

 

http://digital.ni.com/public.nsf/allkb/A3663DE39D6A2C5A86257204005C11CA

Another way is to create your own binary file structure, and then create a reader function at the PERL side. This should be easy to implement using the LV low level binary file functions. 

0 Kudos
Message 15 of 23
(808 Views)
Highlighted

Ok, I just realised PERL is just a script language...so maybe in overall you are stucked with ASCII files...

0 Kudos
Message 16 of 23
(801 Views)
Highlighted

Hi

 

Well, i'm not sure i'm stucked with ASCII file with PERL. It is possible to read binary file with this language.

 

However, i've not been involved in the creation of this script, i'm only using it. It requires input text file, i need to write ASCII file.

 

As i say earlier, it will be possible to change this mechanism, but my LV application is for a customer project in progress, and mostly based on reuse part. 

So i can't modify everything i want, or everything it will be. I need to adapt my appliction to those parts.

 

But, maybe for another use, in another project i try to update the Perl script to read binary, if i gain disk space and reduce the time to read/write the file.

0 Kudos
Message 17 of 23
(790 Views)
Highlighted

In the interest of keeping it fast. Here is one that takes 17s compared to 42s (mcduff's version) on my machine.

WriteToFile.png


CLA CTAChampion
Subscribe to our Test Automation user group: UK Test Automation Group
Message 18 of 23
(784 Views)
Highlighted

Nice work!!

 

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

 

Lastly, the speed comes at a price of more memory, but if thatmay not be a problem for the OP.

 

Cheers,

mcduff

0 Kudos
Message 19 of 23
(768 Views)
Highlighted

@mcduff wrote:

 

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

 


I would expect that the right order is not guaranteed using a parallel FOR loop.


LabVIEW Champion. It all comes together in GCentral GCentral
What does "Engineering Redefined" mean??
0 Kudos
Message 20 of 23
(763 Views)