LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Can i optimize write of large data array into text file ?

Thanks all for your answers and your explanations

I've modified my VI using Mythilt and mcduff examples.

By writting block into my file, i divided my time about 4-5. That's very helpfull for my application...and the user who will to use it Smiley Happy

 

I've also tried the example of Bob_Schor

 

0 Kudos
Message 11 of 23
(747 Views)

I am just courious: what is the reason you need to use ASCII file? At such sizes a binary (TDMS very useful) file is much simpler to handle, and also to read the data out of it later on... Of course if you have a special reason, like the file needs to be read by another software which you cannot modify...?

0 Kudos
Message 12 of 23
(744 Views)

The point of my "example" was to test an algorithm using a specific example.  If it flat-out failed, you stop there.  If it looks promising ("it is also very fast"), then you spend a little bit of time making it "generalizable", i.e. adding the bells and whistles.

 

Bob Schor

Message 13 of 23
(740 Views)

Hi,

 

@Bob_Schor

Don't misunderstood me, i didn't say your example is not good. I've tried it with a few tests.

However, i can't have so much time to modify it to be more used with various size of data array. Well, by 'no much time', i mean that with my skills in LV it will take me some time that i don't really have. 

But i don't forget you help me, and i really appreciate it, and i don't forget your example, because maybe i will come back later on it, if i need.

 

@blokk

You're right. In other applications i've made, i used TDMS file, it is much quicker and easier to handle. However, in my current application, the file is wrote by LV but it used (read) by another application (in fact a Perl script).

 

0 Kudos
Message 14 of 23
(737 Views)

Any computer language should use binary file ops at such file sizes, much faster. You can read tdms from PERL too...

 

http://digital.ni.com/public.nsf/allkb/A3663DE39D6A2C5A86257204005C11CA

Another way is to create your own binary file structure, and then create a reader function at the PERL side. This should be easy to implement using the LV low level binary file functions. 

0 Kudos
Message 15 of 23
(734 Views)

Ok, I just realised PERL is just a script language...so maybe in overall you are stucked with ASCII files...

0 Kudos
Message 16 of 23
(727 Views)

Hi

 

Well, i'm not sure i'm stucked with ASCII file with PERL. It is possible to read binary file with this language.

 

However, i've not been involved in the creation of this script, i'm only using it. It requires input text file, i need to write ASCII file.

 

As i say earlier, it will be possible to change this mechanism, but my LV application is for a customer project in progress, and mostly based on reuse part. 

So i can't modify everything i want, or everything it will be. I need to adapt my appliction to those parts.

 

But, maybe for another use, in another project i try to update the Perl script to read binary, if i gain disk space and reduce the time to read/write the file.

0 Kudos
Message 17 of 23
(716 Views)

In the interest of keeping it fast. Here is one that takes 17s compared to 42s (mcduff's version) on my machine.

WriteToFile.png


CLA CTA
Multics UK Ltd
Multics @ NI Week 2019: Learn about Architecting TestStand Framework for Noncoders Mon, May 20th, 3:15 PM - 4:15 PM. Room 19B
Message 18 of 23
(710 Views)

Nice work!!

 

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

 

Lastly, the speed comes at a price of more memory, but if thatmay not be a problem for the OP.

 

Cheers,

mcduff

0 Kudos
Message 19 of 23
(694 Views)

@mcduff wrote:

 

I did not know writing to an ASCII file could be done in parallel. Did you check that everything is written in the right order?

 


I would expect that the right order is not guaranteed using a parallel FOR loop.


LabVIEW Champion Do more with less code and in less time
0 Kudos
Message 20 of 23
(689 Views)