creating 1 million text files in labview

[lm35]

Dear Sir/Mam, i am trying to read data from serial port and saving it to a text file, i have to create around million of files in a month but my vi is only able to generate around 30,000 to 40000 files and after that the system goes slow, the main problen is that the vi takes a longertime to create a txt file, so when i have a large data sets the vi becomes slow  and when i change the folder location it start to work normal. can any one help me in this regard?

thanks

[pincpanter]

You almost answered yourself. 1 million files per month means about one file every 2 seconds.

Such a huge number of files is unmanageable. You should try to cluster data together, for example generating one file per hour. Into the file, different acquisitions would be properly separated by a "marker" line, reporting e.g. the acquisition start time.

If you really want to save one file per acquisition, separate groups of file into different folders. 1 folder per day would mean about 40000 files in one directory, which in my opinion is still too much.

[Mark_Yedinak]

Or consider using a database. One million files a month would b every manageable. How will you analyze the data? How will user's try to work with that many files. Either try to store fewer files containing more data or use a database.

[altenbach]

If you know that the Folder starts out empty and each file name is incremented by one, generate the file name yourself, keeping track of the current number.  You can use a file name format e.g. as "FILE%010d.dat" or "FILE%010x.dat" or similar. Even if the folder already has some files, it would be trivial to determine the highest suffix once before the program starts and then keep incrementing from there. Alternatively, you could e.g. format the current time stamp into a filename (guaranteed to be unique and sorted, if done correctly).

The tool "Create file name with incrementing suffix" Is the only thing that is holding you back, because with every call it does a "list folder" command, which gets more and more expensive as the number of files increases. Listing tens of thousands of file names to sort out what number comes next is not a trivial operation the way it is implemented. (You can open it to see what it does!).

Still, as the others above, I question the sanity of this approach overall. There are probably better ways to do all that than using the file system to segment data into tiny aliquots.

[RTSLVU]

---

@lm35 wrote:

 the main problen is that the vi takes a longertime to create a txt file, so when i have a large data sets the vi becomes slow  and when i change the folder location it start to work normal. can any one help me in this regard?

thanks

---

Working with 1,000,000 files aside...

Is it the file creation or the data transmission that is taking too long? 

IF it is actually the file creation that is slowing everything down. I suggest using a Producer/Consumer architecture. The producer will collect the data and  pass it to the consumer,  that is now decoupled from the acquisition loop, so it can take its own sweet time creating files.

[altenbach]

---

@RTSLVU wrote:

---

@lm35Is it the file creation or the data transmission that is taking too long? 

---

As I have explained, the only thing slowing things down is the use of "Create file name with incrementing suffix", which does a "list folder" with each call, parsing all existing file names to determine the next suffix. This is the only operation that increases in cost with the number of existing files. The cost of all other operations is constant.

[crossrulz]

---

@altenbach wrote:

The tool "Create file name with incrementing suffix" Is the only thing that is holding you back, because with every call it does a "list folder" command, which gets more and more expensive as the number of files increases. Listing tens of thousands of file names to sort out what number comes next is not a trivial operation the way it is implemented. (You can open it to see what it does!).

---

I haven't even looked at the code and this is instantly what I thought the problem would be.  But agree with everybody else, the shear number of files is insane.  I will side with Mark here: you really should be using a database.

[altenbach]

Even the OS is probably not very happy with so many files in one folder. Just opening an explorer window, potentially sorting, showing the correct icon, etc. Is a big job. Same for the search indexer.

[lm35]

@altenbach

i tried using filename but it generated around 90,000 files in 3 days and after that the system become too slow, windows os was creating trouble and there was a huge data loss because the file generation(create file with increment suffix) was taking a longer time, even i tried to split the files and folder that didn't work because it was even taking a longer time. the main problem is the create file command, as the no files increases the create file command takes longer time and so on...

[Frozen]

As explained others and by yourself, the speed issue is with the OS, not the program. Windows has speed problems with large amounts of files in one directory.