From Friday, April 19th (11:00 PM CDT) through Saturday, April 20th (2:00 PM CDT), 2024, ni.com will undergo system upgrades that may result in temporary service interruption.

We appreciate your patience as we improve our online experience.

LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

creating 1 million text files in labview

Something that might help.... Microsoft recommends "If you use large numbers of files in an NTFS folder (300,000 or more), disable short-file name generation for better performance, and especially if the first six characters of the long file names are similar."

Of course a SSD could help too.

However, I will also recommend you reconsider your approach.

---------------------------------------------
Certified LabVIEW Developer (CLD)
There are two ways to tell somebody thanks: Kudos and Marked Solutions
0 Kudos
Message 11 of 14
(433 Views)

Either use far fewer files with lots of data in them or use a database. The OS will choke once you start creating that many files. Not to mention you are wasting a TON of disk space. A very small file will use a 8192 bytes of disk space. In fact, all files will be broken 8192 byte blocks when saved on disk. So if you have 32 single byte files you will be using 256 KBytes of disk space to store them.



Mark Yedinak
Certified LabVIEW Architect
LabVIEW Champion

"Does anyone know where the love of God goes when the waves turn the minutes to hours?"
Wreck of the Edmund Fitzgerald - Gordon Lightfoot
0 Kudos
Message 12 of 14
(432 Views)

Also review if you really need to save all those data points. Often, it can be enough to save analyzed data, e.g. min/max or a subset representing e.g. a detected peak.

 

And if you really need all that data saved, I suggest using tdms-file(s). You can tag your data and storing to them even when they get huge is no problem.

 

But, it is useless to store lots of data if you do not have any means of searching and viewing/extracting it in a meaningful way. By date-stamping and tagging, that is at least a lot easier, and that can be achieved using tdms-files.

Certified LabVIEW Architect
Message 13 of 14
(392 Views)

I agree with most above, but if you really want to go with 1m small files, store 1000 in 1000 folders. It's a simple Quotient and remainder to calculate which folder and name should be used.

/Y

G# - Award winning reference based OOP for LV, for free! - Qestit VIPM GitHub

Qestit Systems
Certified-LabVIEW-Developer
0 Kudos
Message 14 of 14
(388 Views)