From Friday, April 19th (11:00 PM CDT) through Saturday, April 20th (2:00 PM CDT), 2024, ni.com will undergo system upgrades that may result in temporary service interruption.
We appreciate your patience as we improve our online experience.
From Friday, April 19th (11:00 PM CDT) through Saturday, April 20th (2:00 PM CDT), 2024, ni.com will undergo system upgrades that may result in temporary service interruption.
We appreciate your patience as we improve our online experience.
09-11-2018 01:44 PM
Something that might help.... Microsoft recommends "If you use large numbers of files in an NTFS folder (300,000 or more), disable short-file name generation for better performance, and especially if the first six characters of the long file names are similar.
"
Of course a SSD could help too.
However, I will also recommend you reconsider your approach.
09-11-2018 01:48 PM
Either use far fewer files with lots of data in them or use a database. The OS will choke once you start creating that many files. Not to mention you are wasting a TON of disk space. A very small file will use a 8192 bytes of disk space. In fact, all files will be broken 8192 byte blocks when saved on disk. So if you have 32 single byte files you will be using 256 KBytes of disk space to store them.
09-12-2018 03:13 AM
Also review if you really need to save all those data points. Often, it can be enough to save analyzed data, e.g. min/max or a subset representing e.g. a detected peak.
And if you really need all that data saved, I suggest using tdms-file(s). You can tag your data and storing to them even when they get huge is no problem.
But, it is useless to store lots of data if you do not have any means of searching and viewing/extracting it in a meaningful way. By date-stamping and tagging, that is at least a lot easier, and that can be achieved using tdms-files.
09-12-2018 03:30 AM
I agree with most above, but if you really want to go with 1m small files, store 1000 in 1000 folders. It's a simple Quotient and remainder to calculate which folder and name should be used.
/Y