Recursive File Listing is miserably slow for large directories

[id]

I am using the advanced file IO palette File Recursive.VI that comes with Labview to obtina an array of file paths to load and search thru with my program.

The directory contains hundreds of files and takes 15 minutes to run this VI thru.  Is there a better more faster way of doing this operation in Labview? 

[smercurio_fc]

Are you sure it's the Recursive File List VI that's using up all that time, and not something else, like populating a tree control? As a comparison, on one directory I have that has almost 12,000 file and over 1400 folder the Recursive File List runs for a little less than one minute.

[id]

This VI took >5 minutes to list a single folder dir with 19,000 files in it.  I am using LV 8.1.

[Darren]

Why are you using Recursive File List.vi to list a flat directory?  Seems like all you would need in that case would be the List Folder function.

Nevertheless, I tried this on my machine (2.4 GHz quad core, 4 GB RAM, Windows 7 64-bit).  I had a single folder with 20,000 files in it.  After a reboot, the first run of your VI took 67 seconds to list all the files in the folder.  Subsequent runs took about 2 seconds.  The reason it takes so long on the first run is disk caching.   So if none of those files have been accessed by the operating system (which none of them had since I rebooted), the first folder listing is going to take much longer than others.  There is nothing LabVIEW, or any other program, can do about operating system disk caching.

But if you're seeing consistently long list times, even on non-initial runs, then here are a couple of ideas to think about:

• Is another program (virus scanner, e.g.) using system resources and/or accessing your files?  Or are other VIs running when you try this?
• Are these files on a network drive?  Or is there heavy network traffic that might be bogging down your system?
• Do you have the front panel of Recursive File List.vi open? 

That's all I can come up with off the top of my head.  With normal use, the Recursive File List VI should perform much better than you describe.

[Ben]

---

Darren wrote:

Why are you using Recursive File List.vi to list a flat directory?  Seems like all you would need in that case would be the List Folder function.

Nevertheless, I tried this on my machine (2.4 GHz quad core, 4 GB RAM, Windows 7 64-bit).  I had a single folder with 20,000 files in it.  After a reboot, the first run of your VI took 67 seconds to list all the files in the folder.  Subsequent runs took about 2 seconds.  The reason it takes so long on the first run is disk caching.   So if none of those files have been accessed by the operating system (which none of them had since I rebooted), the first folder listing is going to take much longer than others.  There is nothing LabVIEW, or any other program, can do about operating system disk caching.

But if you're seeing consistently long list times, even on non-initial runs, then here are a couple of ideas to think about:

• Is another program (virus scanner, e.g.) using system resources and/or accessing your files?  Or are other VIs running when you try this?
• Are these files on a network drive?  Or is there heavy network traffic that might be bogging down your system?
• Do you have the front panel of Recursive File List.vi open? 

That's all I can come up with off the top of my head.  With normal use, the Recursive File List VI should perform much better than you describe.

---

And you ruled out building arrays in a loop because... ?

Turning on the Disk Indexing* option can help with speed.

If there are any protected files in there you could be waiting on the system to make a log entry.

Ben

* nine years as a disk specialist for DEc when I had access to the OS source code let me cheat a little.

Message Edited by Ben on 04-23-2010 03:05 PM

[Darren]

---

Ben wrote:

And you ruled out building arrays in a loop because... ?

---

...because the non-initial runs took 2 seconds.

[id]

Ben 

And you ruled out building arrays in a loop because... ?  What do you mean by this?  I am merely using the list folder recursively VI by NI to do a dir listing..

Turning on the Disk Indexing* option can help with speed.  How do you do this with a Win XP on a Dell Laptop internal drive?

[id]

Darren,

Why are you using Recursive File List.vi to list a flat directory?   To provide a simple means of discussing this problem.  I have a multifolder directory structure and this is one of the sub folders that  I am trying to list.

• Is another program (virus scanner, e.g.) using system resources and/or accessing your files?  Or are other VIs running when you try this?  Yes, the antivirus sw is running in the background.
• Are these files on a network drive?  No, not in this instance Or is there heavy network traffic that might be bogging down your system?
• Do you have the front panel of Recursive File List.vi open? No

Would low hard drive free space be an issue here?  

[RavensFan]

---

id wrote:

Ben 

And you ruled out building arrays in a loop because... ?  What do you mean by this?  I am merely using the list folder recursively VI by NI to do a dir listing..

---

I may be mistaken, but Ben's comment on this was directed to Darren.

[Ben]

Windows explorer right-click C: and choose properties >>> check the "Allow Indexing Service ..."

Defrag your drive to see it excesive fragmentation is the issue.

Ben