Recursive File Issues

[ccostello4]

Hi,

I have a rather interesting/frustrating problem I have been dealing with that I am hoping someone here might have a solution for me.

I have a system where multiple computers (one computer for each production line - four total lines) are feeding information back to the main computer (control station). I have complete control over the line PCs in terms of network and firewall settings but do not on the control station as it is connected to the clients network (though their IT is willing to make some changes if we need to).

I pass the information from the line PCs to the control station with shared folders hosted on the line PCs. The control station uses the Recursive File function to determine when the correct number of files are in the folder and then outputs the data for an operator to review and approve. The issue we have been seeing is these files can take anywhere from 3-31 seconds since they are produced to when they are ready to processed on the control station. 

I have done tests with my personal laptop though (it has the full development version of labview but I ran the same runtime build as what the control station runs) and the time was 0-3 seconds for the files to be ready to processed.

I have also done testing mocking one of the other line PCs as the control station to see if it was a firewall issue with the clients PC and again it would take 3-31 seconds for the files to be available for processing. I turned off all firewalls on the Line PC and still saw 3-31 second times. 

The general flow of the program (follow tasks done in parallel while loops):

1. File Lookup on the four different lines - data is queued once the correct files are found

2. Review - data is queued once data is approved by the operator

3. Save - data and images are saved to server based system for historical tracking

Does not appear to be a code issue since it does not happen on all PCs I run it on. Does anyone have any ideas what might be causing this delay? I have done test programs using only my filelookup subVi (see attached) and I still see this delay of 3-31 seconds on the control station but still my personnel PC sees it in 0-3 seconds. I ran that test side by side with both of them plugged into the same switch monitoring the same folder. 

Each line has a GigE switch which all local components are connected and then a Cat6 line that runs to the main GigE swtich by the control station. All tests with my personnel laptop where done by connecting to the same GigE switch the control station is connected too. All cable runs are under 100 ft. 

Line PCs and control station - Windows 7 64-bit

Personal Laptop - Windows 10 64-Bit

Labview Version - 2015 V15.0 64-Bit

Please let me know if you need any more information from me. I can not share the whole code but I can change some variable names and share other portions if needed though I do not know what other portions might be helpful. 

My head hurts from banging it against a wall so many times,

Chris

[billko]

Probably something to do with the way networks cache files?

[Bob_Schor]

Some comments:

• I hate "pictures of code", even when nice and clear, as yours are.  I want to ask "what's in the False case"? (which I just realized was in the second picture).
• Do you know about Conditional Indexing Tunnels?  Much simpler than doing the Build Array to assemble your array.
• Why is there a Clear Error function?  Do you expect Delete to throw an Error for a file that you know exists?
• I love Recursion, so was intrigued by your Subject Line, but you are not doing any recursion -- oh, well.

I'm a little confused by the flow of logic.  Your "Line PCs" seem to be acquiring data and saving it to themselves in files.  What are they communicating to the Control Station?  It would seem, to me, very easy for them to maintain a simple count of their files locally and send the count to the Control PC -- why make the Control PC do repetetive Directory Listings on all of the Line PCs when it could just ask "And how many files do you have now?".

Bob Schor

[ccostello4]

Hi Bob,

Yeah I learned about conditional indexing tunnels on Monday so I have gone back and change several portions of my code to use those now instead haha. One of the features that I always wish existed and it turns out it does.

The clear error is there because sometimes the .db file is being used by some background windows tasks and will not delete so it throws up a "file in use - could not be deleted" error.

 The are communicating two images and image inspection back to the control station. We are currently developing a completely automated version of the rejection for the client but in the meantime we have operator approval of the parts being made. Operator reviews the images and the data and makes selections on individual parts.

So as an update I ran a debugging version of the code yesterday to see where exactly the delay is coming from. There were never any long delays in the Recursive File function so it appears to find the folder just fine on the network. I also see the entire file lookup loop run consistently as well (<50 ms). So it appears that files do not "exist" on the network yet. 

So the reason I am not doing that is because it appears that the files can save locally and "exist" locally but not "exist" for network devices yet. So the line could tell the control station the files "exist" and then when the control station goes to look for them they would not be there yet. 

This is looking to be more and more of an issue with the network then with Labview. Anyone know if Windows 10 is able to see network files faster than windows 7 for some reason? Or a way to speed up how fast network files exist?

[ccostello4 · Accepted solution]

Hi,

So after everything we were able to determine what the issue was. It is in the way windows 7 looks at network files vs windows 10. 

Adding the registry key DirectoryCacheLifetime to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters the windows 7 PC seems to have fixed all the issues we were having. I set it to 1 but you can set it to 0 as well if you would like. 

Thanks to all that have responded to this and hopefully my pain will save someone else some pain in the future.