intermittent report file write error in process model

[dug9000]

Sorry for so many posts, but I just thought of another possible cause. Is the T drive a mapped network drive? If so the cause could be intermittent connection problems with the file server. If the connection is lost and regained at just the right (or wrong depending on your point of view) time it could result it the same error that you are seeing at this same place in the code. This would really be an OS or networking issue though as the OS really shouldn't be having intermittent connection problems to your network filesystem. I still think this is probably less likely than other causes, but I thought it might be worth mentioning.

-Doug

[david_jenkinson]

Hi Doug,
I do remember hitting retry once before modifying the path with the extra "\".  Next time it happens I will try several times to see if it succeeds without the modification.

Early in the process model, we do have a step which takes the existing report file path (which is defined in the report options) and add levels to it according to make/model/sequence being run, and automatically creates the directories if they don't exist.  This directory creation has been in the code for over a year, I have never seen this happen before a couple days ago.  But in this instance, the directory already existed so none was created earlier.  Also, the path looks correct when paused at the error, and did exist when navigating through windows explorer while the code was paused. 

It is a network drive, which is shared and mapped to T:\.  So I guess another possibility might be that the network had some sort of issue, but as I said I navigated through windows explorer fine.  Also, there are 3 other stations running the same code which have not had the issue, but are testing different models. 

If it happens again I'll capture more information and see what we can come up with. 

Thanks for looking into this.

Dave

[dug9000]

No problem,

Let us know if you discover something new.

All it would take is a split second networking problem so even though you can still browse to the directory in explorer it doesn't mean that for a split second we didn't get some incorrect result while examining the filesystem.

Basically all we are doing where this error is being reported is calling GetFileAttributes() to see if the directory component exists (i.e. T:\HTC is the first component of the path we look at). If it doesn't then we are calling CreateDirectory(). If CreateDirectory() fails with the error ERROR_ALREADY_EXISTS then you will get the error message you are seeing. So if GetFileAttributes fails incorrectly (either because of an OS bug or intermittent networking problem) then we might end up trying to create the directory again even though it already exists thus leading to the error. Of course a similar problem could also happen if the directory didn't exist when we first called GetFileAttributes and then somehow did exist or get created before our call to CreateDirectory(). If you're using the batch or parallel model or are otherwise running multiple executions at once and somewhere the directory is being deleted or renamed or didn't exist yet then that could be the problem as well.

One thing you might want to look at is, if the problem only started happening recently, what's changed at about the same time either on the file server or the local machine. Any service packs or updates installed right before the problem started? Any changes to the process model? Something different about the directory structure for the reports?

-Doug

[david_jenkinson]

Just an update in this thread.

We had this happen again a couple times in the last couple days.  Cut/paste of the error message:

An error occurred calling 'Save' in 'Report' of 'NI TestStand API 4.0'A file with the pathname 'T:\Motorola' already exists.. Error writing to file 'T:\Motorola\Model123\R6713_G_71.03.01R\StressTest.seq\StressTest_Report[NO_UUT_SERIAL_NUM38][10 10 35 AM][5 20 2008] - TS2.html'.

So once again it thinks that the complete file path is "T:\Motorola".

When I saw this, I immediately checked to see if I could browse the path with windows explorer, which I could.  I was preparing to take some screen shots of the procedure I did last time to post, but prior to that  I hit retry in the teststand dialog once, and it wrote the file and resumed.   So it seems like teststand temporarily truncates the path for some reason, and errors on the first portion with "already exists".   Strange?  This has happened twice now with this model of phone.  The formation of the path seems ok, because hitting retry once works.

[dug9000]

David,

Again, it is not truncating the path, that is not what that error message means. I've looked at the source code. That error message means it thinks that there is a FILE with the name T:\Motorola and thus it cannot create a DIRECTORY with that name. This error occurs in the code that is trying to verify and create all of the intermediate subdirectories required for the path in which you want to save the report.

It might be incorrectly making this assumption because of a race condition that can occur if multiple threads are trying to create the same directory path at the same time. This will be fixed in a future version of teststand.

Did the directory "Motorola" definitely already exist before the Save call was made? If not and you have multiple executions running and saving reports to this directory than you have likely run into the race condition. If it did already exist then perhaps it is an issue with networked file system incorrectly failing on our internal call into the operating system to the function GetFileAttributes(). If you could narrow down the situations in which this occurs (does it only happen very rarely, only if the directory was deleted and recreated, etc.) it might help us figure out which of these possibilities is the source of the problem.

Hope this helps,
-Doug