intermittent report file write error in process model

[david_jenkinson]

Hello,
We have recently seen a couple errors we have not seen before, that are intermittent.   This occurs in the single pass entry point sequence, the first "write UUT report" step (the non-on-the-fly one).  The error message is as follows:

"An error occurred calling 'Save' in 'Report' of 'NI TestStand API 4.0'
A file with the pathname 'T:\HTC' already exists.. Error writing to file 'T:\HTC\WING\3WWE\StressTest.seq\StressTest_Report[NO_UUT_SERIAL_NUM13][3 23 37 PM][4 28 2008] - TS2.html'."

So when breaking here, and looking at Locals.ReportFilePath (which is the pointed to variable in the specifiy module of this step), it was indeed defined as:
T:\HTC\WING\3WWE\StressTest.seq\StressTest_Report[NO_UUT_SERIAL_NUM13][3 23 37 PM][4 28 2008] - TS2.html

So everything looks correct, but it thinks that "T:\HTC" is the entire path.  So the next thing I did, after much confused pondering, was go to the "variables" tab while in break, and add an extra backslash to the part following "T:\HTC" in Locals.ReportFilePath so that the path is now:
T:\HTC\\WING\3WWE\StressTest.seq\StressTest_Report[NO_UUT_SERIAL_NUM13][3 23 37 PM][4 28 2008] - TS2.html

I hit retry and it worked.  So my question is, if teststand is thinking the "\W" is an escape code, why does it work 99%+ of the time, and suddenly this weird problem? 

Thanks

[dug9000]

"A file with the pathname 'T:\HTC' already exists."

This error indicates that there was a file called "T:\HTC" so the report saving code was not able to create a directory with that name. I doubt this has anything to do with backslash escaping anything since there is no expression evaluation involved and \W is not a valid backslash escapable character anyway.

Why you had a file called HTC I do not know, but I looked and that is what this error is for. The Report saving code attempts to create all of the intermediate directories if they do not already exist and if one happens to exist, but is a file and not a directory you will get this error (it is the error we get from the operating system). It is possible there is a issue with the OS where it incorrectly reports this error in some cases (perhaps with multithreaded access), but that I do not know for certain.

In searching for known OS issues I found the following:

http://support.microsoft.com/kb/q268897/

"On the other hand, an attempt to create a new C:\Development\Bin folder through the use of CreateDirectory() will fail with "ERROR_ALREADY_EXISTS". In this case, the API fails when it should succeed."

Although it does sound like it was fixed in a service pack. Are you by any chance using NTFS on Windows 2000 and are not up to date with the lastest service pack?

Also it's possibly an unknown OS bug, what OS version are you using and can you reproduce the problem in a simple case?

Also, just to be sure you might want to make sure nothing is creating a file called 'T:\HTC' in your code.

Hope this helps.
-Doug




Message Edited by dug9000 on 04-30-2008 10:58 AM

[dug9000]

Actually I was just thinking this could be a synchronization issue with teststand if the directory doesn't already existing and multiple threads are trying to create it at the same time. Is that a likely scenario in this case? Did the directory not already exist yet? One way to work around this is to make sure the directories all already exist before your start, as the code will not try to create them if they already exist.

-Doug

[dug9000]

Another way to workaround this would be to modify the process model to enable the lock synchronization setting on the Report.Save calls. Either that or making sure the directories all already exist should keep this problem from happening again.

Hope this helps,
-Doug

[david_jenkinson]

Hi Doug,
What is most perplexing is that the directory "T:\HTC" already existed and has for months, has not changed.  In fact, the entire path, except the filename itself, already existed also.  There are numerous reports written with that path included there without error in the past.  It looks to me, judging from the error message, that teststand thinks "T:\HTC" is the entire path/file that it is trying to write to, when in fact it was much longer (see my original post).  So just on a hunch I added the extra backslash to the path in Locals.ReportFilePath while paused, and retried the step, in case teststand was interpreting \W as an escape character (which isn't a valid escape character at all?), and it worked.  

Locals.ReportFilePath is being passed to "Write UUT Report".  When in pause, the path in Locals.ReportFilePath looked like it should.  It was not defined as "T:\HTC", although that is the start of the path.

We are running XP sp2. 

If this happens again, perhaps it would be helpful if I can take more screen shots and post for better explanation.  It seems like awfully strange behavior. 

Dave

[david_jenkinson]

Hello Doug,
The directories are all created earlier in the process model, prior to MainSequence call, if they dont' exist.  But in this case, they did already exist anyway.

How do I enable "lock synchronization" on Report.Save calls?  I don't see that anywhere.

Thanks
Dave

[dug9000]

What version of TestStand are you using? In 4.0 on the step settings pane one of the categories is synchronization. Check the Use Lock checkbox for the step that calls Report.Save. In versions earlier than 4.0 this setting is located on the step properties dialog on the synchronization tab.

-Doug

[dug9000]

David,

I've looked at the code involved in TestStand and what happens when we try to make sure all of the directories in a given pathname exist is that we check each component of the path along the way and if it doesn't exist we try to create it. If we get an error that it already exists, then you will get the error with just that path fragment for the part of the path that already exists (not the whole path) which is exactly what you are seeing.

I strongly suspect that if you had just retried the step, even without having added the backslash it would have succeeded as well.

If you create the simple expression:

Locals.String = "\W"

set a breakpoint after it and look at the value of the string you will see it is still "\W". In otherwords, since there is no valid escape sequence "\W" no escaping is done.

If these directories all really already existed, then the only other likely explanation I can think of is some sort of intermittent bug in the OS, but I will look at the code more carefully to make sure.

Do you by any chance have some code that clears out old reports by deleting the subdirectory? If so that could be a cause for the directory not existing.

Another thing you can do if this happens again is to do a retry without modifying the path to test what I've explained above, but I'm pretty sure I'm correct.

Hope this makes things clearer.
-Doug

[dug9000]

David,

I've also checked and determined that this is the only place in TestStand where we report such an error (when making sure directories within a path exist) at least in the latest version of TestStand. Another possibility besides deletion is if the original directory got renamed at some point (that's effectively the same thing as deletion). If that's not the case then the only other alternative I see is an intermittent problem with the OS, but I suspect that is a lot less likely then the directory getting renamed or deleted in some unknown part of the code or manually.

-Doug


Message Edited by dug9000 on 04-30-2008 12:55 PM

Message Edited by dug9000 on 04-30-2008 12:55 PM

[dug9000]

David,

Are you modifying the process model or is this just the default process model? If you added code that renames or removes the report directory then the lock setting will not work to solve the problem, as you might still have a race condition involving that code. Anyway, you might want to check for this possibility just in case. If you discover such code then you can use a named lock around that code (same setting as before but with a specific name - you can also use the lock step type to lock around more than one step) and use that same name for the lock on the report.save steps.

Let me know if you discover anything or if there's anything else you'd like for me to check.
-Doug