TDM files being deleted randomly by preprocessor

[dmurthy]

Hi,

I am seeing a strange issue on our SystemLink server (20.0.0) where preprocessed files are disappearing randomly.

Details:

A preprocessor which uses a custom dataplugin (.dll) combines XLS files based on file naming. 

As soon as the files are streamed to the server, I can see that the TDM file shows up with a .lock extension (this I am guessing is because the preprocessor is still writing to the file).

But then randomly after a few mins the tdm files disappears.

If I repeat the process for the same XLS files, the tdm file does not disappear which probably points to not being file specific issue.

The randomness makes it very difficult to debug.

Has anyone come across such an issue and could provide help? 

Thanks,

Deepak

[Brad_Turpin]

Hi Deepak,

On the face of it, your reported symptoms sound like they're the result of parallel processes fighting each other.  What happens if you adjust the Data Preprocessor to use only 1 execution thread, so that it's never processing more than 1 file at a time?  That's a setting in the Data Preprocessor's config web page, in the "Monitoring Raw Data Areas" Section, the "Number of Parallel Requests to the Computing Nodes" field.  Set that to "1" and dump a new clutch of raw data files in and see if the TDM disappearance stops.  Of course the execution will probably slow down, but we need to first know what's causing this.

I'd like to know a little more about the custom DataPlugin you're using.  You say that it's calling into a *.dll, that must mean that you're either using a C++ DataPlugin or a LabVIEW DataPlugin of your own designs.  Which is it?  What *.dll are you calling into?  I can't recall ever hearing of a C++ DataPlugin that merges multiple Excel files.  What file extension is the Data Preprocessor reacting to?  Do all the component Excel files have the same file extension?  If so, the Data Preprocessor is firing for each component Excel file that should be merged together, and this could well cause one execution thread to step on the toes of another's.

Brad Turpin

Principal Technical Support Engineer

NI