Why do I have to manually reindex before searching?

[testingHotAir]

Hi,

I'm using DIAdem 2012, though this same issue was evident in 2011. 

My system consisits of several workstations either harvesting data or doing post-processing and reporting on the harvested data.  Each station loads files from a central server and saves any new files back to the server.  Pretty standard.

Annoyingly, if I upload to or create a new file on the server, I have to go into the Navigator pane and manually reindex the search areas before a query will return the new file.  Is there a way to ensure that all new files are immediately indexed?

[jallen88]

I am afraid this is a limitation of the DataFinder. You must reindex if there are changes made on the server.

[testingHotAir]

Well that's mildly annoying.  Would running a reindex script after every file creation (at least the ones created by scripts) be a good workaround?  I'm a bit concerned this would make my scripts really slow as the volume of files increases.

[Brad_Turpin]

Hi testing,

I"ve actually had mixed experiences with this scenario.  The DataFinder has a checkbox in "Indexer" tab of its configuration dialog to automatically notice any Windows file events in any of the Search Areas and re-index new or deleted or changed data files as those events occur.  This always works perfectly when the data files are on the same computer as the DataFinder.  When the data files are on one or more remote file servers different from the computer the DataFinder is on, I've seen some scenarios auto-index just fine and others not at all.  Since the trigger here is a Windows file event, there must be some difference in the network or operating system configuration that explains it, but I've never found a satisfactory correlation.  Officially, the DataFinder does not list remote file event detection as one of its features, so the fact that it sometimes works is a bonus of sorts.

The best solution would be to set up a DataFinder Server Edition running on that server computer with all the data.  Then the DataFinder would be on the same computer as the data files appearing periodically, and then any DIAdem (or LabVIEW) clients could query the same always up-to-date DataFinder data base.  If that's not an option, there are functions you can call either in LabVIEW or in a DIAdem VBScript to cause the DataFinder to reindex everything or to index one particular file path.

You could alternatively set up your DataFinder to execute an indexing run periodically, say one per day or once per hour.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Brad_Turpin]

Hi again testing,

I just realized I didn't address one of your questions.  I believe the command to force (re-)indexing of a particular file path is asynchronous, meaning it will return immediately after the command is sent, not wait around for the DataFinder to finish.  So I don't think you need to be worried about your VBScript or LabVIEW program slowing down if you issue lots of these commands, but if those commands start to stack up it could result in not all of the desired files being fully indexed when you want to query them.  If your data files index quickly, then this shouldn't be an issue.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[jallen88]

If you are using DataFinder Server Edition, see the following article on Configuring DataFinder Servers.

The Planning Indexing section is particularly useful. It explains how to setup a schedule that scans for new files, and also how to scan for files on file and folder changes.

[testingHotAir]

Hi Brad,

Thanks for the response.  After I replied I added a reindexing command to the end of my Save File script which did in fact seem to run asynchronously. 

I thought I might have to migrate to server edition eventually, though I've been avoiding it as I'm quite happily managing this resource without corporate IT getting involved.

Two things I'm going to try for now:

1.) Leave the indexing step in my Save File routines and see if it causes/solves any problems.

2.) Call the same script when DIAdem starts up.

Thanks!

[Brad_Turpin]

Hi testing,

The DataFinder Server Edition is designed to be used without IT help/intervention.  It is preferable to run it on a server class operating system, but it does work just fine on a regular Windows 7 computer, if that's what that file server is currently.  Note that non-server-OS installations are limited to the 5 concurrent license option, because Windows XP/VISTA/7 only has a handfull of available communications ports.  If that file server computer is owned by your IT, then I see your point about not wanting to try to install any software on it.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[testingHotAir]

I'm currently storing my data out on the corporate server, so I can't run anything on it without IT.  So far talking them into it looks futile, but we'll see how the workarounds perform.

[testingHotAir]

Well so far the script to force reindexing of my main processed data folder seems not to work.  I use the following call to do it:

Call Navigator.ConnectDataFinder("My DataFinder").Indexer.IndexFolder("I:\Test Engineering\Test Data\Processed Data",TRUE,TRUE)

This leads to the first ~200 files being indexed with the rest displaying the handsome yet frustrating Blue Hourglass icon in the Navigator.  Right clicking on the folder in the Navigator and forcing an index operation does not lead to successful indexing of the hourglassed files; it seems I have to reindex them each individually.

Any suggestions?