DataFinder 2017 not indexing all files

[Siriusly]

I have been using the DataFinder tool for years and recently upgraded to a new PC and moved to from DataFinder 2015 to 2017.  The problem I am having is that the DataFinder tool never seems to complete the indexing process, the icon on the system tray has a red bar on the bottom and the latest files are not found via a custom query.  Attached are a couple images of the DataFinder settings and the task tray icon with the red bar for the DataFinder.

[GBarta]

Good afternoon,

I'm still using the 2015 version - not because of any trouble with the new SW, I'm simple a bit leasy to update. And "never touch the working system".

Anyway: do it slowly and use big timeout values.

I'm using a big data base; more than 200.000 files; and I do it for a long time (I have collected this amount of files in the last 10-12 years).

With all the old versions of Datafinder, starting with DD 2009 and closing with DD 2015 I'm using now I found:

 - do not copy too much files into the database. Measure and convert and such "built in" actions are OK, but copy shall be done slowly.

 - after copy you shall wait and check if the index is ready.

 - do not start a new copy until the indexing is not finished

 - it is better to copy only some hundreds of files (instead of some thousends); then wait until the indexing gets ready. This you can see on the icons.

 - restarting the indexer helps; but I do it only for directories and not for the whole database

 - since I do it by hand I switch off the automatic reindexing (not necessary; file updates will be still cathced by DD)

 - set up high tolarance for the indexing PC. I can see on your screenshot that you are probably at the default selection: 7 and 20 seconds for reading and processing a file. Set them higher - I'm using a range with some minutes. Sometimes it takes a time to reach a file (PC is working on a different SW). I have 60 seconds for "browse and connection".

 - set up high tolerance for searching; I have 300 seconds "Query timeout".

 - old formats may make trouble (.DAT); my solution is a script: it reads the file and stores it again. However it was not necessary to use it in the last time.

As above: I never used DD 2017; but moved my data many times in the last 10-12 years; and do it slowly helped always. Na ja: it takes time.

(you can "restart" it if you move the data from the database; stop and kill the indexer and delete the database. After restart DD will create a new database; and data copy can be started.)

Sorry for the long text; I hope it helps

Bye

[Siriusly · Accepted solution]

GBarta,

I too have been collecting, mining and presenting data using DIAdem and the DataFinder tool for 10+ years.  This time we have moved from the 2015 to the 2017 version as we recently purchased an upgraded PC for this purpose.  I still have the original working system but we want to move forward with the new one.   

I copied all the files then tried indexing them (67,000 =~ 250GB), I will try again copying small groups of data at a time. hopefully this helps.  

On old versions of the data finder we could easily tell if the index was complete by using the system tray icon as an indication (Gear or Gear with Hourglass), however in the 2017 version it is always displays the gear with a red bar below it (see previously posted image "DataFinder Red.PNG ")  This makes it hard to tell if indexing is complete.  In the new version it appears you can click "About My DataFinder" which opens a window that the user can then check the indexing status.  

I appreciate the feedback, and have increased the timeouts as you suggested.  It will take some time before I will know if my issue has been resolved but I will report back.  

[StefanR]

Can you please clearify you're using My DataFinder shipping with DIAdem and not the DataFinder Server product?

[Siriusly]

I'm using v17.0.0f6654 which is the version installed with DIAdem.  I don't think its the server edition.  

Anyways after manually copying and pasting the data in chunks the files are all indexing properly now.   The System Tray icon has not changed and still has a red bar beneath it which to me indicates the indexing is not complete.....  Maybe I'm wrong though.

Thanks

[GBarta]

Morning Stefan,

Yes, I'm using Diadem and not the stand alone Datafinder.

Sorry, I see now: I wrote "DD", which is Diadem, for all; but I think this shall be OK.

Have a good day!

[GBarta]

Morning,

Well, I ment the icon of files in data Database; sorry not to clearify it yesterday.

Shortly: I think the problem could be if you have a slow PC (slow processor; little memory; and only one HDD for everything) - with a rapid HDD. I think - and this is just my opinion, I'm not an expert; neither in NI product development nor in computer sience - the problem is that if you just copy a hugh number of files into the data base then the Indexer recognises it, I think via interrupts. And it starts immediatley to index the changes. However: if the indexing takes longer (slow PC), and runs much slower than the copy (rapid HDD) then it may happen, that some (or many) file accesses get "timeouted", and the indexing terminates (for that files).

This coud be the background.

And one more last note: last time I had to copy my data (over 200.000 files; about 300 GByte) in 2015 - because I got a new PC.

It is a powerfull machine for simulation and testing: Intel I7; desktop PC; 4*2 cores; 16 GByte RAM; with a"small" HDD (500 GB) for the system and a big one (1TB) for the data.

Since that time I have much less problem with Datafinder: the indexing is able to get finished properly even if I copy a few thousends of new files into the data base - may take longer (some hours), but never freezes, what still happened before with the old PC-s.

Have a good day, bye

Gábor

[Brad_Turpin]

Hi Siriusly,

The "My DataFinder" that installs with DIAdem ALWAYS has a red "conveyer belt" ("Fliessband") line at the bottom of its icon.  That does not indicate its state, but it does indicate that it's not the DataFinder Server (which uses a blue one).  If you right-click on that "My DataFinder" icon with the red line, you can choose the "About My DataFinder..." context menu to monitor the indexing status.

Brad

[thomasj1977]

I'm currently working with 'Siriusly' and am experiencing similar problems with 2017 DIAdem not indexing files properly. I have weeks of data that isn't indexing and discovered that if I expand the group that the non-indexed data is in that it seems to force an index. Why is it that these files aren't indexing correctly as they're received? This is a new computer with DIAdem 2017 installed and the TDM's that are being transferred to it are not being indexed.

[NIYiran]

Hi Thomas,

If you right click on MyDataFinder and select to Configure My DataFinder, what options do you have set under the Indexer tab? Can you verify that you have Automatic indexing triggered by set to Data file changes? A screenshot would be appreciated as well.

Thanks!