Polling through multiple files

[falcon98]

Hi,

I have multiple files from our experiment which are named as per sequence no like: T2789_MM_DD_YY_hh_mm_ss.xlsx.

We have around 10000 such files from experiments running for quite some time. For data analysis we are developing a automated LabVIEW based tool to display data and apply some algos.

The issue I am facing is that for few sequence numbers the polling takes a lot of time and I have to manually segregate some data in a folder and then select some sequences.

can anyone suggest any other fast or efficient way to filter required files (based on sequence number) other then polling.

Thanks a lot.

[pincpanter]

What you mean with "polling", precisely?

[GerdW]

Hi falcon,

---

@falcon98 wrote:

The issue I am facing is that for few sequence numbers the polling takes a lot of time and I have to manually segregate some data in a folder and then select some sequences.

---

The problem is either in your VI or in the data file.

You didn't attach anything: what kind of help do you expect?

[Yamaeda]

Assuming you with 'polling' are using some 'List files' there's a couple of things you can do. You have a fixed format, so you can use that as a filter in your List files function to e.g. only list files from the current hour.

You should also separate files into folders so you only have e.g. 1 day (or 1 hour) Worth of files in a single folder.

[falcon98]

I think  I need to add some more details.

By polling I mean to filter out desired file paths from all the available files. Like I said I have 10k files and I need around 100 files at a time for analysis (for the tool).

So for the analysis to begin I need 100 files (selected via sequence numbers) from the available 10k files. 

I am attaching a snippet for your reference.

Can I use some dll file for filtering the required file.

pls let me know if I am not clear.

[GerdW]

Hi falcon,

---

@falcon98 wrote:

By polling I mean to filter out desired file paths from all the available files. Like I said I have 10k files and I need around 100 files at a time for analysis (for the tool).

---

From ListFolder you get an array of filenames.

Use ArraySubset to get an array of the required amount of filenames, then start your "file analysis" with these files…

[pincpanter]

I guess that H: is a network mapped folder. Are you sure that the delay does come from the filtering and not from the folder reading?

In any case, your code is doing a lot of unnecessary work. For 100 selected numbers and 10k files, the total number of iterations is 1M.

Since only one file can match the requirement for a given number, you can do all in a single loop over the 10k files.

My code below removes each found number from the number list: this is probably an overkill optimization that gradually speeds up the search and makes the loop to stop as soon as all required files are found.