Best method to continuously monitor, process, and analyze data files from an RSS feed?

[Paul_E_Mueller]

I need to create a system that will continuously monitor (and analyze) new test result files as tests are performed. The goal is that this system would automatically process test results around the clock. There is an existing  test application which updates an RSS feed (web page) that lists the new test result (data) files. The test result files have a timestamp so that we can determine which new files need to be processed. I am not allowed to modify how the test application notifies when new test files are available. My only option is to parse (read) this RSS feed (webpage) and process the files that are shown on the RSS feed. 

I expect that a simple C# application would be more adept at parsing an RSS feed (web pages) to determine which test results will need to be processed. Would it make sense to parse web pages with a DIAdem script?

And I plan to use DIAdem scripts to analyze (process) the test result files.

There are several ways to approach this problem. Here are some options that come to mind. Feel free to comment on these options or add other options.

1. A simple C# application would parse the RSS feed, create a simple list of files for a DIAdem script to process, then exit after parsing the RSS feed. Every 10 minutes, a scheduler would call the C# app to keep the list up to date. The scheduler, after calling the C# app, would call DIAdem (with a startup script) to process test results . There are other complexities to not reprocess test result files, but this is manageable. This approach seems more robust as the scheduler will have a timeout to ensure that neither application gets stuck (hangs).

2. We could continuously run a DIAdem script (in a loop). This script would call a C# app to parse the RSS feed. Then, it would analyze the test results list that the C# app returns. Our concern with this approach is that if DIAdem needs to run continously, it may become unstable over time (many weeks). And we don't want any of these applications to hang in the middle of the night.

3. We could build a C# app that will run continuously. Every 10 minutes, it would parse the RSS feed. After parsing the RSS, it would call DIAdem to process test results.

Any suggestions?

[Pesmith8]

Paul

Sounds like a nice task.

The closest thing, I have worked/helped build is a DB importer that works 24/7.  This was made to use a C# app that scans for new files in a directory, and then when files are found it puts one file set of them into a specfic directory, then fires up DIAdem to do the putting into DB part. DIAdem closes after the file is imported.

 This has worked well in practice and has been reliable.  There are log files in both C# and DIAdem parts so that can keep tabs on how things are really going. 

I also did not like keeping DIAdem up for long durations, that is why went to the C# app controlling overall process. In my case the C# app is running at a fast update time of  30 seconds. As soon as file is placed in the directory, it gets imported in under 2 minutes and is ready for engineers to look at results. The key was to have a trigger for the C# app to use to know that the importer was ready for next file.   The DIAdem data directory being empty ended up working well as that trigger.

As far as reading the RSS/web page from DIAdem. This should be doable.   VBS is the scripting language used in IE. It has good HTTP_get commands that should be able to directly get the web page, and parse it.    To me the real question  Is DIAdem reliable enough run 24/7?  The Brad Turpin of NI would be my choice for this question.

Just my 2 Cents,

Paul

[Paul_E_Mueller]

Thanks Paul Smith. It seems that your application is quite similar to what I need to accomplish. I may also use C# similar to your technique. This seems straight-forward.

I would like to hear from Brad about the 24/7 reliability of DIAdem.

[Pesmith8]

Paul M.

I am interested in what Brad has to say as well.

Thinking that if you need reading of new files every 5 minutes or so. Then having a scheduled task to fire up DIAdem (every 5 minutes) would work just fine.  DIAdem could then read RSS feed and process the new test data into whatever output needed, and then close.

Having all the code in one environment, versus separate apps in C# and DIAdem helps a lot with the support issues.

For me the maintaince issues for two different apps,  caused numerous logistical issues.  It was needed to get real fast processing time, but if your update rate is longer(in the 5 min range) then the scheduled task route is attractive.

Paul S.

[Brad_Turpin]

Hi Pauls,

I would like to echo Paul Smith's suggestion of using one or more DIAdem instances launched by the Windows Scheduler.  I would not suggest running DIAdem continuously for weeks on end.

Brad Turpin

DIAdem Product Support Engineer

National Instruments