What is the best method for continous long-term datalogging?

[JDfromRI]

Hi Folks,

I have a component life test in progress in an R&D lab.  The component happens to be a DC motor diaphragm pump (5 pumps, actually) that is continuously running on a 10sec ON/5sec OFF duty cycle. The pumps are being loaded and unloaded (backpressure rising and falling) during each 10sec ON interval. I am currently logging current data using a shunt.  I am logging 100% of the time at 100Hz to a text file.

As you are all probably already thinking, using a text file (or even binary) produces huge files very fast and is a major task to sort through after accumulating a few days worth of data. The test must run for 9 months.  What is the best method for datalogging at this rate for such a long period of time?  I'd like to be able to quickly sort through and view hours, days, or weeks worth of data. Is the tdms approach better for this?

Any help is appreciated!

Cheers,

JD

[TWGomez]

You are asking what would be the best format to store the data then?

You want small, but sortable?

I'd like to have my cake and eat it too 😄

You might consider programatically breaking it up into smaller files. For safety purposes I would open, write, close (some of the write functions do that for you).

Design pattern should be a producer/ consumer variant with data being feed into a queue that another loop reads out of the queue and writes to the file(s).

I'm not eniterly sure what would be smaller than writing the file as binary...

EDIT: Add on

I have never really messed with TDS, but it looks like it might be a good idea (again I'd open, write, close -- for safety). Dynamically created file name, a new one for each day or something...

Message Edited by TWGomez on 04-23-2008 03:36 PM

[Jarrod_S.]

You might look into the TDMS functions that ship with LabVIEW 8.2 and above. It's a binary format designed for quick streaming. It's a fairly simple API that's based on organizing your data into separate groups of channels.

You can read this data back easily in LabVIEW. One advantage of a binary format is that it's much easier than text to jump to a certain point. With text you have to continually analyze how many lines you're scanning through to get to a certain time point in the file. With a binary format like TDMS, you can jump directly to the 100,000th sample of any given channel very quickly and easily.

Another huge advantage is that you can leverage other programs like DIAdem, which has a built-in DataFinder fully compatible with TDMS files. This allows you to easily search for certain events or information in the data and properties of multiple files.

[P.S. When doing such long-term logging, as the previous poster said, it is a good idea to regularly flush the data to file or close the file every so often and open a new one. When logging data with TDMS, file sizes can grow very large if you don't follow the simple steps described in this post.]


Message Edited by Jarrod S. on 04-23-2008 09:01 PM

[JDfromRI]

Thanks for both of these replies.  I am switching files every other day at this point to keep files *relatively* small.  I will try the tdms and see how it works.  Jarrod, your point of being able to jump immediately to, using your example, the 100,000th point is key.  I am using ver8.2.1, in case I didn't mention that.

Thanks again,

-JD

[Ben]

Logging to a database running on another machine will give you "your cake" and "let you eat it too". Provided the machine hosting the DB and interconnecting network have adiquit performance, 100 should be very do-able.

Ben

[bef]

Once you solved the data storage, check your code for memory leaks. Running a system nonstop for 9 month will probably stall your Windows-machine at some point.
The waveform-chart has a built in memory leak, so don't use it (Has anyone fixed that in 8.5.1?).  And the OS? Windows. Hmmm.
I have been through all this and ended up using a Linux system. Is that an option?