DAQ fast data storage and time filtered retrieval. Should I use citadel?

[Bob_Schor]

Several years ago, I attended a talk at NI Week about Diadem, which seemed to be designed for searching/isolating/reporting/organizing TDMS data.  I know much less about Citadel, but am not sure why you have rejected TDMS (which NI seems to push as a robust way to log lots of data).  Can others Wiser than I chime in on whether you would be able to do what you want with using TDMS as your logging engine?

Bob Schor

[Sam_Sharp]

Yes, you can put your references in an array and write to them that way - it will be much more efficient than open/write/close! You will need to do something about the JSON - perhaps you could do a find and replace in the JSON string to replace 'NaN' with 'null'.

You could look at another database engine (I'm a bit of out touch but Cassandra rings a bell) that does time-series data.

What about something like SQLite? It's a file-based database - it's pretty efficient so you should get good read/write performance and you can query the data with SQL? Might be good if you don't need a networked/remote solution.

Bob - for the same reason you would use any database for storing data - it's much easier to retrieve/analyse data (even remotely!) from a centralised database than from files.

If you use a file based approach to find data between two timestamps, you need to know (and read in) the files that contain the start of your data, any files in between and the file that contains the end of your data. If you want to do this with TDMS files it would be a pain - although it might be made easier if you use the NI DataFinder.

Database engines often allow you to do calculations on the data before retrieving it and you can store additional meta data .

Also, a good time series database engine allows you to store data with varying dt and can even reduce the data on the fly (e.g. if a temperature doesn't change it just stores the last value).

This isn't true for TDMS, but for other file formats the file is locked while writing so you can't retrieve data at the same time while your logging process is happening.

[Vince_vd]

I changed the references, made a much bigger difference than I expected (CPU usage from +- 3% to +- 0.4% ! ) . Thanks to the poeple suggesting this.

I'll look into other databases designed specifically for time series data, good idea. I already tried MySQL (which should be as performant as SQLite) and that really didn't work out, it was using about 35% of my CPU to write the data (this was the MySQL engine itself, not LabVIEW).

I'm glad you share and understand my idea about TDMS. I have thought about just creating a new TDMS file every minute and entering the time in it's properties (at least that's how I think it works, never used it before). But that would still mean searching through all the files to find the right one like you said.