LabVIEW
Concurrent access to TDMS file with interleaved data and only a single header (no file fragmentation)
Solved!
@Gabriel_H wrote:
@drjdpowell wrote:
wiebe@CARYA wrote:
What I don't get is why you'd keep the data in memory, and write it TDMS, and then read it from TDMS? Makes more sense to me to let all the "file creator actors" read from memory.
I think the OP is planning on NOT keeping data in memory, but instead writing to file from one component and viewing it with another. That's a good architecture.
The measurement files may have 100 to 1000 channels, but only 2 to 5 will ever be viewed during testing (while the files are open and being written) to produce basic plots to visualise how the testing is progressing. I didn't want to have a duplicate set of data sitting in memory and also on disk, when the copy in memory will only ever have a small portion used. That solution may not scale well when the data set becomes large. Keeping it only in memory would mean that application crashes or power failures could cause all data to be lost. Hence writing it to disk immediately then viewing it back appears to be the safest approach for keeping the data intact and minimising memory usage.
A past project I developed used advanced TDMS functions (synchronous write) for a 24/7 data logger, which could produce several files each with volumes of 1~10GB for each day (TBs per year of data). The advanced TDMS functions worked well to reimplement the "one time header" feature, allowing a scalable soltuion for recording huge quantities of interleaved data. That part of TDMS works very well.
I was looking for a similar solution, but with concurrent reading. That's when TDMS falls apart in terms of being a scalable solution (i.e. standard TDMS fragmentation get's out of control, causing 6x larger files full of meta data describing the scattered file structure). I'm not sure if there are any other file types that would also work well and scale well, because the the data needs to be appropriately ordered on disk to avoid the fragmentation when a file is written over a long time. It's easy to solve once a file is complete: I can defragment a TDMS file, or if the data is known (final length of channels) I can write it efficiently to something like HDF5.
My summary is that there isn't an easy, scalable solution for this use case. I've experimented with a lot of variations, but the only thing that looks like it might work would be advanced TDMS reads (in a number of smaller blocks) coupled with knowledge of the file structure (groups, channels, data types per channel). It looks like I'd need to wrap the TDMS functions inside a class that stores a duplicate of the file structure and uses that during reads to deinterleave and correctly type cast the TDMS advanced read outpute. It looks a lot like reinventing the wheel, and not worth my time considering the TDMS API is meant to do that already.
I'd really like to see NI make an example of TDMS advanced reads of interleaved data from multiple channels of mixed types. So far the only LabVIEW examples I've seen do this with the same channel types.
You probably need some mixture between memory and TDMS. Use TDMS for accessing older data, written nicely in chunks. So when writing those chunks, TDMS read won't interfere with TDMS write. During reading older chunks, you might need to stall the writing, so the current chunks don't get fragmented.
To access data in a chunk that's being written, read the data from a buffer.
Not too hard, but I don't think you will get it done in a one liner...
Search LabVIEW like a graph! 
@Gabriel_H wrote:
. It may not scale well when the data set grows to be large, but in that case the copy in memory could be decimated to limit memory usage.
In my experience, keeping the data in memory is a lot easier then displaying it. Displaying anything over 70 MB in a graph will start to render it useless. So I usually keep all the data and apply decimation just before displaying it. That way I can adapt the decimation so that the user does not notice it. There are obviously memory limits as well, but the graph limits are much much tighter.
Thanks wiebe@CARYA,
Based on your advice, and this post from 2014 on the LAVA forum about advanced reading from TDMS, I've put together another example that uses the advanced TDMS synchronous read and write to perform concurrent access using a buffered block of data. The VI is attached.
What's great about this solution is that it performs deinterleaving of the data (value at a time) into the buffered block, then writes the block to file once full. Depending on the block size: a larger block length will reduce the number of read operations required for a given length of data, but with the tradeoff of data not being immediately written to file. The blocked size can be tuned as appropriate for the application.
I've also included a button that selects either standard or advanced TDMS functions.
A comparison of standard vs advanced TDMS:
(20 groups, each with 1 time channel and 50 DBL channels, with 2000 values of data per channel written, or approx. 2 million total values for the file):
- Standard TDMS: Ran for 45.8 seconds to write all data, produced a 92,680kB file, with a read time of 22ms (for 1780 points, 3 channels).
- Advanced TDMS (block length of 10): 1.99 seconds to write all data, produced a 16,306kB file, with a read time of 7ms (~1500 points in, 3 channels).
There's quite a large performance difference, purely based on structuring the file properly. Interesting the read time in both cases was quite small. In my specific case I write the file over a long period of time (many hours) but read in about 5 selected channels of data as the user requests them via the GUI. So fast read time is more important that write time.
Summary of Advanced TDMS solution:
There are four parts:
- TDMS file creation and setup.
- Measurement loop (producer loop).
- TDMS read/write loop (consumer loop), that holds a 1 block buffer that is written to the file once the block is full.
- GUI event loop (produces user commands for the TDMS read/write loop).
TDMS file creation and setup:
As groups and channels are added to the TDMS file, they are also added to an array of clusters containing the same group name and channel name structure.
Measurement loop (producer loop):
This loop is produces measurement data, which is bundled into a cluster and sent via a queue. In a real application, this would be located elsewhere and the queue would be replaced with something like actor messages (to send the data to the TDMS data logging actor).
TDMS read/write loop (consumer loop):
First the buffer is dynamically created from the cluster array of TDMS group and channel names. In this example the format is fixed to have an arbitray number of groups with each group having one time channel followed by an arbitrary number of DBL channels.
Inside the TDMS read/write loops, writes are performed as follows:
The writing is straightforward: overwrite into the buffer (keeping track of the buffer contents), then write the buffer to file once full. The transpose 2D array appears to work here, even though writing 2D data to TDMS doesn't appear to be well documented.
The read operation is more complicated:
The number of blocks in the file needs to be calculated first, then for each block and each channel, a block length of data must be read out. Hence, the smaller the block length, the more reads required. After the read is completed, the contents of the buffer is appended to the array then it is returned to the GUI (in this case, via a dynamically registered user event).
In this case, the only limitation is reading the timestamp channels: these are cast to DBL, although more advanced implementations are possible (using the TDMS data type property to handle the type as requried).
GUI event loop
The GUI event loop is simple. The read request is generated by an array of menu rings (dynamically populated), that allows the selection of an arbitrary number of channels to be read from the file. The channels are displayed in a 2D array, but when sent back to the event GUI only two channels are plotted. This is only a limitation of this GUI event loop, not the TDMS advanced read operation.
Hopefully this helps anyone else in the future who needs to read and write TDMS files with an interleaved structure.
