Are you getting an error messages and if so what are the code numbers?
You mentioned that you wanted to change the U8 to I16 in the compression VIs. However, the only compression VI I saw in the block diagram was to compress the header and this VI does not output any U8 representation values. Please clarify where you would like to make some changes. I know of the Developer Zone titled Data Compaction for High-Speed Streaming to Disk Examples. Please verify that these are the example VIs you want to change. At most, I saw that the Win32 Open file VI (from the posted code) uses U32 as an output. I failed to notice your “notes in parts of the code to give some insight into what I think needed to be done”.
The notes I did see in-regard to lossless compression where very detailed. However, the default for compress in None. Please review the following link titled Data Compaction for High-Speed Streaming to Disk on how to setup lossless compress with a property node. There are also help files in the Labview Help. Please search AI.RawDataCompressionType in the Search tab.
I have tried to play with AI.RawDataCompressionType and AI.LossyLSBRemoval.CompressedSampleSize according to the example you mentioned above?
The issue is that I do not want to log data into binary files using Raw 1D I32. Instead of that I want to save Unscaled 2D I32 array in TDMS file. Is it possible to use lossless (24bits from PXI-4472 in I32 to TDMS) or lossy (20bits from PXI-4472 in I32 to TDMS) compression supported by DAQmx?
Afterwards I would like to open TDMS files directly in Matlab using Dominonilibddc (nilibddc.dll). Also I like the way how data are stored in TDMS files and it can be supported by more application than binary files.
You might have seen my mention of the TDMS logging feature that was added in DAQmx 9.0 on this thread (http://forums.ni.com/t5/LabVIEW/Why-the-TDMS-file-is-larger-than-it-should-be/m-p/836518). Note that this feature will skip Windows buffering and even use an advanced feature of the OS for overlapped (or asynchronous) file I/O. It is the fastest way possible to stream to disk in DAQmx.
Additionally, lossless compression is supported in this feature. That is, if your device supports hardware compression (24-bits represented in 3 bytes) and you enable lossless compression, only 3 bytes will be written to disk per sample.
Let me know if you have any questions on this feature.
It seems that my quite old DSA 4472 does not support hardware compression. I bought it because it has 8 channels and simply I prefer to have 2 times 8-channel DSA instead of newer 449x with 16 channels. Now I can build two GPS-synchronized measurement systems with 8-channels DAQs but unfortunately without hardware compression.
Is there another way to apply lossless and lossy compression (sometimes my sensors sensitivity is bigger than LSB) for measurements using PXI-4472? I would like also avoid binary formats because measurements will be available for many users.
Thanks and best regards,
For lack of hardware compression, you could enable software compression instead by setting attributes for AI.RawDataCompressionType and AI.LossyLSBRemoval.CompressedSampSize.
Note, however, that with the logging feature, software compression is not supported. One of the main reasons for this is that it would add a buffer to that logging operation, which would greatly reduce overall available system bandwidth (by stressing the memory controller); the extra cost for the logging operation far outweighs the byte per sample that you would save (unless of course, file size is your only concern as opposed to streaming performance).
As to your concern about binary file formats, note that disk footprint and performance will suffer if you don't use binary. Additionally, something that greatly helps TDMS is the TDM Excel Add-In, an easy way to open .tdms files in Excel.
And this was my first thought. As I observed my portable measurement set-up can handle long-term multichannel data acquisition with software compression enabled.
When I log raw data using DAQmx Read and TDMS Write Function with 1D I32 representation and software compression enabled is not so straight forward task to read TDMS files afterwards using TDMS Viewer.
If I log data without compression I can use Decimate 1D Array and afterwards save it using TDMS API but in this case there is no difference with Unscaled 2D I32. When I use directly Unscaled 2D I32 there is no difference in files content and sizes but in case of Raw data it seems to work and lossy file (20 CompressedSampSize) is almost 37% smaller than without compression and lossless file is almost 25% smaller than without compression. This would help me to save a lot of disk space.
Is it possible to save data in TDMS files with lossy or lossless software compression and read it directly in TDMS Viewer or Matlab afterwards without any binary data management algorithms?
Many of the DAQmx scaling methods that are used in the DAQmx TDMS logging case are exposed in the standard TDMS API as well via the "Create Scaling Information" VI (Linear, Polynomial, etc). However, there would not be a good way to express some of the complex binary data manipulation that happens with software data compression.
That being said:
1) Here are examples showing how to use software data compression and subsequently do that binary manipulation: http://sine.ni.com/devzone/cda/epd/p/id/4828
2) If you did not use software data compression, you could leverage the "Create Scaling Information" VI to store the scaling information. For example, if your DSA device has a linear calibration of 9.5 (for example), you could describe that with the VI such that the scaling will automatically be applied whenever you read the file.
Thank you for your fast response.
In this case I will stay with TDMS API and Producer/Consumer solutions for long-term one-minute file logging measurement run-time software.
I also create linear scales and it work pretty nice. I am quite sad regarding the software lossy and lossless compression support in TDMS API, but I will just buy few more TB-capacity HDDs.
Right now I am not going to use binary files for compacted streaming purposes. Mainly due to lack of support for external applications. When I have lots of TBs I do not want to convert it in LabVIEW, save new files and afterwards process in third-party software.