Data collection was switched from an AI Config task writing to an hsdl file to synchronized DAQmx tasks logging to TDMS files. Why are different readings produced for the same test?

[bcur]

A software application was developed to collect and process readings from capacitance sensors and a tachometer in a running spin rig. The sensors were connected to an Aerogate Model HP-04 H1 Band Preamp connected to an NI PXI-6115. The sensors were read using AI Config and AI Start VIs. The data was saved to a file using hsdlConfig and hsdlFileWriter VIs. In order to add the capability of collecting synchronized data from two Eddy Current Position sensors in addition to the existing sensors, which will be connected to a BNC-2144 connected to an NI PXI-4495, the AI and HSDL VIs were replaced with DAQmx VIs logging to TDMS. When running identical tests, the new file format (TDMS) produces reads that are higher and inconsistent with the readings from the older file format (HSDL).

The main VIs are SpinLab 2.4 and SpinLab 3.8 in folders "SpinLab old format" and "Spinlab 3.8" respectfully. SpinLab 3.8 requires the Sound and Vibration suite to run correctly, but it is used after the part that is causing the problem. The problem is occuring during data collection in the Logger segment of code or during processing in the Reader/Converter segment of code. I could send the readings from the identical tests if they would be helpful, but the data takes up approximately 500 MB.

[AndrewMc]

First of all, how different is the data?  You say that the reads are higher and inconsistent.  How much higher?  Is every point inconsistent, or is it just parts of your file?  If it's just in parts of the file, does there seem to be a consistent pattern as to when the data is different?

Secondly, here are a couple things to try:

1. Currently, you are not calling DAQmx Stop Task outside of the loop; you're just calling DAQmx Clear Task.  This means that if there were any errors that occured in the logging thread, you might not be getting them (as DAQmx Clear Task clears outstanding errors within the task).  Add a DAQmx Stop Task before DAQmx Clear Task to make sure that you're not missing an error.
2. Try "Log and Read" mode.  "Log and Read" is probably going to be fast enough for your application (as it's pretty fast), so you might just try it and see if you get any different result.  All that you would need to do is change the enum to "Log and Read", then add a DAQmx Read in the loop (you can just use Raw format since you don't care about the output).  I'd recommend that you read in even multiples of the sector size (normally 512) for optimal performance.  For example, your rate is 1MHz, perhaps read in sizes of 122880 samples per channel (something like 1/8 of the buffer size rounded down to the nearest multiple of 4096).  Note: This is a troubleshooting step to try and narrow down the problem.

Finally, how confident are you in the results from the previous HSDL test?  Which readings make more sense?  I look forward to hearing more detail about how the data is inconsistent (all data, how different, any patterns).  As well, I'll be looking forward to hearing the result of test #2 above.

[AndrewMc]

After staring at the code, I have a theory about what could possibly be causing this problem.  If my theory is correct, it's because of a situation encountered only in channel expansion (multiple device synchronization in the same task) and only with a start trigger was takes a while to signal (and thus could occassionally timeout a 10 second wait).

If my theory is correct, then solution #2 should work for you (though it's regrettably not ideal).

It would help me identify whether my theory is correct if you could just send your index file (not the whole tdms file).  If you post that and verify that your scan rate is 204082 (or otherwise clarify), I should be able to identify whether my theory could be correct.

[bcur]

The data from the old HSDL test makes sense while the readout from the new TDMS test doesn't. For testing at 3000RPM, the gap readings in the TDMS file are consistently around 0.065" and the gap readings in the HSDL file are consistently around 0.044. For testing at 5000RPM, the gap readings are 0.083" in the TDMS and 0.056" in the HSDL. The results from the HSDL test are known to be correct from previous experience.

I had to change my program from "Log and Read" to "Log" previously because the generated TDMS file would be corrupt if I did "Log and Read" at 204082 Hz. I have NI case number 1553375 for that very issue.

[bcur]

I am not sure what you mean about channel expansion, but the index file from my previously generated TDMS file is below for a 3000RPM test and a 5000RPM test.

[AndrewMc]

I'm not in the office yet, but something to try in the meanwhile:

Let DAQmx create the TDMS files and do not try to write the custom properties yet.  Again, this is just a troubleshooting step.  I'm wondering if the problem is introduced when combining the two segments into the same file.  We can go from there.

As well, sorry these issues seem to have gone on for a bit for you (in looking at the service requests).  I'll get involved and try to help you get to the bottom of this.  Do we have your correct phone number on file?  If not, please let Applications Engineering know so that they can note a good number to call you at later today, if that's convenient.

[AndrewMc]

Studying and testing some more, the theory I had has been proven impossible.  I found myself re-tracing my steps that I had made to ensure that that problem couldn't happen.

That being said, there's some good news and some bad news:

Good news: The issue you're seeing with "Log and Read" can simply be fixed by reading a specific number of samples in DAQmx Read instead of -1.  When you read for -1, it has horrible performance.  Additionally, you'll end up with a LOT of headers in the file, which will bloat file size and in your case seem to create some file corruption.  When you read a specific size from DAQmx Read, the file will only have 2 headers in the file; everything else in the file will just be raw data.  This means that you greatly reduce the file size and risk of corruption.

Bad news: I can't reproduce this problem.  There are some Applications Engineers that have been working on reproducing it and I have tried, but we have not been able to recreate this behavior, thereby making it difficult to fix any potential problem.  There is a solid workaround of reading a fixed size (note: does not need to be sector aligned; that just helps with performance).  We will continue to work with you through your support request to try and get this reproduced, but at this point, it seems that you have a good workaround.

Let me know if you have any additional problems or concerns.