Measurement file not getting every data hit, presumably due to not processing data fast enough

[JÞB]

And I missed a big problem. 😔

20V÷((2^12)-1)=@4.88mV

Chose range and a trigger level that is at least several LSB above the noise floor. 1.7mV is indistinguishable from 0V at a 20V range on a 12 bit ADC.  From your graphs it looks like a +/-1v range would be ideal.

[Kevin_Price]

I just rewrote this whole reply  The first draft was a playful scolding of Jay, saying that the technique he described was an excellent solution to a slightly different problem.  While it *would* greatly reduce the "eyes closed" time, it would *not* drive it all the way to 0. 

But then I did my own re-read of the first post.  While I still thought my continuous sampling suggestion would be better for the simple test case you described (rapidly tapping the sensor), his suggestion was more likely better for most realistic real world uses.  There'd be a good chance yours is one of them.

You'd only miss events if one happened during the "eyes closed" time between successive runs of a finite sampling task.  In your simple test case, you tap rapidly to make an artificially short time between events.  Most real world scenarios probably produce events-of-interest much less often and perhaps quite rarely.  Coupled with Jay's suggestion, which minimizes the "eyes closed" time down to something probably on the order of a 1 msec (and maybe just a fraction?), there could be some nice advantages to doing re-triggered finite acquisition.

But then I went back and looked at the original post yet again, and I'm back to giving Jay another little raspberry, all in good fun.   😝

So here's the other catch that should probably drive you back to continuous sampling and the software post-processing needed for identifying triggering conditions and time windows for your capture.  That catch is: you are also configuring your task to capture a number of pre-trigger samples with each run.   This fact will effectively lengthen your "eyes closed" time again, because triggering conditions will be essentially ignored if they occur between the time the task starts and the time when you've accumulated the # pre-trigger samples you're asking for.

While it's possible that the extra eyes-closed time won't be a problem for your particular situation, pre-trigger samples *are* another factor that moves you in the wrong direction if you want to be sure to capture *every* event.

Your honor, the defense rests.

-Kevin P

[JÞB]

OK Kevin, I'll bite.

The eyes closed with the OP method contains the following times:

• Resource locking: synchronous action that only returns when locks are acquired- hardware and software dependent 
• Commitment: gain stage switching and ADC stabilization (longer for Sigma Delta ADCs)
• Reverse of both of the above

Essentially not great.

With Either a retriggered Finite Task as described in my approach or continuous Task read at 200msec intervals, accumulated and post processed for Event Detection the ADC never spins down and there is Essentially 0 time lost between iterations. Yes Kevin,  when the ADC is running in the commit state samples ARE being taken and if a trigger is seen immediately after the DAQmx Read.vi is called those samples are stuffed in the transfer buffer.  In either approach, you could miss the first event if the pretrigger samples have not had time to happen but, you won't miss the rest.  In either case. You could Read the device Specs for ADC settling time and add a Commit + delay to assure enough presamples have happened.

Advantages and disadvantages of each:

Continuous samples: can catch multiple Events in the same time window e.g. you have a 1sec record with 100mSec pretrigger and two Events happen 500mSec apart you can see both records and there will be overlapping samples.  Cannot be used with DAQmx Logging.  Must handle buffer size and set loop timing explicitly with samples to read or risk buffer overflow/underflow conditions. Must accumulate Data  Queue/Channel Wire then.post process for trigger conditions.  Bearing in mind that you will be Accumuting by appending the waveform Data to an Array of waveforms for each trigger condition that has yet to contain all samples and calculating the t0 for each wavefom element Accumuting.  Then then pass each full waveform element to Logging and Display queues/Channel wires keeping in mind Task sample size and dt could be changed upstream. IMHO more code than needed for 7-70msec records spaced at human tapping speeds.

Finite Task done as Jay suggests: Can use DAQmx Logging.  Can directly display desired Data from DAQmx Read.vi. CANNOT separate overlapped events.  Must handle Timeout errors when no trigger is seen.

For any situation where less than %99.99999 of the AI signal needs to be recorded use my method.  The 1Sec records of 500mSec spaced Events needs to record %200 of the AI signal (every point is recorded in two records) That's the time to roll your own display and Logging loops with Kevin's method.

[Kevin_Price]

Slow clap.

Followed by a Wayne's World "we're not worthy" bow.

Bookmark that one everybody!

-Kevin P

[JÞB]

---

@Kevin_Price wrote:

Slow clap.

Followed by a Wayne's World "we're not worthy" bow.

Bookmark that one everybody!

-Kevin P

---

Homework time.  Kevin, would you post snippets of each example?  Then we can close the loop with the whole discussion.  SUGGESTION: with the continuous sample method, "Waterfall" the graph@  Just set Stack Plots = TRUE, such that Plot0-n are the 1D Array of waveforms accumulating and delete any complete records from the array as they pass to the Log file.   The X Axis will neatly autoscale as the waveforms move from bottom to top then off the graph.  You could even switch in a Simulate Signal + Ignore all DAQmx errors Case to demo your very cool Display and Logging loops without hardware.

[Kevin_Price]

I'll see what I can do this coming weekend.  Too tied up with other stuff until then.

-Kevin P

[Kevin_Price]

Well, I'm overdue for this so here's an example attached below.  First some comments:

1. Well, it turns out I *did* have to revert to continuous sampling and software-based trigger detection because the NI-9775 device does not support analog triggering

2. I took mcduff's suggestion to use the DAQmx TDMS logging feature.

3. Then I set up a producer-consumer pattern, shipping off raw analog data to a parallel loop for trigger detection.  (Consumer loop terminates upon receipt of empty waveform "sentinel")

4. The consumer saves up all the detected trigger locations, leading to post-processing.

5. In post-processing, the logged TDMS file is re-opened, and all the "interesting" chunks of data are extracted from it, based on # pretrigger samples and total # samples per chunk.

Not shown:  deleting the (possibly very large) original TDMS file after extracting the chunks of interest.

Optional variations: this example is, in a sense, the worst of both worlds.  It consumes processor power to detect trigger conditions in near-real-time, but doesn't do anything useful with them until the post-processing stage.  One could very well do the detection as part of the post-processing.

    I left it doing near-real-time detection b/c in some apps, it'd be helpful to software-trigger some other code ASAP when a hardware signal trigger is detected.  However, be aware that more work would be needed to manage near-real-time extraction of data chunks because the chunk of data where the trigger condition is detected may not contain enough pre-trigger or post-trigger samples.  This example would only be a starting point for *noticing* the trigger condition, not a complete example for also doing the extraction.

-Kevin P

[Harrison_Greene]

Hello Kevin,

It has been some time, but I very much appreciate your work. I have tested the code that you developed, and it works at collecting data, but I'm not seeing the effects of the threshold in the saved TDMS file. Here is a screenshot that I pulled from a test using a level of 0.15 and a hysteresis value of 0 in which I tapped the sensor several times:

And here is another test using a level of 0.1 and a hysteresis of 0.05:

On top of this, the waveform graph is not showing anything during any of my tests. Is there something I'm missing that I need to change or add to the code to make it work as intended?

Thank you again for your time,

-Harrison G

[Kevin_Price]

The TDMS file will contain the entire run of data.  In the code I posted, there's a post-processing step that loops over all the detected trigger locations, extracts the desired # of pre-trigger & post-trigger samples, and builds up separate data series on the graph for each trigger.  No data is graphed until the end of the acquisition and all the post-processing.

Your screencaps look like you might be opening the TDMS file with the example function "TDMS File Viewer".  That's just going to show you the entire dataset, not any of the post-processing results.

Post the TDMS file for the first screencap and I'll make a mod of my earlier code to re-do the processing.  It probably won't happen before the weekend though unless someone else steps in and beats me to it.

-Kevin P

[Harrison_Greene]

Looking back I don't have that specific TDMS file (I assume I overwrote it on accident), but I do have one with the same settings and similar outcome. The file is 22 MB so I just zipped it; hope that's okay. Thank you again.

So am I missing something to be getting that post-processed file, or is it just not working for me?