FFT setup in Diadem -- joint time frequency

[Brad_Turpin]

Hi Jeff,

I now have a version of the DataPlugin that loads 256+2 or 512+2 channels that have as many rows as there were timestamped snapshots.  So we can load all the raw data you want from the X, Y, and Z dimensions of the biggest file you can find.  No problem.  Now what?  The only way to process this data to find frequency components is to run an FFT on each row of values-- all the measurements from a given timestamped snapshot.  But now we're right back in the same conundrum-- the JTFA will creae 1 Frequency channel, 1 TimeStamp channel, and N Amplitude channels, where each Amplitude channel contains the FFT of the Nth snapshot.  If the number of JTFA snapshots exceeds 65,000 then we're back up against the same barrier (though with nearly twice as much room).

I reworked the JTFA.VBS and JTFA.TDR and JTFA.TDV files to work with this new 256/512 channel approach.  You'll need to unregister the original JTFA_XLS DataPlugin and register the new JTFA_CSV DataPlugin, then clobber the above 3 JTFA files you already have with the new ones I'm attaching to this post.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Brad_Turpin]

Hi Jeff,

I forgot to mention that the newest application I sent in the previous post expects you to load your data file first and your NMEA file second.  The newest application then automatically shaves off any snapshots from your data file which lie outside the timestamp range in the NMEA file.  The application should run through without an error even if the NMEA file is missing, but I forgot to have it add a dummy group to the Data Portal (index = 2) so that the VIEW and REPORT layouts can still find the JTFA data in the 4th group where they're looking for it.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[jtw131]

Brad,

What is the procedure to unregister a dataplugin?  I can't find documentaiton anywhere.  It seems like the .uri files are associated with USIREG.EXE, and there are command line options for this app, but I can't get it to work.

Thanks

Jeff

[Brad_Turpin]

Hi Jeff,

Launch DIAdem, then go to either NAVIGATOR or SCRIPT, then select the following menu to launch the DataPlugins dialog:

Settings >> Options >> Extensions >> DataPlugins...

Now select the JTFA_XLS DataPlugin and click on the red "X" icon at the top right of the dialog, or just hit the <Delete> key on your keyboard.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[jtw131]

Thanks, I was able to unregister the old plugin. 

I am really happy with how the solution is shaping up.  I have been looking that the data this weekend and it has led to a few further questions...  OK lots of further questions.   These all relate to VIEW:

1.  Is it possible to have a log scale for the JTFA contour display?   It looks like this is an option for the JFTA in REPORT, but I couldn't find a way to do it in VIEW.

2.  I added a dynamic 2D plot of an individual FFT that changes with the contour cursor (see attached screen shot JPG).  I would LOVE if I could also dynamically view the corresponding raw 256-pt waveform at the same time in another 2D plot.  Is this within the realm of possibility?

4.  I also added map display, which kindof works, but something is wrong with the cursor sync.  The map marker moves when I step through the speed plot, but not when I step through the contour plot.  It also does not update when it hit the "play" button.  I have attached the TDV file.  Data is the same AccelX dataset from this thread coupled with the NMEA.    

5.  Can a route (long/lat 2D) be plotted on the map display? 

6.  As I have been working more in VIEW, i am finding it difficult to rearrange the display "panes".  Is there an easy way to move, or copy and paste a display once you have it setup? Or are you pretty much locked in to the original layout selected, beyond resizing the panes? 

Thanks again for your awesome help!  

Regards

Jeff

[Brad_Turpin]

Hi Jeff,

I tweaked a few of the cursor synchronization settings in your VIEW layout and am sending it back to you.  You'll need to change the map cache location back to your mapped drive-- I had to change it to a folder on my computer to play with the GPS synchronization.  So it now works better, but it looks like there are just too many things being synchronized too quickly for my laptop to keep up.  If I intentionally lame the JTFA graph by choosing a channel that doesn't exist, then all the other areas, including the GPS map keep up nicely, even at faster replay speeds than you had.  As soon as I set the JTFA channel back to the correct one, then the JTFA and FFT snashot synchronize nicely again, but the GPS map doesn't update until you stop the replay.  Note that you can stop the replay with the <Ctrl-Q> keystroke on your keyboard.

Your best bet is to select the speed graph and run the synchronization from there.  You can also hit the left and righy arrow keys ON YOUR KEYBOARD to toggle the speed graph one data point back or forward-- and in that case ALL the areas update nicely.

I also added a Latitude vs. Longitude area to show the whole course traversed.

For Log display in VIEW we're going to have to explicitly calculate the Log values as part of the JTFA.vbs-- so, did you mean Log Frequency or Log Amplitude?

One of the side benefits of loading the raw data the new way (JTFA_CSV) is that now it is ready to be shown in a contour area in VIEW, and you could display the snapshot time series data.  You're going to be running out of room in your VIEW area, and I'm not sure how you want to rearrange things, so I didn't attempt to add that, but it's just the same as the JTFA contour display and FFT snapshot.

You can't really rearrange the VIEW panes interactively.  You can add new ones and delete existing ones, and you can even copy and paste particular graph configurations bewteen existing areas, but there are no general copying or moving methods for an entire area.  There are a couple of requests for this already at www.ni.com/DIAdemIdeas if you want to vote for them.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Brad_Turpin]

Hi Jeff,

I answered my own question regarding bumping into the limit of DIAdem channels in the Data Portal.  I decided to stop creating 1900 individual JTFA channels, each potentially 256 values long, and instead to create up to 256 JTFA channels that each have 1900 values, one for each TimeStamp.  Since we expect your number of TimeStamps to always be way more than the number of sampled values in a given snapshot, this should be MUCH more efficient for DIAdem.  In fact, we could even start entertaining the idea of loading and calculating the X, Y, Z components in the Data Portal at the same time.  The open question to you is how you want to display 3x the information we currently have.  On 3x the number of sheets?  Calculate 3D vector resultant values?

Now that the JTFA results fit in only about 40 channels, I changed the JTFA.vbs to calculate both a regular "JTFA" group as well as a "JTFA Log" group that stores the log of the amplitudes.  This enables us to show the log of the JTFA amplitudes in VIEW, and it makes the log display slightly more efficient in REPORT.

I also went ahead a reshuffled the VIEW layout area to show both the raw snapshot data as well as the JTFA data.  The raw data contour is WAY slower than the JTFA contour, but it only needs to draw once, then you can reuse the image as you slide the cursor (still on the speed graph) back and forth.  The reson it's slower is that we're deleting the high frequency parts of the JTFA, so that we're only plotting the first 35 FFT frequencies (minus the 0th one) out of a possible 256.  So the JTFA is plotting 34/512 = 7% as much data as the raw snaphot contour.  You could change the allowed lower and upper limits of the JTFA frequency to see more or less of the JTFA frequency range-- more would update slower, less would update faster.  The raw data contour is stuck showing all the data all the time.  Glancing over the raw data snapshots, it looks like your acceleration sensor is railing a fair amount at the top and bottom of the voltage range-- you might want to lower the amplification.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Brad_Turpin]

Hi Jeff,

It turns out it was your acoustic sensor (microphone?) that's railing-- the acceleration data looks clean.  I made a last tweak to the snaphshot graphs so that they don't get confused if the order they're created in flip-flops (Group Name referencing instead of Group Index referencing).

It looks to me like your acoustic data has interesting amplitudes between 900Hz and 1300Hz pretty much across the whole measurement.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[jtw131]

Brad,

This is very very nice, i love that I can see the raw data snapshot at the same time as the FFT.  However, I think there is a time sync problem between the contour graphs. Notice the screen capture below -- I changed the frequency contour (JFTA) back to the linear scale for this example.  See how the raw data sample is +/- 400, but the corresponding FFT is nowhere near this amplitude? 

Another thing that I notice is that the timestamps that are not in the NMEA file are no longer discarded.  That was actually a very useful feature because the times at the beginning of the accel/audio files are often bad because there is not a good satellite fix yet.  The NMEA plugin automatically discards all GPS samples without a valid fix,

Going forward it would be great to be able to display accel X,Y,Z and Audio on four different worksheets, but have the cursor sync'd between all sheets.   Is this possible, or is cursor sync local to one sheet only?

The ideal way for us to browse the data would be by moving the cursor through the intensity plots rather than the speed plot.  The intensity plots allow you to quickly pinpoint areas of interest.  Stepping manually through the JFTA intensity plot almost works, but the cursor in the raw data intensity plot is not visible, nor does the long/lat plot update. 

The addition of the log JFTA values makes the JFTA intensity plot easier to read.  The problem is that the live updated 2D FFT plot now does not have the correct Y Axis value anymore since we took the log10 of it (Makes for difficult comparison to raw data).  Maybe there is another way to skin the cat -- can we just define a non-linear color pallete for the intensity plot? 

Finally, you are correct about the audio data being saturated.  The mic amplifier cirucit had too much gain.  This has been fixed since my original post.

Once again, I want to say thanks for all your ongoing help.  I have learned a lot about diadem over the past two weeks and I am encouraged with the high quality support from NI, 

[Brad_Turpin]

Hi Jeff,

You were right, my latest edits lamed the JTFA time axis clipping-- I fixed that in the attached JTFA.vbs.  I also renamed the VIEW areas in that busy VIEW sheet so it will be easier to find them by name programmatically, and so that the names of the automatic Groups that get created for the Raw and FFT slice graphs make more sense.  I also added the raw data to the first 2 REPORT sheets and made all the VIEW and REPORT sheets reference all the channels (except from the 1st Group) by Group name rather than Group index.

How will the VBScript know which loaded Groups are X, Y, Z, Sound data?  Can we rely on the name of the data files and their loaded Group names?  If so, what are the naming conventions, if not, what else can the script key off of?  It IS possible to have the VIEW cursor maintain its value between sheets, but we'll need to employ a UserCommand to automatically kick in every time you switch between VIEW sheets.  I've done it before and still have the code.

I'm also working on a UserCommand to force the Route graph to update when one of the intensity/contour graph cursors is moved.  The reason it's not updating automatically is that it's a second-order cursor correlation.  That graph just has "Longitude" and "Latitude" cursors.  The GPS map correlates the TimeStamp cursor with the Longitude and Latitude cursors.  I'm not sure why the Speed cursor updates the Longitude and Latitude cursors and the Contour plot cursors don't, but we're asking quite a bit of the built-in automatic update services at this point.

It occurs to me that it would be cool to zoom all 3 of the left VIEW areas into a selected time window (and out again) to take a closer look at it.  We can add add custom menus or icons to make that possible if you like.

This really is above and beyond the normal support level.  You're benefitting from my personal interest in employing new DIAdem technology for this exact type of use case.  Perhaps we could turn the resulting application into a User Solution we could post to the NI web site-- if that's OK with your company (not a precondition for further support).

Brad Turpin

DIAdem Product Support Engineer
National Instruments