Generating histogram of property values?

[az_ltr]

The data finder works really well for pre-determined property values or logical ranges.  For example find all channels whose property.Result <> "Pass" or result > 8...  You can get the results back pretty efficiently.  But in the case that you need to determine the unique properties before you can perform a search than this is not as efficient.  So using the result example above if we generalize to a non fixed set then we need to query all the channels in the dataset to determine the unique values for that property.  How about we assume that result is now a 16 bit number of random values.  But I need to summarize my dataset fail count by Result.   You would start by creating a query that finds all relevant channels and having a result property column in the search.  Then you would take all the result values into a new channel and find the unique values or do this while iterating through the query element list.  

But given my dataset of 92 files each with 12 x 80 captures that produces 88k channels per dataset which is above the 32k limit for a query response list.  So I cannot have just a simple search and count... So to work around this I am finding all the files and performing 92 separate searches and building a dictionary as I iterate through the element list.

What are some approaches that people have used to perform this type of search?

Some thoughts we had were

1) Store in file properties the values that are unique to file itself.  Then the query is more simple and there should be a speed up.  But this approach pushes more logic into the test program itself and essentially is making me write a mini indexer but should work in some cases.

2) Use the index file itself?  Thinking about idea 1 made me think that the properties are already indexed so can't I leverage the indexer in some way?  I did not see a property only query/function that would do this.  Did I miss it?

- Looking in the index file itself seems like all the properties are stored in the index file already.  I see an entry for each channel.  So for the channel property Result number I see 960 entries which is 1 per channel.  So the data is already indexed just not sure how to get to it.

3) Search for more files at a time rather than 1 by 1... This probably a speed up since you are performing less queries but the iteration count is still the same.  Assuming your channel count per file is consistent then you could search one file and then scale up to the max channel count.  But generically you probably cannot assume all datasets have the same number of channels.

4) ?

Talk Soon and regards,

[az_ltr]

4) build a negative search...Have not tried it yet but perhaps a negative case search.  

      Assume property result has values of E,B,A,D,C...  

    - Find the first channel.  Can limit search result to something small.

    - Find the first property value and store it into the query list with a <>… Query is now Result<>"E"

    - Search again and add the next value to the query...  Result <>"E" and Result <> "B"   

    - Search again and add the next value to the query...  Result <>"E" and Result <> "B" and Result <>"A" and Result <> "D"

    - Search again and add the next value to the query...  Result <>"E" and Result <> "B" and Result <>"A" and Result <> "D" and Result <> "C" 

until search result is 0...  I like this approach as it is leveraging the DIAdem query.  You only need to load one item from the query and the contents can be whatever.  If you need the histogram you loop through the conditions that you added...  So it goes from 92 searches and iterating to 5 searches without any iterating.  I will give this a go to see if it is faster.

[gsklyr]

I like option 4.  Also could process the first X results in the query, then mark them as processed in a property that way they don't come up next time your run a query.  This will be slower the first time but easier once you add new files.

[az_ltr]

Example Data set and VBS code attached.  

[16704] NI: getUniquePropertyValues loaded...
[16704] NI: >>>>unit_test
[16704] NI: >>>>clearSearchItem
[16704] NI: <<<<clearSearchItem
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<type = addSearchItem @ file
[16704] NI: >>>>findUniquePropertyValues
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(1) Name=serial_number Value=22
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(2) Name=serial_number Value=33
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(3) Name=serial_number Value=44
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(4) Name=serial_number Value=66
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(5) Name=serial_number Value=0
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(6) Name=serial_number Value=2
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(7) Name=serial_number Value=4
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(8) Name=serial_number Value=5
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 1 tdm_channel items.
[16704] NI: localDic.Added(9) Name=serial_number Value=100
[16704] NI: >>>>addSearchItem
[16704] NI: <<<<serial_number <> addSearchItem @ tdm_channel
[16704] NI: Found serial_number 0 tdm_channel items.
[16704] NI: ResultDic.Added(1) Name=serial_number
[16704] NI: <<<<findUniquePropertyValues
[16704] NI: property=serial_number:
[16704] NI: item: 22
[16704] NI: item: 33
[16704] NI: item: 44
[16704] NI: item: 66
[16704] NI: item: 0
[16704] NI: item: 2
[16704] NI: item: 4
[16704] NI: item: 5
[16704] NI: item: 100
[16704] NI: <<<<unit_test

The bold response is the unique values..  (DebugView shows 2.4 seconds to complete script on this very small dataset) 

Noticed that the query builder interface has a "..." section that allows you to select from a list with filter...  So DIAdem has this functionality built in... asked via ticket if this built in function is exposed somehow...But until then this could be used.

Attached code is use at your own risk, no warranties.

[Brad_Turpin]

Hey az,

I haven't digested this full discussion thread, so forgive me if I'm repeating what others have said.  I just want to make sure that you know that there is a way to ask the DataFinder for all the DISTINCT values of an optimized string property.  You can use the max/min request in a loop for datetime and numeric properties with a little more effort to enumerate their DISTINCT values as well.

Also, there's a shipping example in DIAdem that makes it easy to pivot from queried property columns to standard report types such as a histogram or a Yield% graph, etc.

Let me know if you're interested,

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[az_ltr]

Hello Brad,

Interested.  Did not know about the DISTINCT.  Searching for DISTINCT in help does not yield any results.  Which example?

Thanks,

LR

[Brad_Turpin]

Hi az,

Here's an example of requesting all the DISTINCT values of the ChannelGroup property "Measurement_Month" (string data type):

Set DataFinder = Navigator.Display.CurrDataFinder
Set QueryForm  = DataFinder.QueryForm
Set Query = QueryForm.GetCurrQuery
Set GroupProps = QueryForm.GetIndexedProperties(eSearchChannelGroup)
Set MonthParam = GroupProps.Item("Measurement~Month")
Set MonthList  = MonthParam.GetValueList()
iMax = MonthList.Count
FOR i = 1 TO iMax
  LogFileWrite MonthList.Values(i)
NEXT ' MonthVal

DISTINCT is an SQL term that is analogous here,

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[az_ltr]

Do not know SQL so thanks for the clarification. I was able to run the attached code and get values back for optimized index properties. But I am getting back all of the values.  I created four sample files and attached here.  But essentially contain:

Meas~Month    DATASET

chGrp                File

JUNE                1
JULY                 2
AUG                  1
MAY                  2

If I run: 

Option Explicit 'Forces the explicit declaration of all the variables in a script.
Dim DataFinder, QueryForm, Query, GroupProps, MonthParam, MonthList, iMax, i

Set DataFinder = Navigator.Display.CurrDataFinder
Set QueryForm = DataFinder.QueryForm
Set Query = QueryForm.GetCurrQuery

for i = 1 to Query.Conditions.Count
     LogFileWrite Query.Conditions.Item(i).Name & " " & Query.Conditions.Item(i).Property & " " & Query.Conditions.Item(i).Operator & " " &         Query.Conditions.Item(i).Value & " " & Query.Conditions.Item(i).Type
next

' Tried this no diff Call DataFinder.GetDataFinder.SearchElements(Query)
Set GroupProps = QueryForm.GetIndexedProperties(eSearchChannelGroup)
Set MonthParam = GroupProps.Item("Measurement~month")
Set MonthList = MonthParam.GetValueList()
iMax = MonthList.Count
FOR i = 1 TO iMax
LogFileWrite MonthList.Values(i)
NEXT ' MonthVal

The output is:  All Values...

C1 DATASET=2

AUG

JUL

JUN

MAY

How to apply the conditions in the query form to provide a subset of the values returned by GetValueList?

[Brad_Turpin · Accepted solution]

Hi az,

Sorry, I left out the (Query) parameter in this line of my code:

Set MonthList  = MonthParam.GetValueList(Query)

Brad Turpin

DIAdem Product Support Engineer

National Instruments