"Reverse Lookup" using Variants?

[Kevin_Price]

FWIW, I never did any benchmarking on the string reversal.  I assumed (maybe incorrectly) that string reversal doesn't actually have to move bytes around just like array reversal and reshape don't.  I figured there's some metadata parameter that gets set to define traversal or somesuch.

I brought up the idea of reversal because it "seems right."  Reversal should offer some theoretical lookup advantage when a lot of the leading bytes are shared by a lot of the lookup keys.   Such as a bunch of arrays of the same size.  Whether there's any discernible *practical* advantage I do not know. 

Another tidbit on use of variant attributes as a lookup table:

    There was a thread talking about the crossover point where variant attributes become faster than using a pair of arrays with a linear search, depending on the size of the lookup table.  Somewhere in the low 10's of items in the set seemed to be the crossover point in that thread.

However, most of the benchmarks there merely found that the lookup item existed.  When I had done some brief similar benchmarking, I made sure to include the cost of converting from variant back to native datatype.  That conversion made a big difference, moving the crossover point by an order of magnitude from 10's of items to  100's.   I'm sure this would vary with the specific datatype and the specific cost of converting a variant to it.

So while variant attributes provide a nifty mechanism for a lookup table, they aren't *automatically* faster or better.  Getting back to the original topic, a single lookup table based on a pair of arrays accomodates forward and reverse lookups equally well without requiring duplication.

-Kevin P

P.S.  Found that thread with discussion about the crossover point between arrays and variant attributes.

[Yamaeda]

Depending on the value range and granularity you could simply have an Array of strings and index them out with a value. If you have decimals you could e.g. multiply with 1000 to use as an index.

Ofcourse if the value range is big and/or the strings many, this might not be an option, but it's fast. 😄

But, let me get this straight, you have an Array of values, all connected to 1 string, so basically a named Array, right? You want to find this name from the Array instead, right? Is all the numbers in the Array unique, or just the total?

One option to the variant attribute is to store the Array in a queue with your preferred name. Then create a queue with the Array type cast to string or flattened, used as name with the original name as data.

It's basically the same idea, just through queues. 🙂

/Y

[wiebe@CARYA]

---

@Yamaeda wrote:

It's basically the same idea, just through queues. 🙂

/Y

That could make a huge difference. To\From Variant conversion cost time. The variant parameter lookup is very efficient though. Queues are as efficient as it gets.

A downside would be that named queues are global. So if you make a class that uses them, you can only use one instance of that class. That could be a feature of course. But you can only use the trick once in an application. You can get a bit more slack by prepending the queue names, but that is just a (useful) hack.

[paul.r]

Flattening the array to a string and using that as the name seems horribly inefficient as soon as that array has a decent amount of data in it. A quick benchmark showed using DVRs is orders of magnitude faster once you get beyond an array with ~100 or so elements.

[wiebe@CARYA]

---

@paul.r wrote:

Flattening the array to a string and using that as the name seems horribly inefficient as soon as that array has a decent amount of data in it. A quick benchmark showed using DVRs is orders of magnitude faster once you get beyond an array with ~100 or so elements.

---

Better post that benchmark then.

AFAIK, DVRs are a big no-no. Simply because they are a pain to debug.

[abvenk]

---

@Yamaeda wrote:

But, let me get this straight, you have an Array of values, all connected to 1 string, so basically a named Array, right? You want to find this name from the Array instead, right? Is all the numbers in the Array unique, or just the total?

---

I have a bunch of string-and-1D array pairs. Each string corresponds to a unique 1D array. So when I feed in a 1D array, I want the code to find the corresponding string. 

[paul.r]

---

wiebe@CARYA wrote:

---

---

Better post that benchmark then.

AFAIK, DVRs are a big no-no. Simply because they are a pain to debug.

---

Benchmark attached. Of course, the problem with this solution is that you need to keep track of the DVR and array, but depending on the use case that may or may not be a problem. If all the array operations can be done in place, the performance difference becomes much larger. 

[abvenk]

Thanks everyone, for the lively discussion. Keep the ideas coming 

So far, I have two ideas I could try:

1. Flatten to String

2. DVRs

I clarified my need again in the post above... I have a bunch of string-and-1D array pairs. So when I feed in a 1D array, I want the code to output the corresponding string. So there is a bunch of unique strings and a bunch of unique 1D arrays, just to be clear.

[Kevin_Price]

Good point on "Flatten to String".  The times I've used it, I had data structures occupying more like dozens of bytes rather than 100's or more.  For that matter, most of the lookup tables I've made have been small enough that the lookup speed of variant attributes probably wasn't important in the app.

Either DVR's or single-element Queues seem like appropriate possible solutions to me, as long as the original data wire terminates without branching when the DVR/Queue is created.  (If it branches such that a copy needs to be made, then it seems like we'd be back to the same inefficiency as Flatten to String.)   I'd probably opt for a Queue first due to familiarity, but don't have any specific reason to prescribe that as a better option for others.

-Kevin P

[Kevin_Price]

To the OP:  a few fairly important questions

1. About how big a lookup table do you expect to need (what # of key-value pairs)?   If you're in the realm of 10's or 100's of pairs, variant attributes may not be a particular advantage over a conventional pair of arrays that are searched linearly.

2. How big can the unique arrays get?   The inefficiency of Flatten to String may not matter a lot if the arrays are pretty small.

3. Consider whether developer (you alone?) and maintainers of the code might benefit from a conceptually simpler approach even if it wastes some memory and microseconds.     To me, the simple pairs of arrays makes for the most conceptually simple and transparent of all these methods.   That's worth something for debugging and maintenance, and may be worth a little tradeoff against raw execution efficiency.

Note also that methods based on references such as DVR's or Queues have an additional failure mode not shared by data-based methods.  Between parallel code and dynamically launched vi's, there are ways for those references to become invalid while parts of your code still depend on the data they (used to) refer to.  Good programming practices can prevent and avoid this pitfall, but it *is* "out there".

-Kevin P