Convert Complex (Pink) Cluster to Byte Array

[Tanner!]

There are some good ideas here with regards to using variants, you might check out this library I made for creating, modifying, and interrogating cluster variants:

https://github.com/tannerblair/advanced-labview-variant-tools

It should make some of what you're wanting to do a lot easier.

[ajf200]

---

@Yamaeda wrote:

You can as an alternative convert the cluster to a JSON string and send it as byte Array. As it's a string it's easy to read and interpret and handles Changes better.

---

This has exactly the same issues as Flatten to String:

1. It does not handle complicated data types. In fact it's even less flexible than the regular Flatten To String because it does not handle datatypes like Enums.

2. It adds tons of extra tags and data (i.e. bytes) that are not compatible with the interface requirements.

[ajf200]

---

@Yamaeda wrote:

Maybe we should attack this from Another angle. What's the definition you need to adhere to on the DLL side, and/or, can't you add a function to handle some LV format?

---

No. It's a defined interface with requirements already set.

[ajf200]

---

@Kevin_Price wrote:

So if the data structures change during development, the external app has to be changed in order to digest them properly again, right?

My thoughts are with Yamaeda.  Complicated, hierarchical data structures that are subject to change don't seem to map well to raw byte-mapping manipulations.  JSON, XML, or some analogously hierarchical and self-defining scheme sure *sound* like a more natural approach overall.

I realize this doesn't solve the problem you asked about, but I for one sometimes find myself locked in on a difficult solution path where a few impertinent questions from a colleague help me realize that I should try a whole different approach.

-Kevin P

---

I get what you're saying, but going back to the team and saying "let's change the interface because LabVIEW cannot easily comply with it" is not really an option. The interface is set.

And just to be clear, there is an interface description langauge. All of the data is defined in a text document, similar to JSON. However, the data itself is passed into this particular inteface as bytes of data only. The applications parse the bytes, based on the IDL.

If I could reframe the problem, I would. But this is the requirement we are dealing with. Just looking for an easier way to do it, and I think I developed one.

[ajf200]

---

@Tanner! wrote:

There are some good ideas here with regards to using variants, you might check out this library I made for creating, modifying, and interrogating cluster variants:

https://github.com/tannerblair/advanced-labview-variant-tools

It should make some of what you're wanting to do a lot easier.

---

Thanks. I'll check these out later tonight.

[Tanner!]

Cool. In particular, that library has the ability to parse a variant, partially parse a variant given a path, dynamically build clusters at runtime, insert elements at an arbitrary path, etc. It basically lets you put off defining the data structures until runtime. It also includes some helper VIs to:
- parse JSON to/from a variant

- show the contents of a variant in a tree
- create a Control VI from a variant that you can insert into a subpanel to dynamically define UI elements from an external source.

Let me know if you find it helpful, and I'm always taking PRs!

[ajf200]

---

@Tanner! wrote:

Cool. In particular, that library has the ability to parse a variant, partially parse a variant given a path, dynamically build clusters at runtime, insert elements at an arbitrary path, etc. It basically lets you put off defining the data structures until runtime. It also includes some helper VIs to:
- parse JSON to/from a variant

- show the contents of a variant in a tree
- create a Control VI from a variant that you can insert into a subpanel to dynamically define UI elements from an external source.

Let me know if you find it helpful, and I'm always taking PRs!

---

I think the next big challenge that I've been putting off for awhile is to populate a cluster of any time with data from an array of bytes or doubles, in which the array contains data only (no extra bytes or values for description or sizes). Basically going in the opposite direction as this problem. We are able to dynamically create clusters already.

I tried doing this in the past using Variant functions, but it never seemed to be fast enough (this is for a RT system, my loop timing would fail to keep up).

The thought on this is to use VI scripting to dynamically create a VI for each cluster definition that will use a for loop/case structure combo to iterate through all of the cluster elements and the array at the same time and populate each using typecasting and then bundle by name. I think I remember hitting a wall on this trying to get references to the elements in subclusters correctly. I have another thread on that I'll bump and send to you when I get some example code to upload.

[RavensFan]

It seems like you are trying to solve a problem that doesn't need to be solved very often.

1.  You want something that takes an arbitrary complicated  (better word then complex) structure, and rebuilds it to a byte array automatically.

2.  You don't want to hard code anything because someone might come along in 3 months and change the structure.

3.  You say the structure is already determined and hard-coded in a dll.  That implies it is unlikely to change.

So exactly why would someone come along and change something in 3 months?

Is it worth the effort to automate this?  It reminds me of this:

https://xkcd.com/1205/

[ajf200]

---

@RavensFan wrote:

It seems like you are trying to solve a problem that doesn't need to be solved very often.

1.  You want something that takes an arbitrary complicated  (better word then complex) structure, and rebuilds it to a byte array automatically.

2.  You don't want to hard code anything because someone might come along in 3 months and change the structure.

3.  You say the structure is already determined and hard-coded in a dll.  That implies it is unlikely to change.

So exactly why would someone come along and change something in 3 months?

Is it worth the effort to automate this?  It reminds me of this:

https://xkcd.com/1205/

---

I might not be doing a great job of explaining the problem and constraints well, but there is only so much I can convey via forum. The short answer is yes, it's worth solving. I'm aware of the xkcd comic, and we've already debated this, but consider:

1. We've already spent probably 4-5 days worth of man-hours on this specific problem (and related problems).

2. The changes will happen. They have happened, they continue to happen. We don't know when, but as issues arise or features are added, they get put in. Those changes can result in hours, if not days of re-work, and again, it is error prone since it's all done manually.

3. This is a tool and method that will be used on future projects which currently don't have IDLs defined.

4. The DLLs change as the IDLs change, but they are able to be auto-generated. However, the LabVIEW test system is just one component that is connected to this project. There are dozens of other nodes that are written in various languages that are working just fine. Should we change the protocol just because LabVIEW can't easily adapt to it?

[Kevin_Price]

[Edit: wrote this before seeing OP's recent reply]

Nice one, RavensFan!  That same xkcd comic hangs on cube wall of a colleague here.

With that thought in mind, could a less automated, more hard-coded approach (with versioning) make sense?

My LVOOP skills are pretty meager, but it seems like there could be a decent fit where your data structures go into concrete child classes, conversion to and from byte arrays are overridden member functions, and the block diagram code is written in terms of an abstract parent class that the children are derived from.

When a data structure changes, the main framework of the code which is written around the abstract parent functions doesn't need to change.  You just derive a new child class from the abstract parent, hardcode the byte array conversion functions, and change the code in the main app where you choose which child to instantiate.

Automating is elegant, and there are many times it pays off.  But there are other times where a bit of ugly grunt work or operator training achieves results faster.   Hence the comic I guess.

-Kevin P.

P.S.  Adding this after reading OP's recent reply.   And I mean this in a spirit of commiseration.   On the one hand, it seems reasonable not to change the byte-mapping protocol when LabVIEW is seemingly the only language that doesn't handle the changes nimbly.  On the other hand, at a higher organizational level, there should be a recognition that it might make more sense to settle on a protocol that caters to the least nimble language.

    Why not just accept oddball LabVIEW's way of storing the data?  You won't earn any extra love for LabVIEW from the text programmers, but presumably it shouldn't be hard for them to compensate with the byte-padding needed to match LabVIEW's memory layout.

    Yes, the data structure will get a little larger.  That's one tradeoff, though hopefully a pretty small one.  But organizationally, most of the change work is deferred to the "nimble" languages with more natural auto-coding support.  Seems like a potential win.   Just good luck convincing all the stakeholders.  😞