Actor Framework Documents

cancel
Showing results for 
Search instead for 
Did you mean: 

AQ Character Lineator (working draft version of serialization/deserialization)

[Edit] Vastly improved version 0.3 released Aug 31, 2012. Now with more data types and cleaned up error handling.

[Edit] Sept 4, 2012. Fixed the linking problem that was causing VIs to be broken on your machines.

[Edit] Sept 12, 2012. Full support for variants. More array data types. Problem with serializing time stamps solved so they maintain full precision in JSON. Still haven't tackled the UTF+8 issue for JSON.

[Edit] Oct 30, 2012. All data types now included. Test suite refined. INCLUDES CIRCULAR GRAPH OF DATA VALUE REFERENCES (unexpectedly easy solution).

[Edit] August 2, 2013. Large number of changes to framework. Expect documentation and changelog to come. Package built for LabVIEW 2013.

[Edit] March 4, 2014. Continued refinement of framework. 2D arrays and Filter removed to reduce bloat of the framework (2D arrays still addable as extended types if you want one in your serialization). Binary Serialize/Deserialize added. Split into two packages.


Introducing draft version of the AQ Character Lineator, a library for taking LabVIEW objects and reducing them to character sequences and then restoring the objects back later. Why the strange name? Because I got multiple people all having different opinions on what I should call this and I went with a name that has zero hits on Google (a true Googlewhack until someone mirrors this post).

These VIs or their evolution may become part of LabVIEW in the future. National Instruments reserves all rights necessary to allow their distribution as part of LabVIEW. Any modifications made may only be shared by posting back to ni.com. You agree that all modified versions posted may become part of LabVIEW in the future without any compensation to you.

Package is built for 2013 due to a new vi that the framework takes advantage of. It is possible to save it in previous versions but we lose some speed and efficiency.

So... what does this library do?

  • You can create new LabVIEW classes that inherit from Serializable.lvclass, give those classes ONLY TWO FUNCTIONS -- "To Serial Form.vi" and a "From Serial Form.vi" -- and then be able to flatten and unflatten from any supported file format. These two VIs use a could-be-easily-scripted pattern. Autogeneration of these two VIs is a desired feature for the future.
  • In the library, you will find serializers for JSON and Funky XML. The XML is a weird dialect that I made up to test edge cases of file formatting. The JSON is actual UTF-8 encoded JSON.
  • You will find a deserializer for JSON only. Parsing strings is very hard to do in any programming language because of the extreme attention to detail required. The Lineator helps as it gives you a good framework for knowing when you've covered all the cases that a parser must handle. My goal in creating this library is not to support all the world's file formats but rather to enable others to support file formats as needed. I made the JSON work. Writing a deserializer for any other format is something I hope someone else will take on. I have done a proof of concept for Binary String enough to know that it is possible to support unnamed formats like that. If I write another deserializer, it will be for the binary format. A proper XML serializer and deserializer is a substantial project that I hope someone volunteers for.
  • The following data types are supported:
    1. Analog Waveform
    2. Digital Waveform
    3. Digital Data
    4. Boolean
    5. Complex (Single/Double/Extended)
    6. Float (Single/Double/Extended)
    7. All Integer Types (signed & unsigned)
    8. Path
    9. Serializable (recursively)
    10. DVR of Serializable (recursively AND circularly linked if necessary)
    11. String (No need to include Picture String since the Type Cast node will allow you to cast one as the other)
    12. Timestamp
    13. Variant (containing any of the other types except Serializable)
    14. AND ARRAYS OF ALL OF THE ABOVE
  • The Formatter class can provide you with facilities to reformat data for localization. The FXML serializer uses the formatter. The JSON serializer ignores the formatter (JSON defines strict rules for all its data types).
  • You can do load from previous and save to previous mutations in your Serializable's To Serial Form and From Serial Form. Mutations can include renaming the classes.
  • The class paths can be embedded in the string in a few different ways, and there is built-in support for dynamic loading of classes into memory as the string is parsed, provided all the classes that you want to dynamically load are under a common root path.

There's pretty much zero documentation. VIs and classes are named with a fair amount of detail, but after that, well, it's a barren landscape. I will be filling this in in the next revision.

You're going to see some fairly non-standard error handling on these VIs... I've become opposed to putting "error in" on dynamic dispatch VIs, so I've got a lot more "merge error" nodes than normal whose only purpose is to preserve warnings. I hate warnings propagating on error wires... I'd throw them out of the language if I had my way. The error handling on these diagrams would be a lot cleaner if I stopped trying to preserve warnings. The whole error system may get reworked in a future revision of this library, but it works fully as it stands and I may leave it, depending upon your feedback.

How do you start:

Package installs into the vi.lib under the folder NI. Open the enclosed project file. Inside, you'll see a virtual folder for "Test". That is where you will find a partial test plan for the library. Open and run "BIG TEST To String.vi" for a sample of most of the data types in both FXML and JSON and to see how the mutation works across version bumps of your classes. Open any of the "TEST_ME.vi" files to test specific subsets of the type functionality. I am still finishing out the test suite. Any type that does not have a named test suite is not fully tested and is much more likely to have bugs at this time. The ones with named test suites are pretty solid.

The Test....lvclass files are examples of the types of classes you would create to be serialized. That's the part 90% of you are interested in. The two inherited functions in those classes are the ones that we want to have a nice scripting tool to generate (the magic scripting tool mentioned in the design document).

The JSON Serializer class and the JSON Deserializer class are of interest to the next 9% of you who want to create custom file formats. You can judge for yourselves the amount of labor that will be needed to support a new case.

The next 0.9% of you may be interested in localization for human cultures. Take a look at the Formatter class.

The remaining 0.1% who actually want to figure out *how* it all works and make comments about the under-the-hood stuff are free to wander where you will. I have added limited comments at this point, but certain sections, like the handling of variants, will require some study to understand.

Comments welcome!

Comments
Proven Zealot
Proven Zealot

Version 0.3 posted Aug 31, 2012.

Active Participant DanyAllard Active Participant
Active Participant

AQ, I got broken VIs with the new Error Ring.

"Error Ring: Library version is incompatible with XNode version"

Anybody else have those broken VI's

Dany

Dany Allard

Active Participant Daklu Active Participant
Active Participant

Yep, I get that error too.

Proven Zealot
Proven Zealot

Ah... I modified the Error Ring on my local machine... I was playing around with an idea and forgot to put it back. this is the equivalent of the "conpane doesn't match" error for subVI calls. For the moment, just delete the nodes and drop a constant that returns the same error code number. I'll try to put up a relinked version soon.

Proven Zealot
Proven Zealot

Ok... new version should have the linking issue fixed up. Sorry about that... a hazard of my experiments mixing with each other.

Active Participant DanyAllard Active Participant
Active Participant

Good Thank you

Dany Allard

Member jarcTek
Member

I am very interested in the JSON serializer and deserializer.

Thank you!

Proven Zealot
Proven Zealot

jarcTek wrote:


                       

I am very interested in the JSON serializer and deserializer.

Thank you!


                   

Not while I'm developing. I work in ine version to avoid cross links and to keep my workload down (this is an outside-of-work project). I work in the latest LV version to exercise it and identify issues. When the library is complete early next year (I hope), I'll investigate back porting. No promises. In the meantime, anyone else who wants to do the backport of the interim files is welcome to do so.

Active Participant LVB
Active Participant

Any plans to include support for converting a JSON string to XML string?

CLA, CTA
Active Participant stbe
Active Participant

Hi AQ,

I really like your approach on this one. Hope to see it in any future LV version ...

During Beta-Testing of LV 2012 I also wrote a small LV<->JSON serializer that also did nice with UTF encoded+encoding strings. I used an intermediate reprensentation of the data in terms of recursive LV Objects (for any supported JSON datatype - fortunately only 6 - I had a class derived from a generic "Value" class; Array and Object would store an array of "Value"s - with/without labels)

LV_JSON_hierarchy.png

In addition, I managed to write a very cool feature to "LabVIEW-script-create" clusters from a JSON description, however this renderer is working only in the development environment (but could be worth a nice QD plugin ).

Basically, JValue knows 2 parser and 2 renderer, so it's possible to read/write JSON text and read/write LV clusters. I never planned it to extend to XML (I would need to create all the datatypes and write the parser/renderer for each) but there's already a toolkit for that .

If you're (still) interested in my version, I'd be happy to post it to a proper location in the NI community.

-Benjamin

_________________________
CLA
Proven Zealot
Proven Zealot

LVB wrote:


                       

Any plans to include support for converting a JSON string to XML string?


                   

Not directly, but if you unserialize the JSON string you can serialize the objects as XML (assuming someone completes the XML serializer).

Proven Zealot
Proven Zealot

stbe wrote:


                       

If you're (still) interested in my version, I'd be happy to post it to a proper location in the NI community


                   

I think I'm past the need for that kind of parsing code at this point. I'll get back to you if I have problems. Once I get the cast off my hand, I'll get back to the project.

Proven Zealot
Proven Zealot

Latest version is a full complete draft with all data types included, a partial test plan, significantly cleaned up code base and some really nice unexpected features such as DVR circular graph serialization.

Member DavidAMoore Member
Member

What's the intended use for the version stuff?

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Member DavidAMoore Member
Member

FYI, I'm 4 hours into a serializer to an ini format similar to what MGI's Read/Write Anything uses.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Member DavidAMoore Member
Member

Finished the serializer, which seems OK, but I need to write the deserializer to test properly. Only minor issues so far, which I could provide privately.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Proven Zealot
Proven Zealot

DavidAMoore wrote:


                       

What's the intended use for the version stuff?


                   

The version number allows you to define a version for your protocol so that if you are unserializing information from an earlier version of your set-of-classes, you can mutate it forward. Likewise, you can flatten for previous. Predominantly these become interesting when a server revs but not all clients rev (or vice versa, as is the case with browsers).

By "set-of-classes", I mean all the classes that are participating in your particular protocol -- so any class that goes into your communications stream, which may include many unrelated class hierarchies.

Active Participant cirrusio
Active Participant

AQ, why is this in the Actor Framework forum? It doesn't necessarily look like this related to the AF and it seems like this would benefit from wider distribution.  Matt

Proven Zealot
Proven Zealot

mtat76 wrote:


                       

AQ, why is this in the Actor Framework forum? It doesn't necessarily look like this related to the AF and it seems like this would benefit from wider distribution.  Matt


                   

Because it started life as the mechanism for streaming AF messages across the network in a way that could support independent revision of sender actor and receiver actor. And then it grew from there. :-)

I've posted it by itself over on LAVA and a few other key users were pinged to make sure it didn't get missed when it outgrew its original goals.

Member DavidAMoore Member
Member

AQ,

If you try to run Big Test From String.vi while TestData.lvclass:From Serial Form.vi has its front panel open, you hang LabVIEW. Verified on a virgin copy of the code. Are we bitten by a LV bug or just missing a timeout?

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Proven Zealot
Proven Zealot

David: Bug verified. I'll see what I can find.

Proven Zealot
Proven Zealot

Hah... I went away and 15 minutes later, LV unhung itself. Of course, it did so with a "VI Failed To Compile" error that broke the hierarchy, so there is still a bug.

Proven Zealot
Proven Zealot

Grr. Argh. BLARGH.

I have no idea what the bug actually is, but the problem seems (emphasis "seems") to be springing from the recursion. If you remove the add-on data type classes (Serializable Variant, Serializable Analog Waveform, etc) and edit the test cases to not use them, the problem seems to evaporate.

Thank you for reporting this. I've pushed it over to the appropriate developers.

Active Participant cirrusio
Active Participant

Sorry to hear about the bugs.  I have successfully deployed this on an RT machine.

Proven Zealot
Proven Zealot

The easy workaround to the bugs appears to be "don't open the front panels". Obviously, that's pretty easy for RT. :-)

Member DavidAMoore Member
Member

I think I’m hitting another recursive hang when I try to save TestNestedData.ctl with a new default value.

Formatter.lvclass:Timestamp To String.vi produces a data mismatch with the default timestamp value, I think because %20u has too much data. I’m trying to test alternatives to 20 and see if there’s one that’s generally better. Since I couldn’t change the default value, I tried putting a test VI in the formatter class, hit another hang, and cancelled. I couldn’t recover without hitting another hang. Now my formatter class claims that it’s missing a VI, but nothing shows up in the project window as missing. Grr. Argh. BLARGH.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Member DavidAMoore Member
Member

Figured out the Timestamp issue. String to Timestamp.vi called String to Extended.vi for the fractional seconds (possibly because that VI had the wrong icon, looking like String to Double.vi. Replacing with String to Double.vi and setting Timestamp to String.vi to use %16u, get reliable matches. (Oops, only reliable right now, other days have issues).

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Proven Zealot
Proven Zealot

Remove the secondary data type formatter classes from the project and from the Poly VI. Then they won't even load into memory. That should eliminate it. I'll keep my eyes open for a better solution.

Member DavidAMoore Member
Member

timestamp bug snippet2.png

OK, it's timestamp to datetime record conversion that's the culprit.

Snippet above easily yields a false except with integer numbers of seconds. Format to string and from string with timestamp codes has the same limitation. I think TestNestedData.lvclass:Compare elements.vi can't work as designed. It's too bad we don't have operator overloading on the Equal? function so that we could just put a tolerance in for timestamps.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Member DavidAMoore Member
Member

I've posted my code here AQ Character Lineator INI Serializer/Deserializer.

Now I figure out what I learned for an upcoming user group discussion.

(Is Austin near a wormhole? I post this message and the timestamp is 10 minutes after the local time and 45 minutes before some existing messages.)

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Proven Zealot
Proven Zealot

Oh... yeah... I could've told you that. I discovered that when writing my test cases. Sorry... I focused on the compilation bug and not the comparison issue. You can see the hoops I jumped through in my test cases code for comparing before and after data... I filed the CAR and it got closed as "expected behavior", so I refiled the CAR to have a "does not preserve value" added in blinking red 48pt text to the Context Help window for those primitives. I've been told they'll likely add a sentence but not in blinking, red or 48pt.

Proven Zealot
Proven Zealot

You should also be aware that there is *no* mechanism in LabVIEW to take a floating point value and generate a decimal string such that you get a decimal with enough precision to guarantee that the original decimal can be reconstructed unless you specify the absurd precisions that I am currently specifying in the code, and even then there are some edge case values for which you just cannot compensate. It would require a custom primitive in LabVIEW to produce the value, distinct from the existing Flatten To String because some bit signatures map to the same decimal values beyond the declared precision of the data types. I've talked to the mathematicians on the LV team and they tell me it is a long standing problem they face reguarly. I am totally unhappy with the handling of floating point values at this time.

Active Participant David_Staab
Active Participant

AristosQueue wrote:

               

I filed the CAR and it got closed as "expected behavior", so I refiled the CAR to have a "does not preserve value" added in blinking red 48pt text to the Context Help window for those primitives. I've been told they'll likely add a sentence but not in blinking, red or 48pt.

                   

And dozens more developers will fail to see that discreet new sentence, costing their employers hundreds of engineering hours more while they each debug the problem. :\

Proven Zealot
Proven Zealot

Yeah. But this isn't a case of the CAR being closed because it is too much work or not worth doing. There's no solution for it no matter how many developers you want to throw at the problem. There's no way to encode the original timestamp value precicely as a floating point number of partial seconds. It just mathematically isn't possible ... it will always be lossy. The format of the time record would have to be changed to be an extended precision instead of a double precision, which is a larger breaking change, a hold over from when timestamp itself could only be as precise as a double precision variable.

Active Participant cirrusio
Active Participant

OK...so I may have jumped the gun on the RT thing.  I appear to be having some issues building an exe and it seems to be centered around the Remove Serializable Array vi (I believe the error is 1502 - can't build with bad VI or something to that effect).  This may be a problem on my end, but whenever I have had it, I have always had to dig around and remove the problem and reimplement so I am not sure if I will be able to move forward from here....

Active Participant cirrusio
Active Participant

Ugh...now at a dead end with this 1502 build error.  Can't seem to track down where it is coming from.  The error seems to originate with Remove Serializable Array.vi, but cutting bits and pieces out leads me through the core to Deserialize.vi and ultimately to From Serial Form.vi (in which there is nothing left to cut out).  I have tried changing the build properties on this (not removing the block diagram or front panel, disconnecting type defs, etc, etc), but to no avail.  I have even tried to replace the original code with a freshly downloaded code set.  Nothing seems to be fixing this problem.  Will now have to start winding back...sigh.  This is an awesome idea, just wish it would build. m

Active Participant cirrusio
Active Participant

OK - so I am running into the same problem when I build an .exe from the test project using the BIG Test from String.vi as the source.  Can someone else take a look and see if they can build an exe from the test VIs supplied by AQ?  Maybe this is related to my LV installation (fingers crossed).

I don't seem to be having a problem building an .exe from the BIG Test to String.vi.

Member DavidAMoore Member
Member

I got error 1502 regarding Deserialize.vi.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Active Participant cirrusio
Active Participant

Sigh.  That doesn't bode well.

Proven Zealot
Proven Zealot

Try the same workaround I offered to David of removing the extended data type classes ... yes, that means removing them from the poly VI, the project and all the points in the test where they're used. Not trivial, but straightforward. I believe there is something about the recursion causing problems.

Active Participant cirrusio
Active Participant

So does this mean that without the recursion, the library no longer supports nesting of serializable types?  (I believe the answer is yes unless I am missing something)

Active Participant cirrusio
Active Participant

Starting to get somewhere... I think the problem lies in the execution settings.  I am now no longer getting the 1502 error on Remove Serializable Array.vi.  I changed 'Shared clone reentrant execution' to 'Preallocated...' for this vi and the Remove Serializable.vi.  I am now getting a 1502 on Deserialize.vi.  I am going to keep digging.

Active Participant cirrusio
Active Participant

Never mind...either it was never fixed or magically back.

Active Participant cirrusio
Active Participant

Hallelujah!  I finally got what you were saying AQ and removed the extra "serializable" add ons and got a good build.  I should probably make a more concerted effert to listen and understand those wiser than I...

Active Participant cirrusio
Active Participant

AQ - can you say something about the the Name Substitutions in the Serializer and Deseralizer classes?  I am assuming that they are available so that you can retarget the location?

Active Participant cirrusio
Active Participant

So, as it stands, the deserialization is not compatible with RT.  Thus far, Deserializer::Get Class Default Value By Name uses a property node that is not compatible with RT (ProjectItem->Path).  And, I am not really understanding what the code is doing.  In Deserializer::Load Top-Level Dynamic Paths, it appears that the deserializer is getting the default values based on the paths available through the ClassPaths identifier (and Deserializer::Load this Level Dynamic Path appears to do the same for the "One path with each object" formatter specification).  But, then this vi does nothing with the default values.  Am I missing something?

Member DavidAMoore Member
Member

I've tried a real world example in a large project (~5600 files). It chokes on 46 calls to Deserializer.lvclass:Get Class Default Value by Name.vi which average about 0.7s each, for a grand total of over 32s. I'd want it to run 50x faster to give reasonable performance.

In BIG TEST From String.vi there are 24 calls averaging 0.15s, and again it's the dominant time by a wide margin. Any hope for an optimization?

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Proven Zealot
Proven Zealot

> Any hope for an optimization?

This is a first draft wherein I was trying to get the functionality right. I haven't done any work on performance yet, other than keeping in mind throughout the development trying to make the system not create data copies of the elements during serialization. I have not actually confirmed that I was completely successful with that, nor have I done any profiling.

Can you do profiling without using any particular file format? That would give us a good clue about whether this is something about the particulars of the JSON parsing that I coded (which I will happily admit is totally outside my skill set) or in the serialization framework proper.

Proven Zealot
Proven Zealot

And I forgot to mention... Deserializer.lvclass:Get Class Default Value by Name.vi will be a lot faster in LV 2013. I had to ammend the underpinnings of LV to accelerate that function, so I can't backport it to LV 2012 or earlier.

Member DavidAMoore Member
Member

I think that 24 and 46 were reasonable numbers of calls to Get Class Default Value by Name in each case, so cutting down the time for each call, which you are on top of, is the essential thing.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
Contributors