Actor Framework Documents

cancel
Showing results for 
Search instead for 
Did you mean: 

AQ Character Lineator (working draft version of serialization/deserialization)

[Edit] Vastly improved version 0.3 released Aug 31, 2012. Now with more data types and cleaned up error handling.

[Edit] Sept 4, 2012. Fixed the linking problem that was causing VIs to be broken on your machines.

[Edit] Sept 12, 2012. Full support for variants. More array data types. Problem with serializing time stamps solved so they maintain full precision in JSON. Still haven't tackled the UTF+8 issue for JSON.

[Edit] Oct 30, 2012. All data types now included. Test suite refined. INCLUDES CIRCULAR GRAPH OF DATA VALUE REFERENCES (unexpectedly easy solution).

[Edit] August 2, 2013. Large number of changes to framework. Expect documentation and changelog to come. Package built for LabVIEW 2013.

[Edit] March 4, 2014. Continued refinement of framework. 2D arrays and Filter removed to reduce bloat of the framework (2D arrays still addable as extended types if you want one in your serialization). Binary Serialize/Deserialize added. Split into two packages.


Introducing draft version of the AQ Character Lineator, a library for taking LabVIEW objects and reducing them to character sequences and then restoring the objects back later. Why the strange name? Because I got multiple people all having different opinions on what I should call this and I went with a name that has zero hits on Google (a true Googlewhack until someone mirrors this post).

These VIs or their evolution may become part of LabVIEW in the future. National Instruments reserves all rights necessary to allow their distribution as part of LabVIEW. Any modifications made may only be shared by posting back to ni.com. You agree that all modified versions posted may become part of LabVIEW in the future without any compensation to you.

Package is built for 2013 due to a new vi that the framework takes advantage of. It is possible to save it in previous versions but we lose some speed and efficiency.

So... what does this library do?

  • You can create new LabVIEW classes that inherit from Serializable.lvclass, give those classes ONLY TWO FUNCTIONS -- "To Serial Form.vi" and a "From Serial Form.vi" -- and then be able to flatten and unflatten from any supported file format. These two VIs use a could-be-easily-scripted pattern. Autogeneration of these two VIs is a desired feature for the future.
  • In the library, you will find serializers for JSON and Funky XML. The XML is a weird dialect that I made up to test edge cases of file formatting. The JSON is actual UTF-8 encoded JSON.
  • You will find a deserializer for JSON only. Parsing strings is very hard to do in any programming language because of the extreme attention to detail required. The Lineator helps as it gives you a good framework for knowing when you've covered all the cases that a parser must handle. My goal in creating this library is not to support all the world's file formats but rather to enable others to support file formats as needed. I made the JSON work. Writing a deserializer for any other format is something I hope someone else will take on. I have done a proof of concept for Binary String enough to know that it is possible to support unnamed formats like that. If I write another deserializer, it will be for the binary format. A proper XML serializer and deserializer is a substantial project that I hope someone volunteers for.
  • The following data types are supported:
    1. Analog Waveform
    2. Digital Waveform
    3. Digital Data
    4. Boolean
    5. Complex (Single/Double/Extended)
    6. Float (Single/Double/Extended)
    7. All Integer Types (signed & unsigned)
    8. Path
    9. Serializable (recursively)
    10. DVR of Serializable (recursively AND circularly linked if necessary)
    11. String (No need to include Picture String since the Type Cast node will allow you to cast one as the other)
    12. Timestamp
    13. Variant (containing any of the other types except Serializable)
    14. AND ARRAYS OF ALL OF THE ABOVE
  • The Formatter class can provide you with facilities to reformat data for localization. The FXML serializer uses the formatter. The JSON serializer ignores the formatter (JSON defines strict rules for all its data types).
  • You can do load from previous and save to previous mutations in your Serializable's To Serial Form and From Serial Form. Mutations can include renaming the classes.
  • The class paths can be embedded in the string in a few different ways, and there is built-in support for dynamic loading of classes into memory as the string is parsed, provided all the classes that you want to dynamically load are under a common root path.

There's pretty much zero documentation. VIs and classes are named with a fair amount of detail, but after that, well, it's a barren landscape. I will be filling this in in the next revision.

You're going to see some fairly non-standard error handling on these VIs... I've become opposed to putting "error in" on dynamic dispatch VIs, so I've got a lot more "merge error" nodes than normal whose only purpose is to preserve warnings. I hate warnings propagating on error wires... I'd throw them out of the language if I had my way. The error handling on these diagrams would be a lot cleaner if I stopped trying to preserve warnings. The whole error system may get reworked in a future revision of this library, but it works fully as it stands and I may leave it, depending upon your feedback.

How do you start:

Package installs into the vi.lib under the folder NI. Open the enclosed project file. Inside, you'll see a virtual folder for "Test". That is where you will find a partial test plan for the library. Open and run "BIG TEST To String.vi" for a sample of most of the data types in both FXML and JSON and to see how the mutation works across version bumps of your classes. Open any of the "TEST_ME.vi" files to test specific subsets of the type functionality. I am still finishing out the test suite. Any type that does not have a named test suite is not fully tested and is much more likely to have bugs at this time. The ones with named test suites are pretty solid.

The Test....lvclass files are examples of the types of classes you would create to be serialized. That's the part 90% of you are interested in. The two inherited functions in those classes are the ones that we want to have a nice scripting tool to generate (the magic scripting tool mentioned in the design document).

The JSON Serializer class and the JSON Deserializer class are of interest to the next 9% of you who want to create custom file formats. You can judge for yourselves the amount of labor that will be needed to support a new case.

The next 0.9% of you may be interested in localization for human cultures. Take a look at the Formatter class.

The remaining 0.1% who actually want to figure out *how* it all works and make comments about the under-the-hood stuff are free to wander where you will. I have added limited comments at this point, but certain sections, like the handling of variants, will require some study to understand.

Comments welcome!

Comments
Synaesthete
Member
Member
on

It looks like this thread has gone quiet for a couple months.  Is there still work on this?  The implementation in LV is pretty frustrating and tends to give very poor performance (I ran in to many of the same issues described in this thread when attempting my own implementation).  It seems that the 'right' solution would be to do this down in the VI Server architecture.  I'm guessing there must be work towards making this happen.  Is this up in the Ideas forum?

Active Participant
Active Participant
on

Since AQ announced in another thread (unrelated to this one) that he's on vacation for the next month, I'll try to throw out what I "know" to help answer your questions:

  • He's still very interested in working on this and needs more feedback from people trying to use it.
  • Performance and code size can, I think, be optimized by only including code for the datatypes needed. (I haven't used it yet.)
  • I strongly doubt that anything is going to be added to VI Server or any other non-G part of the language to solve this problem. I suspect AQ started working on this in G precisely because that's the only way he'll be able to add the feature to the language at all. (Personal opinion.)
  • If AQ has made a pet project of this, there's no need to add anything to the Ideas Exchange. Getting his attention is, for all intents and purposes, getting the attention of LV R&D's leadership.
DavidAMoore
Member Member
Member
on

Actually, there ARE behind the scenes changes in 2013 that should drastically improve performance, but AQ has not yet had time to refactor Character Lineator to leverage them. The lookup/load that is the performance killer in the current version can now be done more quickly. The current 2013 beta includes the changes, so anyone should be capable of trying the refactor, but I'm a bit foggy on what exactly needs to be done.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
AristosQueue (NI)
NI Employee (retired)
on

I am back in communications range.

Remember, please, that the Char Lineator is a personal project, not a work assignment, so it makes progress only when I have downtime at home. Adding to the Idea Exchange will just make a lot of my coworkers ask, "What the heck is this user talking about?" 🙂

> It seems that the 'right' solution would be to do this down in the VI Server architecture.

Actually, that would be exactly the wrong solution. I'm building this in G because it needs to be G extensible to have maximal utility. The goal is to get the functionality working first and then get the performance powered up. The only performance tuning I've done thus far are the broad architecture level tunings to avoid module dependencies that would trigger excess data copies. But data copies on the individual function level and slow function calls (like the name lookup function that David Moore identified) remain.

I expect to have more time to work on this in June. There are some users that have privately expressed interest to me about doing some work to tune the Lineator... they might make this library take some steps forward sooner than I would get to it (I know, for example, that they've been putting together a VIPackage to make it easier to install and maintain versions). And after Beta 2 is released for LV 2013, I may try to find a day or so to drop in the fix for the name lookup.

JonJay
Member
Member
on

I have taken on the code from AristosQueue and am working with him closely to continue the progress of this project. I'll be uploading an updated package within a few weeks, hopefully with more documentation. If anyone has been using this and has comments, questions, or examples of how they have used this package I'd love to read about it.

AristosQueue (NI)
NI Employee (retired)
on

Many thanks to Jon Jay, a contractor working for National Instruments, who took my Character Lineator and substantially refined and expanded its capacities. The VIPackage is saved for LV 2013... which most of you do not have access to until later this week. Why did we pick 2013? Because that's the LV version where this library is actually viable to use because of performance improvements I added to LabVIEW. Everything is functional in LV 2012 but SLOW. There are some workarounds that Jon came up with to make the performance reasonable in 2012, but those workarounds interfere with some of the usability of the APIs to some degree. 2013 provides a pair of VIs for looking up the name of a LabVIEW class that make this library truly useful for all the use cases for which it was designed.

Since the vast vast vast majority of LV users have the SSP enrollment, most of you will be able to upgrade shortly, so I felt that the right thing to do was to package this for the version of LabVIEW that really supports it.

cirrusio
Active Participant
Active Participant
on

I never posted anything here, but I believe I did in the other forum where we were discussing this, but there seemed to be some issues with the original code and running on RT.  (I eventually had to rip out all reference to the lineator).  Has Jon attempted to run this on embedded system?  The frustrating problem is that the code seemed to work initially and then just got to the point where I couldn't actually deploy any code that referenced the AQ Lineator.  Matt

JonJay
Member
Member
on

I have not had the bandwidth to try this on an RT system.

AristosQueue (NI)
NI Employee (retired)
on

A new version of the Lineator was posted this morning. Jon and I both worked to clean up a lot of code.

KazP
Member
Member
on

Thanks for your time in posting this. Works great, just thought I'd mention in the example "BIG TEST To String.vi" I had to replace the input object TestableData with TestData for the sample data types output.

JonJay
Member
Member
on

Ah, I see that. I'll have to add a static Test Serializable type instead so you can run it out of the box. If you ever see some sort of serialization where it just states the ClassName that means that that Serializable has not had its To/From Serial Form VIs configured.

stbe
Active Participant
Active Participant
on

Hi AQ,

after meeting you at the European CLA summit, I understand your ideas better and will give the Lineator another try

Overall impression is great, I start liking it. I have a few comments to share, I hope they are usefull.

* One thing that annoyes me is that for inserting and removing the fields you need to specify the field names separately - sounds very error-prone for me .

Also, I couldn't find a scripting tool and had to build up all the in-place element structures (for the From/To Serial Form VIs) - is it available or planned?

* The JSON Serializer :: Encode String does not properly handle the control characters (\n, \t, ...). attached you find a version I use for encoding JSON strings - it's functionally equivalent to the Lavag version http://lavag.org/files/file/216-json-labview/ albeit a bit slower (~ 8%, but a lot more readable ). One can definitely tweak to achieve same or better performance. One would also need to update the decode string version to be able to read all json formatted strings ...

LV_JSON_Escape_String.png

* The JSON Serializer ::Build Indent can be simplified (by just using the concatenate strings primitive):

JSON Serializer _ Build Indent.png

* I derived my own version of the JSON Serializer class, in order to tweak only a few things instead of having to write my own full JSON serializer.

** For the Encode Double as String, I used a format string with "%#.24e" (other length for Single/Extended) as formatting, to get rid of the trailing zeros. I believe that with this format string, also the trim whitespace could be omitted (because it is not filled with spaces).

* After some tweaking, I found that I cannot implement all the features I wanted without modifying code in vi.lib: I desire to override a few more VIs of the Serializer -> for me it would be great if the "Start/End Compound Value.vi" had dynamic dispatch terminals (so that they really can be overridden) and if the member variables had accessors (ie. to set a custom indent character from a derived class, or modify the indentation behavior - like I did)

* The JSON Deserializer would need a configuration flag to ignore excessive fields in the string input (and not throw an error) - I guess I need also derive my own version here

-Benjamin

_________________________
CLA
AristosQueue (NI)
NI Employee (retired)
on

stbe wrote:

Overall impression is great, I start liking it. I have a few comments to share, I hope they are usefull.


                   

Thanks.

stbe wrote:

* One thing that annoyes me is that for inserting and removing the fields you need to specify the field names separately - sounds very error-prone for me .

Also, I couldn't find a scripting tool and had to build up all the in-place element structures (for the From/To Serial Form VIs) - is it available or planned?

The scripting tools are planned, but I haven't built any. I was waiting until I was sure that the system that I had proposed was tested out before writing any scripting to build it. And, I admit, kind of hoping that someone else would write that tool. 😉

The names are interesting... in order to support version mutation, you're going to be supplying the names of the fields as needed for previous versions. And even in the current version, you may be breaking apart fields for custom serialization (take a look at what I did to the timestamp, for example). Single sourcing those names is something you could choose to do in any given class (i.e. create a subVI that returns the array of names), but the flexibility to have custom names at each place seems to me to be necessary.

Originally I was thinking that there would be a single top level class "Serialize Either Way" with methods "Do Something On Type_Int32.vi" (and so on for the other types) and "Serializer" and "Deserializer" might both inherit from that, allowing a single VI to do both Insert and Remove, but the conpanes for Insert and Remove turned out to be very different, and the mutation schemes are very different, such that it really is better -- as far as I can see -- to have two separate VIs.

I don't think the names are any more or less error prone than any other paired system. At least they are easy to check.

I'll look into the rest of your mods.

AristosQueue (NI)
NI Employee (retired)
on

One more point regarding

    Encode Double as String

I've been told by multiple people that the "e" notation is not readable and is definitely not desirable if a human is going to be looking at the values. I get that many people like this notation, but I don't think it is generally accessible, especially if multiple values have different lengths of numbers. For myself, understanding a single lone number is fine, but I can't really compare two numbers unless I write both of them out "properly" to visually put the decimal point in the right place. So unless we make that a setting on the JSON Serializer itself, I was steering clear of it.

stbe
Active Participant
Active Participant
on

Hi again,

I'm now using the Lineator more often.

When reading some additional configuration data while the app is running, I found the following issues with LV2013 (this is a dump from my error logger within the application):

2014-06-13 14:09:16errorError 1026 occurred at Call By Reference in Class Retrieval.lvlib:Get LV Class Name.vi:6510002
2014-06-13 14:09:16errorSerializer.lvclass:Insert Class Name As Field.vi:890001
2014-06-13 14:09:16errorSerializer.lvclass:Add Qualified Name.vi:1060001
2014-06-13 14:09:16errorJSON Serializer Improved.lvclass:Add Qualified Name.vi:1060001
2014-06-13 14:09:16errorSerializer.lvclass:Insert Name And (Optionally) Path.vi:5900001
2014-06-13 14:09:16errorSerializer.lvclass:Serialize Core.vi:3440001
2014-06-13 14:09:16errorJSON Serializer Improved.lvclass:Serialize Core.vi:3440001
2014-06-13 14:09:16errorSerializer.lvclass:Serialize.vi:2550001
2014-06-13 14:09:16errorJSON Serializer Improved.lvclass:Serialize.vi:2550001
2014-06-13 14:09:16errorConfig.lvlib:Config.lvclass:to String.vi
2014-06-13 14:09:16errorConfig.lvlib:Config.lvclass:Update GUI.vi
2014-06-13 14:09:16errorConfig.lvlib:Write Cfg Msg.lvclass:Do.vi:3750001
2014-06-13 14:09:16errorActor Framework.lvlib:Actor.lvclass:Receive Message.vi:1040001
2014-06-13 14:09:16errorActor Framework.lvlib:Actor.lvclass:Actor Core.vi:5880004
2014-06-13 14:09:16errorConfig.lvlib:Config.lvclass:Actor Core.vi:5880001
2014-06-13 14:09:16errorActor Framework.lvlib:Actor.lvclass:Actor.vi:6640010
2014-06-13 14:09:16errorActor Framework.lvlib:Actor.lvclass:Actor.vi.ACBRProxyCaller.EB700272

I don't fully understand the problem here yet, but the strange thing is, that the "Class Retrieval.lvlib:Get LV Class Name.vi" works properly after a second call (at least sometimes). It seems like the VI is already locked by some other VI (through the vi ref?) and cannot be opened a second time?

I'm using the Lineator several times in the same project (reading initial config, reading separate Actor's config, displaying the loaded - and possible altered - config to the operator)

I also wonder, what's the difference between this version of the Class Retrieval library (vi.lib\NI\Class Retrieval\Name to Path Conversion\Get LV Class Name.vi) vs. the vi we can find in LV2013's palettes (vi.lib\Utility\LVClass\Get LV Class Name.vi).

Looks like the LV2013 version has been included in your Class Retrieval library and you added the fall-back for 2012.

What's "bad" on the explicit version for LV2012 (the pure G-code without the hooks into LabVIEW that are hidden from us)?

Can I safely omit the LV2012 dependency and remove the Class Retrieval dependeny?

A second issue arises when I start the app from a fresh LV startup

1. start the Launcher.vi from the Actor project

2. while reading the config, I get:

Error 1446 occurred at Preserve Run-Time Class in Deserializer.lvclass:Deserialize Core.vi:2540002->JSON Deserializer.lvclass:Remove Serializable Core.vi:2920001->Serializable.lvclass:Remove Serializable.vi:7370001->SAM Settings.lvclass:From Serial Form.vi:580001->Serializable.lvclass:From Serial Form Wrapper.vi:5180001->Deserializer.lvclass:Deserialize Core.vi:2540001->Deserializer.lvclass:Deserialize.vi:7360001->Config.lvlib:Config.lvclass:Parse String.vi->Config.lvlib:Config.lvclass:Read File.vi->SAM.lvlib:SAM.lvclass:Read Config File.vi->SAM.lvlib:SAM.lvclass:Load App.vi->Launcher.vi

Class Hierarchy Source:

...

...SAM Settings

...SerializableNo TagNo Tag

Possible reason(s):

LabVIEW:  The class of object in did not match the class of target object at run-time.

When I run the launcher a second time, it's able to load the config ...

I have the same issue when I only run the Deserializer with the config string (not with the whole project)

I will look into this more closely next week.

Thanks, Benjamin

_________________________
CLA
AristosQueue (NI)
NI Employee (retired)
on

Benjamin: First question... are you using the March 4 2014 version of the library?

I  suggest that you modify your code to get rid of the Class Retrieval library entirely. It's a shim layer to make the lineator work with LV 2012 because in 2012 the high speed name lookup function didn't exist. If you're not in 2012, get rid of that library entirely. The version of the Lineator that I have put into the GitHub is already updated to 2013 exclusively.

JonJay
Member
Member
on

I agree that library should be removed as it was a lot of extra stuff. One thing I did want to point out is that there is a somewhat efficient method in that library that builds a pre-load VI for applications. This is due to the fact that you can't load a class into memory by name, it has to be in memory already. The library gives a method to dynamically generate a pre-load VI that places a Class constant on a block diagram for all Classes used in the project. This was a method we used to make sure everything was loaded properly.

stbe
Active Participant
Active Participant
on

Thanks for the quick reply.

Yes, I have the March version.

Good to know that also this library is in the Git repos. I'll check this one out and link my code against it.

@JonJay: I get your point, that's also the main reason we're not (really) building executable applications, but run in development with a project explorer most of the time.

_________________________
CLA
rolfk
Knight of NI Knight of NI
Knight of NI
on

JonJay,

I'm currently trying to wrap my head around LVOOP and mainly the factory pattern for now, for implementation of drivers with different underlaying communication paths. Your comment that one cannot load a class into memory by name made me wonder what you mean with this. What do you mean with this, since the Get Default Class Value.vi is in fact exactly loading a class into memory based on a name that has to somehow be evaluated at runtime? Am I missing something here?

Rolf Kalbermatter
My Blog
JonJay
Member
Member
on

What I mean by this, is that for the Get Default Class Value.vi to work the Class has to be in memory. When you are working in the development enviroment the Project Explorer automatically does that for you. However when you attempt to use the AQ Character Lineator in an application with multiple dynamically called classes it wont work. Unless each class that could be called is somewhere on a block diagram or statically called it won't load into memory with the application. If its not in memory the Get Default Class Value.vi won't work.

Hopefully Aristos can comment on this further.

rolfk
Knight of NI Knight of NI
Knight of NI
on

If that is true, that kind of perverts things like the Factory Pattern quite a bit.

Rolf Kalbermatter
My Blog
stbe
Active Participant
Active Participant
on

Just for clarification, the "Get Default Class Value.vi" will work, as it accepts a full path to the .lvclass file. the "Get Default Class Value By Name.vi" requires the LV-object already be in memory (either by project explorer, or static linking or any other means)

For my factory patterns, I usually have the loader-VI at a common root of all classes I would want to load and then derive a relative path from the given input (usually my lvclass files are just located in separate directories) and thus am able to load the object.

_________________________
CLA
rolfk
Knight of NI Knight of NI
Knight of NI
on

Thanks for the clarification. That makes then sense of course. I have a hunch where the "By Name.vi" variant may be used but that is not of any concern in my case. My intended use case is not going to instantiate objects many times during the life cycle but typically once for every time an IO session is started, and the derived classes will be indeed relative to the parent class on disk, so that the class path can be easily computed at runtime (well with some extra caveats in the case of build applications that might be using the 8.x application format).

Rolf Kalbermatter
My Blog
AristosQueue (NI)
NI Employee (retired)
on

Yeah, Jay misspoke -- "Get LV Class Default Value.vi" is not the same as "Get LV Class Default Value By Name.vi". The latter only works if the class is in memory. When deserializing, we usually only have names, not paths.

BUT... the Character Lineator has the ability to dynamically load classes!

When you want to Deserialize, pass in a Formatter object that has the absolute path to a directory of classes. Then use the option to embed the paths in the stream.

In the next revision of the Lineator, I'll be adding the ability to do the load from just the embedded name... no path needed.

stbe
Active Participant
Active Participant
on

Hi,

thanks for granting me access to the ni-dev repo.

I already have a minor suggestion: please put *.aliases and *.lvlps patterns into the .gitignore file (and especially, delete those files from the repo(s)).

Both file types are "local" files (I generally don't care what your IP-address is, and I want my project explorer always on the same spot )

Is there also a dedicated forum/community/... to discuss such issues/questions/... with ni-dev repo?

thanks, Benjamin

_________________________
CLA
AristosQueue (NI)
NI Employee (retired)
on

I added the .gitignore files to the master branch. I have wiped out some but not all of the aliases and lvlps files. I'll get around to the rest of them eventually.

cirrusio
Active Participant
Active Participant
on

OK...so I am officially done with the AQ Character Lineator for now.  Seems that I had a total collapse building an RT exe.  Here are some of the errors for your edification that kicked off a long list of "now VI is bad on..."

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable Core.vi" (0x06543e00)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Insert Serializable Core.vi" (0x06543e00)]

vi->flags=17375872, vi->flags4=0, vi->flags6=134219776

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable Core.vi" (0x06543e00)]

SetVIBad on vi [VI "Serializer.lvclass:Insert Serializable Core.vi" (0x06543e00)]

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable Core.vi:5110001" (0x06547510)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Insert Serializable Core.vi:5110001" (0x06547510)]

vi->flags=598656, vi->flags4=0, vi->flags6=0

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable Core.vi:5110001" (0x06547510)]

SetVIBad on vi [VI "Serializer.lvclass:Insert Serializable Core.vi:5110001" (0x06547510)]

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable.vi" (0x06624ca0)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Insert Serializable.vi" (0x06624ca0)]

vi->flags=17375744, vi->flags4=0, vi->flags6=134219776

VI_BROKEN (0): [VI "Serializer.lvclass:Insert Serializable.vi" (0x06624ca0)]

SetVIBad on vi [VI "Serializer.lvclass:Insert Serializable.vi" (0x06624ca0)]

VI_BROKEN (0): [VI "Serializer.lvclass:Serialize Core.vi" (0x061fe930)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Serialize Core.vi" (0x061fe930)]

vi->flags=17375872, vi->flags4=0, vi->flags6=134219776

VI_BROKEN (0): [VI "Serializer.lvclass:Serialize Core.vi" (0x061fe930)]

SetVIBad on vi [VI "Serializer.lvclass:Serialize Core.vi" (0x061fe930)]

VI_BROKEN (0): [VI "Serializer.lvclass:Serialize Core.vi:3440001" (0x061fdaa0)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Serialize Core.vi:3440001" (0x061fdaa0)]

vi->flags=598656, vi->flags4=0, vi->flags6=0

VI_BROKEN (0): [VI "Serializer.lvclass:Serialize Core.vi:3440001" (0x061fdaa0)]

SetVIBad on vi [VI "Serializer.lvclass:Serialize Core.vi:3440001" (0x061fdaa0)]

VI_BROKEN (0): [VI "Serializer.lvclass:Serialize Core With String.vi" (0x068fc3d0)]

VILinkObj::SetSelfErrorsCORE - VI is actually bad on [VI "Serializer.lvclass:Serialize Core With String.vi" (0x068fc3d0)]

vi->flags=17375872, vi->flags4=0, vi->flags6=134219776

...and it goes on and on and on and on...

stbe
Active Participant
Active Participant
on

Hi AQ,

I'm using the serialization framework a lot to exchange serialized JSON data with an external library.

We identified that the "[...]\Deserializer\Get Class Default Value By Name.vi" is very slow, when it loads the classes.

By implementing a cache (you must set the VI properties to non-reentrant), the deserialization gets about a 100x faster (yes, 2 orders of magnitudes) in our example (about 70 classes).

Deserializer.lvclass__Get Class Default Value By Name.png

cheers, Benjamin

_________________________
CLA
iannicholson
Member
Member
on

I just started using this library, and while it is very useful, I have some questions...

  1. Why do the numeric JSON functions use Scientific notation? "3.5" becomes "3.500000000000000000000000E+0" in the JSON representation. This seems like unnecessary bloat for small numbers. What would be the best way to work around this? So far I am converting each number to a string before outputting to the serializer, but that seems like a step backwards.
  2. It seems that the framework requires you to manually define the serializing and deserializing of your classes. This is flexible, but extremely time consuming and error-prone. Consider this: your names in From Serial Form.vi and To Serial Form.vi must match exactly, but there is no built-in error checking. You must manually check each string attached to the serialize and deserialize functions to make sure they are identical. Would it make more sense to store these in a global variable and reference from both From Serial Form.vi and To Serial Form.vi? What are other options for guaranteeing they are the same? I'm a little disappointed that as mature as LabVIEW is, none of the serialization options will serialize/deserialize a class without having to build it one block at a time (barring the XML serializer, whose output is less than desirable).
  3. I'm having a heck of a time building an EXE using this library. If "Remove unused polymorphic VI instances" is checked, as is default, the build always fails with Error 1502 (LabVIEW: Cannot save a bad VI without its block diagram). Am I missing something? I've checked the standard suggestions for dealing with 1502 but I haven't been able to build without disabling that option.
  4. I converted a class to inherit from this library, and I use it in (I think) 3 custom probes. It used to take 6 seconds to load these probes when right-clicking a wire, but now it takes about 45 seconds for the context menu to show up. Is there something I should be doing to limit the impact of this library?

Thanks!

DavidAMoore
Member Member
Member
on

2. This one is the killer to me. The OpenG and MGI tools for serialization let us get away from hand coding all this stuff, and with this tool AS IS we're back where we started. What is needed to make this usable is someone to write all the accompanying scripting to handle most of that work. Volunteers?

For my purposes, I wanted human readable class data. MGI Read/Write Anything often ends up rendering class data as hex, and class mutation often seems to struggle. I got to a solution that's working for me by creating MGI Robust XML. I get class data as at least semi-readable text, and when the built in XML reader struggles with changes to my classes, I give it some help.

Lack of scripting is one way this tool is still just a beta, and some of the other problems you encountered show the other places that need work. AQ does not lack for other priorities, so I don't begrudge the limited polish here.

David A. Moore, Ph.D.
President
Moore Good Ideas, Inc.
AristosQueue (NI)
NI Employee (retired)
on

iannicholson wrote:

  1. Why do the numeric JSON functions use Scientific notation? "3.5" becomes "3.500000000000000000000000E+0" in the JSON representation. This seems like unnecessary bloat for small numbers. What would be the best way to work around this? So far I am converting each number to a string before outputting to the serializer, but that seems like a step backwards.

You're free to edit the format if you wish (either edit the source code of the library itself or supply a custom formatter object), but this is the only format that completely retains the exact value of the double. All other formats that I tried resulted in a significant rate of "two numbers different in memory getting written down as the same value as text which means that deserialization is a loss of precision". Since numeric data is the lifeblood of the scientists and engineers who are LabVIEW's primary users, I opted for a format that would completely preserve the value.

When I researched this, I found many people across the Internet complaining about this issue and that IEEE had never defined a format for textifying floating point in a way that would guarantee unique values.

iannicholson wrote:

  1. I'm a little disappointed that as mature as LabVIEW is, none of the serialization options will serialize/deserialize a class without having to build it one block at a time (barring the XML serializer, whose output is less than desirable).


                   

Building it one block at a time is the whole point of the character lineator. 🙂 If you wanted it done as a block, use the Flatten To String primitive (or Flatten To XML). I'm sorry, but there does not seem to be any middle ground... I did a lot of research into other serialization systems in other OO languages to put the Lineator together. *This* is the Best Of All Possible Worlds, unfortunately, where each class is responsible for declaring its blocks. Now -- having said that -- there is some tooling that could be built to automate the creation and maintenance of the fields as you edit the class. That drudge work of coding could definitely be brought down. But at the end of the day, there are many cases where you're going to need to step in and manually code those fields, and when one of those arises, suddenly all the autogeneration becomes useless.

iannicholson wrote:


                       

  1. Consider this: your names in From Serial Form.vi and To Serial Form.vi must match exactly, but there is no built-in error checking.


                   

Nor can there be except in trivial cases. If you just look at a simple class, yes, those names are the same, but as soon as you start having different versions of the class being deserialized, the deserializer has many names that the serializer knows nothing about -- the old names of the fields that were deprecated out. Yes, your functions have to be written symetrically. The error checking is by testing a serialized string can be deserialzied and the objects before and after match.

iannicholson wrote:

  1. I'm having a heck of a time building an EXE using this library. If "Remove unused polymorphic VI instances" is checked, as is default, the build always fails with Error 1502 (LabVIEW: Cannot save a bad VI without its block diagram). Am I missing something? I've checked the standard suggestions for dealing with 1502 but I haven't been able to build without disabling that option.

I haven't seen that behavior with the CL, but I don't claim exhaustive testing on that aspect of the library. If I had to guess, I'd guess it has to do with the parts of the library that are dynamically loaded. Try adding the VIs that are part of the library to the "always included" section. App Builder is not my area of expertise, but since this is all plain G code, there shouldn't be anything magical happening... if you can't figure it out, contact an NI application engineer and get it looked at. That sounds like a bug, but I hesitate to say so because I don't know that system well.

iannicholson wrote:

  1. I converted a class to inherit from this library, and I use it in (I think) 3 custom probes. It used to take 6 seconds to load these probes when right-clicking a wire, but now it takes about 45 seconds for the context menu to show up. Is there something I should be doing to limit the impact of this library?


                   

Hm... I haven't tried using it for probing... post the block diagram of your custom probe. I might be able to suggest something.

AristosQueue (NI)
NI Employee (retired)
on

PS: You converting the numerics to a string before writing them is NOT inefficient! The serializer converts the numbers to a string at some point... you're just moving that point earlier. 🙂 You probably want to be using the formatter, but if that's your solution, it isn't hurting your performance.

iannicholson
Member
Member
on

AristosQueue wrote:


                       

iannicholson wrote:

  1. Why do the numeric JSON functions use Scientific notation? "3.5" becomes "3.500000000000000000000000E+0" in the JSON representation. This seems like unnecessary bloat for small numbers. What would be the best way to work around this? So far I am converting each number to a string before outputting to the serializer, but that seems like a step backwards.

You're free to edit the format if you wish (either edit the source code of the library itself or supply a custom formatter object), but this is the only format that completely retains the exact value of the double. All other formats that I tried resulted in a significant rate of "two numbers different in memory getting written down as the same value as text which means that deserialization is a loss of precision". Since numeric data is the lifeblood of the scientists and engineers who are LabVIEW's primary users, I opted for a format that would completely preserve the value.

When I researched this, I found many people across the Internet complaining about this issue and that IEEE had never defined a format for textifying floating point in a way that would guarantee unique values.

iannicholson wrote:

  1. I'm a little disappointed that as mature as LabVIEW is, none of the serialization options will serialize/deserialize a class without having to build it one block at a time (barring the XML serializer, whose output is less than desirable).


                   

Building it one block at a time is the whole point of the character lineator. 🙂 If you wanted it done as a block, use the Flatten To String primitive (or Flatten To XML). I'm sorry, but there does not seem to be any middle ground... I did a lot of research into other serialization systems in other OO languages to put the Lineator together. *This* is the Best Of All Possible Worlds, unfortunately, where each class is responsible for declaring its blocks. Now -- having said that -- there is some tooling that could be built to automate the creation and maintenance of the fields as you edit the class. That drudge work of coding could definitely be brought down. But at the end of the day, there are many cases where you're going to need to step in and manually code those fields, and when one of those arises, suddenly all the autogeneration becomes useless.

iannicholson wrote:


                       

  1. Consider this: your names in From Serial Form.vi and To Serial Form.vi must match exactly, but there is no built-in error checking.


                   

Nor can there be except in trivial cases. If you just look at a simple class, yes, those names are the same, but as soon as you start having different versions of the class being deserialized, the deserializer has many names that the serializer knows nothing about -- the old names of the fields that were deprecated out. Yes, your functions have to be written symetrically. The error checking is by testing a serialized string can be deserialzied and the objects before and after match.

iannicholson wrote:

  1. I'm having a heck of a time building an EXE using this library. If "Remove unused polymorphic VI instances" is checked, as is default, the build always fails with Error 1502 (LabVIEW: Cannot save a bad VI without its block diagram). Am I missing something? I've checked the standard suggestions for dealing with 1502 but I haven't been able to build without disabling that option.

I haven't seen that behavior with the CL, but I don't claim exhaustive testing on that aspect of the library. If I had to guess, I'd guess it has to do with the parts of the library that are dynamically loaded. Try adding the VIs that are part of the library to the "always included" section. App Builder is not my area of expertise, but since this is all plain G code, there shouldn't be anything magical happening... if you can't figure it out, contact an NI application engineer and get it looked at. That sounds like a bug, but I hesitate to say so because I don't know that system well.

iannicholson wrote:

  1. I converted a class to inherit from this library, and I use it in (I think) 3 custom probes. It used to take 6 seconds to load these probes when right-clicking a wire, but now it takes about 45 seconds for the context menu to show up. Is there something I should be doing to limit the impact of this library?


                   

Hm... I haven't tried using it for probing... post the block diagram of your custom probe. I might be able to suggest something.


                   

Aristos, thank you for again providing a thorough and reasonable answer to all of my questions! I commend NI for allowing you to handle the user communications this in-depth.

Your answer has persuaded me to look into the Formatter class, and it looks like this is exactly what I need to be able to tweak the number formats for my applications. I guess I didn't catch on when I read your original instructions.

Is it possible to override Enums in the Formatter class? If so, how is that done? I can see formatters for most of the basic data types but not Enums. I would like to output the String representation of my Enums. I am already doing this by preconverting to String, and that works. I'm just curious if that can be encapsulated in the Formatter or if it makes more sense to do it in my To and From routines.

As for the probing, I found that if I put the probe inside the class instead of my user probes folder, the right-click no longer takes forever. This has the added benefit of letting me define the default probe for my classes, which I was not aware was possible before this. This is a great feature!

iannicholson
Member
Member
on

AristosQueue wrote:


                       

PS: You converting the numerics to a string before writing them is NOT inefficient! The serializer converts the numbers to a string at some point... you're just moving that point earlier. 🙂 You probably want to be using the formatter, but if that's your solution, it isn't hurting your performance.


                   

I am less concerned about performance (this will be written to disk occasionally so performance is not a huge issue). I'm more concerned that this is undermining strong typing (which is admittedly not a big deal considering JSON is a string format anyway), and it surrounds numeric values in the JSON output with quotes, but I've read that this is invalid in JSON implementations. I believe a Formatter subclass would solve these issues though, and I'm currently trying to implement this.

UPDATE:

As I'm looking into this, I'm finding comments that say "JSON has a fixed format for numerics, so this serializer does not use the formatter at all" and it seems to be using hardcoded formatting rules. What is the reason for this? Is this something you meant to revisit? JSON seems to be happy with floating point numbers. I can modify your library, but I don't want to cause problems if you release more revisions later that I'd like to upgrade to.

AristosQueue (NI)
NI Employee (retired)
on

You don't override enums through the Formatter, but you can create a custom Serializable class for enum types... take a look at the "Serializable Analog Waveform" class for one example. The waveforms, variant, and arrays thereof are not treated as primitive data types by me. They're compound types built out of the more primitive layers. That means that someone plugging in a new serializer (JSON, XML, Binary, etc) doesn't have to write custom routines for those types.

You can plug in a custom serializable for your enum type as well. Duplicate one of those classes and then modify it. You should be able to do that for typedefs, also.

You know something... as soon as I typed all of that... it occurred to me that I might have never tested a cluster containing a waveform or any of the other plugin types. I'm not sure if that'll work out right. I don't recall writing any code that will check the plug-ins for types contained in other types. That may be a bug, and if your enums are inside a cluster, they might not pick up your custom serializable. You'll have to test it and find out... I haven't focused on the Lineator in a while.

iannicholson wrote:

As for the probing, I found that if I put the probe inside the class instead of my user probes folder, the right-click no longer takes forever. This has the added benefit of letting me define the default probe for my classes, which I was not aware was possible before this. This is a great feature!


                   

You're welcome.

AristosQueue (NI)
NI Employee (retired)
on

JSON demands, for example, the period be used for the decimal point, never a comma. JSON would not allow "3.141592" to be recorded as "pi". A customer Formatter allows arbitrary string formatting (that's its point). JSON says, "I define the format and there is no other if you're going to call this a number." So the Formatter is ignored because the JSON standard says to ignore custom formatters.

Rather than edit my library, I think you could just create a new child class of JSONSerializer... say "NicholsonJSONSerializer" and then override the key methods. That should protect you from future version issues.

10Things_Rob
Member Member
Member
on

AQ,

 

Great work; thanks for sharing it! I've been able to start experimenting with it successfully fairly quickly but I've run into a problem and am interested in hearing your thoughts.

 

I have a Serializable class. When I serialize it, it looks correct. Example:

{
"ClassName":"ScaleRecord_Table",
"Type":"Table",
"RawPoints":[],
"ScaledPoints":[]
}

I put this as a data member of ANOTHER Serializable class, and serialize that and I get the correct output EXCEPT the ScaleRecord class is missing the ClassName, and hence I cannot Deserialize it.

{
"ClassName":"Signal_Scaled",
...

"ScaleRecord":{
"Type":"Table",
"RawPoints":[],
"ScaledPoints":[]
}
}

 

Any idea what might cause the Serializer to "lose" the ClassName? I can prove that it RECOGNIZES the class, because it is invoking the specific To Serial Form method of the child class ScaleRecord_Table.

 

stbe
Active Participant
Active Participant
on

Hi AQ,

 

I'm running an application with multiple Actors in parallel where each of them is continuously deserializing data (e.g. API requests).

As soon as I have more than 1 parallel Deserializers working, I get undeterministic errors every few seconds.

To me it looks like the Lineator has some issues with re-entrancy (I never get such errors with only 1 deserializing instance)

 

Error 1373 occurred at Get LV Class Default Value By Name.vi

 

LabVIEW:  Class could not be loaded.

Complete call chain:
Get LV Class Default Value By Name.vi
Deserializer.lvclass:Get Class Default Value By Name.vi
Deserializer.lvclass:Deserialize Core.vi:2540002
Deserializer.lvclass:Deserialize.vi:7360002
API Call.lvclass:Deserialize API Call.vi:2300001
Test Lua.lvlib:Test Lua.lvclass:Function Call.vi:3730002
Test Lua.lvlib:Function Call Msg.lvclass:Do.vi:3750002
Actor Framework.lvlib:Actor.lvclass:Receive Message.vi:1040004
Actor Framework.lvlib:Actor.lvclass:Actor Core.vi:5880003
Test Lua.lvlib:Test Lua.lvclass:Actor Core.vi:5880002
Actor Framework.lvlib:Actor.lvclass:Actor.vi:6640015
Actor Framework.lvlib:Actor.lvclass:Actor.vi.ACBRProxyCaller.1B90012D

LabVIEW attempted to find the class with this name:
.lvclass

Possible reason(s):
Lua string received:
{"ClassName":"AC test.timer.lvclass","arguments":[{"ClassName":"LUA String","value":"tmr2"},{"ClassName":"LUA Integer","value":10}]}

 

The string below indicates, that I successfully received a valid JSON formatted object.

Somewhere between API Call.lvclass:Deserialize API Call.vi:2300001 and Deserializer.lvclass:Get Class Default Value By Name.vi the ClassName string is lost and an empty string is passed (which is expanded to ".lvclass").

 

For testing reasons, I have stripped the Name and Path substitutions, but no changes so far.

 

Has anyone tested re-entrant use (massive parallel de-serializing) of this library?

 

-Benjamin

_________________________
CLA
Elgsis
Member
Member
on

Can someone share their working implementation of this AQ Character Lineator? Because DEMO's are missing in project explorer and I cant find them 😞

It is not clear to me how can I pass my class and have it serialized.

stbe
Active Participant
Active Participant
on

@Elgsis: You should go into the "Test" folder of the main Character Lineator directory.

also read the "How do you start" section from AQ's post.

_________________________
CLA
stbe
Active Participant
Active Participant
on

@Aristos Queue: I'm pretty much sure now, that re-entrant use of the library does not work.

I still cannot figure out why and how, but I have a simple test-bench that plugs directly into the library (no external dependencies):

 

CL-reentrancy.png

 

When running this VI, within a couple of seconds (when some critical section is accessed), one of the 2 loops will stop with the error message:

 

Get LV Class Default Value By Name.vi<APPEND>

<b>Complete call chain:</b>
     Get LV Class Default Value By Name.vi
     Deserializer.lvclass:Get Class Default Value By Name.vi:6580001
     Deserializer.lvclass:Deserialize Core.vi:2540001
     Deserializer.lvclass:Deserialize.vi:7360002
     CL-reentrancy.vi

<b>LabVIEW attempted to find the class with this name:</b>
<Empty Name>

 

I'll keep you posted on my findings.

 

-Benjamin

 

_________________________
CLA
Elgsis
Member
Member
on

 

I have successfully managed to implement AQ Character Lineator to my project, but unfortunately I am having trouble building an .exe file. I get Error 1502. The same error as some users here mentioned. Has anyone managed to solve this error and successfully build and run executable application?

stbe
Active Participant
Active Participant
on

as far as I know, you have to use the version from the NI Github repository. the version here is only slightly outdated (LV 2012 vs. LV 2013)

 

further, you need all classes that you use statically linked in your app.

I usually drop all these class constants without wiring on my block diagram.

 

the seconds possibility is to use ppl's

_________________________
CLA
Elgsis
Member
Member
on

I am using NI AQ Character Lineator v0.9.3.47. Isn't this the newest version? If not could you please give a link to newest?

Could you please explain in more detail what you mean by "you need all classes that you use statically linked in your app."?

And what is "ppl's"?

iannicholson
Member
Member
on

I'm not sure if this is a LabVIEW problem or a problem with the AQCL (I'm not sure how it could be my code), but I'm getting Error 1446 when I first run my app which utilizes the AQCL. After a run or two or three, the error goes away, and I can start and stop my app with no problems until I close the project and re-open it. I then get the error for a couple runs.

 

It was quite hard to track down the bug, but I have provided screenshots below. LabVIEW seems to be getting confused thinking that the Deserializer and JSON Deserializer classes are not compatible types, but only temporarily. It forgets that the JSON Deserializer is a child of Deserializer for some reason. After a couple runs, it remembers.

 

Does anyone have any insight?

 

LabVIEW 1446 error.png

Gary_Chen
Member
Member
on

Hi, guys!

The AQ Character Lineator tool is very good, and I'm using it. But, I met a difficult problem when using it. I tried to read resource code to resolve it.But it's difficult for me. So, I want your help if possible.

A class named "Map" which inherit from Serializable class includes a variant. Class Map can be serialized by using JSON serializer.

Map.jpg 

Map Serialized.jpg

 

Another class named Abstract Power Meter contains a element Map class. And also, Abstract Power Meter is inherited from Serializable class. This means Abstract Power Meter  is serializable, and its element Map class is also serializable. So, Abstract Power Meter class should can be serialized by using JSON in theory. But, in fact, it can't. When serialize it, the Serialize.vi is always running but not complete. 

PM.jpg

I don't know where is incorrect. I hope can get help from you to resolve this problem.  Could anyone help me? Thanks.

AristosQueue (NI)
NI Employee (retired)
on
@Gary_Chen Did you ever resolve your issue? I've been less and less able to monitor the forums as the number of posts on ni.com spike upward, so I missed your post. The only easy guess I have about your problem is to check that all your VIs in the serialization call chain are reentrant... you may be reentering a function that is deadlocked on itself. I made that mistake multiple times when designing this library.
stbe
Active Participant
Active Participant
on

it always bugged me that you must use the "ClassName" field in the root object in order to specify the class you want you use. Many times however, the serial data I provide to the deserializer is known and possible missing fields are anyway caught at run-time.

 

Therefore, I've made another small tweak to the Deserializer.lvclass:Deserialize.vi, where I read the classname of the Target Type (Serializable.lvclass) input:

 

use_given_target_type.png

 

Now you may just provide

 

{
"a": 5,
"b": "test"
}

to your deserializer 🙂

 

KR, Benjamin

_________________________
CLA
pawhan11
Active Participant
Active Participant
on

Hi guys!

 

Where I can find this NI github repository for this library that people are talking about?

I have looked at https://github.com/ni but there is no  public Character Lineator repo

AristosQueue (NI)
NI Employee (retired)
on

pawhan11: It's a private repo that you have to have signed a license agreement with NI to access.

What are you interested in?

Contributors