Example Code

CognoscentUI - XControl, LVOOP, Animation, Unicode, and Image Processing

Code and Documents

Attachment

Download All

Summary

CognoscentUI is a user interface framework intended to host modules within a SubPanel within an XControl. Instances of modules are able to run concurrently, and the framework allows for easy development of new modules to perform any sort of UI task.

CognoscentUI was developed as a contest entry for the LV2010 Coding Challenge (LVXCC), and it is specifically distributed as a reference source code example, demonstrating the following:

  • How to design attractive user interfaces in LabVIEW that mimic popular web/mobile applications on devices such as the iPad, iPhone, and Android.
  • How to achieve extensibility using the Dynamic Dispatch and Inheritance capabilities of LVOOP

Function

CognoscentUI is composed of three discrete frameworks (plus some eye candy):

  • Object Kinematics class hierarchy - The base class, Object Kinematics, defines a PI motion controller process and interfaces to the motion controller. Child classes implement those interfaces, allowing an extensible object animation framework. The implementations that come with the base package are ControlPosition, XControlSize, PanelSize, PaneOriginPosition, and PicRingMotion (which is specific to the Pic Ring Module). Since the motion control uses imaginary CDB as the process variable, you can extend any type of 1D or 2D animation as a new child class with overrides. (It takes about 20 minutes create a brand new animation class, icon and all!)
  • Interprocess Messaging class hierarchy - This is a no-frills interprocess messaging framework. I started with a single messaging class, but soon remembered XControls do not "wake up" for User Events to which they have dynamically registered (d'oh!). The solution was a messaging base class with two child implementations, one which implements messaging through User Events, the other implements messaging using the Val(Sgnl) property of a FP control.
  • Module Manager class, CognoscentUI XControl, and Module class hierarchy - These components work together to comprise the essence of the CognescentUI package - the ability to manage and display dynamically instantiated modules (or apps, or plug-ins, or add-ons, or extensions, whichever name you fancy). The CognescentUI ZIP download includes two Modules for starters: the Pic Ring, and the Pic List.
  • Bonus: Image Processing class - Includes a homebrew version of a Gaussian blur for images, and a method to resample images. You can also find examples of using 32-bit PNG images with alpha transparency to customize controls, and a hijacked Classic Numeric Simple Horizontal Slide to achieve "vector resizing" of custom images on the title bar and status bar.
  • Bonus: Unicode Localization (Language Translations) - I didn't have time to develop a full framework for on-the-fly language switching, but the Pic List module demonstrates that localization is as simple as writing a String Value to a String Control! (Importing Unicode as UTF-16 rather than the ubiquitous UTF-8? That's another story )

All that blabber - in true LabVIEW style, let's just look at the diagram:

CognoscentUIClassHierarchy.png

Steps to Execute Code

  1. Run CognoscentUI_Example.vi, the top level demo in the project. Watch this video (http://bit.ly/CognoscentUI-Demo) to see how to interact with the demo.
  2. Check out the virtual folder in the project called "Bookmarks" which highlights some interesting portions of code. With the top-level CognoscentUI_Example.vi in memory, you can use "Find All Instances" on each of the bookmarks.

CognoscentUI Container

CognescentUIContainer.png

This is the heart of the framework. The container is made of three distinct parts: the Titlebar, the SubPanel, and the Status Bar

Highlights of this framework:

  1. XControl - The container is an XControl, meaning it can be dropped on any front panel. Additionally, you may have multiple concurrent instances of the XControl on a Front Panel, each with its own distinct modules
  2. SubPanel - Since the XControl contains a SubPanel, it acts as a plug-in container for any imaginable user interface module you develop.
  3. Messaging - When the Container registers a new module, it establishes two-way communication with the module with Messaging classes. All messaging is asynchronous and non-blocking. A Messaging base class defines the messaging protocol, and the container has two child classes that override the implementation using User Events and Val (Sgnl) (since XControls do not "wake up" for a user event, but they will respond to a Signalling Value Change).
  4. Extensible - The framework was written with LVOOP, and all important aspects have an overrideable base class. This includes the Module class, the Object Kinematics class, and the Interprocess Message class.
  5. Mobility - Most LabVIEW Front Panels are static, but this XControl can be moved and resized during run-time (just like you can move and resize controls in Edit Mode). Overrideable messaging for different Object Kinematics actions will broadcast to all active modules for important events they may need to respond to in order to update their displays (e.g., Resize)
  6. Graphics - Customizing controls with alpha-channel PNG images provides transparency, and hijacking Slide Numerics provides "vector resizing" of these graphics (note the corners of the titlebar and status bar do not pixelate or deform as it gets wider).
  7. Dynamic Processes - All processes are non-blocking and asynchronous. You will find extensive use of the VI Server launching reentrant clones of processes.

Pic Ring Module

PicRing.png

This plugin displays pictures in a ring the user can spin around using a mouse (or finger if using a touchscreen).

Highlights of this module:

  1. Animation - Since animation is controlled by a non-blocking parallel process, user interaction is seamless. Animation parameters can be adjusted via the Object Kinematics class interface to tweak the animation's response to user input
  2. Image processing - Two routines for image processing are introduced: the ability to add a Gaussian blur, and the ability to resize an image.

Pic List Module

PicList.png

This module plug-in displays a linear list of text and icons. The user can pan the list using a mouse (or finger if using a touchscreen).

Highlights of this module:

  1. Unicode Text Display - A method of the parent Container "Set Language" will broadcast a language tag to each active module. This module reponds by updating it's display to reflect the correct language translation of the list item. Future work might include a UTF-8 to UTF-16 converter, because it's not trivial importing Unicode text into LabVIEW.

Dependencies

LabVIEW 2010 - No additional drivers, add-ons, or toolkits required.


Video Demo

http://bit.ly/CognoscentUI-Demo

It's embarrassing how choppy the animations are in the the video. I would recommend that as you watch the video on one monitor, recreate the demonstration yourself on your other monitor. (Harder said than done if, like me, you only have one monitor ) Use the link below to download the 2.2MB Zip file, then run "CognoscentUI_Example.vi"

Selected Code Screenshots

XControlFacade.png

From the XControl Façade. When the XControl receives a message (from the Val(Sgnl) on the 'Interprocess Message' class control), the message is interpreted and handled by a case structure. The case shown handles the action for a module requesting to be inserted into the container.

CogMSG_Send_UserEventImplementation.png

The base messaging class has a method called "Send Message", and this screenshot shows the override implemented in the User Event child class. (At first, I only had one messaging class based on User Events, and it ended up genericized into a messaging base class since XControls won't "wake up" for User Events.)

CogMod_Activate.png

When the "Activate Module" method is called, this method launches a reentrat clone of the module. Ugh, if only we had better dynamic launching.

CogKin_MouseControlProcess.png

Notice this guy has no Error terminals - it's a dynamically launched process that runs whenever a "Mouse Down?" event is detected on objects that can be manipulated with the mouse (the title bar moving the Container, the resize handle on the Container, the Pic Ring rotation...). The Object Kinematics class defines (overrides) how an object will move, and as the mouse moves "SetTarget" is called on the class to continually update the set point of the animation controller. The process continues until the button is no longer held.

CognescentUI_ImageProcessing_Blur.png

One goal for this project was to not require anything more than a clean LabVIEW 2010 installation. The result? A few homebrew image processing routines. Note the interface only allows an input called "Sharpness (0.0 to 1.0)". One theme I tried to carry around this project is abstraction: develop interfaces through the eyes of the end user. The end user only cares about how blurry or sharp the image is: a value of 1.0 means no blurriness whatsoever, and a value of 0.0 means the entire image is smeared to oblivion.

So, How Do I Create My Own Module?

I'm glad you asked!

1. Save a new instance of the CogMod_Template.lvclass

Step1SaveNewTemplate.png

2. Give it a kickin' new name. Create a "Copy" of the Template - don't "Rename" the Template! Add it to your project.

Step2NewModuleName.png

3. At this point, you have the skeleton of a new module that's ready to be instantiated and used! Just drop an Invoke Node "Activate Module" for the container on your calling diagram and insert an instance of your Kickin' New Module Class. To add functionality to the new module, edit the "CogMod_Display.vi" class member. Consider this VI loosely analogous to the Façade member of an XControl - it must be present, and it contains the FP display of the module. Within this small VI, you will find brief instructions on how the VI should react to messages sent to it by the containing CognescentUI XControl. Feel free to add whatever class members you want, but don't change the inheritance of the class (the Template already made your new class inherit from Module base class).

Step3CreateNewInstance.png

No Typedefs Used

There are compelling arguments against using typedefs. Madness?! You decide. I think I pulled it off with this project, and if someone can show where typedefs would produce more robust code, I'd love to discuss it in the comments.

Credits

  1. Icons in the Pic List module were from the "Black Neon Agua" collection from ToffeeNut Design.
  2. Pictures from the Pic Ring module were from Add-On Screenshots displayed on the LabVIEW Tools Network.
  3. Thanks to Andrey Dmitriev for the concept of Gaussian Blurring.

LV2009 Version

Based on demand, I have added an additional download for a downconvert to LV2009.

Disclaimer

This project is not complete and buggy. Like any framework, if it's going to be legit, it's got to be vetted.

The future of this project is demand-driven.

Kindly disregard all profane comment blocks I forgot to erase from source code prior to distribution.

Released under Creative Commons Attribution Unported 3.0 license - please be aware of the terms prior to further development or redistribution of this framework.

Creative Commons License

Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.

Comments
shred444
Member
Member
on

interesting take on a Multi Document Interface. 

wirebirdlabs
Member
Member
on

I haven't thought of this framework as an MDI, but you're right, good comparison.

Alternatively... you could hide the Active Module Selector on the top-left title bar and have multiple instances of dedicated Containers running one module apiece (SDI).

Thinking aloud - It would also be straightforward to put an array of pictures on the title bar, which would effectively turn the Container into a TDI. Each module then would provide a title bar icon as part of it's contract. (Using the Pic Ring as the Module Selector is impractical, yet makes an attractive demo. I would lean toward TDI given more time.)

(And "document" is used loosely here, meaning module or plugin or extension or app)

altenbach
Knight of NI Knight of NI
Knight of NI
on

I wonder if the blurring would be simpler with a direct 2D convolution (DBL) with a small gaussian kernel and the output size set to size=x. You can create the kernel with a plain 1D Gaussian fed to both inputs of "outer product".

Try it.

Blur.png

wirebirdlabs
Member
Member
on

altenbach, I first started with Convolution, and it was MUCH (!!!) slower. As in, when I realized it wasn't going to respond within 30 seconds, I went and made some tea, and came back and it was still crunching.

Perhaps I was doing something wrong.

Then, lo and behold, the next day Andrey posted his Deblur contest entry. It was the answer to a much faster Gaussian blur (~50msec for a few hundred pixels square) using Fourier transforms. I spent a few hours wondering why I could not adapt the method, and then found it it only works for 2D arrays where both dimensions are a power of 2

Still, I'm unhappy with how long it takes to render the Pic Ring. I think the answer is a Box Blur, since it does not even require a trek into the Frequency Domain (but it comes with the price of a lower fidelity blur). Given more time, I definitely would have experimented with the Box Blur.

PaulUrbanski
Member
Member
on

Looks very interesting.  Do you have a version that works in Labview 2009  ?

altenbach
Knight of NI Knight of NI
Knight of NI
on

Using Andreys images, I get a loop rate of about 20ms for B&W and about 120ms for color using convolution (after removing the wait, on a 4+ year old Laptop :D). I wonder why yours took so long....

Here are two very quick and dirty, "back of the envelope" drafts using a 2D cosine (but a 2D gaussian would be equally easy), modify as needed. I am sure many improvements are possible. For example, each color plane could be done on a different CPU core.

CosineBlurBW.png

CosineBlurColor.png

wirebirdlabs
Member
Member
on

Paul and others - if you send me a PM on the LabVIEW Forums (http://bit.ly/LabVIEW-PM-to-JRD) containing your email address, I will send you a zip file saved for LV2009. I don't want to widely distribute older source code versions, but I have no problem sending it to individuals.

shred444
Member
Member
on

Jack, can you give me an example of when to use a framework like this for automation and test?  It seems like this is a framework most often used in something like a mobile app. Have you used this for a project/customer? 

I've built 1 unusual Multi Document Interface for a client of mine where they needed multiple windows open at once.  Other than that, i've tried to steer clear of it in LV

wirebirdlabs
Member
Member
on

>> It seems like this is a framework most often used in something like a mobile app.

I ought to update the document to better define the three distinct frameworks: one is for animating objects (the Object Kinematics class hierarchy), one is for asynchronous interprocess messaging (the Interprocess Message class hierarchy), and the final framework is a general purpose container meant to host plugins (the Module Manager class, the CognescentUI XControl, and Module class hierarchy). All three frameworks are intertwined to form the "CognescentUI package".

Tuning down the animations to a reasonable level (high Speed, low Overshoot) works great for touchscreen HMI's at test or control stations. I've had a high level of success using suble animations to make user interaction more intuitive. I work in an environment where that "namby pamby" graphics BS isn't required or tolerated, but undeniably the subtle animations are preferred over "things just popping up and disappearing from nowhere". (Note: the benefit-to-cost tips positive when hundreds or thousands of people might need to become acquainted with your interface, not just a few dedicated operators. Also, if you're optimizing for speed - taking orders on a touchscreen at a fast-food restaurant - animations might not be a good choice)

As for stylized controls in industry, one requirement for my largest embedded touchscreen project to date was "Don't make it look like Windows, and don't make it look like LabVIEW".

>> Have you used this for a project/customer?

All the code was written from scratch after LVXCC was announced, so it's unencumbered (hence the CC license). Recreational programming for this contest was a good creative release to test out some concepts and test out some beers.

>> can you give me an example of when to use a framework like this for automation and test?

I stuck loosely to the contest theme of "example code and iPAD" and used that as a springboard for the project. A contest has a dramatically different scope than a contract, so I just ended up settling on an entry that could showcase the widest smattering from my bag-o-tricks. I think an important take-away from this example for the ATE apps is how easily extensibility can be achieved with LVOOP.

>> I've built 1 unusual Multi Document Interface for a client of mine where they needed multiple windows open at once.  Other than that, i've tried to steer clear of it in LV

I really hesitate to classify this framework as an MDI. Fulfilling the role of a MDI as a subset of the framework's capabilties? Sure, that sounds better. (Also, if they have multiple windows open at once, that's not an MDI, which only has one window) And I'll say again, the Pic Ring is an annoying method of switching through plugins, I would definitely go TDI with icons on the the title bar.

wirebirdlabs
Member
Member
on

Alright, altenbach, I haven't been ignoring you, I finally got the results from Convolution vs. Fourier in this unusually messy snippet. On my computer, I see Fourier taking between 65 and 80 msec dependent upon blur level, and Convolution takes between 80-260msec, highly sensitive to the multiple of the kernel size. I'm sure we could optimize both methods even further, but I doubt either method would smoke a box blur. I was disappointed I couldn't get up and make tea while using your convolution method.

Since the results were not very close, I just used the "Run Continuously" button for this benchmark, without taking any of the usual benchmarking precautions.

FourierVsConvolutionImageBlur.png

altenbach
Knight of NI Knight of NI
Knight of NI
on

Yes, its a matter of code simplicity vs a slight speed edge.

However, you are comparing apples with oranges! For my method, don't do the padding (just use the raw picture as upper input!) and skip the trimming at the end and it will be about twice as fast as Andrey`s.

In any case, try the FFTs from the NI Labs High Performance Analysis library 2.0 on Andreys solution, on my dual core, I improve about 20+% with them.

I am also puzzled by your parallel loop configuration. I would probably only parallelize the outer loop. Stacked parallel loops don't make sense to me.

EDIT:

OK, the final verdict: convolution is up to 3x faster!

Have a look at the attached modification, comparing different equivalent blur levels.

(not sure why the forum does not rescale the snipped....)

BlurBenchmark2.png

wirebirdlabs
Member
Member
on

The plot thickens: I wrote the padding routine to go ahead and upscale it to the greater power of two when the dimension already was a power of two. In other words, the picture we have been starting with - 256x256 - was actually getting resampled to 512x512, which is why Convolution was faster than the Fourier. (The reason I wanted the extra padding is so I could specify the blend color for the edges. Given more time, I was going to add a "Background Color" input to this blur routine)

Now, if we shave off a pixel, so that Convolution is running on a 255x255 and Fourier is padded up a pixel to 256x256 (fair enough) Fourier again beats Convolution by about 30%.

One thing I have learned for sure: the speed of both methods is highly sensitive to input array size.

FourierVsConvolutionImageBlurRound3.png

altenbach
Knight of NI Knight of NI
Knight of NI
on

Yes, you don't need to pad. The FFT has been optimized for virtually all sizes quite a while ago (6.0 maybe?), so, YES, without padding Andrey's algorithm is again faster for that particular size.

Quote from Wikipedia:

"The most well known FFT algorithms depend upon the factorization of N, but (contrary to popular misconception) there are FFTs with O(N log N)complexity for all N, even for prime N."

Very long ago, I extensively studied it for 1D fft and made a nice graph of speed vs size for all sizes. Typically, it's probably not worth the extra rescaling. You could make a lookup table of all speed vs. sizes and then pad to the next higher fastest size, also accounting for the rescale penalty.

Both methods basically do the same things. The convolution method probably costs slightly more because it also involves an FFT of the broadening function. I still like it better for code clarity.

As mentioned above, don't forget to substitute the high performance library from NI Labs for Andrey's code.

wirebirdlabs
Member
Member
on

>> Very long ago, I extensively studied...

My last comment fully documents the extents of my studies (), and at this point I defer to you. Thanks for the helpful criticisms - you've given me enough good ideas to probably decrease the rendering time of the Pic Ring by 50%.

>> I still like it better for code clarity.

Totally agree. The code is clearer, and the sliding blur kernel concept is more intuitive.

>> don't forget to substitute the high performance library from NI Labs for Andrey's code.

Based on future demand, will do!

altenbach
Knight of NI Knight of NI
Knight of NI
on

Also, Andrey's code works especially well for the "special case" of a Gaussian, because the FFT of a Gaussian is again a Gaussian. The convolution approach lets you more easily play with other broadening functions (try a motion blur ;)).

Anyway, congratulations for being a finalist. Well deserved!

I haven't studied your code yet (probably will vote on the last day) after looking at everything.

Since you are dealing with relatively small images and a limited set of blur levels, it's probably worth to generate and cache e.g. 5 blurs for each image. In this case you only need one forward transform and N back transforms, saving you again almost half (over N forward and N back transforms).

(Here we get the real advantage over the convolution where forward and back FFT are merged and we cannot tap into the "middle").

crelf
Trusted Enthusiast Trusted Enthusiast
Trusted Enthusiast
on

Nice background image of the Melbourne Citylink freeway: http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=Melbourne,+Victoria,+Australia&sll=37.06...





Copyright © 2004-2023 Christopher G. Relf. Some Rights Reserved. This posting is licensed under a Creative Commons Attribution 2.5 License.
JMkokott2
Member
Member
on

This is an amazing tool you've created.

The runtime resize-ability/moveability feature was something I was going to look into creating, and you've done a superb job of accomplishing this.

Any particular bugs I should be aware of?

Awesome example

~Jon

wirebirdlabs
Member
Member
on

>> This is an amazing tool you've created.

Thanks!

>> Any particular bugs I should be aware of?

Here's what I know about:

  • Bug - Showstopper: Right before I made the demo video, I tried for the first time to drop two XControls on one Front Panel. Something wasn't working right.... they didn't play well together. I haven't looked at the source code since the contest entry, but I don't think the problem is going to require a fundamental change
  • Not Implemented - Rejected: When using the Active Module Selector, I did not program a "Select Module" function into the Pic Ring module. Hence, you can view the active modules, but there's not a way to re-insert an active module. I hate the Pic Ring as a module selector, so I didn't bother writing this function - I would go to a TDI icons-on-the-toolbar with further development (search this page for TDI and you'll see my comment above)
  • Bug - Rejected: The Active Module Selector Pic Ring does not respond to the Resize message, but the Tools Network Pic Ring on the Demo works just fine. I won't fix this, because I don't think the Pic Ring makes a good module selector.
  • Bug - Nuisance: Look at the Façade on the XControl, and you'll find a "State Change = T" in the Exec State Change event case. Notice the Diagram Disable around this. This State Change flag is required for the Display State to persist, but it causes the Dirty Dot each time the XControl is compiled, run, saved, ... I have tried putting that "Initialization Stuffs" in other places such as the Init Ability of the XControl, but with no success. I just need to do more research and possibly redesign to get this booger in the right spot.
  • Bug - Nuisance: Intermittently, when I shut down, one of the dynamically launched processes keeps running, which causes a few Classes to be locked. This should be easy to track down. The current fix is to exit the project to the GSW, and immediately open back up the CognoscentUI project.

Alright, full disclosure, that's all the dirty laundry I got on myself. Feel free to comment when you find more bugs

JMkokott2
Member
Member
on

I've built the example into an executable, but I've lost some of the xcontrol's features.  I can no longer move/resize the xctntrl by clicking on it.  I can still use the controls at the bottom of the screen however. Any thoughts on why this is happening?

~Jon

JMkokott2
Member
Member
on

Something was not being picked up by the compiler.  Always included for whole project fixed this.  I bet you are loading something off the path somewhere.

wirebirdlabs
Member
Member
on

JMkokott2 wrote:

Something was not being picked up by the compiler.  Always included for whole project fixed this.  I bet you are loading something off the path somewhere.


Can you report back which VI was the culprit (I have not investigated)? I thought there were static references to each of the important modules, but it's clear I need to add more. +1 for the buglist.

wirebirdlabs
Member
Member
on

I had a way-homer about the Messaging class hierarchy. It makes more sense to differentiate "Message" from "Messenger". It never felt right having the child classes ("Messengers") inherit from Message base class, with the most noticeable "code smell" coming from the "Send" child implementations (see the screenshot above, and also the class hierarchy. It's strange how the Messenger Ref is unbundled from the same data that is being sent).

While maybe not a bug, this should be considered a design flaw that needs reconsideration.

Contributors