Community Documents

cancel
Showing results for 
Search instead for 
Did you mean: 

Browse NI Community User Groups & Sort documents as you want!

LabVIEW Interface for Amazon S3 is available on the LVTN and in VIPM!

Released on the LVTN and is available in the VIPM package list.

For any questions, issues or comments check the Support group for this toolkit!

First Round:

Summary:

I'm sure all of you have read a lot of documents on this Community. There is however a big problem I've experienced many times here: I cannot sort the documents. There are some filtering options, but it is impossible to browse popular, much valuable content. That's not the case any more!

With this project you can browse NI Community groups, get all documents of the group you select, and the more importantly you can sort them by their score.

The score is based on the number of views, average rating and likes. At least by default. What's better than that? You can specify your own scoring formula.

Of course you can open each document from the program in a web browser.

 

This project uses many parts of this community site, it communicates with web services, downloads HTML pages, parses HTML and JSON (JavaScript Object Notation) data using Regular Expressions.

For Inter-process communication purposes I use my previous qualifying code. The project is fully event-driven.

It is highly documented, so I'm sure you will find the answer for your questions, if any.

 

This project uses a .NET assembly, which is open source and free to use and redistribute. I included the binaries in the zip file below, however sources have been trimmed out to save space. Documentation to it is included and there is a Readme.txt which contains author's information and the license.

 

I also used some graphical/decoration elements from the NI Community site, which I hope does not violate copyright rules, as this is not a commercial application and in fact the purpose of this is to serve this community.

Function:

  • Browse NI Community Groups
  • Display Group's documents
  • Sort the documents by scoring them
  • Write your own score calculation formula
  • Open documents in your browser
  • View document's body along with the attachments in a 2nd window
  • Create report of the selected group in Excel

 

Second Round - Transfer data to other VI:

The application is extended with a new functionality: it can now display the selected document in a new LV window.

Only the document's body and the attachment are displayed. No eye-candy, page headers and footers, other side panels appearing on the community page are displayed.

 

There are also minor bugfixes:

  • The Author name did not appear in the document list when the name contained other than alpha characters.
  • The number of Views was 0 if there were more than 999 views. That was caused by the page of the document displaying views with thousands separator (like: 1,278).

Communication method:

 

1. For communicating with the display VI I use a queue.

 

The queue holds clusters as elements. The cluster contains an enum, which defines the Action the display has to take; and an optional variant that is the data which may be passed to the display as well.

There are actions, which need data to operate on, but there are others where data is unnecessary.

The de-queueing part unbundles the action and the data part of the cluster. The action is wired to the selector of a case structure, so each cases define the tasks to do on specific action commands. If an action needs the data, it may be converted to the specific type inside the case structure.


If you need a fast response to a command and that command makes unnecessary for the receiver to execute previous ones (eg: Close, Hide Panel), you can flush the queue in the transmitter side before queueing the next command.

 

I chose queue as it is a perfect communication method inside an application. It allows to use any data types, does not use a lot of resources, does not need the receiver to poll for available data.

 

The transmitter side of the transfer is the Main.vi, the receiver is the DocumentSidePanel.vi located in Display\DocumentSidePanel\ folder.

 

2. Main.vi already used a notifier to transfer data between VIs.

 

That example however shows how to combine a notifier (or even a queue) with user events. That way the received data may be transmitted to an event structure allowing the structure to stay "fully event-driven" (it is not necessary to set a timeout and poll for available data on each timeout).

 

Third Round - Process and display data in another Software Package:

 

The application is extended with the functionality to generate a report of the selected group.

I was going to use DIAdem, but after playing with it for a few hours, it seemed to be so much more complicated to create reports from non-TDMS data, than M$ Excel, that I chose Excel instead. To transfer data to Excel I choso to use the Report Generation Toolkit for Microsoft Office. It's usage is really straightforward, and works very well. It is not locked to specific Office versions (like when I would use ActiveX).

 

To create the report, I transfer the harvested data to an Excel report created from a template. The data is then processed via VBScripts and Excel formulas (most of the formulas are also generated "on-the-fly" with scripts to allow adopting to the amount of data). Scripts are needed, because there are functions in Excel you can use only manually or via scripts (like filtering data). Using scripts to generate the formulas also allows you to use formulas on cells that really need them.

 

Minor bugfixes and modifications were made to speed up the fetching of groups.

 

Fourth Round - LabVIEW to the Cloud

 

Cloud services and cloud storage are going to be more and more popular nowadays. Cloud storage is a perfect way for sharing data (like measurement results, generated reports), creating backups, etc.

 

Amazon with it's AWS (Amazon Web Services) is probably the most popular and recognized provider of this kind of technology. S3 (Simple Storage Service) is a very powerful one, as it allows the cheap, safe storage of virtually infinite amount of data. As a very popular service, there are countless implementations of it's API in many languages (like Python, PHP, .NET, Ruby, etc.).

 

Even though I decided against them, and I did implement the S3 API from scratch. This way I was able to make it using only native LabVIEW functions. The VIs implementing the API are grouped to an lvlib. These together make it possible to use this library very easily on every platform LabVIEW supports. This might be very useful on targets like cFPs and cRIOs, where storage space is very limited, but they usually gather data 24/7/365.

 

The only third party code I used to implement the library is the SHA-1 Cryptographic Hash Function's implementation found in the NI Community as the service needs HMAC SHA-1 to sign requests.

 

The Amazon_S3 library I created in the project does not implement all the existing API calls, but the complete VIs are enough for many tasks. Also after understanding Amazon's REST API and the build-up of my code, it is a really straightforward process to realize other calls, as the building blocks necessary are there already as Private members of the library.

 

The Amazon_S3 library's code does not come with much documentation, becuse it would make no sense at all trying to place 10s of pages of documentation inside the block diagrams. Amazon has two very useful documents on S3 REST API:

 

I used these documents during development, so all the VIs are created according to them. They are very detailed, and really easy to understand, so if someone is going to extend this code, he/she should start with reading them.

Eevn if you want to use it in a project, the first part of S3 Developer Guide would worth to read.

 

Behind the scenes:

  • Communicating with webservices
  • Fetching simple HTML pages
  • HTML Client VIs
  • Data parsing via regular expressions
  • Converting JSON data to XML
  • Parsing XML documents
  • Using math formulas and evaluating them
  • Event-driven architecture
  • Using notifiers combined with user events to send notifications to an event structure
  • Using queues to send commands and data between VIs
  • Using Report Generation Toolkit for Microsoft Office

 

Steps to execute code (latest version):

  1. Open attached project
  2. Open and Run Main.vi
  3. Press the Get Groups button
  4. After fetching groups you can browse them
  5. Click on one of the groups - Documents of the group will be fetched
  6. Change the formula as you want or leave default, then press Calculate button
  7. Press the Sort - Subject button
  8. Click on any document to display it in an other window
  9. Click on Hide Display button to hide or Show Display to show the document window
  10. Double-click a document to open it in your default browser
  11. Click on Create Report button to generate a report of the selected group in Excel
  12. Repeat any of Step 4 - 11 as many times as you want & Have Fun!
  13. Press Stop button
  14. Vote for my project!

Screenshots:

Main Front Panel:

NI_groups_front_panel.png

 

Main Front Panel after task 2:

NI_groups_front_panel_task_2.png

 

Report generated in Task 3:

NI_Groups_report_1.png

Main Block Diagram:

This diagram is quite large, but I hope it will show at least the architecture of the program.

main_vi.png

VI Snippet:

This program is quite complex, and there is not much use of a snippet from it, however to suit the rules I include one here.

This snippet may be used to display the likes of a document of an NI Community Group. After placing it, fill in the URL with a Documents URL and run the VI.

Get_Documents_likes_from_Documents_site_snippet.png

VIs attached below:

Note: I just upgraded to LV 2010 SP1 and noticed that fetching of the groups does not work. After identifying the root of the error I discussed it with Grant (Thank you for your help!), and he found that this is an already reported bug of the HTTP Client VIs. CAR number is 284156 if someone might be interested in it.

There is a workaround however: A path constant (eg. cookies.txt) should be wired to the OpenHandle.vi's cookie file input on the block diagram of the ConnectToCommunitySite.vi to allow the HTTP Client VIs to store the received cookies on the file system.

This bug is already repaired in LV 2011.

 

Bugfix: As Christina pointed out there was a bug in the application, as I assumed that all the groups have a description property. I corrected the VI that was responsible for this issue, it is attached below. To resolve the problem, simply overwrite the old VI in the Harvesting\NICommunitySite\GetGroups\ folder with the new Get_Group_data_from_XML_string.vi.

Comments
RedLion
Member
Member
on

Wow Peter, I love this!

It's really fun to play with this program. I also wished many times if I could sort docs in a group the way I want (eg. by rating).

It would be nice to allow the use of number of comments in the formula though.

G-Money
NI Employee (retired)
on

Nice work! You probably know more about parsing our community than some of our developers now

peter_smith
Member
Member
on

Thanks Grant!

Yes, I really needed quite a few hours to find out where real, useful data come from. However Wireshark was my best friend during development. Jive makes things really complicated with it's mysterious javascripts and webservices.

MonicaR
Member
Member
on

Nice work! It's really better to browse groups and find interesting documents this way.

I hope this will move you on in the contest!

Monica

SpecialK
NI Employee (retired)
on

Wow!

This is pretty amazing Peter! Keep up the good work and I really hope you win

-Karthik

peter_smith
Member
Member
on

Thank you Karthik!

I'm happy you liked my work. I'm sure the most valuable part comes this round, though!

Peter

SpecialK
NI Employee (retired)
on

Peter,

You are right. Part 4 is pretty darn amazing

Thoric
Trusted Enthusiast Trusted Enthusiast
Trusted Enthusiast
on

I think the code is fantastic, great style, attractive UI, very intuitive.

(Just saw your note on fetching groups - applied the fix - it works great now)

Thoric (CLA, CLED, CTD and LabVIEW Champion)


Christina_R
Active Participant
Active Participant
on

Congratulations on the win! This is a very impressive example app.

I noticed a small problem while playing with it, though. Some of the descriptions do not match the groups. It looks like Get_Group_data_from_XML_string.vi is assuming the Names and Descriptions arrays are the same length, but there are actually fewer Descriptions (presumably because some of the groups don't have descriptions). For example, scroll down to "Large LabVIEW Application Development" and notice that it has the description from Linux Users.


Christina Rogers
Principal Product Owner, LabVIEW R&D
peter_smith
Member
Member
on

Hi Christina!

Thanks for the gratulations.

I did not notice this issue, and currently it's a little past midnight here in Hungary, so I'm not going to debug the code right now. But I will do it defenetely tomorrow. Currently I suspect the problem will be with the JSON->XML conversion process. I think the empty strings in JSON data won't get through to the XML string. This might cause that there will be fewer description elements in the XML, so when I combine each groups' data in the FOR loop using the indexing inputs, the group descriptions might shift to the previous groups which did not have a description. The conversion is done via a 3rd party .NET assembly, so I absolutely don't know it's behaviour.

I'll make sure to check it tomorrow.

Thank you for your interest in my code, posting this issue and digging into the code trying to identify the problem.

Peter

peter_smith
Member
Member
on

Hi again!

You were correct, that the VI was assuming that the arrays of each gruop attribute are the same size. Even though I think they should be, as if a group does not have a description, the empty string would be a perfect choice to represent it . But after debugging the app, I realized that the root of the problem is that the webservice which serves the JSON data does not list the description property at all for groups which do not have the description specified.

So I redesigned the part processing the XML document. As I see it works now as it should.

I attached the corrected VI to the document, so you can create correct reports on eg. the "Large LabVIEW Application Development" group to the cloud from now.

Peter

jcarmody
Trusted Enthusiast
Trusted Enthusiast
on

I'm interested in your 4th round "LabVIEW to the Cloud" example.  Is that available?

Congratulations on your victory, by the way.

Jim
You're entirely bonkers. But I'll tell you a secret. All the best people are. ~ Alice
For he does not know what will happen; So who can tell him when it will occur? Eccl. 8:7

peter_smith
Member
Member
on

Thanks jcarmody!

What do you mean by "available"? It is attached to this document named NI_Groups_LV2010_task_4.zip. However I corrected a bug today, and that's attached too (Get_Group_data_from_XML_string.vi), and you should overwrite the corresponding VI inside the zip file with this new one.

Regards,

Peter

peter_smith
Member
Member
on

Hi everyone,

 

I just would like to inform you that as an evolution of my last round's work, I started to develop an Amazon S3 Toolkit for LabVIEW, which is now in a state ready for beta-testing.

I'd like to ask everyone who is familiar with Amazon S3 to participate in the testing. It would be great to make the toolkit error-free and get some feedback from end users.

 

If you are interested in the testing, please send me a PM, and I will send you the VIPM package file.

 

After the Toolkit is ready to be released, it is going to be distributed on the LV Tools Network.

 

Thank you for all your help in advance!

 

Regards,

 

Peter