LabVIEW 2015 - Fast File Format and PlugIn Architecture

[Andrey_Dmitriev]

In LabVIEW 2015 the new "fast file format" was introduced:

"Improving Load Time for LabVIEW-Built Applications and Shared Libraries.
You can build stand-alone applications (EXE) and DLLs that load faster by using the fast file format in LabVIEW. <...> When you enable the fast file format, LabVIEW does not use the Application Builder object cache"

The question about PlugIn architecture, when lot of SubVIs called dynamically (for example, from external LLBs) from relative small core application. What I observed is - the size of the executable with this option was significantly reduced (roughly twice) and the core application itself starts faster, but PlugIns load time is the same regardless from this option (I understand why - when application fully loaded then Dynamic calls just "normal" LabVIEW code and therefore this option takes no effect). Unfortunately in code distribution build spec (as well as in packed libraries) this option is not available.

Is it possible to get "fast" application also for PlugIn-based architecture and get LLBs or packed libraries in "fast file format"? Can someone explain, what means "LabVIEW does not use the Application Builder object cache" and how this "fast load" mechanism technically working?

Thank you in advance,

Andrey.

[GerdW]

Hi Andrey,

you are two weeks to early!

Please post to the LabVIEW2015 beta tester forum… 😄

[Andrey_Dmitriev]

---

@GerdW wrote:

Hi Andrey,

you are two weeks to early!

Please post to the LabVIEW2015 beta tester forum… 😄

---

Ups... Its not me. NI is too early this year. 😄

Thanks to pointing on that - I was sure that it was already released (and wondered why its released before NI Week)

I'll ping this topic after NI Week 2015.

[Rob_Calhoun]

OK, now that NI Week is over, can we talk about the "fast file format"? This (welcome) feature is just about the only significant change in LV2015 that is not editor-related. However it is very sparesely documented for such a big change.

Doing some quick testing I got the following results (after launching once to heat up the windows cache):

LV 2014 SP1: app size 20.3 MB, launch time 9-10 sec

LV 2015: app size 11.6 MB, launch time 8-9 sec

Smaller size and faster launch are well-worth additional compile time. I have not seen any downsides yet; anyone else using this in production?

[dan.bailey]

---

@Andrey_Dmitriev wrote:

In LabVIEW 2015 the new "fast file format" was introduced:

Is it possible to get "fast" application also for PlugIn-based architecture and get LLBs or packed libraries in "fast file format"? Can someone explain, what means "LabVIEW does not use the Application Builder object cache" and how this "fast load" mechanism technically working?

Thank you in advance,

Andrey.

---

Hey Andrey!  It's good to see that you guys have been getting some good mileage out of this project.  I'll go through your questions in order ...

1) These optimization are actually always enabled by default for packed project libraries. In fact, the load time benefits for packed libraries should generally be better than what you observe for .EXEs and .DLLs!

2) The app builder cache is something that is enabled for EXEs and DLLs that works to cache the results of previous compiles when the source has not been updated.  It is somewhat analogous to the .obj object files generated by a C++ compiler.

3) I won't get into too many nitty gritty details but the gist of it is... LabVIEWs various non optimized file formats are treated somewhat similarly to the way that we treat 'loose' VIs. In a DLL/EXE/LVLibp we know at build time what the contents and dependencies of a given built binary are going to be.  With this knowledge we can go ahead and construct something that is more similar to a statically linked PE or ELF file (clearly we're not using either of those) while the analogy is not 100% perfect it's the best I can do without going into a couple pages worth of description 😉  In addition to these basic file format changes we did a large amount of work on implementing a new loader which is able to take advantage of the large amounts of precomputed file data that is now included in the format which enabled us to cut a lot of corners that were there previously.

[dan.bailey]

Hey Rob, while I can't speak for everyone out there I do know that there are many groups here internally at NI who are using or will be using this in the future!

If you have some specific questions regarding the format I'd be happy to address them!

[Rob_Calhoun]

Thanks, Dan.

The reason I am looking for more information is that thus far the only documentation for this feature appears in "LabVIEW 2015 Features and Change". (Fast File Format doesn't appear at all in the help screen for "Advanced Page (Application Properties Dialog Box".)

Improving Load Time for LabVIEW-Built Applications and Shared Libraries

You can build stand-alone applications (EXE) and DLLs that load faster by using the fast file format in LabVIEW.

To use the fast file format, place a checkmark in the Use fast file format checkbox on the Advanced page of the Shared Library Properties dialog box or the Application Properties dialog box. This option is disabled by default. When you enable the fast file format, LabVIEW does not use the Application Builder object cache. Therefore, stand-alone applications and shared libraries may take longer to build.

Note To use the fast file format, ensure that the Enable debugging checkbox on the Advanced page does not contain a checkmark.

From the above I gathered that debugging EXEs is not supported (which is fine) but little else about it. So I appreciate you filling in some details about how it is associated with improvements with the loader/linker, and that NI intends to use it internally. (My other big worry was "this option is disabled by default"!)

The work on the new loader is evident in LV 2015. Getting rid of the seemingly-endless "find missing VI" dialogs when e.g. loading a DAQ application without having installed DAQmx first is a big improvement, and it's always been a bit weird to watch an EXE go through the same kind of dynamic-link process as the development environment. These are nice improvements.

More details on our specific experience:

I've switched our builder over to use the Fast File Format using this project property tag:

TgtF_fastFileFormat=true

Contrary to my expectations, this did not increase build time, possibly because we use a "clean" build process that starts from zero each time. (Our build system blasts the VI object cache before each build, checks out a copy of the source code, builds each EXE, builds each installer, adds the LV runtime, zips up the whole set along with intall scripts, checks in the zips and tags the source.)

Build time for a 6 EXE + 6 installer project was actually a touch faster with the Fast File Format enabled:

Total build time (hours: min)

0:54 for LV 2014 SP1

1:02 for LV 2015

1:00 for LV 2015 + FFF

Since we include the LV runtime in the archive the total archive size is dominated by the runtime installer, but we did see a some improvement there as well:

259 MB for LV 2014

286 MB for LV 2015

254 MB for LV 2015 + FFF

Launch times seem a little faster and peak private memory usage a little lower, but it is hard to give quantitative values without instrumenting the launch more than I've done. Everything looks to be in the right direction, however, and thus far we've seen no downsides to using the fast file format.

Rob

[dan.bailey]

Thanks Rob, it's good to see some actual numbers.

The missing documentation on the advanced page has had an internal bug report created for it and should be taken care of at some point soon (TM).

There was a fair amount of debate about whether this should be enabled by default for exes.  Eventually, we decided that we would rather not 'rock the boat' too much when it came to them.  You should probably note that it is the default (and only!) format for PPLs now, in fact there are a few extra things that we did for PPLs that make them even (and in some cases, much) more performant than exes, especially when it comes to class heavy projects.

If you're doing a 'clean' build of your .exes every time then it's definitely not surprising that you're not seeing any degradation in your build times.

It's also not surprising that you would be seeing lower peak memory usage, remember that PE files (which includes things like DLLs and EXEs) keep an image memory mapped into the process and since they're now noticeably smaller ... 😉

[Cepera]

Here's a link to a post about a minor cosmetic bug related to the fast file format:

http://forums.ni.com/t5/LabVIEW/2015-Fast-File-format-disrupts-Window-Run-Time-Position/m-p/3238306

[Rob_Calhoun]

Hi Dan,

I'm resurrecting this old thread because there is still no documentation of the Application Builder's Fast File Format option in the LabView help files or ni.com..

Overall I am quite pleased with Fast File Format. It both reduces application size (a factor of 2 in my case) and improves launch time. I've been using it exclusively for several years now. However, my builds recently broke due to an incompatibility between the Fast File Format and the new malleable VIs introduced in LabView 2017.

After some experimentation, I found that an application cannot be built if all of the following conditions are true:

* a VI calls a .NET class that uses a local (vs GAC) .dll

* a malleable VI is in use in the project

* Fast File Format is enabled on the project.

Since I value Fast File Format more than malleable VIs, I removed the malleable VIs. This fixed my build.

Please see the attached stripped-down example of the problem, and the application builder log below showing the failure. I am using LabView 2017.0f2 64-bit for Windows. I have reproduced this on multiple machines, but I have not tried other LabView versions or architectures.

Also please apply a double-noogie to whomever wrote the error message below, as the failure to including either the name of the dll that that failed to load or the calling VI made isolation much more difficult than it should have been.

I will file a Support Request with NI about the Fast File Format problem, and reference this thread.

Sincerely yours,

Rob Calhoun

6/6/2018 5:35:57 PM
[General]
Project: C:\BuildBug\FastFile.lvproj
Target: My Computer
Build Specification: FastFile
OS: Windows x64 6.1

[Open Top Level VIs]
C:\BuildBug\FastFileDotNet.vi	6/6/2018 5:18:58 PM


[Source Files]
C:\BuildBug\Microsoft.Diagnostics.Tracing.EventSource.dll	6/3/2018 2:06:41 PM
C:\BuildBug\RabbitMQ.Client.dll	6/3/2018 2:06:41 PM
C:\Program Files\National Instruments\LabVIEW 2017\vi.lib\Array\Shuffle 1D Array.vim	3/19/2017 11:29:06 AM	29
C:\BuildBug\FastFileDotNet.vi	6/6/2018 5:18:58 PM	10

[ERROR]
Code:13
Invoke Node in AB_Engine_Write_Linker_Wrapper.vi->AB_Build.lvclass:Copy_Files.vi->AB_Application.lvclass:Copy_Files.vi->AB_EXE.lvclass:Copy_Files.vi->AB_Build.lvclass:Build.vi->AB_Application.lvclass:Build.vi->AB_EXE.lvclass:Build.vi->AB_Build.lvclass:Build_from_Wizard.vi->AB_UI_Frmwk_Build.lvclass:Build.vi->AB_UI_FRAMEWORK.vi->AB_Item_OnDoProperties.vi->AB_Item_OnDoProperties.vi.ProxyCaller
<APPEND>
Method Name: <b>Linker:Write Info To File</b>

[Output Files]


[Errors]
Click the link below to visit the Application Builder support page. Use the following information as a reference:

Error 13 occurred at Invoke Node in AB_Engine_Write_Linker_Wrapper.vi->AB_Build.lvclass:Copy_Files.vi->AB_Application.lvclass:Copy_Files.vi->AB_EXE.lvclass:Copy_Files.vi->AB_Build.lvclass:Build.vi->AB_Application.lvclass:Build.vi->AB_EXE.lvclass:Build.vi->AB_Build.lvclass:Build_from_Wizard.vi->AB_UI_Frmwk_Build.lvclass:Build.vi->AB_UI_FRAMEWORK.vi->AB_Item_OnDoProperties.vi->AB_Item_OnDoProperties.vi.ProxyCaller

Possible reason(s):

LabVIEW:  Failed to load dynamic library because of missing external symbols or dependencies, or because of an invalid file format.
=========================
LabVIEW:  Failed to load dynamic library because of missing external symbols or dependencies, or because of an invalid file format.

Method Name: Linker:Write Info To File
6/6/2018 5:35:59 PM