LabVIEW 2020 crashes while rebuilding source-only DQMH Modules' Main.vi files

[Shatoshi]

Hello,

I am developing a relatively large application using LabVIEW 2020 Professional (32-bit) with DQMH Toolkit 7.1.0 running on Windows 10 Enterprise. I am also using GitLab with TortoiseGit for source control. I am experiencing reproducible LabVIEW crashes after enabling Separate compiled code for all existing project items. The issue often becomes visible while running DQMH module validation. However, validation itself does not appear to be the root cause: LabVIEW can also crash immediately when I try to open an affected Module's Main.vi directly. The most important observation is that the same project validates successfully when an existing local VIObjCache is present, but crashes after the cache is removed and LabVIEW has to rebuild the compiled objects from source-only VIs.

I would appreciate advice on this issue.

Project history:

The project has been under active development for approximately nine months. Initially, the project was developed without SCC and software revisions were stored locally. Later a colleague and I migrated the project to Gitlab. At that point we enabled: Separate compiled code from new project items. However, we did not change this setting for existing VIs, libraries and modules because we were not yet aware of the mass-marking option for existing project items. As a result, the project was developed in a mixed configuration: newly created VIs had Separate compiled code from source file enabled, while existing VIs remained unmarked. The project remains functional in this mixed state.

We later tried to enable Separate Compiled Code for all existing project items because Gitlab commits in the mixed configuration contained many unrelated changes. We have now tried this three times. Each time, after some development and several commits, LabVIEW started crashing while validating or opening certain DQMH modules' Main.vi. Clearing the Compiled Object Cache did not resolve the issue, so we rolled the project back and continued working in the mixed configuration. During the third attempt, I created a separate experimental branch and performed more controlled tests. 

The project contains four MainController modules: MainController(Plant 1), MainController(Plant 2), MainController(Plant 3), MainController(Plant 4) and a bunch of secondary Modules. Three of MainControllers Modules were cloned from the original MainController(Plant 1) approximately halfway through the project development. They have since diverged because each module is used for a different test stand configuration. Their front panels and functionality are similar but not identical.

The most recent failure initially occurred after the following sequence:

PC1: 

1. Created a separate Git branch for the experiment
2. Enabled Separate Compiled Code for all existing project items
3. Cleared the Compiled Object Cache
4. Restarted LabVIEW
5. Ran Validate All Modules successfully
6. Made small cosmetic changes in MainController(Plant 2) and PSController
7. Committed and pushed the changes to GitLab

PC2:

1. Pulled the same branch from GitLab
2. Made a small cosmetic change in PSController
3. Committed and pushed the change to GitLab

PC1: 

1. Pulled the latest revision from GitLab
2. Opened the project
3. Ran Validate All Modules
4. LabVIEW crashed while loading MainController(Plant 2): Main.vi

PC2:

1. Pulled the same latest revision from GitLab
2. Opened the same project revision
3. Ran module validation
4. LabVIEW crashed while loading a different module: MainController(Plant 3):Main.vi

This was confusing because both PCs used the same revision from the same GitLab repository, but the affected module was different on each machine. The crash did not necessarily occur in a module modified immediately before the failure.

Crash details:

The LabVIEW error dialog reports:

LabVIEW 2020 encountered a problem and needs to close.
Exception: Access violation (0xC0000005)
Version: 20.0 (32-bit)

When the crash is triggered during DQMH module validation, the dialog may report:

Possible Cause:
C:\Program Files (x86)\National Instruments\LabVIEW 2020\
project\Delacor\DQMH\_DQMH Validate Module\
Parse Module Elements.vi

However, DQMH validation is not required to reproduce the issue: LabVIEW can also crash immediately when I try to open an affected Main.vi directly from the project. A typical stack is:

GoDoit: VI "MainController(Plant 3).lvlib:Main.vi"
(D:\...\Libraries\MainController(Plant 3)\Main.vi), HeapClass "FPHP", Act 55 
#** Loading: "D:\...\Libraries\MainController(Plant 3)\Main.vi"

When validation triggers the crash, the stack may contain not only Main.vi, but also several standard DQMH Module VIs:

GoDoit: VI "MainController(Plant 3).lvlib:Main.vi"
(D:\...\Libraries\MainController(Plant 3)\Main.vi), HeapClass "FPHP", Act 55, 
#** Loading: "D:\...\Libraries\MainController(Plant 3)\Main.vi"
#** Loading: "D:\...\Libraries\MainController(Plant 3)\Get Module Main VI Information.vi"
#** Loading: "D:\...\Libraries\MainController(Plant 3)\Stop Module.vi"
#** Loading: "D:\...\Libraries\MainController(Plant 3)\Test MainController API.vi"

Nevertheless I believe Main.vi alone is responsible to the failure.

Workaround: standalone open and save:

I found the following workaround:

1. Close LabVIEW completely
2. Move the affected Main.vi file outside the module directory, for example to the Desktop
3. Open Main.vi as a standalone VI (ignoring all items while loading)
4. Immediately save it without any changes
5. Close LabVIEW
6. Copy the VI back into its original module directory with replacement
7. Reopen the project
8. Run module validation

After this procedure, the module opens normally and validation completes successfully. Initially, I also performed Force Compile (Ctrl + Shift + click on the Run button). However further testing showed that Force Compile is not required: opening the VI standalone and saving it is enough. 

The standalone open-and-save workaround changes the binary size of the affected VI, for example: MainController(Plant 3)/Main.vi: 323432 bytes -> 296568 bytes. When the VI is later opened and saved again inside the project with its dependencies available, its size increase to approximately its previous value. 

I understand that saving a VI after ignoring dependencies may not be a safe long-term repair procedure. I used it only as a diagnostic workaround.

Different failing modules on two PCs:

I pulled the same Git revision on both PCs and ran validation. With the existing local ViObjCache validation completed successfully on both machines. I then closed LabVIEW and renamed the user cache directory on both PCs. After restarting LabVIEW PC1 crashed while validating MainController(Plant1) and MainController(Plant 4) and PC2 crashed while validating MainController(Plant2) and MainController(Plant 3).

Therefore, with the same Git revision and empty local caches, the two PCs failed on different modules. Restoring the previous VIObjCache directory made the same project revision validate successfully again. 

Workaround with all Main.vi files:

Then I performed the standalone open-and-save workaround for all module Main.vi files:

PC1:

1. Applied the standalone open-and-save workaround to all module Main.vi files
2. Renamed VIObjCache
3. Restarted LabVIEW
4. Validation of entire project completed successfully
5. Committed and pushed the changes to GitLab

PC2:

1. Renamed VIObjCache and pulled the latest revision from GitLab
2. Opened the project
3. Validation of entire project completed successfully
4. Did an additional change: moved one front-panel element in each Main.vi
5. Committed and pushed the changes to GitLab

PC1:

1. Renamed VIObjCache and pulled the  latest revision from GitLab
2. Opened the project
3. Validation of entire project completed successfully

I then performed actual refactoring in two modules (MainController(Plant 1) and SequenceEngine), these were not only cosmetic front-panel or block-diagram changes. I committed and pushed the modified project to GitLab. On the second PC I pulled the same commit, opened the project and ran Validate All Modules which completed successfully.

Selective mixed-mode test:

I also tried to disable the Separate compiled code from source file for the four Main.vi files only but it resulted in LabVIEW crashing as well. 

Current observations:

1. After renaming or clearing VIObjCache, the same project revision may crash during rebuild
2. Two PCs using the same Git revision and empty caches can initially fail while loading different modules
3. Moving an affected Main.vi outside the module directory, opening it standalone, saving it and copying back can restore validation
4. After applying the workaround to all module Main.vi files, the current Git revision passed cache validation on both PCs

Questions:

1. Has anyone bumped into a similar issue with source-only DQMH module VIs?
2. Does "HeapClass "FPHP", act 55" provide any useful indication?
3. Is a mixed configuration considered safe in practice for long-term development?
4. Would you recommend clearing or renaming ViObjCache before each git pull?

Unfortunately, I can not provide the entire project, but only anonymized versions of the affected Main.vi files. However, these Main.vi contain Requests to approximately 20 other submodules. Therefore I am not sure whether isolated Main.vi files alone would be sufficient to reproduce the issue.

Thank you for reading through, and please let me know if you have any thoughts!

[joerg.hampel]

Hello Shatoshi, thank you for posting here and welcome to the DQMH Community. I'm sorry to hear that you're currently facing problems with LabVIEW and/or DQMH. Here are a few first thoughts off the top of my head:

1. We have been using DQMH with compiled code separated for nearly a decade now without those symptoms you are seeing right now.
2. Sadly your exact problems don't ring a bell - I can't say that I have seen that exact same behaviour before. 
3. The HeapClass "FPHP" message points at a corrupt front panel object (often a lvclass). There are ways to investigate more deeply, but usually we have little success with that. Someone from NI might be able to chime in and tell more.
4. Maybe, more logging might give better insight (but this is more of a Hail Mary, really)
5. Have you tried recreating one of those modules from scratch (i.e. from a fresh, new module)?
6. Btw: I don't believe this solves anything, but the latest version of DQMH is now v7.1.2 and it's always a good idea to stay up to date...

I'm curious to hear if others have more ideas.

[Shatoshi]

Hello, Joerg! Thank you for the quick response.

---

@joerg.hampel wrote:
Have you tried recreating one of those modules from scratch (i.e. from a fresh, new module)?

---

Regarding this point: we have already encountered a somewhat similar situation earlier in the project. At that time we were already using SCC and needed to create a new controller module for another plant, so we cloned MainController(Plant 1) into MainController(Plant 2).

Approximately two weeks later, during normal development, opening MainController(Plant 2)\Main.vi started causing LabVIEW to crash. At that point we had not yet changed any Separate Compiled Code settings or other project-level configuration. To recover, we recreated MainController(Plant 2) from scratch using a fresh DQMH module and continued development.

Later we cloned MainController(Plant 1) again to create MainController(Plant 3) and MainController(Plant 4). These two newer modules did not cause similar problems during subsequent development.

One potentially relevant detail is that the modules were not all stored in the same Separate Compiled Code state: Plant 1 & Plant 2 were unmarked, while Plant 3 & Plant 4 - marked.

We continued development for several months in this mixed configuration without further crashes. However Git commits remained partially noisy, this was the main reason why we later tried to convert the entire project to the marked state. Previous attempts always failed, so this time I decided to log the events to identify any patterns.

I will also update DQMH to the recent version in a separate test environment. I understand that this may not directly solve the issue, but it is worth excluding before further investigation.

Thank you for the suggestions!

[Shatoshi]

Actually, I also remember encountering several weird visual issues after merging LabVIEW VIs via TortoiseGit Merge. 

In one case, after pulling changes from the repository, a VI required a three-way merge using the base/yours/theirs versions. After the merge, the text-editing area inside MHL Case Structure's Subdiagram Label extended beyond the visible boundaries of the label itself. As a result, part of the text was no longer visible.

I also encountered other strange effects after merging VIs. For example, some block-diagram elements appeared to be dublicated inside certain cases. The duplicated elements partially repeated existing code fragments, but they were not connected to anything.

Because I could not be confident that the merged VI remained structurally valid, I recreated that module from scratch as well.

However, I have not observed these particular artifacts in the MainController modules discussed above, and I guess the cases are irrelevant.

For reference, I configued TortoiseGit to use LabVIEW Compare and LabVIEW Merge according to the following article. My .gitattributes file currently contains:

gitconfig
*.vi binary
*.ctl binary
*.vit binary
*.ctt binary
*.lvproj binary
*.lvlib binary
*.lvclass binary
*.vim binary
*.xctl binary
*.lvsc binary
*.txt text eol=crlf

[Olivier-JOURDAN]

Like @joerg.hampel, I've been using DQMH with the separate compile code option enabled for 7+ years. I never came across the issue you are describing.

One thing that rings a bell when I read your very detailed messages is that you seem to use LVCompare and LVMerge. This is something I never used, and one thing I know is that when you enable the separate compile code option, moving from one commit to another (when switching branches or checking out a specific commit) can cause weird LabVIEW behavior and loading crashes.

I don't know how LVCompare and LVMerge handle the cache, but I may have some concerns. Even more in the case of a mixed, separate compile code option activation.

Are you using those tools often?

[Shatoshi]

---

@Olivier-JOURDAN wrote:

One thing that rings a bell when I read your very detailed messages is that you seem to use LVCompare and LVMerge. This is something I never used, and one thing I know is that when you enable the separate compile code option, moving from one commit to another (when switching branches or checking out a specific commit) can cause weird LabVIEW behavior and loading crashes.

I don't know how LVCompare and LVMerge handle the cache, but I may have some concerns. Even more in the case of a mixed, separate compile code option activation.

Are you using those tools often?

---

Hello, Olivier!

Yes, we used LVMerge a couple of times when we had to perform a three-way merge tool using yours/base/theirs versions. Unfortunately, this resulted in several visual (at least) artifacts, so we rebuilt the module from scratch. Since then we have completely avoided using LVMerge. We also used LVCompare a few times to see the "diff with previous version" just to understand how the tool works, but that was a long time ago.

Could you please describe your usual workflow when pulling changes from other developers? For example:

1. What do you do when a VI conflict occurs, if you have encountered one?
2. Do you close LabVIEW before switching branches, checking-out commits or pulling changes?
3. Do you clear VIObjCache after switching branches or pulling changes?

[joerg.hampel]

For HSE, the rule clearly states that merging of individual files is not allowed (see https://dokuwiki.hampel-soft.com/kb/bestpractices/scc/rules).

To answer your questions specifically:

1. What do you do when a VI conflict occurs, if you have encountered one?
   1. This does not usually occur. If it does, we discard one of the two
2. Do you close LabVIEW before switching branches, checking-out commits or pulling changes?
   1. Not LabVIEW, but the LabVIEW project(*) - always! This is mandatory.
3. Do you clear VIObjCache after switching branches or pulling changes?
   1. Not by default, no.

(* edited for clarity)

[Olivier-JOURDAN]

Same rules at Wovalab 🙂

[Shatoshi]

---

@joerg.hampel

---

I have narrowed the issue down further. Initially, I localized the cold-cache crash to the waveform chart objects inside the MainController VIs. When I removed all charts from affected Main.vi files, the modules opened successfully after clearing VIObjCache. Then I localized the issue further to the chart Digital Display. Note: all the description below applies to Plant 2, Plant 3 and Plant 4 as well.

Test 1: Existing project, Main.vi:

In MainController(Plant 1)\Main.vi I placed a newly created Waveform Chart outside the existing Tab Control and enabled: Visible Items -> Digital Display. Then I:

1. Saved the VI
2. Closed the LabVIEW project
3. Renamed the user VIObjCache directory
4. Reopened the project and opened Main.vi

Result: Main.vi opened successfully

Then I moved the same chart inside a page of the existing Tab Control, keeping Digital Display enabled. After repeating the same cold-cache procedure LabVIEW crashed while loading Main.vi. If the chart remains inside the Tab Control but Digital Display is disabled, the VI opens successfully.

Test 2: Separate empty LabVIEW project:

I created a new empty LabVIEW project and added a new VI containing Waveform Chart with Digital Display enabled inside Tab Control. After closing the project, renaming ViObjCache, reopening the project and opening the VI LabVIEW did not crash.

Test 3: Separate new DQMH project:

I created a new DQMH project and repeated the same test - LabVIEW did not crash.

Test 4: New Tab Control and new chart inside the existing project:

I returned to the original application project. Inside an existing Main.vi I added new Tab Control and placed inside new Waveform Chart with Digital Display enabled. Note: I tested different combinations of Tab Control and Waveform Chart styles. After clearing ViObjCache and reopening the project LabVIEW crashed.

Test 5: New standalone Test.vi inside the existing project:

I created a separate new Test.vi inside the original application project. The VI contains the same elements: Waveform Chart with Digital Display enabled inside Tab Control -> after the same procedure LabVIEW crashed while loading Test.vi.

At this point, the issue appears to depend on the context of the existing project or on one of the items loaded with that project. I am providing the minimal set to reproduce the issue: 

1. Clear or rename ~\User\Documents\LabVIEW Data\VIObjCache
2. Open the project: 1.vi successfully loads, while loading 2.vi results in a crash.

I wonder if it is possible to repair the project..

Let me know if you have any thoughts! Thank you.

[Shatoshi · Accepted solution]

If anyone runs into this issue, check this thread. It seems to be a persistent bug that affects at least LabVIEW 2019 and 2020. I am practically certain that you must avoid the following combination: enabling "Separate compiled code from source file" on a VI while using a Waveform Chart with a Digital Display inside a Tab Control page.