Hi Brad,
I checked both the "Remove unused members from project libraries" and "Modify project library file after removing unused members" and the build now takes just seconds. So it works fine now.
Previously I unchecked "Modify project library file after removing unused members" because I follow the instruction in the FAQ - since it says "disable" this option if you use a version of LV earlier than LV2010. I am using LV2010. Strange but nevermind... problem solved!