One way to get rid of the garbage is to open the main VI and then do a Save With Options>Development Distribution. This will create an llb file with the main VI and all subVIs (exclusive of VIs in vi.lib). Once the llb is created, delete all of the files in the folder except for the llb. You can then use Tools>VI Library Manager to convert the llb back to a folder. If you have multiple main VIs, you'll have to create the llb for each before you do any deleting. And, if in any of your VIs, you call other VIs dynamically, the dynamically called VIs will not be included. Make a backup of the folder before doing this!
A programatic way is to write a small VI and in there, use the application property All VIs in Memory. Open your main VI(s) and run the small one to get a list of all VIs loaded. You could then use that list and the File I/O function List Directory. Whatever file is listed in both, you keep. If a file only shows up the in the List Directory result, delete it.
The first method is fine for a one time operation. The attached example gets a list of unused VIs.