I have several thoughts for you. First, I recommend splitting up your 5000 variable library into a set of smaller libraries. In contrast with the LabVIEW DSC 7 engine, the LabVIEW 8 Shared Variable Engine is optimized for multiple smaller libraries. I would recommend having at most about 1000 variables per library. You'll see some different recommendations from different people at NI, but in my personal experience I think 1000 is about as many Shared Variables that can be easily managed at once. For your case this would mean having 1000 variables in 5 different libraries. --Splitting up your library gives you the added benefit of being able to undeploy/disable individual libraries without affecting the other libraries. Unfortunately, the SCF migration tool will not do this for you manually so it may take a little time to get everything organized.
Your comment about ever-increasing memory usage is concerning. Do you see this while deploying your library, or while writing values? If you write to values very fast (like in an untimed loop) for an extended period of time memory usage will increase as the variable buffers increase. Please elaborate more on this topic and let me know what happens if you separate your variables into multiple libraries. I was not able to reproduce this on a similarly spec'd computer using 10 libraries of 500 variables each.
You can programmatically undeploy libraries using the delete process VI on the DSC Engine Control palette in LabVIEW. Use this VI in combination with the get process list VI to quickly remove all currently deployed libraries.
Also, regarding your question about why variables sometimes show up the Published Variable Monitor window; the different utilities you described may use different methods to get that variable list. These methods behave differently depending on how stressed the Shared Variable Engine is. If you're in the middle of loading a library with 5000 variables I'm not surprised that the list won't get populated immediately in the Published Variable Monitor window. After the library successfully loads though, you should be able to refresh the view and see all the variables.
Were you able to split your 5000 tags into multiple libraries? How did the system behave? If I understand you right, you are saying that the tagsrv increases in size even if you are not updating it with values from your vi. Does this 'memory leak' stop if you reduce the number of deployed variables (to say about 2500, 1000, 100)?