Hi Rob, thanks for your suggestions.
I eventually sussed it though, in a place I would not have expected...
I coded a routine to defer front panel updates when doing lots of graphical changes. The VI takes a control reference as an input (any control on the front panel to be deferred) and from this it determines the owning VI and from this it can get a reference to the panel. It seems that these references never get closed, and as I was calling this function many times it was creating new (and different) references to the panel each time. The memory usage increased ever so slightly over time (I had not noticed this before).
When the app finally closed I suppose all these references got cleaned up, and this was what was taking a proportional length of time.
The solution was to use a more sophisticated routine (not terribly clever though) that only derives the reference once, and then stores it in a feedback node for each other time the VI is called.