Given that you say that when this happens, it happens hard, and the only way to recover is to restart the app, that sure sounds a whole lot like some resource is being used up. When it expires that resource, it cannot answer connection requests anymore.
My situation was the same way. Even though the problem was with TCP connections, quitting the app apparently freed those up, so you could run again.
I would look long and hard at a memory leak (some queue not being emptied??) or the connection-not-being-closed issue.
Steve Bird
Culverson Software - Elegant software that is a pleasure to use.
Culverson.com
Blog for (mostly LabVIEW) programmers: Tips And Tricks