System Web Server Error: Overload - No Free Workers

[BDA_VAN1]

Deployed several remote front panels on CRIO-9067 via LV 2015.  Generally connection works fine but after some time, connection to html files becomes intermittent and eventually fully unable to access.  Error is 503 - Server Overload error.  Went digging into the log files and found the following in /var/local/natinst/log/SystemWebServer.log:

appweb: Error: No free workers. Increase ThreadLimit. (Count 10 of 10)

Any thoughts? Could expand free workers in SystemWebServer.conf but uncertain of other impacts.

[GreyGrey]

Hi BDA_VAN1,

Can you attach the entire log file?

[BDA_VAN1]

Was only line in the file.  Controller was reset and log is now empty.  I only retained screenshot from config and monitoring page. I did capture error log - attached.

[GreyGrey]

Can you give me a more complete picture of what you are doing? Do you have multiple front panels that are being connected to from multiple clients or is it just one front panel opened from multiple clients?

[BDA_VAN1]

I've got four panels that are accessible from multiple clients.

[Sam_Sharp]

I would imagine that for system performance reasons, there is a limited number of clients that can connect simultaneously to a remote front panel. I would imagine that each time a client connects, it creates a thread to serve the data to that client and there is a limit to the maximum number of threads (10).

Having too many clients connected would likely cause high CPU which in turn would affect the 'RT'-ness of the cRIO target.

If you need to have so many clients connected, you might want to look at having an intermediate 'relay' server which can serve more clients (e.g. single connection from cRIO to server, many clients can connect to a single server). You could do this using LabSockets (3rd party toolkit), or implement your own system (e.g. based on HTTP, TCP/IP, WebSockets etc.).

[BDA_VAN1]

Thanks Sam for the reply.  Indeed you are right in that we've got a limited number of threads set to 10.  What we do have is clients that connect intermittent to monitor RT status & download data.  

Is there a way to "dump" a thread after a client disconnects / times out? I will never have 10 clients connect at the same time.

[GreyGrey · Accepted solution]

Hi BDA_VAN1,

I couldn't find a way to terminate the connections but the webserver should terminate the connections if they reach the timeout limit (which I believe is 60 seconds by default). 

You can go into the NISystemWebServer.conf file and increase the number of connections allowed. There is a KnowledgeBase article that walks through this process. This may only serve to postpone the error but it also may give the other connections that have been established a chance to timeout.