Software tab grayed out

[wrkcrw00]

I have a dev target (cRIO-9045 with client 18.2) that is connected to SystemLink (server 18.5). The short history is:

1. Connected to SystemLink
2. Deployed a package, all was well
3. Connected to the target from LV proj to do some dev deployment and testing
4. Next time I tried to deploy from SL it just hung
5. Restarts had no effect
6. I removed the target from SystemLink and re-added, still not able to deploy
7. I upgraded the target to client 18.5
8. Now the software tab on the SL page for that target is grayed out

What could cause the software tab to be grayed out?

Does development testing and deployment mess with the ability of SL to install packages? I wasn't even able to restart the target from SL after I did that (although the target does show connected).

[GreyGrey]

Development testing shouldn't affect the ability to install packages, however, I'm not sure what could be going on. If you can provide the salt logs from the cRIO (this KB lists the location), we can take a look at those to see if the salt client isn't working properly. Also, the skyline or salt logs on the server may have some information too. If you want to send these to me in a direct message, I can take a look at them and respond back here.

Alternatively, you could try reformatting and reinstalling and this would likely get you past the issue as I don't think this is normal. However, I would prefer to take a look at the logs so that we can try and determine whether this is something reproducible.

[JoshuaP]

What you are describing almost sounds like the target may be running low on memory. In order to refresh the list of installed software, deploy updates, or even reboot the system our service needs to fork its process to call other applications like opkg.  If the system is running low on memory or if the memory is highly fragmented the new process will immediately fail.  In some cases, the job history will show an out of memory exception, but it depends on when the exception occurs.

When you initially add a target, we automatically kick off a refresh job to populate the list of installed software and feeds in the database.  If this data doesn't exist, we grey out the Software tab.

You may want to look at the Health view for the target and look at the memory attributes for available and percent used.  This could be an indication that there isn't enough free memory for our service to execute jobs.  If this data isn't working either you may have to use MAX.  If the system is low on available memory, you can use MAX to disable the startup application and reboot the system.  Once rebooted, check to see if SystemLink is working.

[wrkcrw00]

The health tags have not updated since yesterday morning (when all this started). I opened up the targets web browser and can see ~40% memory usage. I would imagine that over 1GB free memory should be plenty enough. 

I've rebooted several times with no change in behavior. 

[JoshuaP]

Would it be possible for you to SSH into the target and pull the /var/log/salt/minion log off the target and post it or direct message it to me.

[wrkcrw00]

For those of us non-linux savy users, could you tell me how to open / download the minion file from Putty? Previously I had gone in through the web interface file browser. The minion file isn't shown in the file browser so I thought it didn't exist. I do see the file when navigating to the salt folder through SSH though. I've tried a few commands to open the file but NI Linux RT has been stripped of a lot of commands.

[JoshuaP]

I typically use something like pscp https://it.cornell.edu/managed-servers/transfer-files-using-putty to get the files from a Windows PC.

[JoshuaP]

FileZilla is also a good alternative if you want a GUI for dragging and dropping files off an RT target.

https://filezilla-project.org/

[andre.buurman@carya]

My tool for the moment is WinSCP https://winscp.net/eng/download.php

As protocol you can use scp over port 22 (same as ssh).

It provide you with a commander-style explorer with your PC on the left and the cRIO on the right.

[GreyGrey]

I took a look through your logs and didn't see anything super telling as to why you're seeing the grayed out behavior that you are. I did see a couple of entries where it looks like you tried to add a system as a managed system and the username and password provided weren't accepted by the target. I'm working with Josh to look at the salt client logs that you provided to see if there is anything more informative there. 

I do want to add that from looking at your logs, you have been running into an issue that we discovered with the 18.5 version of Jupyter that we will be fixing in an upcoming patch. In the logs, you'll see that Jupyter seems to get stuck in a loop of starting its service. For now, I would recommend that you uninstall Jupyter so that if it gets into that loop again, it won't hog any memory resources or flood the message bus.