Available software not updating when new packages uploaded to repository

[wrkcrw00]

SystemLink isn't showing new packages in a feed, actually it's not showing any of the packages in a feed other than the package that's installed. The options in the "Available" tab under Change Version are "Uninstall". That's it. I have several new packages in that repo so it should say "Upgrade" and give me the option to go back versions. 

One fix is to remove the feed from the target and re-add it. I can see the latest package if I do that. This of course defeats the purpose of having a feed at all. 

The following correlation isn't conclusive, but so far it's only 904x series cRIO's that exhibit this behavior. I checked several of my 9035's and they show all available packages. Also, the 904x's have either SystemLink Client 18.2 or 18.5 installed. The 9035's that I checked are all still back on 17.5 or earlier. 

[KathrynTheGreat]

Hi wrkcrw00,

That is surprising behavior, thanks for bringing it to our attention! Based on the behavior you're describing and the amount of testing you've already done, it sounds like something we will want to look into in more depth than is practical to do on the forums.

Would you be willing to give us a call at 866-275-6964, so that we can work together to further narrow down the cause of the behavior?

[GreyGrey]

Have you looked in the logs to see if there are any errors that seem relevant (both client and server)? Also, are these newer targets ones that you've been having trouble with on your network? To show the upgrades, the target is checking through its registered feeds to see if there are any upgrades and if this is taking awhile, and then there is also latency in the communication, I could see it not responding in time.

Here's the KB that lists where all the logs are at.

[wrkcrw00]

Yes, these targets are the same that I was having deployment issues with until we found the issue with Jupyter. Here's the minion log from one of the targets. Apparently it's running into trouble with the hostname. SystemLink server shows that it's connected. FYI this particular target is not one I've ever seen the communication issue on, only the deployment issue that was fixed by removing Jupyter. I can see health data and if I remove/add the feed I can deploy SW. The server master log is not as easy to understand. I will PM you.

2019-02-28 14:04:45,392 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:46,395 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:47,398 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:48,402 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:49,404 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:50,407 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:51,410 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:52,413 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:53,416 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:54,419 [salt.loaded.ext.beacons.nisysmgmt_startup :57 ][WARNING ][2889] Exception occurred when trying to detect network: [Errno -5] No address associated with hostname
2019-02-28 14:04:54,420 [salt.loaded.ext.beacons.nisysmgmt_startup :65 ][WARNING ][2889] Network not detected after 10 seconds. Continuing.

[GreyGrey]

Can you check to see if your cRIOs have the same time settings as your SystemLink Server? In some cases, this can cause intermittent issues with various parts of SystemLink. 

From looking at the logs, the error that we saw ("Exception occured when trying to detect network") was due to the cRIO taking too long to start the network stack. This can be caused by a slow/unusual network configuration or a slow cRIO. We're not getting this same error from all your cRIOs are we? Are their RT Apps running on these cRIOs? If so, are they resource intensive?

[wrkcrw00]

I have 4 904x's deployed right now. All 4 have the same exact time and timezone as my SystemLink server (well, technically 1 of them is about a minute off).

I do have rtexe's deployed to these cRIO's (done through SystemLink actually). Nominally when they start up they are running anywhere from 15% to 35% cpu (I have one 9049 that is barely being tapped yet) and using < 1GB of RAM. I have noticed another issue in that the cpu usage grows over time...like over days. I looked at one that ran for a day at ~20% then jumped up to 35% for a couple days, then back down, and before I restarted the target today it was at nearly 50% cpu (target is 9045). CPU growth aside, the memory usage is pretty constant and all targets have well over 1GB of free memory.

I will look at the logs for all 4 targets to check for this same error message.

[GreyGrey]

Given those numbers, it doesn't seem like it's a low resource issue. Let me know how those logs look. Feel free to email them to me and I and the team can take a look at them. 

[wrkcrw00 · Accepted solution]

To close out this thread with a solution, I worked NI's SystemLink team to identify the issue and resolve it. There exists in SL 18.5 (likely earlier versions also, but we didn't go into that depth) a bug where the available packages won't display properly when there are too many packages in the target's feeds. I believe the limit is 500 packages. My custom feed only has about a dozen packages, but there are three feeds added by default to RT targets when the SW is installed. These three feeds list every available RT package, each feed containing 100's of packages.

The simple solution was to delete the 3 default feeds and just stick with my custom feed. The issue was resolved and all upgrades and previous package versions display as expected. 

The SystemLink team informed me that a CAR had already existed for this issue and has already been addressed. The next version of SystemLink server (19.0) will include the fix.