Strategies for updating remote devices?

[Daklu]

I'm looking for some ideas on how we can manage and update the low level software on our devices.  I'm specifically thinking about the NI-Linux kernal, the LV runtime engine, optional NI-RT components installed via NI Max, and anything else in those layers I haven't thought of.  (We manage the application-level software using SaltStack and opkg.)
 
Background
We have hundreds of sbRIOs deployed around the world running our application. Using NI Max is not a practical solution for us, either on-site or with a remote connection.
 
VxWorks devices execute entirely in memory, so our strategy for those was have an update routine that downloaded an image (which includes all the low-level software), deleted all the old files, copied over the new files, and rebooted the system.  My understanding is this process will not work with linux-based devices because they do not reside entirely in memory.

Ideally we would like use the same tools and distribution mechanisms for the low level software that we do for the application level software--SaltStack and opkg--as much as possible.

Questions

1. I compared the list of ipks installed on a freshly reformatted device in safe mode to a list of the packages installed on a device with only the basic NIRT components installed via Max.  It appears Max does use opkg to do at least some of its work.  It is feasible to eliminate Max altogether and install everything via opkg?  (I'm thinking of a scenario where we want to upgrade our devices in the field from LV2015 to, for example, LV2018.)
2. Does NI update the kernel with each yearly distribution?  If so, I assume this is installed to the device via the Max software installation process.  Can this also be replaced with an opkg updating mechanism?
3. I've been poking at the linux_safemode.itb to try and get a better understanding of the boot process and what customizations I can make.  I see you're using u-boot, but I haven't found any way to decompress the itb file and examine the components.  Is this possible? 

[BradM]

A couple of points:

To question 1., the listing of packages that exist on the target in either run- or safe-mode don't reflect anything that is installed (in the normal way) through MAX, they simply represent the base system image that was created through the image-building process that is used internally and closely replicated at the NI github repo https://github.com/ni/nilrt. Any difference you see between safe- and run-mode is a difference in that base image.

With that clarification out of the way, based on what both MAX and OPKG do, it is possible to create packages for NI software installation through ipks/OPKG. As it was hinted at in the puppet/chef/salt discussions on these forums, NI is considering different approaches to software installation that make more sense for current targets. This doesn't change that NI has to support the current software and installation targets, so it is difficult to move to usch an approach quickly lest we leave some customers in the lurch.

All of this to say, there's nothing preventing you from creating an .ipk that can be used to install NI software instead of the package that is installed through MAX, a so-called "smart tarball", which is strikingly similar in theory and operation to an ipk. NI doesn't currently support that and any hints/tips/suggestions would come at the cost of NI devs being able to do other work.

On to 2. The kernel is indeed updated on each release, and it comes packaged with the base image. It is not installed through .ipks currently, rather through the base image "smart tarball". Again, there's nothing preventing you from building and packaging up a newer kernel in whatever installation package you want for distribution, but again we are treading on ground that is not currently supported and the majority of the burden would be on you to implement.

Finishing with 3., what is it that you wanted to check in the .itb? You already have access to the root filesystem when you boot the controller into safemode. If you really need to dig further into the image, a cursory googling turns up the dumpimage command for u-boot utilities that seems to ... fit *rimshot* the bill. Be forewarned: modifying the safemode itb may result in a bricked target

[Daklu]

Brad,

Thank you for the clarification wrt the images.  I know NI is looking at different approaches; unfortunately, our business requirements dictate that we have a solution within a couple months.  We can't wait for NI's solutions.

I remember your warning about potentially bricking the device by messing with the image, hence my question about ways to unpack the image.  I was /hoping/ to find a simple tool (like zip) that could unpack the existing image, allow me to add the additional services we require, and repack it.  Starting with a known good image seems much safer than trying to replicate your entire image creation process. 

I spent several hours reading information about flat-image trees, but I did not find the dumpimage program.  Thank you for that info... I'll start investigating it to see if it is a viable option.

Working off your comments about MAX's smart tarballs (which I assume is simply a tarball with postinstall/preuninstall scripts included), I found a bunch of .tar files in the National Instruments\RT Images folders on my pc.  Is there a switch in MAX that can be activated that will log the details of what it is doing while installing the NI software?  (Or something else that will help me map functionality in MAX to the tarballs?)

In the interest of /not/ bricking my device by overwriting the safemode image, are there any issues with testing a custom safe mode image by using it as the runmode image?

[BradM]

Daklu wrote:

...

I remember your warning about potentially bricking the device by messing with the image, hence my question about ways to unpack the image.  I was /hoping/ to find a simple tool (like zip) that could unpack the existing image, allow me to add the additional services we require, and repack it.  Starting with a known good image seems much safer than trying to replicate your entire image creation process. 

I spent several hours reading information about flat-image trees, but I did not find the dumpimage program.  Thank you for that info... I'll start investigating it to see if it is a viable option.

...

Sadly, it's not quite so simple. To modify the rootfs used for safemode, you've got to extract the ramdisk from the itb then mount that on a loopback on a Linux system. If, during modification, you run out of space in the ramdisk image (kept close to the needed image size to save on memory use), you'll need to create a new ext2 image (or something that supports extended attributes in the filesystem). If you need to increase the ramdisk size, you risk creating a safemode that won't boot on certain targets or will exhaust the memory during certain operations.

Daklu wrote:

...

Working off your comments about MAX's smart tarballs (which I assume is simply a tarball with postinstall/preuninstall scripts included), I found a bunch of .tar files in the National Instruments\RT Images folders on my pc.  Is there a switch in MAX that can be activated that will log the details of what it is doing while installing the NI software?  (Or something else that will help me map functionality in MAX to the tarballs?)

...

The CDF files link the MAX installable to the .tar.gz's. They're plaintext XML.

Daklu wrote:

... 

In the interest of /not/ bricking my device by overwriting the safemode image, are there any issues with testing a custom safe mode image by using it as the runmode image?

Normally, that would be the way to go, but once again, resource limitations on Zynq targets spoil our fun. The boot partition (where u-boot will look for the itbs) only has enough room for a safemode, a runmode, and a little bit more. Safemode's itb is fairly large. Two of them won't fit in the boot partition, which means the "safemode-as-runmode.itb" trick won't work.

[Daklu]

---

BradM wrote:

If you need to increase the ramdisk size, you risk creating a safemode that won't boot on certain targets or will exhaust the memory during certain operations.

Can you expand on this a little bit?  Would increasing the ramdisk size prevent safemode from booting on certain targets for reasons other than running out of available memory for the operating system to use?

Do you have any idea how much bigger Id have to make the ramdisk before I start running into problems?  My goal is to create a safemode image that includes a salt minion, which I'm guessing is in the tens of MB (including all the dependencies.)  Given the target (9627) has 512 MB of RAM, I'm thinking that risk is low...

BradM wrote:

The CDF files link the MAX installable to the .tar.gz's. They're plaintext XML.

I detect a significant amount of regex fu and grep fu in my future...

Edit:  On the off chance that a device does get bricked, is there *any* way to restore the existing safemode image to the device?

[BradM]

---

@Daklu wrote:

---

@BradM wrote:

If you need to increase the ramdisk size, you risk creating a safemode that won't boot on certain targets or will exhaust the memory during certain operations.

Can you expand on this a little bit?  Would increasing the ramdisk size prevent safemode from booting on certain targets for reasons other than running out of available memory for the operating system to use?

Do you have any idea how much bigger Id have to make the ramdisk before I start running into problems?  My goal is to create a safemode image that includes a salt minion, which I'm guessing is in the tens of MB (including all the dependencies.)  Given the target (9627) has 512 MB of RAM, I'm thinking that risk is low...

...

No, it would be precisely due to running out of memory during the boot process. Regarding increasing the ramdisk size, if you only ever intend on supporting targets with 512MiB of RAM or more, it is pretty safe to increase the size of the ramdisk. Keep in mind that salt requires python, python can be big.