Unable to access or format cRIO 9068

[Oksavik]

Hello,

I'm struggling with a cRIO 9068. I've had problems with this particular device before, but I've always been able to fix them. This time I'm lost.

What I basically want to do at this time is reset the cRIO completely. I'm not able to discover it in MAX, so I'm trying to do this using a "recovery.cfg" on an USB drive.

I've copied the file "C:\Program Files (x86)\National Instruments\Shared\Firmware\cRIO\76D6\cRIO-9068_4.0.0.cfg" to a formatted (FAT 32) USB drive, and renamed it to "recovery.cfg".

Powered off, hold reset button, reconnect power, release reset button when status light is solid on. After a second or two the status light starts blinking continuously. I've tried connecting the USB drive via an USB hub, as advised by NI.

I'm trying to log the console using a null-modem cable and Putty, but there is not much output. At one point I got the following when connecting the power. Maybe this makes sense to someone, I've tried googling the term "Kernel panic - not syncing", but the answers doesn't immediately help me.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.03.27 14:30:45 =~=~=~=~=~=~=~=~=~=~=~=
Booting LabVIEW RT...
[    1.721130] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[    1.721140] CPU1: stopping
[    1.721151] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.1.15-rt17-ni-4.0.0f1 #1
[    1.721154] Hardware name: Xilinx Zynq Platform
[    1.721160] Backtrace: 
[    1.721192] [<c00135d8>] (dump_backtrace) from [<c00137b8>] (show_stack+0x20/0x24)
[    1.721205]  r6:c0804b88 r5:00000001 r4:00000000 r3:00200040
[    1.721227] [<c0013798>] (show_stack) from [<c0569ca4>] (dump_stack+0x78/0x98)
[    1.721242] [<c0569c2c>] (dump_stack) from [<c00158bc>] (handle_IPI+0x140/0x2e8)
[    1.721249]  r4:c0813f18 r3:00010002
[    1.721260] [<c001577c>] (handle_IPI) from [<c0009400>] (gic_handle_irq+0x68/0x70)
[    1.721275]  r8:0000406a r7:df53bfac r6:c07b3a00 r5:df53bf78 r4:f8f00100 r3:00000005
[    1.721286] [<c0009398>] (gic_handle_irq) from [<c0014340>] (__irq_svc+0x40/0x88)
[    1.721291] Exception stack(0xdf53bf78 to 0xdf53bfc0)
[    1.721298] bf60:                                                       dfbe1f80 00000000
[    1.721309] bf80: df53bfd0 c0020fe0 00000001 c07a745c 10c0387d c0813f30 0000406a 413fc090
[    1.721321] bfa0: 00000000 df53bfcc df53bfd0 df53bfc0 c0010900 c0010904 60000113 ffffffff
[    1.721332]  r6:ffffffff r5:60000113 r4:c0010904 r3:c0010900
[    1.721356] [<c00108c8>] (arch_cpu_idle) from [<c005b1f0>] (cpu_startup_entry+0x26c/0x31c)
[    1.721370] [<c005af84>] (cpu_startup_entry) from [<c00154f8>] (secondary_start_kernel+0x154/0x178)
[    1.721381] [<c00153a4>] (secondary_start_kernel) from [<000094ac>] (0x94ac)
[    1.721388]  r4:1f50c06a r3:c0009494
[    1.869082] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[    2.042289] ------------[ cut here ]------------
[    2.070803] WARNING: CPU: 0 PID: 23 at /builds/penguin/nilinux/kernel/nilrt-zynq/trunk/4.0/objects/nilrt_zynq_kernel/linuxU/armv7-a/gcc-4.7-oe/release/source/kernel/workqueue.c:2004 process_one_work+0x74/0x434()
[    2.109377] Modules linked in:
[    2.122435] CPU: 0 PID: 23 Comm: kworker/1:0 Not tainted 4.1.15-rt17-ni-4.0.0f1 #1
[    2.130021] Hardware name: Xilinx Zynq Platform
[    2.144425] Backtrace: 
[    2.146897] [<c00135d8>] (dump_backtrace) from [<c00137b8>] (show_stack+0x20/0x24)
[    2.164420]  r6:c068c6ca r5:00000009 r4:00000000 r3:00000000
[    2.170129] [<c0013798>] (show_stack) from [<c0569ca4>] (dump_stack+0x78/0x98)
[    2.187305] [<c0569c2c>] (dump_stack) from [<c0025864>] (warn_slowpath_common+0x98/0xc4)
[    2.205289]  r4:00000000 r3:00000000
[    2.208891] [<c00257cc>] (warn_slowpath_common) from [<c00258bc>] (warn_slowpath_null+0x2c/0x34)
[    2.227552]  r8:dfbe4c00 r7:c080494f r6:dfbe1240 r5:dfbdfcac r4:df49b300
[    2.244184] [<c0025890>] (warn_slowpath_null) from [<c003ae68>] (process_one_work+0x74/0x434)
[    2.262606] [<c003adf4>] (process_one_work) from [<c003b54c>] (worker_thread+0x2e8/0x460)
[    2.270802]  r10:dfbe129c r9:00000008 r8:df49b318 r7:c07b0400 r6:dfbe1268 r5:dfbe1240
[    2.288547]  r4:df49b300
[    2.291101] [<c003b264>] (worker_thread) from [<c00404a8>] (kthread+0xf0/0x100)
[    2.308337]  r10:00000000 r9:00000000 r8:00000000 r7:c003b264 r6:df49b300 r5:00000000
[    2.326096]  r4:df49a3c0 r3:df540c00
[    2.329699] [<c00403b8>] (kthread) from [<c000ffa8>] (ret_from_fork+0x14/0x2c)
[    2.346855]  r7:00000000 r6:00000000 r5:c00403b8 r4:df49a3c0
[    2.362422] ---[ end trace 0000000000000002 ]---

How should I proceed to completely reformat the cRIO at this point?

I should say I'm also in contact with NI support. I've followed this procedure:

Procedure to follow:
1. Obtain a FAT16 or FAT32 USB drive with FAT on the first partition.
2. Copy the attached recovery.cfg file which matches your controller type from this article to the thumb drive.
3. Plug in the USB drive to the controller.
4. Power off the target by unplugging the power.
5. Press and hold the reset button while completing the next two steps.
6. Power on the target by plugging the power back in.
7. Once the status light turns on, release the reset button.
8. Wait for the controller to boot. This should take less than a minute.
9. If a customer has software installed (run mode) on the target it may be necessary to force the controller to boot into safemode. Refer to the controllers documentation on how to do this. Due to CAR 545444 it may require a second attempt to put the controller into safemode before the controller is accessible.
10. The customer should format and reinstall software to the target as most of the configuration files that are created for LabVIEW Real-Time will be removed in this process.

If you see continuous flashing of the status light after attempting to change the password the target is either in an unrecoverable error state or it simply couldn't read the USB drive. If this occurs try the following steps,
1. First, try forcing the controller into safemode to see if the utility was successful (Step 9, sub item 1). Console out should be enabled while the controller is booting.
2. Try connecting the USB drive to the target through a USB Hub. Situations where the USB drive is not recognized by the Zynq target it is normally an issue related to timing with the recovery tool and not the USB drive itself. Connecting through a USB hub should resolve this timing issue.
3. If it was not successful please try using a new USB drive- in case the problem is limited to the controller's ability to read the drive.
4. If new USB drives does not work, then you may need to repair the safe mode installation on your target. Please get a console out log for the controller before proceeding to the next step. Please see: Repairing Safe Mode on a NI Linux Real-Time OS Controller for further instructions. (see below)


Repairing Safe Mode on a NI Linux Real-Time OS Controller

To repair the safe mode on your controller you will need a FAT16 or FAT32 USB drive with FAT on the first partition. Follow the steps below in order to repair the safemode on your controller. While the file listed below has the same name as the recovery file used to reset the password for ARM-based targets, the safe mode recovery.cfg files are very different. The easiest way to tell these files apart is the file size, as the files used to repair safe mode are about 30-40 MB and the password-resetting files are just ~2 KB.
1. Find a USB drive and format it into FAT32.
2. Copy the recovery.cfg file to the empty USB drive.
3. Power off the device.
4. With device powered off, plug in the USB drive to the controller.
5. While holding down the reset button, power on the controller. Continue holding down the reset button until the Status LED turns on before releasing the reset button.
6. Wait for the controller to boot; this should take less than a minute.
If you see continuous flashing of the Status LED after attempting to recover safemode, the target may not have read the USB drive successfully. If this occurs try the following steps:
1. Try connecting the USB drive to the target through a USB Hub. Situations where the USB drive is not recognized by the Zynq target are normally related to timing with the recovery tool and not the USB drive itself. Connecting through a USB hub should resolve this timing issue.
2. If the previous step was not successful please try using a new USB drive in case the problem is limited to the controller's ability to read the drive.

Any advice is appreciated.

[mferguss]

Hi Oksavik,

After using the recovery file and carrying out the actions have you then tried forcing the device into safe mode as per page 2 of the following:

http://www.ni.com/pdf/manuals/376007b_03.pdf

You may have to do this a couple of times to do so.

When you have tried the recovery what is the actual issue that you are seeing? You are not able to see it in MAX still?

Kind Regards,

Matthew Fergusson

Applications Engineer UK

Matthew 

[Oksavik]

When using the recovery file, the cRIO status LED immediately starts blinking continuously. I assume this means that there is something wrong with the recovery file, the usb drive, or something more fundamental with the cRIO.

I've also tried the "repair safe mode", by renaming "cRIO-9068_4.0.0.cfg" to "recovery.cfg", and follow the same procedure. Continuous blinking then also.

I'm not able to discover the device in MAX.

Thank you for your reply.

[a_clucker]

Hey Oksavik,

The repair safe mode ("recovery image") is a different file than the cRIO-9068_4.0.0 file. True, they both share the same extension, but they contain very different things. cRIO-9068_4.0.0 is firmware.

Matthew Ferguson - can you contact Oksavik and start a Service Request to troubleshoot and potentially get Oksavik the recovery image?

[Oksavik]

---

@a_cluckerThe repair safe mode ("recovery image") is a different file than the cRIO-9068_4.0.0 file. True, they both share the same extension, but they contain very different things. cRIO-9068_4.0.0 is firmware.

---

Really? I guess I just assumed it was this file, since I wasn't corrected by NI support when I said I used it (they didn't attach any file), and the file size seemed correct:

The easiest way to tell these files apart is the file size, as the files used to repair safe mode are about 30-40 MB and the password-resetting files are just ~2 KB.

 I would appreciate the correct recovery image, if someone could get in touch.

Thanks!

[ikaiser]

---

@Oksavik wrote: 
 I would appreciate the correct recovery image, if someone could get in touch.

Looks to me like you already have it, because it is mentioned in the procedure from NI support(?) you quoted in your first posting: "Copy the attached recovery.cfg file [...]".

As far as I know this recovery.cfg file is the "recovery image" you are looking for. Do you have it?

[a_clucker]

Really. There are three main .CFG files used with cRIO: password stuff (only for certain OS)[recovery.cfg]; safe mode recovery (only for certain targets and OS)[different recovery.cfg]; and firmware [cRIO-90xx_y.y.y.cfg]. Think about it like any other file format - .TXT, .MP3, .VI - you can have very different things stored in these file containers, like both a Shakespeare play and a school essay can both go in a TXT file.

I chatted with the AEs that are working with you (Matthew and Elmunzir/Razvan through the Service Request). They'll coordinate on this.

[Oksavik]

---

@ikaiserAs far as I know this recovery.cfg file is the "recovery image" you are looking for. Do you have it?

---

There was no attachments unfortunately, I think the procedure was copy-pasted from an article on the web, and the files were attached there perhaps.

I have received the small recovery.cfg however, for password reset.

[Oksavik]

---

@a_clucker wrote:

Really. There are three main .CFG files used with cRIO: password stuff (only for certain OS)[recovery.cfg]; safe mode recovery (only for certain targets and OS)[different recovery.cfg]; and firmware [cRIO-90xx_y.y.y.cfg]. Think about it like any other file format - .TXT, .MP3, .VI - you can have very different things stored in these file containers, like both a Shakespeare play and a school essay can both go in a TXT file.

---

Thank you, I understand now.

---

@a_cluckerI chatted with the AEs that are working with you (Matthew and Elmunzir/Razvan through the Service Request). They'll coordinate on this.

---

Thank you very much. I will post a reply here when the problem is resolved.

[tuenis]

I am curious to know if this issue was ever resolved...

I ran into the exact same scenario when I click the box to force a boot into safe mode from NI-Max.

Upon reboot, the cRIO never communicated again and the status LED blinks continuous.

I did the same steps of copying the cRIO-9068_6.0.0.cfg file and renaming it as recovery.cfg.

It sounds like however there is a separate cfg file used specifically for recovery.

If this is the case, how do I go about obtaining the recovery.cfg file for the cRIO-9068?

Thanks for your help