RabbitMQ database corruption

[JKSH]

I first posted this at https://forums.ni.com/t5/SystemLink-EAR-Support-Community/Which-services-are-required-to-run-the-Sys..., but it nobody replied so I'm re-posting in case it was missed.

I was having trouble getting the NI Skyline Service Manager service up and running again. Every time I tried to start it, it would silently crash again within a minute.

It turns out that I was experiencing the same problem as the OP at https://groups.google.com/forum/#!topic/rabbitmq-users/sMubP52NfHs. Here's an excerpt from C:\ProgramData\RabbitMQ\log\rabbit@localhost.log: 

=INFO REPORT==== 20-Jul-2017::14:19:44 ===
Error description:
   {could_not_start,rabbit,
       {{badmatch,
            {error,
                {{{badmatch,
                      {error,
                          {not_a_dets_file,
                              "c:/PROGRA~3/RabbitMQ/db/RABBIT~3/recovery.dets"}}},

...

I had somehow ended up with a corrupted copy of C:\ProgramData\RabbitMQ\db\rabbit@localhost-mnesia\recovery.dets (it was 0 bytes). Each time I started NI Skyline Service Manager, it in turn tried to start the RabbitMQ service. However, RabbitMQ encountered the bad recovery.dets file and instantly crashed.

After I deleted recovery.dets, I was able to start the NI Skyline Service Manager again. As far as I can tell, my SystemLink is now working properly again.

This is a particularly tough issue, because the system fails silently, it persists between reboots, and needs to be resolved by modifying non-NI files. SystemLink would benefit from a troubleshooting tool that can detect and resolve this -- a good start would be a tool to scan the RabbitMQ logs to detect errors.

[JoshuaP]

Thanks for the additional information.  We will look into creating a troubleshooting tool and specifically try to catch this issue and automatically fix it in a future release.

Can you think of anything that you may have done when installing the software that could have caused this issue?  

[JKSH]

Great to hear, JoshuaP!

---

@JoshuaP wrote:

Can you think of anything that you may have done when installing the software that could have caused this issue?  

---

I'm not sure that the problem was introduced during installation, because after installation I was able to run the SystemLink server just fine.

However, Windows Automatic Updates decided to kick in soon after, and I had to reboot. It was after this reboot that I had trouble starting the SystemLink server.

According to the comments at https://groups.google.com/forum/#!topic/rabbitmq-users/sMubP52NfHs, it sounds like an improper shutdown can cause this particular issue.