01-12-2016 11:34 AM
Device started booting into safe mode. Discovered that the internal drive was full. Ssh'd into the device and discovered the following:
(safemode) admin@NI-cRIO-9066-0306cade:/# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 127.9M 64.4M 57.0M 53% /
devtmpfs 102.3M 0 102.3M 0% /dev
tmpfs 64.0M 320.0K 63.7M 0% /var/volatile
ubi0:bootfs 49.0M 33.7M 15.3M 69% /boot
tmpfs 115.8M 0 115.8M 0% /dev/shm
ubi0:config 2.1M 192.0K 1.8M 10% /etc/natinst/share
ubi1:rootfs 386.4M 386.4M 0 100% /mnt/userfs
ubi0:config 2.1M 192.0K 1.8M 10% /mnt/userfs/etc/natinst/share
tmpfs 64.0M 320.0K 63.7M 0% /mnt/userfs/var/volatile
(safemode) admin@NI-cRIO-9066-0306cade:/# ls -l
total 35
lrwxrwxrwx 1 admin administ 13 Jan 12 17:23 C -> /mnt/userfs/C
-rw-r--r-- 1 admin administ 1419 Oct 8 2014 README_File_Paths.txt
-rw-r--r-- 1 admin administ 1373 Oct 8 2014 README_File_Transfer.txt
drwxr-xr-x 2 admin administ 1024 Oct 2 2014 bin
drwxrwxrwx 4 admin administ 368 May 14 2015 boot
lrwxrwxrwx 1 admin administ 13 Jan 12 17:23 c -> /mnt/userfs/c
drwxr-xr-x 12 admin administ 13780 Jan 12 17:23 dev
drwxr-xr-x 36 admin administ 2048 Jan 12 17:23 etc
drwxr-xr-x 4 admin administ 1024 Oct 8 2014 home
drwxr-xr-x 8 admin administ 2048 Jan 12 17:23 lib
drwx------ 2 admin administ 16384 Oct 8 2014 lost+found
drwxr-xr-x 11 admin administ 1024 Jan 12 17:23 media
drwxr-xr-x 3 admin administ 1024 Jan 12 17:23 mnt
drwxrwxrwx 3 admin administ 1024 Jan 12 17:23 natinst
dr-xr-xr-x 106 admin administ 0 Jan 1 1970 proc
lrwxrwxrwx 1 admin administ 8 Jan 12 17:23 run -> /var/run
drwxr-xr-x 2 admin administ 2048 Oct 8 2014 sbin
drwxr-xr-x 12 admin administ 0 Jan 12 17:23 sys
lrwxrwxrwx 1 admin administ 8 Jan 12 17:23 tmp -> /var/tmp
drwxr-xr-x 10 admin administ 1024 Oct 8 2014 usr
drwxr-xr-x 7 admin administ 1024 Jan 12 17:23 var
(safemode) admin@NI-cRIO-9066-0306cade:/# cd mnt/userfs
(safemode) admin@NI-cRIO-9066-0306cade:/mnt/userfs# ls -l
total 3456
lrwxrwxrwx 1 admin administ 2 May 14 2015 C -> /c
-rw-r--r-- 1 admin administ 1419 Dec 16 2014 README_File_Paths.txt
-rw-r--r-- 1 admin administ 1373 Dec 16 2014 README_File_Transfer.txt
drwxr-xr-x 2 admin administ 4920 Jan 11 12:45 bin
drwxr-xr-x 2 admin administ 240 May 14 2015 boot
drwxrwxr-x 3 lvuser ni 536 Jan 12 16:38 c
drwxr-xr-x 3 admin administ 5032 May 14 2015 dev
drwxr-xr-x 38 admin administ 6136 Jan 12 17:22 etc
drwxr-xr-x 5 admin administ 488 Jan 11 17:11 home
-rwxrwxrwx 1 admin administ 3528005 Jan 11 12:45 info.zip
drwxr-xr-x 8 admin administ 6120 Jan 11 17:13 lib
drwxr-xr-x 11 admin administ 816 Jan 11 18:41 media
drwxr-xr-x 2 admin administ 416 Dec 9 2014 mnt
drwxr-xr-x 4 webserv ni 304 May 14 2015 natinst
drwxr-xr-x 2 admin administ 160 Dec 9 2014 proc
lrwxrwxrwx 1 admin administ 8 May 14 2015 run -> /var/run
drwxr-xr-x 2 admin administ 7472 Dec 21 18:58 sbin
drwxr-xr-x 2 admin administ 160 Dec 9 2014 sys
lrwxrwxrwx 1 admin administ 8 May 14 2015 tmp -> /var/tmp
drwxr-xr-x 10 admin administ 744 Jan 11 18:44 usr
drwxr-xr-x 7 admin administ 960 Jan 12 15:27 var
It looks like the root file system is mounted within /mnt/userfs instead of at the root /.
Don't have any other 9066's in the office to look at, but compared to a healthy 9064:
admin@NI-cRIO-9064-0308e8a7:/# df -h
Filesystem Size Used Available Use% Mounted on
ubi1:rootfs 850.9M 530.9M 315.2M 63% /
devtmpfs 242.7M 30.8M 211.9M 13% /dev
tmpfs 64.0M 2.2M 61.8M 3% /var/volatile
ubi0:bootfs 49.0M 35.6M 13.4M 73% /boot
tmpfs 242.7M 0 242.7M 0% /dev/shm
ubi0:config 2.1M 2.1M 0 100% /etc/natinst/share
I noticed the way partitions are mounted is quite different... there is no /mnt/userfs mount. I'd like to try and understand how this condition arose, how to prevent it, and afterwards how to restore this particular unit to health (nisystemformat threw an error resulting from the drive being full).
Thanks in advance
01-12-2016 11:49 AM
The difference in mount points for the rootfs is a difference between safe-mode and run-mode. If you boot your 9064 in safe-mode, it should look the same.
01-12-2016 11:53 AM
davegravy2 wrote:
...and afterwards how to restore this particular unit to health (nisystemformat threw an error resulting from the drive being full).
From a console or shell as admin in safe-mode, run this to manually format:
nisystemformat -f -t ubifs
nisystemformat -f -c -t ubifs
01-12-2016 12:01 PM
Good to know.
So then this is likely just an issue of a full drive... Strange because my application has run fine for months and doesn't (shouldn't) write anything to the internal drive except an error log which is not the cause of the full drive.
If you have any pointers as to where to look for extraneous data that's been written I'd be grateful.
01-12-2016 12:13 PM
davegravy2 wrote:
Good to know.
So then this is likely just an issue of a full drive... Strange because my application has run fine for months and doesn't (shouldn't) write anything to the internal drive except an error log which is not the cause of the full drive.
If you have any pointers as to where to look for extraneous data that's been written I'd be grateful.
You can use the "du" utility to identify where the space is being used.
Also make sure that logrotate is successfully running. I've seen it fail when the device has no real-time clock. You can test the logrotate command like this:
logrotate -f /etc/logrotate.conf
When I run it on an affected target it reports:
error: bad year 1970 for file /var/log/auth.log in state file /var/lib/logrotate.status
I have a fix built for my target running LVRT 2015.
01-12-2016 01:26 PM
I've noticed that there is a huge number of files called info.zip scattered throughout the filesystem. They are within almost every (if not every) subdirectory, and each is 3.4M in size.
The archives contain two files:
IMG001.scr (4.3M)
information.vbe (1.3K)
Both appear to be binary files with no readable text. On a fresh cRIO file system these files don't seem to exist. A quick search on google reveals nothing about the file.
Are these important files? Can they safely be deleted?
01-12-2016 01:29 PM
Just found this:
https://malwr.com/analysis/MWJmZjBhNDcwNWJkNDhkM2EzZjMwYzU0ODI1NzQ2MDQ/
I believe this may be caused by a virus.
01-12-2016 01:37 PM
Now, what is interesting is that the .vbe and .scr extensions, popular with virus writers, are a windows-only filetype.
I am wondering if someone has software installed to mount the 9066's disk as a local disk over the network. The Windows system that had the 9066's disk mounted as a network share or drive was infected and proceeded to save these .zip files all over the disk.
01-12-2016 06:57 PM
Was the FTP server installed?
01-12-2016 09:57 PM