NI Linux Real-Time Discussions

cancel
Showing results for 
Search instead for 
Did you mean: 

9066 strange internal disk issue - drive full?

Device started booting into safe mode. Discovered that the internal drive was full. Ssh'd into the device and discovered the following:

(safemode) admin@NI-cRIO-9066-0306cade:/# df -h

Filesystem                Size      Used Available Use% Mounted on

/dev/root               127.9M     64.4M     57.0M  53% /

devtmpfs                102.3M         0    102.3M   0% /dev

tmpfs                    64.0M    320.0K     63.7M   0% /var/volatile

ubi0:bootfs              49.0M     33.7M     15.3M  69% /boot

tmpfs                   115.8M         0    115.8M   0% /dev/shm

ubi0:config               2.1M    192.0K      1.8M  10% /etc/natinst/share

ubi1:rootfs             386.4M    386.4M         0 100% /mnt/userfs

ubi0:config               2.1M    192.0K      1.8M  10% /mnt/userfs/etc/natinst/share

tmpfs                    64.0M    320.0K     63.7M   0% /mnt/userfs/var/volatile

(safemode) admin@NI-cRIO-9066-0306cade:/# ls -l

total 35

lrwxrwxrwx    1 admin    administ        13 Jan 12 17:23 C -> /mnt/userfs/C

-rw-r--r--    1 admin    administ      1419 Oct  8  2014 README_File_Paths.txt

-rw-r--r--    1 admin    administ      1373 Oct  8  2014 README_File_Transfer.txt

drwxr-xr-x    2 admin    administ      1024 Oct  2  2014 bin

drwxrwxrwx    4 admin    administ       368 May 14  2015 boot

lrwxrwxrwx    1 admin    administ        13 Jan 12 17:23 c -> /mnt/userfs/c

drwxr-xr-x   12 admin    administ     13780 Jan 12 17:23 dev

drwxr-xr-x   36 admin    administ      2048 Jan 12 17:23 etc

drwxr-xr-x    4 admin    administ      1024 Oct  8  2014 home

drwxr-xr-x    8 admin    administ      2048 Jan 12 17:23 lib

drwx------    2 admin    administ     16384 Oct  8  2014 lost+found

drwxr-xr-x   11 admin    administ      1024 Jan 12 17:23 media

drwxr-xr-x    3 admin    administ      1024 Jan 12 17:23 mnt

drwxrwxrwx    3 admin    administ      1024 Jan 12 17:23 natinst

dr-xr-xr-x  106 admin    administ         0 Jan  1  1970 proc

lrwxrwxrwx    1 admin    administ         8 Jan 12 17:23 run -> /var/run

drwxr-xr-x    2 admin    administ      2048 Oct  8  2014 sbin

drwxr-xr-x   12 admin    administ         0 Jan 12 17:23 sys

lrwxrwxrwx    1 admin    administ         8 Jan 12 17:23 tmp -> /var/tmp

drwxr-xr-x   10 admin    administ      1024 Oct  8  2014 usr

drwxr-xr-x    7 admin    administ      1024 Jan 12 17:23 var

(safemode) admin@NI-cRIO-9066-0306cade:/# cd mnt/userfs

(safemode) admin@NI-cRIO-9066-0306cade:/mnt/userfs# ls -l

total 3456

lrwxrwxrwx    1 admin    administ         2 May 14  2015 C -> /c

-rw-r--r--    1 admin    administ      1419 Dec 16  2014 README_File_Paths.txt

-rw-r--r--    1 admin    administ      1373 Dec 16  2014 README_File_Transfer.txt

drwxr-xr-x    2 admin    administ      4920 Jan 11 12:45 bin

drwxr-xr-x    2 admin    administ       240 May 14  2015 boot

drwxrwxr-x    3 lvuser   ni             536 Jan 12 16:38 c

drwxr-xr-x    3 admin    administ      5032 May 14  2015 dev

drwxr-xr-x   38 admin    administ      6136 Jan 12 17:22 etc

drwxr-xr-x    5 admin    administ       488 Jan 11 17:11 home

-rwxrwxrwx    1 admin    administ   3528005 Jan 11 12:45 info.zip

drwxr-xr-x    8 admin    administ      6120 Jan 11 17:13 lib

drwxr-xr-x   11 admin    administ       816 Jan 11 18:41 media

drwxr-xr-x    2 admin    administ       416 Dec  9  2014 mnt

drwxr-xr-x    4 webserv  ni             304 May 14  2015 natinst

drwxr-xr-x    2 admin    administ       160 Dec  9  2014 proc

lrwxrwxrwx    1 admin    administ         8 May 14  2015 run -> /var/run

drwxr-xr-x    2 admin    administ      7472 Dec 21 18:58 sbin

drwxr-xr-x    2 admin    administ       160 Dec  9  2014 sys

lrwxrwxrwx    1 admin    administ         8 May 14  2015 tmp -> /var/tmp

drwxr-xr-x   10 admin    administ       744 Jan 11 18:44 usr

drwxr-xr-x    7 admin    administ       960 Jan 12 15:27 var

It looks like the root file system is mounted within /mnt/userfs instead of at the root /.

Don't have any other 9066's in the office to look at, but compared to a healthy 9064:

admin@NI-cRIO-9064-0308e8a7:/# df -h

Filesystem                Size      Used Available Use% Mounted on

ubi1:rootfs             850.9M    530.9M    315.2M  63% /

devtmpfs                242.7M     30.8M    211.9M  13% /dev

tmpfs                    64.0M      2.2M     61.8M   3% /var/volatile

ubi0:bootfs              49.0M     35.6M     13.4M  73% /boot

tmpfs                   242.7M         0    242.7M   0% /dev/shm

ubi0:config               2.1M      2.1M         0 100% /etc/natinst/share

I noticed the way partitions are mounted is quite different... there is no /mnt/userfs mount. I'd like to try and understand how this condition arose, how to prevent it, and afterwards how to restore this particular unit to health (nisystemformat threw an error resulting from the drive being full).

Thanks in advance

0 Kudos
Message 1 of 30
(7,831 Views)

The difference in mount points for the rootfs is a difference between safe-mode and run-mode. If you boot your 9064 in safe-mode, it should look the same.

0 Kudos
Message 2 of 30
(4,774 Views)

davegravy2 wrote:

...and afterwards how to restore this particular unit to health (nisystemformat threw an error resulting from the drive being full).

From a console or shell as admin in safe-mode, run this to manually format:

nisystemformat -f -t ubifs

nisystemformat -f -c -t ubifs

0 Kudos
Message 3 of 30
(4,774 Views)

Good to know.

So then this is likely just an issue of a full drive... Strange because my application has run fine for months and doesn't (shouldn't) write anything to the internal drive except an error log which is not the cause of the full drive.

If you have any pointers as to where to look for extraneous data that's been written I'd be grateful.

0 Kudos
Message 4 of 30
(4,774 Views)

davegravy2 wrote:

Good to know.

So then this is likely just an issue of a full drive... Strange because my application has run fine for months and doesn't (shouldn't) write anything to the internal drive except an error log which is not the cause of the full drive.

If you have any pointers as to where to look for extraneous data that's been written I'd be grateful.

You can use the "du" utility to identify where the space is being used.

Also make sure that logrotate is successfully running. I've seen it fail when the device has no real-time clock. You can test the logrotate command like this:

logrotate -f /etc/logrotate.conf

When I run it on an affected target it reports:

error: bad year 1970 for file /var/log/auth.log in state file /var/lib/logrotate.status

I have a fix built for my target running LVRT 2015.

0 Kudos
Message 5 of 30
(4,774 Views)

I've noticed that there is a huge number of files called info.zip scattered throughout the filesystem. They are within almost every (if not every) subdirectory, and each is 3.4M in size.

The archives contain two files:

IMG001.scr (4.3M)

information.vbe (1.3K)

Both appear to be binary files with no readable text. On a fresh cRIO file system these files don't seem to exist. A quick search on google reveals nothing about the file.

Are these important files? Can they safely be deleted?

0 Kudos
Message 6 of 30
(4,774 Views)

Just found this:

https://malwr.com/analysis/MWJmZjBhNDcwNWJkNDhkM2EzZjMwYzU0ODI1NzQ2MDQ/

I believe this may be caused by a virus.

0 Kudos
Message 7 of 30
(4,774 Views)

Now, what is interesting is that the .vbe and .scr extensions, popular with virus writers, are a windows-only filetype.

I am wondering if someone has software installed to mount the 9066's disk as a local disk over the network. The Windows system that had the 9066's disk mounted as a network share or drive was infected and proceeded to save these .zip files all over the disk.

0 Kudos
Message 8 of 30
(4,774 Views)

Was the FTP server installed?

0 Kudos
Message 9 of 30
(4,774 Views)

Legacy FTP server was installed, yes.

David Grant, B.A.Sc. P.Eng.

davidg@aercoustics.com<mailto:davidg@aercoustics.com>

Tel: +1-416-249-3361;291

www.aercoustics.com<http://www.aercoustics.com/

0 Kudos
Message 10 of 30
(4,774 Views)