PDA

View Full Version : disk usage problem



timblack
28th October 2009, 05:14 AM
Hi all. My root file system is full and I can't figure out why. I've looked at the disk usage using baobab, and below is the output from several pertinent commands. It doesn't add up and I'm not sure why. / is a logical volume (/dev/mapper/VolGroup00-LogVol00) of size 35GB, but the sum of its components' usage (output from du) is around 11GB, yet df shows / is full! I have emptied the trash in Gnome desktop... Any ideas?


[tim@servy ~]$ sudo df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
35645048 33535440 269688 100% /
/dev/sda1 194442 22358 162045 13% /boot
tmpfs 1019328 268 1019060 1% /dev/shm
/dev/mapper/vg1-lv0 976625664 922792016 53833648 95% /media/raid
[tim@servy ~]$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/md1 vg1 lvm2 a- 931.51G 0
/dev/sda2 VolGroup00 lvm2 a- 37.06G 32.00M
/dev/sdb1 vg2 lvm2 a- 931.51G 0
[tim@servy ~]$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz--n- 37.06G 32.00M
vg1 1 1 0 wz--n- 931.51G 0
vg2 1 1 0 wz--n- 931.51G 0
[tim@servy ~]$ sudo lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol00 VolGroup00 -wi-ao 35.09G
LogVol01 VolGroup00 -wi-ao 1.94G
lv0 vg1 -wi-ao 931.51G
lv0 vg2 -wi-a- 931.51G
[tim@servy ~]$ ls /
./ bin/ home/ media/ opt/ selinux/ tmp/
../ boot/ lib/ misc/ proc/ srv/ usr/
.autofsck dev/ lib64/ mnt/ root/ stripey/ var/
.bash_history etc/ lost+found/ net/ sbin/ sys/
[tim@servy ~]$ sudo du -hs /bin
7.7M /bin
[tim@servy ~]$ sudo du -hs /boot
17M /boot
[tim@servy ~]$ sudo du -hs /dev
348K /dev
[tim@servy ~]$ sudo du -hs /etc
186M /etc
[tim@servy ~]$ sudo du -hs /home
2.1G /home
[tim@servy ~]$ sudo du -hs /lib
197M /lib
[tim@servy ~]$ sudo du -hs /lib64
31M /lib64
[tim@servy ~]$ sudo du -hs /lost+found
20K /lost+found
[tim@servy ~]$ sudo du -hs /misc
0 /misc
[tim@servy ~]$ sudo du -hs /mnt
24K /mnt
[tim@servy ~]$ sudo du -hs /net
0 /net
[tim@servy ~]$ sudo du -hs /opt
8.0K /opt
[tim@servy ~]$ sudo du -hs /proc
du: cannot access `/proc/25977/task/25977/fd/4': No such file or directory
du: cannot access `/proc/25977/task/25977/fdinfo/4': No such file or directory
du: cannot access `/proc/25977/fd/4': No such file or directory
du: cannot access `/proc/25977/fdinfo/4': No such file or directory
0 /proc
[tim@servy ~]$ sudo du -hs /root
40M /root
[tim@servy ~]$ sudo du -hs /sbin
17M /sbin
[tim@servy ~]$ sudo du -hs /selinux
0 /selinux
[tim@servy ~]$ sudo du -hs /stripey
16K /stripey
[tim@servy ~]$ sudo du -hs /sys
0 /sys
[tim@servy ~]$ sudo du -hs /tmp
26M /tmp
[tim@servy ~]$ sudo du -hs /usr
7.2G /usr
[tim@servy ~]$ sudo du -hs /var
1.5G /var

Hlingler
28th October 2009, 05:22 AM
For added clarity, try:
sudo du -chs --max-depth=1 /

Temporary fix:
sudo yum clean all

Looks like way too much junk in /var (1.5 GB is too much):
sudo du -chs --max-depth=1 /var

V

timblack
28th October 2009, 05:29 AM
Thanks (again) V!

Already did yum clean all.

1.5GB is a lot for logs, but I tend to use verbose logs. Anyway, this doesn't account for the 24GB discrepancy.

[tim@servy ~]$ sudo du -chs --max-depth=1 /
du: warning: summarizing conflicts with --max-depth=1
Try `du --help' for more information.

Hlingler
28th October 2009, 05:31 AM
OOPS! omit the 's' option.
sudo du -ch --max-depth=1 /
sudo du -ch --max-depth=1 /var

V

timblack
28th October 2009, 05:36 AM
Thanks, figured that one out. Dropped the -s, but of course it's the same output in nicer format. Took a while bc it's about a terabyte of data in sum. Note most of this is in /media/raid. I'm concerned with what is in / that is not in /media/raid or /boot.

Are there any hidden, non-obvious places I can check for disk usage that will account for the 24GB discrepancy between the output of df and the summed output of du?

Hlingler
28th October 2009, 05:41 AM
To be honest, I'm at a bit of a loss here as to what might be causing the discrepancy, and I know nothing about RAID. Let's see if one of the resident experts comes along... .

<BUMP>

V

kyryder
28th October 2009, 05:49 AM
Have you cleaned /var/tmp ?
Edit: Would bad sectors cause this?

diamond_ramsey
28th October 2009, 05:54 AM
...Are there any hidden, non-obvious places I can check for disk usage that will account for the 24GB discrepancy between the output of df and the summed output of du?

:) timblack, interesting story/mystery. :)

If possible then, cd to / and do a ls -la which may be posted as a snippet. :)

I would like to see the listing.

Also, anywhere in /home and/or /usr which may be cleaned up for space? :)

timblack
28th October 2009, 05:55 AM
Below is some more readable du output. The only Gs in the list are for /home, /media, /usr, and /var. We can ignore /media which is on its own logical volume. As I mentioned before the others only sum to ~11GB. But the / partition is 35GB as shown in the df output of the OP.


[tim@servy ~]$ sudo du -h --max-depth=1 /
26M /tmp
du: cannot access `/proc/26890/task/26890/fd/4': No such file or directory
du: cannot access `/proc/26890/task/26890/fdinfo/4': No such file or directory
du: cannot access `/proc/26890/fd/4': No such file or directory
du: cannot access `/proc/26890/fdinfo/4': No such file or directory
0 /proc
du: cannot access `/home/tim/.gvfs': Permission denied
2.1G /home
186M /etc
40M /root
17M /boot
20K /lost+found
cc902G /media
8.0K /opt
0 /selinux
0 /net
24K /mnt
7.2G /usr
0 /sys
17M /sbin
31M /lib64
160K /dev
1.5G /var
0 /misc
16K /stripey
197M /lib
8.0K /srv
7.7M /bin
913G /

Hlingler
28th October 2009, 05:59 AM
This I don't understand:
du: cannot access `/home/tim/.gvfs': Permission deniedV

timblack
28th October 2009, 06:01 AM
Thanks for your input kyryder and diamond_ramsey! Sure, cleanup of /home or /var or /usr would reduce usage, but I'm mostly concerned with the massive discrepancy between the summed output of "du --max-depth=1 /" and "df". Maybe I will look into what lvm diagnostics there are to run next to see if there are any disk problems...

And here is the ls output you requested:

[tim@servy ~]$ ls -la /
total 206
drwxr-xr-x 25 root root 4096 2009-10-26 08:27 ./
drwxr-xr-x 25 root root 4096 2009-10-26 08:27 ../
-rw-r--r-- 1 root root 0 2009-10-26 08:27 .autofsck
-rw------- 1 root root 5796 2008-02-07 20:12 .bash_history
drwxr-xr-x 2 root root 4096 2009-10-23 21:10 bin/
drwxr-xr-x 5 root root 1024 2009-10-19 15:54 boot/
drwxr-xr-x 17 root root 4900 2009-10-26 19:07 dev/
drwxr-xr-x 137 root root 12288 2009-10-27 20:19 etc/
drwxr-xr-x 8 root root 4096 2008-09-06 03:13 home/
drwxr-xr-x 12 root root 12288 2009-10-27 20:19 lib/
drwxr-xr-x 8 root root 12288 2009-10-23 21:10 lib64/
drwx------ 2 root root 16384 2008-01-11 18:16 lost+found/
drwxr-xr-x 6 root root 4096 2009-10-26 19:07 media/
drwxr-xr-x 2 root root 0 2009-10-26 08:27 misc/
drwxr-xr-x 4 root root 4096 2008-09-06 03:13 mnt/
drwxr-xr-x 2 root root 0 2009-10-26 08:27 net/
drwxr-xr-x 2 root root 4096 2008-09-06 03:13 opt/
dr-xr-xr-x 278 root root 0 2009-10-26 08:27 proc/
drwxr-x--- 27 root root 4096 2009-10-24 20:54 root/
drwxr-xr-x 2 root root 12288 2009-10-19 22:04 sbin/
drwxr-xr-x 7 root root 0 2009-10-26 08:27 selinux/
drwxr-xr-x 2 root root 4096 2008-09-06 03:13 srv/
drwxr-xr-x 3 root root 4096 2009-04-14 18:52 stripey/
drwxr-xr-x 12 root root 0 2009-10-26 08:27 sys/
drwxrwxrwt 15 root root 4096 2009-10-27 21:37 tmp/
drwxr-xr-x 14 root root 4096 2009-09-02 07:56 usr/
drwxr-xr-x 22 root root 4096 2009-10-19 12:35 var/

timblack
28th October 2009, 06:04 AM
A side note: I have just come out of a massive update to FC10 from FC8, where I had about a thousand packages to fix/update due to piles of arch package conflict poop left behind by PreUpgrade. I thought yum clean all would clean up any temporary package files. Are there any other yum cleanup tools that I should run?

timblack
28th October 2009, 06:08 AM
This I don't understand:V

Me neither. I'm assuming this has to do with the "GNOME virtual file system, a userspace virtual filesystem". Could I have some hidden and massive virtual filesystem occupying all this space? You'd think the non-virtual filesystem would at least have the decency to report its usage...

Hlingler
28th October 2009, 06:11 AM
Are there any other yum cleanup tools that I should run?sudo package-cleanup --dupes

Ideally, there should be none.

V

timblack
28th October 2009, 06:13 AM
[tim@servy ~]$ sudo package-cleanup --dupes
Setting up yum
Loaded plugins: refresh-packagekit
[tim@servy ~]

Hlingler
28th October 2009, 06:13 AM
"A swing and a miss".

No dupes. Hmm...

V

stevea
28th October 2009, 07:18 AM
Yes the .gvfs is part of the funked-up Gnome VFS and it's entirely possible this is inaccessible to root. Part of the Gnomes-gone-wild set of bad ideas.

Yes, it really looks like you are missing a good bit of your file sysytem and the reason is mysterious based on this evidence..


I DO suggest you examine the root directory with ...
ls -CFla /[/B

The other commands you've used would not check any hidden file or directory in root (file or dirnames that begin with a "."). At a minimum you should see a "/.dbus/" directory.

I can't imagine why such a file or directory in root would grow uncontrolled, but it's not impossible - check it out.

In the unlikely event that sone monster files exist there - you can probably delete them w/o system functional consequence.

=======

More likely your filesystem needs a fsck check. Somehow some glitch made a lot of blocks disappear. For root "/" you can only do this from the reboot (or from a live CD).


To do this ....[B]
sudo touch /forcefsck
sudo reboot

when your system comes back up it will spend some minutes checking root file system.
It should hopefully recover your lost blocks.

SlowJet
28th October 2009, 08:27 AM
/media looks a bit odd (902g + (others) ~= /913GB
If media was supposed to be a separate file system?
If itt was never mounted, then 902GB has been added to the overall / f/s.

SJ

timblack
29th October 2009, 01:18 AM
Slowjet's last comment turned on a light bulb over my head. It turned out that during the upgrade to FC10, anaconda overwrote my /etc/fstab, omitting the mount point for /media/main, which is another huge logical volume in my system that had, interestingly, ~24GB of data on it. Because it wasn't being mounted, ever since l installed FC10, everything I wrote to /media/main went onto the / 35GB volume. You know, the one that became mysteriously full?

So what I did was to mount the volume on /media/main2, move the data (~24GB) from /media/main to /media/main2, update /etc/fstab to mount the volume now on /media/main.

I suppose if I had originally posted my /etc/fstab, someone would have caught this sooner. Thanks, everybody.

Now all is as it was before the FC8->FC10 upgrade. Well, almost. I wish this was the last scab I have to slough off, but I know it's not, bc of the FC10 DNS problems (http://fedoraforum.org/forum/showthread.php?t=232292) I still have to contend with.