SUMMARY: /var out of disk space

As covered in 5.10 of the FAQ, space is not freed up until the process
exits or closes the file.

In my case it turned out that clearcase needed to be restarted to free
up the space. Lsof was the tool suggested by most and what I used to
find the problem process.

Thanks to all the replies, there were far too many to list them all but
Ill list the ones that helped me locate the problem process and a few
others which may help in other circumstances.

Most people suggested restarting syslogd (pkill -HUP syslogd) first to
make sure none of the log files were the problem as this is an quick and
easy check.

Rob McMahon:
This is FAQ 5.10.  Although it mentions lsof in there, it doesn't give
you a cookbook recipe.  The best way to handle this is to get lsof and
do
lsof -a +L1 /var

Robert Collishe
A process probbly has a file lock (flock) open on the file you have
deleted.
Use the fuser -c command (I think it writes the flags to standard error
so 'fuser -c /var 2> /dev/null' works nicely to get just the procs
themselves) on /var to see what's running - you can then run 'pfiles'
against processes in the output to see what's opened - kill the process
and hey presto all your space will return.  If this is a common issue,
scripting the above stuff is pretty straightforward.


Adeday:
We can do a 'find' like:

find /proc/*/fd -links 0 -type f -ls

or look for those above a certain size:

find /proc/*/fd -links 0 -type f -size +2000 -ls

and see which process are have files open and stop/re-start them.


Michael Schulte:
You may not be able to do anything except reboot.
There are several file in /var/ that are opened when the system comes up
   and only closed when the system is shut down.  The process that
opened
   them can not be closed down during operation (closing the process
would
   also close the file).
In particular, some of these files are: /var/adm/{utmp,utmpx,wtmp,wtmpx}
   all of which can get very large (on a system with a lot of users,
   hundreds of MB).

The problem is that until the file is closed, the space on the disk is
not
   released, even if there is no name for the file in the directories.
   "df" report on disk space (from internal partition tables), "du"
traverses
   the directory system.  In the case above, "du" will not count the
file
   and its space, "df" will count the space; numbers differ by quite a
bit.

In the future, the best way to recover the space in such file is to do:
   "cat /dev/null > file".  Once the file name is deleted, you are
stuck!


Raf:
Try:
 find /var -type f | xargs du -h | sort -n |tail -n 15 > topfive.txt

Will show you your 15 biggest files.  Could help in identifying what has
filled up /var.


Simon J. Loewenthal:
I think you or a process might have deleted a file that was still being
written to by another process.  If this is the case, then the process
would have carried on writing to the inode entry and won't stop writing
until the process closes the file, or the system is rebooted.

If you cannot wait you could move all the files under /var/sadm to
another partition, delete /var/sadm and symbolically link the directory
in the other partition to /var/sadm.  This could bide you some time
until you have to reboot.  However, rebooting the computer is the
easiest, although unwanted in your case, option.


Pascal Grostabussiat:
I have encountered those kind of inconsistency in the following case:

If you mount a partition/volume on a mounting point that is not empty,
data that this mounting point contains (along with all sub-directories)
will be masked. This will lead to an inconsistency between what the "du"

command will see and report, and what the file-system ("df" command)
knows and reports.

I am not sure it is your case, but it might be worth double-checking.


-----Original Message-----
From: sunmanagers-bounces@sunmanagers.org
[mailto:sunmanagers-bounces@sunmanagers.org] On Behalf Of Plant, Dean
Sent: 27 November 2006 13:05
To: sunmanagers@sunmanagers.org
Subject: /var out of disk space


On one of my Solaris 8 machines a 512MB /var is showing full. After
clearing up some of the log files from /var/adm the disk still shows
full even though a du -ks /var only shows 74MB usage. Where has the disk
space gone? I have come across this before and a reboot fixed the
problem but I would like to be able to fix it without having to reboot.
Any idea's?

Thanks

Dean.

$ df -k /var
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d2        494235  485224       0   100%    /var

$ sudo du -ks /var
74503   /var

$ sudo df -o i /var
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d2          2847  244449     1%   /var
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers