We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

==> $ df -h /home/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/sys-home  137G  123G  7.2G  95% /home

$ sudo du -shc /home/
34G     /home/
34G     total
===
Looks like some process  is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

With Thanks in Advance
--Zaman

We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

==> $ df -h /home/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/sys-home  137G  123G  7.2G  95% /home

$ sudo du -shc /home/
34G     /home/
34G     total
===
Looks like some process  is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

bzaman wrote:
We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

==> > $ df -h /home/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/sys-home  137G  123G  7.2G  95% /home

$ sudo du -shc /home/
34G        /home/
34G        total
===
Looks like some process  is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

Zaman,
This can happen if you have a lot of small files that occupy a fraction
of a block.  Du is showing the file size but not the actual disk
space.  For example, if you have a file containing 2 bytes, it still
takes up one block plus an inode.  With a lot of files, also check
that you have enough inodes (df -i).  I'm not sure if this is your
problem, but something to check.  If this is your problem, the only way
to fix it that I know of is to reformat the file system with a smaller
block size.

Roger

We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

===
$ df -h /home/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sys-home 137G 123G 7.2G 95% /home

$ sudo du -shc /home/
34G /home/
34G total
====

Looks like some process is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

===
$ df -h /home/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sys-home 137G 123G 7.2G 95% /home

$ sudo du -shc /home/
34G /home/
34G total
====

Looks like some process is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

We are seeing a strange problem on one of our Linux production host.

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

===
$ df -h /home/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sys-home 137G 123G 7.2G 95% /home

$ sudo du -shc /home/
34G /home/
34G total
====

Looks like some process is not releasing space which it was using .
But not sure how to identify the process which is taking space.
We tried to restart one process which is running for many time , but
is not helping .

Can you guys can suggest some troubleshooting space , commands to
identify and resolve the issue .

With Thanks in Advance
--Zaman

bzaman <bzaman.las... at (no spam) gmail.com> writes:
We are seeing a strange problem on one of our Linux production host.
df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

It could be a file which you have deleted but is still open. Those files are
removed from the directory, but the file content is still there and will be
removed when the program closes the file. Eg, you might has erased some huge log
file from /var/log but forgotten to restart the program that had it open.

Solution-- restart that program  or Reboot the system.

bzaman <bzaman.las... at (no spam) gmail.com> writes:
Thanks everybody for sharing the suggestions.
On Oct 14, 9:40=A0pm, Unruh <unruh-s... at (no spam) physics.ubc.ca> wrote:
bzaman <bzaman.las... at (no spam) gmail.com> writes:
We are seeing a strange problem on one of our Linux production host.
df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

It could be a file which you have deleted but is still open. Those files > >are
removed from the directory, but the file content is still there and will > >be
removed when the program closes the file. Eg, you might has erased some h> >uge log
file from /var/log but forgotten to restart the program that had it open.

Solution-- restart that program =A0or Reboot the system.

Yep .  We restarted all the process which we are having doubt  that
may not be releasing space but it was not helping us. Eventually we
rebooted the host which resolved the problem .
probably next time if the problem occurs , we will try to mount/
umount the partition instead of a reboot.

It will just refuse to let you unmount because the partition is in use.
Have you figured out which partition it was that was the problem. You need to
discover which program it was that did this so you can prevent it next time.
It clearly was not one of the ones that you restarted. Which was it and why did it
have such a huge file open?

Thanks everybody for sharing the suggestions.

On Oct 14, 9:40=A0pm, Unruh <unruh-s... at (no spam) physics.ubc.ca> wrote:
bzaman <bzaman.las... at (no spam) gmail.com> writes:
We are seeing a strange problem on one of our Linux production host.
df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

It could be a file which you have deleted but is still open. Those files =
are
removed from the directory, but the file content is still there and will =
be
removed when the program closes the file. Eg, you might has erased some h=
uge log
file from /var/log but forgotten to restart the program that had it open.

Solution-- restart that program =A0or Reboot the system.


Yep . We restarted all the process which we are having doubt that
may not be releasing space but it was not helping us. Eventually we
rebooted the host which resolved the problem .
probably next time if the problem occurs , we will try to mount/
umount the partition instead of a reboot.

... Also , will try
to figure out which processes are using /home and are not releasing
space. Probably , `fuser' and 'lsof' should help us in this regard.

On Oct 15, 9:58=A0am, Unruh <unruh-s... at (no spam) physics.ubc.ca> wrote:
bzaman <bzaman.las... at (no spam) gmail.com> writes:
Thanks everybody for sharing the suggestions.
On Oct 14, 9:40=3DA0pm, Unruh <unruh-s... at (no spam) physics.ubc.ca> wrote:
bzaman <bzaman.las... at (no spam) gmail.com> writes:
We are seeing a strange problem on one of our Linux production host.
df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had
taken the rest of space.

It could be a file which you have deleted but is still open. Those fil=
es =3D
are
removed from the directory, but the file content is still there and wi=
ll =3D
be
removed when the program closes the file. Eg, you might has erased som=
e h=3D
uge log
file from /var/log but forgotten to restart the program that had it op=
en.

Solution-- restart that program =3DA0or Reboot the system.

Yep . =A0We restarted all the process which we are having doubt =A0that
may not be releasing space but it was not helping us. Eventually we
rebooted the host which resolved the problem .
probably next time if the problem occurs , we will try to mount/
umount the partition instead of a reboot.

It will just refuse to let you unmount because the partition is in use.
Have you figured out which partition it was that was the problem. You nee=
d to
discover which program it was that did this so you can prevent it next ti=
me.
It clearly was not one of the ones that you restarted. Which was it and w=
hy did it
have such a huge file open?

The partition was /home . We have some native applications which
reads from file and transfer the data to a different machine . The
process is not a daemon but runs continuously reading and writing
files. We are suspecting those processes only. We restarted those
processes , but it was not helping . So , we rebooted the host which
resolved the problem . Next time , if the problem reoccurs , we
thought of trying a mount/umount instead of a reboot. We will stop
the process before trying to umount /home partition . Also , will try
to figure out which processes are using /home and are not releasing
space. Probably , `fuser' and 'lsof' should
help us in this regard.

Thank You
Zaman

df -h is showing 95% full ( Used- 123G ) for home directory but du is
showing only 34G used for /home. Could not able to identify who had