Cannot ‘ls’ /mnt directory

directoryfilesfilesystemsls

I have a CentOS-6 machine that I'm working on that is running a live database. It's having a problem with the /mnt directory. I cannot ls the directory for some reason. stat is working and shows this output:

  File: `/mnt/'
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: ca02h/51714d    Inode: 16321       Links: 9
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2011-09-23 06:50:20.000000000 -0500
Modify: 2012-10-29 11:46:33.000000000 -0500
Change: 2012-10-29 11:46:33.000000000 -0500

I can also cd into the directory and issue both the touch and mkdir commands successfully (or at least without errors. However, if I try to run ls in the directory it just hangs. At that point neither Ctrl+C nor Ctrl+Z have any effect and my system locks up. I have to close my ssh window and reconnect to do anything else.

I'm at a bit of a loss here, does anyone have any ideas what might be causing this? Or, does anyone know how I can try to fix it, without rebooting the machine (I've encountered this once before on the same directory on a different machine and it was corrected by rebooting) since this is a live machine, I can't actually reboot.

These are the filesystems which are mounted in that directory:

blob.XXXXX.com:/blend on /mnt/blend type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/new_log on /mnt/new_log type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/new_backup on /mnt/new_backup type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/vz on /mnt/vz type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
blob.XXXXX.com:/git on /mnt/git type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

Interestingly enough, the output of strace ls /mnt shows the mount directories in there.

Best Answer

One thing is, ls attempts to stat() every file within the directory (which includes, in this case, the root of every mounted file system), which neither of the commands you attempted successfuly do so. So the problem is likely with one of the mounted filesystems, and not with /mnt itself.

To know more, you need to figure out what filesystems you have mounted in /mnt, and try each of them individually (ls -d /mnt/foo) to see which one causes the problem.

Related Question