Umount failed: device busy

I have a folder in /tmp that is mounted as ramfs.
After some action that my script does, I delete everything inside said folder with the command:

rm -R -f "$tmp_dir"/{*,.*}

Then, I try to unmount the directory, but on the first try it doesn't work because the device is busy.
After sleeping for a 0.5sec, the unmount succeeds.

I've verified that no process is using the folder or anything inside that folder with any of the following commands:

fuser -m "$tmp_dir"
fuser "$tmp_dir"
lsof +d "$tmp_dir"
lsof "$tmp_dir"

Why would the device be busy in the 1st try?

Edit #1 (30 Sep, 18:32 UTC):
When I execute find "$tmp_dir" -delete, the unmount succeeds on the 1st time!
But then the find command complains about $tmp_dir being busy.

Edit #2 (30 Sep, 18:45 UTC):
With stat I noticed a change in the size of the folder, before an after the success in the unmount:

$ stat '/tmp/tmp.nbljlVcmix'
  File: `/tmp/tmp.nbljlVcmix'
  Size: 0           Blocks: 0          IO Block: 4096   directory
Device: 17h/23d Inode: 121188      Links: 2
Access: (0700/drwx------)  Uid: ( 1000/     dor)   Gid: (    0/    root)
Access: 2013-09-30 20:37:51.430769893 +0300
Modify: 2013-09-30 20:37:51.430769893 +0300
Change: 2013-09-30 20:37:51.430769893 +0300
$ umount '/tmp/tmp.nbljlVcmix'
umount: /tmp/tmp.nbljlVcmix: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
$ sleep 0.5
$ umount '/tmp/tmp.nbljlVcmix'
$ stat '/tmp/tmp.nbljlVcmix'
  File: `/tmp/tmp.nbljlVcmix'
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 806h/2054d  Inode: 2401825     Links: 2
Access: (0700/drwx------)  Uid: ( 1000/     dor)   Gid: (    0/    root)
Access: 2013-09-30 20:37:47.600513531 +0300
Modify: 2013-09-30 20:37:47.600513531 +0300
Change: 2013-09-30 20:37:47.610513892 +0300

Edit #3 (1 Oct, 11:04 UTC):
I've copied all the code (single file) to: http://pastebin.com/RJP6eQiy (Valid for 1 Month)

The relevant umount is in the cleanup procedure, line #346, that is umount "$DEST_DIR".