Canceled SSHFS (fuse) mount – cannot remove directory

I tried to mount an SSHFS directory locally, failed, and now I cannot use my original directory anymore.

Following a tutorial, I just installed the sshfs package, executed sudo modprobe fuse, and then created a local directory remote where I would like to mount the remote directory.

I then ran sshfs -o idmap=user "myuser@<host>:/home/myuser" remote, as indicated in the tutorial, but my connection was slow so I Ctrl+C'd it.

Now, if I do ls on its parent directory, I obtain this entry (as an example, I left the next entry as well):

...
d?????????  ? ?      ?      ?    ?            remote
drwxr-xr-x  2 myuser myuser 4096 Mar 13 11:02 Videos
...

If I try cd remote, I get:

cd: transport endpoint is not connected: remote

Trying to remove the directory does not succeed:

rmdir: failed to remove ‘remote’: Device or resource busy

How can I get rid of this directory, preferably without having to reboot?