Microsoft has an Article about hard to delete files.
Most of the time for me, prepending the path with \\?\ does exactly what I want.
So you could try
del "\\?\c:\path\Kanno, Yoko - Where Does This Ocean Go?.mp3"
There is also \\.\ and I don't know what the exact difference to \\?\ is.
Both prefixes seem to disable some checks and/or make del use lower level APIs than the usual Win32 API.
RecycleBinEx is a simple application for Windows that does exactly what you ask. See: http://www.fcleaner.com/recyclebinex
On Mac OSX, Hazel does the same thing (among the others): http://www.noodlesoft.com/
KDE Plasma ships this feature as default, so if you're running Kubuntu, Arch, Chackra Linux or any other distro with KDE, you already have this feature. Just look at Dolphin configuration window.
On Ubuntu Unity, Gnome or any other gnu/linux desktop environment providing a standard FreeDesktop.org Trash feature you can use AutoTrash to do this thing: http://www.logfish.net/pr/autotrash/
Similar behaviour can be accomplished also with trash-cli, that could be used also to send files to trash can right from the command line. See: https://github.com/andreafrancia/trash-cli
Most email apps out there also have this feature for their "trash can".
On Android there isn't any "trash can" by default (when you delete it, it's gone forever), but you can install apps like Dumpster to (somehow) get similar features: http://www.dumpsterapp.mobi/
As said above, I think that automatically removing old files from trash can is a great feature to make it more usable, since it reduces clutter (are those files you trashed 3 months ago still relevant to you? And ALL those old revisions of the same file?) and makes easier to find what you want to recover (this is the reason for having a "trash can" on our computers, after all), still being safe.
It's even more useful if you work a lot with text files (code or prose), that most of the time are small and don't need a lot of space (so may never reach your trash can quota). This way you won't even need to periodically "empty your trash can". You just know that you have a window of time for recovering your "trashed" files if you need to.
Looking at most cloud services out there (Dropbox, Google Drive, Simplenote, ...), most of them seem to have a similar policy for deleted files.
I really think it's the right thing to do with your files, and they seems to think so.
Best Answer
That is by design. If a file is created with the name of a just-deleted file, timestamps, attributes, and security are carried forward.
Reason: Plenty of apps delete and recreate on saving, as opposed to truncating the existing file and writing the new contents. This feature fixes the (for the user unexpected) behaviour that security settings and all that suddenly disappear.