How to decompress a password protected rar file without the password

You can bypass the hash extraction altogether and use cRARk instead. cRARk is a freeware command-line RAR password cracking utility available for Windows, Mac and Linux. It is also designed to work with CUDA so you may want to take advantage of that if you have a powerful GPU.

One caveat is that it will take very long to crack an archive if you know nothing about the password, and it is > 6 characters in length. If you do know a small detail about the password, such as the approximate number of characters, it allows you to input that as a switch to dramatically shorten cracking time. Even though cRARk is one of the fastest RAR crackers out there using extremely optimized MMX & SSE code, this holds true for any brute force application. When cracking longer passwords, it will take a substantially longer amount of time due to the myriad of possibilities to try.

Here's a sample run of cracking the password 'john':

C:\>crark.exe -c -l4 -g4 Chap7.rar
     cRARk 3.2d (CUDA enabled) Freeware
    Copyright 1995-2001, 2006-09 by P. Semjanov,
         http://www.crark.net
     portions (c) 1993-2005 Eugene Roshal
 (c) PSW-soft Password Cracking Library PCL v. 2.0d by P. Semjanov


Testing archive Chap7.rar : version 2.9
Testing     Chap7.rtf
Choosing best crypto functions.................................................
Chosen: ASM  (Prescott/AMD), SSE2 (P4/Core 2) (-f1114)
Ticks per password expected = 40438280, theoretical = 27000000, CPU rate = 0.67

Processing line 56 of password definition file...
Testing 4-chars passwords ...
ckdk
Passwords tested = 42000 (time = 3:45.00, rate = 186 p/s)
elka
Passwords tested = 78000 (time = 6:58.99, rate = 186 p/s)
john - CRC OK
In hex (PCL style): \6A \6F \68 \6E
Passwords tested = 167844 (time = 15:02.38, rate = 186 p/s)
Total     tested = 167844, slow tests = 20914

Not too shabby ;)

Out of the box, no, you can not. Version 3 of the RAR file format (implemented first in WinRAR 2.9) encrypts the actual data itself, as well as the file headers (if requested) using AES-128 encryption. With just WinRAR, it is impossible to simply "remove" the password from an archive, since the data itself is encrypted with the password.

You could make a quick batchfile implementing a "remove password" feature, which could simply unrar the archive, and then re-compress the files without a password.

Technically, the data is compressed before being encrypted. This indicates that, given enough knowledge of the RAR file format itself, one could create a tool to AES-decrypt the datastream of the compressed files, and then save it into a new RAR archive. It should be noted, however, that this requires extensive knowledge of the file format itself.

Given the number of open-source tools that support password-protected RAR files (e.g. unar), one could learn how to do this by reading existing source code, and then using the decrypted, but still compressed, bitstream to generate a new RAR archive. However, this is far from a trivial task, as you would then have to rebuild the RAR header manually as well (or at least ensure the file format's compatibility).