Ubuntu – Your Firefox profile cannot be loaded. It may be missing or inaccessible

I think you're better off figuring out why Firefox 21 isn't working for you. The problem might not be with Firefox 21, but with a corrupted profile that will affect all Ubuntu versions of Firefox.

That aside, you can load Firefox 20 from the lucid-security repo:

sudo apt-add-repository 'deb http://us.archive.ubuntu.com/ubuntu/ lucid-security main'
sudo apt-get update
sudo apt-get install firefox/lucid-security

On my system running 12.04 (precise), part of the resulting output shows:

dpkg: warning: downgrading firefox from 21.0+build2-0ubuntu0.12.04.3 to 20.0+build1-0ubuntu0.10.04.3.
(Reading database ... 332312 files and directories currently installed.)
Preparing to replace firefox 21.0+build2-0ubuntu0.12.04.3 (using .../firefox_20.0+build1-0ubuntu0.10.04.3_i386.deb) ...

This seems to a known, system-dependent bug. Have a look at the link added at end (your precise error message). Following another link on that page, only suggestions I saw were to disable AppArmor if you run it; or roll FF back: Bug Report.

If nothing works, maybe give Chrome a try.