Yes, there is at least one major pitfall when considering git to manage a home directory that is not a concern with subversion.
Git is both greedy and recursive by default.
Subversion will naively ignore anything it doesn't know about and it stops processing folders either up or down from your checkout when it reaches one that it doesn't know about (or that belongs to a different repository). Git, on the other hand, keeps recursing into all child directories making nested checkouts very complicated due to namespace issues. Since your home directory is likely also the place where you checkout and work on various other git repositories, having your home directory in git is almost certainly going to make your life an impossible mess.
As it turns out, this is the main reason people checkout their dotfiles into an isolated folder and then symlink into it. It keeps git out of the way when doing anything else in any child directory of your $HOME. While this is purely a matter of preference if checking your home into subversion, it becomes a matter of necessity if using git.
However, there is an alternate solution. Git allows for something called a "fake root" where all the repository machinery is hidden in an alternate folder that can be physically separated from the checkout working directory. The result is that the git toolkit won't get confused: it won't even SEE your repository, only the working copy. By setting a couple environment variables you can tip off git where to find the goods for those moments when you are managing your home directory. Without the environment variables set nobody is the wiser and your home looks like it's classic file-y self.
To make this trick flow a little smoother, there are some great tools out there. The vcs-home mailing list seems like the defacto place to start, and the about page has a convenient wrap up of howtos and people's experiences. Along the way are some nifty little tools like vcsh, mr. If you want to keep your home directory directly in git, vcsh is almost a must have tool. If you end up splitting your home directory into several repostories behind the scenes, combine vcsh with mr for quick and not very dirty way to manage it all at once.
The location of the X cookie file can be configured with the XAUTHORITY environment variable. The default is ~/.Xauthority.
Of course, the location that you pass to applications has to match the location where the cookie is stored. SLiM doesn't offer a way to add the cookie to a different file: it has ~/.Xauthority hard-coded. If you want to use a different file, patch SLiM or use a display manager that happens to have this configuration option. For example, Gdm stores X cookies under /var/run/gdm.
I think you can make .Xauthority a symbolic link, if you don't want the modifiable file to be in your home directory.
Making your home directory immutable is an exercise in futility. You're likely to encounter many other similar issues. The standard place for configuration files and state files is your home directory — that's where dot files get their name, because they start with a . so that ls won't list them by default.
Best Answer
You can just temporarily displace them.
That will find all files in your
$HOMEdirectory - without recursing into child directories - which have not been accessed for a year. It will update the access time for all of them to right now, and then move all of them into a directory named.trash. If you encounter any problems between the time you run it and whatever time you decide to start deleting old files in~/.trashthen you can try moving some of them back and see if any of those you put in the trash were the cause.