You need to set your HOME environment variable to whatever directory you selected for creating the .ssh folder (it doesn't have to be necessarily Users\username\).

%HOME% isn't a native environment variable (%HOMEDRIVE%%HOMEPATH% or %USERPROFILE% is), and the mingw Unix-like session of msysgit is entirely based on the $HOME value.

I have put together a step-by-step guide to get Git setup for windows using PuTTY's Plink application for SSH authentication.

Follow along below:

Setup

1. Install putty.zip which is available at the PuTTY Download Page or you can download individually.

   • PuTTY: putty.exe (or by FTP)

     The SSH and Telnet client itself.

   • Plink: plink.exe (or by FTP)

     A command-line interface to the PuTTY back ends.

   • Pageant: pageant.exe (or by FTP)

     An SSH authentication agent for PuTTY, PSCP, PSFTP, and Plink.

   • PuTTYgen: puttygen.exe (or by FTP)

     An RSA and DSA key generation utility.

2. Generate RSA and PPK Keys

   1. Using the Git Bash, use ssh-keygen to generate a pair of RSA public/private keys. More information on how to do this can be found on the official Generating SSH keys article.
   2. In PuTTYgen, import your existing ~/.ssh/id_rsa (private) key, via Conversions → Import key.
   3. Save the imported key via the Save private key button as ~/.ssh/id_rsa.ppk.

   4. You should now have the following keys in your ~/.ssh directory:

      • id_rsa: Private (OpenSSH) RSA key
      • id_rsa.pub: Public (OpenSSH) RSA key
      • id_rsa.ppk: Private (PuTTY) key

3. Install Git for Windows.

   Make sure that you choose to use Plink.

   

   Note: If you have already installed Git, you can just run the installer again and set Plink to be your default SSH application.

4. Set your Environment paths.

   1. In Control Panel, navigate to the System view.
   2. Choose Advanced system settings.
   3. In the System Properties window, click the Advanced tab.
   4. Click Environment variables….

   5. Add the following System variables (if not already set):

      • GIT_HOME: C:\Program Files\Git
      • GIT_SSH: C:\Program Files (x86)\PuTTY\plink.exe

   6. Append the Git binary directory to the system path.

      • Path: %Path%;%GIT_HOME%\bin

5. Open Pageant and load the ppk key located at ~/.ssh/id_rsa.ppk.

   Note: Once Pageant has started, you can click on its icon in the system tray located in the taskbar, next to the time, on the right.

6. Open Putty and connect to test your connection via SSH and add the server's key as a known host.

   

   Examples hostnames:

   • GitHub: git@github.com:22 (or via ssh-agent ssh -Tv git@github.com)
   • BitBucket: git@bitbucket.org:22 (or via ssh-agent ssh -Tv git@bitbucket.org)

7. Start Git Bash.

   You should be able to push and pull from your remote host without entering a password each time.

Shortcut

You can place a shortcut in your startup directory to auto-load your key each time you log into your Windows account.

Via Batch Script

This idea was inspired by an answer to this question:

Super User: How to make a shortcut from CMD?.

REM |==================================================================|
REM | Pageant Autoload.bat                                             |
REM |                                                                  |
REM | This script creates a shortcut for auto-loading a PPK (key) in   |
REM | Pageant by writing a temporary VB script and executing it. The   |
REM | following information below is added to the shortcut.            |
REM |                                                                  |
REM | Filename  : Pageant Autoload                                     |
REM | Target    : pageant.exe                                          |
REM | Arguments : id_rsa.ppk                                           |
REM | Start in  : ~/.ssh                                               |
REM |==================================================================|
@echo off

REM |==================================================================|
REM | Global Values - Do not touch these!                              |
REM |==================================================================|
SET VBSCRIPT="%TEMP%\%RANDOM%-%RANDOM%-%RANDOM%-%RANDOM%.vbs"
SET STARTUP_DIR=Microsoft\Windows\Start Menu\Programs\Startup
SET STARTUP_USER_DIR=%APPDATA%\%STARTUP_DIR%
SET STARTUP_ALL_USERS_DIR=%PROGRAMDATA%\%STARTUP_DIR% REM Alternative

REM |==================================================================|
REM | Shortcut Values - You can change these to whatever you want.     |
REM |==================================================================|
SET FILENAME=Pageant Autoload.lnk
SET TARGET=%PROGRAMFILES(x86)%\PuTTY\pageant.exe
SET ARGUMENTS=id_rsa.ppk
SET START_IN=%%USERPROFILE%%\.ssh
SET DESCRIPTION=Autoload PuTTY key with Pageant on startup (Ctrl+Alt+S)
SET HOTKEY=CTRL+ALT+S

REM |==================================================================|
REM | Write a new VB script, on the fly; execute and delete it.        |
REM |==================================================================|
ECHO Set oWS = WScript.CreateObject("WScript.Shell") >> %VBSCRIPT%
ECHO sLinkFile = "%STARTUP_USER_DIR%\%FILENAME%" >> %VBSCRIPT%
ECHO Set oLink = oWS.CreateShortcut(sLinkFile) >> %VBSCRIPT%
ECHO oLink.TargetPath = "%TARGET%" >> %VBSCRIPT%
ECHO oLink.Arguments = "%ARGUMENTS%" >> %VBSCRIPT%
ECHO oLink.WorkingDirectory = "%START_IN%" >> %VBSCRIPT%
ECHO oLink.Description = "%DESCRIPTION%"  >> %VBSCRIPT%
ECHO oLink.HotKey = "%HOTKEY%" >> %VBSCRIPT%
ECHO oLink.Save >> %VBSCRIPT%
CScript //Nologo %VBSCRIPT%
DEL %VBSCRIPT% /f /q

Via Windows Explorer

1. Navigate to the startup directory in Windows Explorer.

   • User Startup/ directory (preferred) is located at:

     %AppData%\Microsoft\Windows\Start Menu\Programs\Startup
     

   • All Users Startup/ directory is located at:

     %ProgramData%\Microsoft\Windows\Start Menu\Programs\Startup
     

2. Right-click inside the folder and select New → Shortcut

3. In the Create Shortcut dialog, enter the following information.

   • Location: "C:\Program Files (x86)\PuTTY\pageant.exe"
   • Name: Pageant Autoload

4. Right-click the new shortcut and choose Properties from the context menu.

5. Modify the following fields under the Shortcut tab:

   • Target: "%PROGRAMFILES(x86)%\PuTTY\pageant.exe" id_rsa.ppk
   • Start in: %USERPROFILE%\.ssh

    
   Notes:

   1. If you are using a 32-bit Windows OS, you should use the %PROGRAMFILES% environment variable instead of %PROGRAMFILES(x86)%.

   2. If you placed your shortcut in the All Users startup directory, make sure that the current user has an id_rsa.ppk key in their ~/.ssh directory or the key will not auto-load.

Closing Remarks

There you have it. Next time you log into your Windows profile, you will be greeted with a Pageant prompt to enter the password for your key. If you did not set a password on your key, then your key should be loaded automatically without a prompt.

If you are not sure if your key loaded view the current keys in Pageant by selecting View Keys from the context menu for Pageant in the system tray.