Gpg with gpg-agent never asks for passphrase

Without gpg-agent running, I can enter my passphrase to sign things:

$ echo 123 | gpg -s

You need a passphrase to unlock the secret key for
user: "Mr. Ops <ops@bxxx.com>"
2048-bit RSA key, ID 20F31903, created 2014-03-13

gpg: gpg-agent is not available in this session
Enter passphrase:

Works fine. But I don't want to type my passphrase all the time, so I run gpg-agent:

$  eval $(gpg-agent --daemon)

Now I would expect to be prompted for my passphrase at least once, but I never am, and all operations using gpg fail.

$ echo 123 | gpg -s

You need a passphrase to unlock the secret key for
user: "Mr. Ops <ops@bxxxx.com>"
2048-bit RSA key, ID 20F31903, created 2014-03-13

gpg: cancelled by user
gpg: no default secret key: bad passphrase
gpg: signing failed: bad passphrase

How do I get my passphrase stored in the agent? Once I have it there, how do I keep it there across login sessions? (Ideally I never want to be prompted again.) This is on Ubuntu 12.04.4 with the standard apt-get gpg packages, in case it matters.