Build binary and dependencies without sudo

I want to install a piece of software (rtorrent) from source to my home folder. It depends on ncurses, which is not installed. I've installed ncurses to my home folder by using the PREFIX option during the configuration step, but this doesn't seem to work when I try to do the same with rtorrent, as it keeps telling me that it requires ncurses.

The last few lines of output during make:

checking if more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... no
checking for NcursesW wide-character library... no
checking for Ncurses library... no
checking for Curses library... no
configure: error: requires either NcursesW or Ncurses library

How can I get this to work?