Ntp using ipv6 address instead of ipv4

I have a laptop running a version of Linux ( 3.12 ). I have ntpd installed and configured to sync with time.google.com. The problem I am having is if I am plugged into a network that only provides an IPV4 address, ntp's dns is using the ipv6 address:

    root@Node00b01973d6cc:~# ntpq -pn                                                                                           
     remote           refid      st t when poll reach   delay   offset  jitter                                              
==============================================================================                                              
 127.127.1.0     .LOCL.          10 l    7   64    1    0.000    0.000   0.002                                              
 ff0e::101       .MCST.          16 M    -   64    0    0.000    0.000   0.002                                              
 2001:4860:4806: .INIT.          16 u    -   64    0    0.000    0.000   0.000  

Now, if I force ntpd to use only ipv4 DNS with the -4 option, everything works fine

root@Node00b01973d6cc:~# ntpq -pn 172.16.17.111                                                                             
     remote           refid      st t when poll reach   delay   offset  jitter                                              
==============================================================================                                              
 127.127.1.0     .LOCL.          10 l   60   64    1    0.000    0.000   0.002                                              
 216.239.35.12   .GOOG.           1 u   57   64    1   29.278  341.883   0.002     

But I don't want to do this because if I go to a network that only gives ipv6 addresses this will fail. Is there a way to configure ntp to only use ipv4 ( or ipv6 ) if my interface actually has an ip address in that family?

The version of ntpd is 4.2.8p12. This is the ntp.conf file:

tinker panic 0                                                              

driftfile /usr/local/etc/ntp.drift                                          
disable auth                                                                   

# Update the realtime clock and override its default                         
# stratum of 0.                                                                   
server time.google.com prefer #Real Time Clock                             
server 127.127.1.0 #Real Time Clock                           
server 127.127.1.0 #Real Time Clock                                         
fudge 127.127.1.0 stratum 10                                      
broadcast ff0e::101 iburst ttl 7              

I have figured out the root cause of why this started to happen. I recently disabled ipv6 forwarding on my system. Once I turn ipv6 forwarding back on, ntp starts using the ipv4 address as I would expect.

Why would turning on ipv6 forwarding do this?

By default the system boots with ipv6 forwarding disabled. To turn it on, in one of the boot scripts I have the line:

sysctl -w net.ipv6.conf.all.forwarding=1