Ubuntu – ntpd does not sync clock while ntpdate does

ntp

My issue is bit simmilar to those below, but still different. ntpdate syncs clock successfully, but ntpd does not get any data back after transmitting queries to same time server(s). My clock's drifft is 1.0-1.5s per hour!

ntpdate

$ ntpdate -qv 194.177.4.2
16 Sep 21:45:42 ntpdate[21836]: ntpdate 4.2.6p5@1.2349-o Thu Feb 11 18:30:41 UTC 2016 (1)
server 194.177.4.2, stratum 2, offset 17.656685, delay 0.06981
16 Sep 21:45:48 ntpdate[21836]: step time server 194.177.4.2 offset 17.656685 sec

and with more details

$ ntpdate -qd 212.33.77.42
16 Sep 21:48:32 ntpdate[21841]: ntpdate 4.2.6p5@1.2349-o Thu Feb 11 18:30:41 UTC 2016 (1)
Looking for host 212.33.77.42 and service ntp
host found : 212.33.77.42
transmit(212.33.77.42)
receive(212.33.77.42)
transmit(212.33.77.42)
receive(212.33.77.42)
transmit(212.33.77.42)
receive(212.33.77.42)
transmit(212.33.77.42)
receive(212.33.77.42)
server 212.33.77.42, port 123
stratum 2, precision -20, leap 00, trust 000
refid [212.33.77.42], delay 0.03363, dispersion 0.00159
transmitted 4, in filter 4
reference time:    db86ca25.1cf0982a  Fri, Sep 16 2016 21:44:37.113
originate timestamp: db86cb28.8dda4c1d  Fri, Sep 16 2016 21:48:56.554
transmit timestamp:  db86cb16.da7f48f5  Fri, Sep 16 2016 21:48:38.853
filter delay:  0.03363  0.03381  0.03365  0.03368 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 17.69400 17.69494 17.69572 17.69653
         0.000000 0.000000 0.000000 0.000000
delay 0.03363, dispersion 0.00159
offset 17.694009

16 Sep 21:48:38 ntpdate[21841]: step time server 212.33.77.42 offset 17.694009 sec

ntpd

$ sudo ntpd -d4L
ntpd 4.2.6p5@1.2349-o Thu Feb 11 18:30:40 UTC 2016 (1)
16 Sep 21:16:47 ntpd[21468]: proto: precision = 2.793 usec
event at 0 0.0.0.0 c01d 0d kern kernel time sync enabled
Finished Parsing!!
16 Sep 21:16:47 ntpd[21468]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
16 Sep 21:16:47 ntpd[21468]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
16 Sep 21:16:47 ntpd[21468]: Listen normally on 1 lo 127.0.0.1 UDP 123
restrict: op 1 addr 127.0.0.1 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: Listen normally on 2 eth0 xxx.116.4.142 UDP 123
restrict: op 1 addr xxx.116.4.142 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: Listen normally on 3 as0t0 172.27.224.1 UDP 123
restrict: op 1 addr 172.27.224.1 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: Listen normally on 4 as0t1 172.27.228.1 UDP 123
restrict: op 1 addr 172.27.228.1 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: Listen normally on 5 as0t2 172.27.232.1 UDP 123
restrict: op 1 addr 172.27.232.1 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: Listen normally on 6 as0t3 172.27.236.1 UDP 123
restrict: op 1 addr 172.27.236.1 mask 255.255.255.255 mflags 00003000 flags 00000001
16 Sep 21:16:47 ntpd[21468]: peers refreshed
16 Sep 21:16:47 ntpd[21468]: Listening on routing socket on fd #23 for interface updates
restrict: op 1 addr 0.0.0.0 mask 0.0.0.0 mflags 00000000 flags 000005d0
16 Sep 21:16:47 ntpd[21468]: restrict: error in address '::' on line 45. Ignoring...
restrict: op 1 addr 127.0.0.1 mask 255.255.255.255 mflags 00000000 flags 00000000
16 Sep 21:16:47 ntpd[21468]: restrict: error in address '::1' on line 49. Ignoring...
key_expire: at 0 associd 45622
peer_clear: at 0 next 1 associd 45622 refid INIT
event at 0 194.177.4.2 8011 81 mobilize assoc 45622
newpeer: xxx.116.4.142->194.177.4.2 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
key_expire: at 0 associd 45623
peer_clear: at 0 next 2 associd 45623 refid INIT
event at 0 212.33.77.42 8011 81 mobilize assoc 45623
newpeer: xxx.116.4.142->212.33.77.42 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
key_expire: at 0 associd 45624
peer_clear: at 0 next 3 associd 45624 refid INIT
event at 0 193.25.222.240 8011 81 mobilize assoc 45624
newpeer: xxx.116.4.142->193.25.222.240 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
key_expire: at 0 associd 45625
peer_clear: at 0 next 4 associd 45625 refid INIT
event at 0 192.86.14.67 8011 81 mobilize assoc 45625
newpeer: xxx.116.4.142->192.86.14.67 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
key_expire: at 0 associd 45626
peer_clear: at 0 next 5 associd 45626 refid INIT
event at 0 91.189.94.4 8011 81 mobilize assoc 45626
newpeer: xxx.116.4.142->91.189.94.4 mode 3 vers 4 poll 6 10 flags 0x1 0x1 ttl 0 key 00000000
event at 0 0.0.0.0 c016 06 restart
event at 0 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
event at 0 0.0.0.0 c011 01 freq_not_set
transmit: at 1 xxx.116.4.142->194.177.4.2 mode 3 len 48
auth_agekeys: at 1 keys 1 expired 0
transmit: at 2 xxx.116.4.142->212.33.77.42 mode 3 len 48
transmit: at 3 xxx.116.4.142->193.25.222.240 mode 3 len 48
transmit: at 4 xxx.116.4.142->192.86.14.67 mode 3 len 48
transmit: at 5 xxx.116.4.142->91.189.94.4 mode 3 len 48
[...]
^C16 Sep 21:17:37 ntpd[21468]: ntpd exiting on signal 2

I stopped it with Ctr+C. The observed errors seem not to be an issue as are ignored.

restrict: op 1 addr 0.0.0.0 mask 0.0.0.0 mflags 00000000 flags 000005d0
16 Sep 21:16:47 ntpd[21468]: restrict: error in address '::' on line 45. Ignoring...
restrict: op 1 addr 127.0.0.1 mask 255.255.255.255 mflags 00000000 flags 00000000
16 Sep 21:16:47 ntpd[21468]: restrict: error in address '::1' on line 49. Ignoring...

They origin from these lines of /etc/ntp.conf

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

After commenting them out errors disapear but server still do not receive any packages back. And while server is running, ntpq -np shows it is in INIT state (here with different servers pool).

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 94.154.96.7     .INIT.          16 u    -   64    0    0.000    0.000   0.000
 158.75.5.245    .INIT.          16 u    -   64    0    0.000    0.000   0.000
 193.219.28.147  .INIT.          16 u    -   64    0    0.000    0.000   0.000
 193.219.28.2    .INIT.          16 u    -   64    0    0.000    0.000   0.000
 91.189.89.198   .INIT.          16 u    -   64    0    0.000    0.000   0.000

I guess this proofs I have no firewall related issue. I also asked my ISP. They do not block anything and do not offer local time server.

cron

The obvious workaround I am currently using is hourly scheduled ntpdate. However, as explained by Tim Bielawa here, ntpd works by adjusting the length of a second on your system by a slight bit so that you slowly get the correct time, while ntpdate may adjust clock backwards, which might freak out some programs.

# m h  dom mon dow   command
@hourly /usr/sbin/ntpdate -u ntp.tp.pl

Update

nmap

executed on the server

$sudo nmap -p 123 -sU xxx.yyy.zzz.142
Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-19 14:30 CEST
Nmap scan report for example.com (xxx.yyy.zzz.142)
Host is up (0.00021s latency).
PORT    STATE SERVICE
123/udp open  ntp
Nmap done: 1 IP address (1 host up) scanned in 1.12 seconds

$ sudo nmap -p 123 -sU example.com
Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-19 14:27 CEST
Nmap scan report for example.com (127.0.1.1)
Host is up.
PORT    STATE         SERVICE
123/udp open|filtered ntp
Nmap done: 1 IP address (1 host up) scanned in 2.09 seconds

$ sudo nmap -p 123 -sU localhost
Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-19 14:26 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00018s latency).
Other addresses for localhost (not scanned): 127.0.0.1
rDNS record for 127.0.0.1: localhost.localdomain
PORT    STATE SERVICE
123/udp open  ntp
Nmap done: 1 IP address (1 host up) scanned in 1.08 seconds

Executed on a host in different country

$ sudo nmap -p 123 -sU xxx.yyy.zzz.142
Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-19 14:51 CEST
Nmap scan report for example.com (xxx.yyy.zzz.142)
Host is up (0.046s latency).
PORT    STATE         SERVICE
123/udp open|filtered ntp
Nmap done: 1 IP address (1 host up) scanned in 0.78 seconds

iptables

Although nmap reported 'STATE open|filtered' my iptables are cleaned up for the exercise.

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

tcpdump

Executed while below was running.

$ sudo ntpd -d
transmit: at 2 xxx.yyy.zzz.142->94.154.96.7 mode 3 len 48
transmit: at 3 xxx.yyy.zzz.142->194.177.4.2 mode 3 len 48
transmit: at 67 xxx.yyy.zzz.142->94.154.96.7 mode 3 len 48
transmit: at 70 xxx.yyy.zzz.142->194.177.4.2 mode 3 len 48

gave this result

$ sudo tcpdump udp port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:16:29.733037 IP example.com.ntp > 96-7.cpe.smnt.pl.ntp: NTPv4, Client, length 48
15:16:30.733095 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:17:34.733013 IP example.com.ntp > 96-7.cpe.smnt.pl.ntp: NTPv4, Client, length 48
15:17:37.733139 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48

Here is dump for ntpdate -q 194.177.4.2, which was successful.

15:31:16.790206 IP example.com.48658 > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:31:16.834517 IP pscolka.of.pl.ntp > example.com.48658: NTPv4, Server, length 48
15:31:18.790268 IP example.com.48658 > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:31:18.834546 IP pscolka.of.pl.ntp > example.com.48658: NTPv4, Server, length 48
15:31:20.790210 IP example.com.48658 > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:31:20.834336 IP pscolka.of.pl.ntp > example.com.48658: NTPv4, Server, length 48
15:31:22.790253 IP example.com.48658 > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:31:22.834572 IP pscolka.of.pl.ntp > example.com.48658: NTPv4, Server, length 48

And when I executed this

sudo ntpdate 194.177.4.2 
19 Sep 15:36:19 ntpdate[8123]: no server suitable for synchronization found

the dump was

15:36:11.856663 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:36:13.856654 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:36:15.856640 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48
15:36:17.856765 IP example.com.ntp > pscolka.of.pl.ntp: NTPv4, Client, length 48

Remark

While one of many tests $ sudo ntpd -d I could observe the below.

receive: at 1618 xxx.yyy.zzz.142<-178.39.91.211 mode 3 len 48
transmit: at 1618 xxx.yyy.zzz.142->178.39.91.211 mode 4 len 48
receive: at 1618 xxx.yyy.zzz.142<-178.39.91.211 mode 3 len 48

Server 178.39.91.211 is not in my configuration. I guess some other host asked my server 'what is the time?' Meaning, incoming communication on port 123 is possible? I do not have tcpdump log for above, but ntpd listens only on port 123.
I have dump for similar event, unfortunately without answer:

15:27:29.313748 IP 209.126.136.2.42440 > example.com.ntp: NTPv2, Reserved, length 12

I have impression, it is not possible to send out packets from my server on port 123. ISP asked again still denies they would block anything.

Update 2

Finally, my ISP confirmed that precedent ISP blocks source port 123 on my IP address. I followed below suggestion from Bill Thor and added NAT rule to my iptables which changes source port 123 to another one if destination port is also 123.

$ sudo iptables -t nat -A POSTROUTING -p udp -s xxx.yyy.zzz.142 --sport 123 --dport 123 -j SNAT --to-source xxx.yyy.zzz.142:12345

Now, my ntp server receives answers from other time servers.

$ sudo tcpdump udp port 123
14:31:29.875903 IP example.com.12345 > pscolka.of.pl.ntp: NTPv4, Client, length 48
14:31:29.921176 IP pscolka.of.pl.ntp > example.com.12345: NTPv4, Server, length 48
14:32:30.875809 IP example.com.12345 > ntp.task.gda.pl.ntp: NTPv4, Client, length 48
14:32:30.882699 IP ntp.task.gda.pl.ntp > example.com.12345: NTPv4, Server, length 48
14:32:33.875963 IP example.com.12345 > pscolka.of.pl.ntp: NTPv4, Client, length 48
14:32:33.920863 IP pscolka.of.pl.ntp > example.com.12345: NTPv4, Server, length 48

And my ntpreports as per below.

$ ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 2001:67c:1560:8 .STEP.          16 -    - 1024    0    0.000    0.000   0.000
*153.19.250.123  212.244.36.227   2 u   20   64  377    6.785   -0.791   9.129
 194.177.4.2     80.50.231.226    2 u   64   64    0    0.000    0.000   0.000

Best Answer

The fact all the servers are stuck with an .INIT. refid indicates you are NOT able to establish a connection with the server. Once you connect to a server this value changes to the preferred source for that server.

It does appear you can't reach the pool servers. Try adding a server with the IP address 194.177.4.2 to your ntp.conf file.

Since amplification attacks became an issue, I've found I have issues connecting to ntp servers over IPv4. I have an IPv6 tunnel that allows me to connect to pool servers over IPv6.

This configuration may work for you. I've omitted my key and stats configuration.

driftfile /var/lib/ntp/ntp.drift

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
pool 2.ubuntu.pool.ntp.org iburst
server  194.177.4.2.       maxpoll 17  iburst

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
restrict 10.0.0.0       mask 255.0.0.0       kod nomodify notrap notrust noserve limited
restrict 172.16.0.0     mask 255.31.0.0      kod nomodify notrap notrust
restrict 192.168.0.0    mask 255.255.0.0     kod nomodify notrap notrust
restrict 2001:470:c3c::0 mask ffff:ffff:ffff:: kod nomodify notrap notrust

# Needed for adding pool entries
restrict source notrap nomodify noquery

Some options will cause ntpdate to use an unprivileged port which may result in different behavior than if you are connecting from port 123. ntp will always use port 123. Your tcpdump indicate the servers are not replying to your requests when they originate from port 123. This will break amplification attacks.

I've been running into your issue on IPv4 for a while now. I verified traffic is not blocked when both ports are 123 with the command traceroute --sport=123 -p 123 94.154.96.7. I found two behaviors:

Servers that are reachable refuse to serve time; or
A network gateway blocks traffic when both ports are 123.

In either case, time is served if one of the source-port is not 123.

I found a work-around using a second IP I have on the host and ip-tables masquerading. I use shorewall to build my firewall so I added a masquerade rule:

#INTERFACE    SOURCE     ADDRESS                       PROTO   PORT(S) 
eth0               192.0.2.4  192.0.2.5:200-300:persistent  udp     123

This appears to result in the following rules.

-A POSTROUTING -o eth0 -j eth0_masq
-A eth0_masq -s 192.0.2.4 -p 17 --dport 123 -j SNAT --to-source 192.0.2.5:200-300 --persistent

I haven't fully tested this, but it is working for for shorter timeouts.