I used to be able to start nginx on my AWS EC2, but now I get bind() to 0.0.0.0:3008 failed (13: Permission denied)
. This happens when nginx calls bind() in response to the configuration listen 3008 default_server
, in /etc/nginx/nginx.conf
.
Possible causes I have looked for are that AWS is blocking port 3008, that the port is in use or that the user running the service has insufficient privileges.
The answers to the following two related questions all come down to the second or third of those possibilities:
https://stackoverflow.com/questions/48478869/cannot-bind-to-some-ports-due-to-permission-denied
https://stackoverflow.com/questions/39586692/nginx-error-bind-to-0-0-0-080-failed-permission-denied
As for the first of the three possible causes, I checked whether AWS is blocking port 3008 by deleting all but one security group other than the default (which allows all traffic). The inbound rules for this security group allow incoming TCP traffic on port 3008:
HTTP TCP 80 0.0.0.0/0
HTTP TCP 80 ::/0
SSH TCP 22 0.0.0.0/0
SSH TCP 22 ::/0
Custom TCP Rule TCP 3000 - 3030 0.0.0.0/0
Custom TCP Rule TCP 3000 - 3030 ::/0
HTTPS TCP 443 0.0.0.0/0
HTTPS TCP 443 ::/0
All ICMP - IPv4 All N/A 0.0.0.0/0
All ICMP - IPv4 All N/A ::/0
Custom TCP Rule TCP 8080 - 8084 0.0.0.0/0
Custom TCP Rule TCP 8080 - 8084 ::/0
To check whether port 3008 is in use, I ran $ sudo netstat -anp | grep 3008
and there is no output.
To ensure that privileges are sufficient, I added User=root
and Group=root
to the service file, ran ls -l
on the service file to verify that its user and group are already root, and I ran systemctl start
with sudo. Details below, in the form of commands and output, should answer any routine questions about the above, and about the overall environment in which this problem occurs. Thanks in advance for any suggestions.
Operating system:
$ uname -a
Linux ip-172-31-40-184.ec2.internal 3.10.0-957.12.1.el7.x86_64 #1 SMP Wed Mar 20 11:34:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Attempting to start nginx:
$ sudo systemctl start nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
Checking status of nginx:
$ sudo systemctl status nginx
â— nginx.service - nginx - high performance web server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since mié 2019-09-04 13:59:24 UTC; 32s ago
Docs: http://nginx.org/en/docs/
Process: 24450 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)
sep 04 13:59:24 ip-172-31-40-184.ec2.internal systemd[1]: Failed to start nginx - high performance web server.
sep 04 13:59:24 ip-172-31-40-184.ec2.internal systemd[1]: nginx.service failed.
Errors logged by nginx (port 80 error was fixed by rebooting):
$ tail /var/log/nginx/error.log
2019/09/03 19:58:09 [emerg] 17319#17319: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 19:58:59 [emerg] 17381#17381: bind() to 0.0.0.0:80 failed (98: Address already in use)
2019/09/03 19:58:59 [emerg] 17381#17381: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 21:18:17 [alert] 7491#7491: unlink() "/var/run/nginx.pid" failed (2: No such file or directory)
2019/09/03 21:25:32 [emerg] 11207#11207: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 22:30:21 [emerg] 16333#16333: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 22:50:51 [emerg] 15980#15980: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:31:57 [emerg] 9819#9819: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:32:07 [emerg] 10095#10095: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:32:12 [emerg] 10264#10264: bind() to 0.0.0.0:3008 failed (13: Permission denied)
The nginx service file (with User and Group added by hand):
$ ls -l /usr/lib/systemd/system/nginx.service
-rw-r--r--. 1 root root 420 sep 3 22:50 /usr/lib/systemd/system/nginx.service
$ cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
User=root
Group=root
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
[Install]
WantedBy=multi-user.target
Configuration of nginx (including /etc/nginx/conf.d/default.conf, which specifies ):
$ cat /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
server {
listen 3008 default_server;
root /home/ec2-user/webapp/debug/build;
server_name search-demo.net;
index index.html index.htm;
location / {
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/search-demo.net-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/search-demo.net-0001/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
}
Other than the first part shown below, the included default nginx configuration is commented out:
$ cat /etc/nginx/conf.d/default.conf
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
...
}
Best Answer
To unlock port 3008:
The question could be marked as a duplicate. The way I searched was not bringing up solutions involving SELinux. After I started to suspect SELinux, a colleague found:
https://stackoverflow.com/questions/18480201/ubuntu-nginx-emerg-bind-to-0-0-0-080-failed-13-permission-denied#answer-39971725
Even after the port fix, SELinux was not done messing with nginx. Visitors would get a 403 (forbidden) error from nginx. To enable the nginx user to read the directory: