Bash – No Bash Prompt after SSH login to rPI (Pi-Hole)

I have had an rPi serving up pi-hole DNS for some time now. The issue has existed for at this point years. I would like to be able to understand and potentially resolve without rebuild.

After an OS update I no longer get a prompt when I login over SSH on any user.

Authentication goes through, the prompt is blank, if I leave it there, it will eventually timeout. If I ctrl+c it will drop me to a basic bash shell.

Linux blackholedns 4.19.66+ #1253 Thu Aug 15 11:37:30 BST 2019 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jun 25 11:27:59 2021 from x.x.x.x

^ This is all I will see, unless I throw ctrl +c at the session.

like so:

Last login: Fri Jun 25 11:35:04 2021 from x.x.x.x
^C
-bash-4.4$ 

If I whoami, its the pi user:

-bash-4.4$ whoami
pi

If I su to pi, it then returns to the familiar prompt and I can go about my business.

-bash-4.4$ su pi
Password: 
pi@blackholedns:~ $ 

For the past 2 years. I was content with just manually bypassing the issue.

Now, I would like to setup syncing for a backup pi and in order to setup gravity sync. The replica pi, requires the ability to ssh into this pi. That presents the issue shown above. The session attempts to connect and eventually times out.

What have I tried to fix this issue? (none of which has worked)

• Set hosts entry for both systems in /etc/hosts on the troubled system.
• "UseDNS no" in sshd
• "GSSAPIAuthentication no" in sshd
• "QOS = IPQoS 0x00" in sshd
• Created a new user, ssh with this new user, has the same result. no prompt.
• Tried logging in via ssh keys instead of pw
• Attempted to restore .bashrc or .profile so I went and copied from /etc/skel the .bashrc and the .profile file back into /home/pi

Here is a debug of the connection with -vvv

OpenSSH_7.9p1 Raspbian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 10.10.10.5 is address
debug2: ssh_connect_direct
debug1: Connecting to 10.10.10.5 [10.10.10.5] port 22.
debug1: Connection established.
debug1: identity file /home/pi/.ssh/id_rsa type 0
debug1: identity file /home/pi/.ssh/id_rsa-cert type -1
debug1: identity file /home/pi/.ssh/id_dsa type -1
debug1: identity file /home/pi/.ssh/id_dsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa type -1
debug1: identity file /home/pi/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519 type -1
debug1: identity file /home/pi/.ssh/id_ed25519-cert type -1
debug1: identity file /home/pi/.ssh/id_xmss type -1
debug1: identity file /home/pi/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Raspbian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Raspbian-10+deb9u7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.10.10.5:22 as 'pi'
debug3: hostkeys_foreach: reading file "/home/pi/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/pi/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 10.10.10.5
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:3jpMhHMt9SbH1sv9XDxKm+f/ONgSf3BhpjNAHn41dqc
debug3: hostkeys_foreach: reading file "/home/pi/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/pi/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 10.10.10.5
debug1: Host '10.10.10.5' is known and matches the ECDSA host key.
debug1: Found key in /home/pi/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/pi/.ssh/id_rsa RSA SHA256:PtWlqIMMnTLjjxDnqoVlfumTiemrv93ksoT92Jp1Qj0
debug1: Will attempt key: /home/pi/.ssh/id_dsa 
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa 
debug1: Will attempt key: /home/pi/.ssh/id_ed25519 
debug1: Will attempt key: /home/pi/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/pi/.ssh/id_rsa RSA SHA256:PtWlqIMMnTLjjxDnqoVlfumTiemrv93ksoT92Jp1Qj0
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/pi/.ssh/id_rsa RSA SHA256:PtWlqIMMnTLjjxDnqoVlfumTiemrv93ksoT92Jp1Qj0
debug3: sign_and_send_pubkey: RSA SHA256:PtWlqIMMnTLjjxDnqoVlfumTiemrv93ksoT92Jp1Qj0
debug3: sign_and_send_pubkey: signing using ssh-rsa
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 10.10.10.5 ([10.10.10.5]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env NO_AT_BRIDGE
debug3: Ignored env PWD
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env HOME
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LS_COLORS
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env XDG_SESSION_CLASS
debug3: Ignored env TERM
debug3: Ignored env USER
debug3: Ignored env SHLVL
debug3: Ignored env GS_INSTALL
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env SSH_CLIENT
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env SSH_TTY
debug3: Ignored env TEXTDOMAIN
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Linux blackholedns 4.19.66+ #1253 Thu Aug 15 11:37:30 BST 2019 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jun 25 12:17:24 2021 from 10.10.10.40

Any guidance is greatly appreciated.

Updates: Check for references to files that no longer exist in the .profile or .bash_profile.
I do not see anything referenced such as awk, grep, sed in the file.

.profile

pi@blackholedns:~ $ cat .profile
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
set -x

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
        . "$HOME/.bashrc"
    fi
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

/home/pi:

pi@blackholedns:~ $ ls -la
total 72
drwxr-xr-x 6 pi   pi    4096 Jun 25 13:02 .
drwxr-xr-x 6 root root  4096 Jun 25 11:22 ..
-rw------- 1 pi   pi    9465 Jun 25 12:27 .bash_history
-rw-r--r-- 1 pi   pi     220 Mar 13  2018 .bash_logout
-rw-r--r-- 1 pi   pi    3523 Dec 12  2020 .bashrc
-rw-r--r-- 1 pi   pi    3523 Mar 13  2018 .bashrc.old
drwxr-xr-x 2 pi   pi    4096 May  2  2018 .nano
-rw-r--r-- 1 pi   pi     682 Jun 25 12:59 .profile
-rw-r--r-- 1 pi   pi     675 Mar 13  2018 .profile.old
-rw-r--r-- 1 root root 12288 Jun 25 12:51 .profile.swp
drwx------ 2 pi   pi    4096 Jan 21  2019 .rapid7_tmp
-rw-r--r-- 1 pi   pi       0 May  2  2018 .selected_editor
drwxrwxrwx 2 pi   pi    4096 Dec 18  2019 .splunk
drwx------ 2 pi   pi    4096 Dec 11  2020 .ssh
-rw-r--r-- 1 pi   pi     171 Oct 11  2019 .wget-hsts

The .bashrc referenced in the output was restored from /etc/skel back in Dec of 2020. The last time I attempted to fix this issue on my own.

Update: Set trace on .profile results with this:

    pi@blackholedns:~ $ bash .profile
    + '[' -n '4.4.12(1)-release' ']'
    + '[' -f /home/pi/.bashrc ']'
    + . /home/pi/.bashrc
    ++ case $- in
    ++ return
    + '[' -d /home/pi/bin ']'
    + PATH=/home/pi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
pi@blackholedns:~ $ 

I assume since I am back at a prompt, the .profile did not get hung up.

Update: Request for contents of /etc/profile

-bash-4.4$ cat /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).
export SPLUNK_HOME=/opt/splunkforwarder
source /etc/profile
echo $SPLUNK_HOME

if [ "`id -u`" -eq 0 ]; then
  PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
else
  PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games"
fi
export PATH

if [ "${PS1-}" ]; then
  if [ "${BASH-}" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi