Centos – Install PostgreSQL client (psql) only on CentOS

To protect your data against physical access to the server, you need to move the encryption/decryption at least partially off the server. If your server can boot and mount its encrypted partitions without external help, then it's vulnerable.

If you only care about physical removal, involving a reboot of the server, then disk encryption might be sufficient, if the encryption key is stored off the server; for example, using network encryption (a key unlocking server somewhere else on the network, for example Tang and Clevis), or as grochmal suggested, a key entered manually at boot (but in the latter case your server won't boot unattended, which could be a significant inconvenience).

If you care about physical access in general, then you should perform the encryption and decryption using information stored on the clients, for example using pgcrypto, or even performing encryption and decryption solely on the clients, coded in your client application. In these cases though the encrypted data is no longer usable in queries, unless you implement homomorphic encryption. There are quite a few subtleties involved, for example with data representation (NULL values in particular), so you should probably talk to a security expert (without re-inventing the wheel...).