Preliminarily, I hope you didn't actually do what you showed. Except for the labels, all the data used in TLS key derivation is binary data that cannot accurately be given as argument to a shell echo command or any other command, except possibly the builtin version in zsh if you disable escapes, and certainly cannot be 'typed' (even cut&pasted) in a literal argument with singlequotes.
If you actually have the data in files, those can be used. Binary data can be read from, and written to, files -- at least on the platforms OpenSSL supports. If you don't have files (but do have pipes) you can work-around this by passing the data in hex, and except for -macopt hexkey: which already takes hex, using either a program like xxd -r, or printf with the hex formed into escapes, or echo with the hex formed into non-portable escapes, or hacks like GNU sed s///e, or a more general program awk or perl, to convert the hex to binary for input to openssl. And when necessary similar hacks to convert binary output to hex, but dgst -mac hmac can output in hex as long as you remove the blahblah=(sp) from the front.
More substantively, running the first HMAC output back through itself only gives you the 'A' blocks, which you then run through another layer of HMACs to get the actual output 'P_hash'. See https://crypto.stackexchange.com/questions/46549/tls-1-2-prf-with-hmac-sha256 which is effectively the same question using a test vector published at https://www.ietf.org/mail-archive/web/tls/current/msg03416.html except I answered in Java. Here is the OpenSSL commandline equivalent:
$ echo $key; echo $lbsd # start from hex
9bbe436ba940f017b17652849a71db35
74657374206c6162656c a0ba9f936cda311827a6f796ffd5198c
$ echo $lbsd | xxd -r -p >a0
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a0 -binary >a1
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a1 -binary >a2
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a2 -binary >a3
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a3 -binary >a4
$ cat a1 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k1
$ cat a2 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k2
$ cat a3 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k3
$ cat a4 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k4
$ cat k1 k2 k3 k4 | head -c100 | xxd
0000000: e3f2 29ba 727b e17b 8d12 2620 557c d453 ..).r{.{..& U|.S
0000010: c2aa b21d 07c3 d495 329b 52d4 e61e db5a ........2.R....Z
0000020: 6b30 1791 e90d 35c9 c9a4 6b4e 14ba f9af k0....5...kN....
0000030: 0fa0 22f7 077d ef17 abfd 3797 c056 4bab .."..}....7..VK.
0000040: 4fbc 9166 6e9d ef9b 97fc e34f 7967 89ba O..fn......Oyg..
0000050: a480 82d1 22ee 42c5 a72e 5a51 10ff f701 ....".B...ZQ....
0000060: 8734 7b66 .4{f
If you can get your code to replicate this, it should work for the actual TLS1.2 handshake also, adjusted for the correct lengths: 48 for the master secret, and depending on the ciphersuite for the working keys.
First, your mseed file (the label+seed value in the PRF) should be 77 bytes, not 69. You must have messed up the client and/or server nonces somehow.
Second, -hmac spre.key is badly wrong. It uses the actual characters s p r e . k e y i.e. the octets 73 70 72 65 2e 6b 65 79 as the HMAC key. You need to use the value of the decrypted premaster secret, which is the contents of your spre.key file. And because that is binary data that can include bytes which are special character codes like null, tab, dollarsign, quote, backslash, delete, etc. you can't safely pass it directly, as -hmac {key} or even -hmac '{key}'; instead you need to use -mac hmac -macopt hexkey:{hex key value} as I showed in the previous answer, except using the actual hex key value you showed in this Q namely 0303CB30...2CDC.
Third, as I showed in the previous answer, you concatenate the results of the second layer of HMACs i.e. in my notation k1,k2,... to form the output (which in that example was 100 octets):
$ cat k1 k2 k3 k4 | head -c100 | xxd
but as I went on to say:
... for the actual TLS1.2 handshake also, adjusted for the correct lengths: 48 for the master secret, and depending on the ciphersuite for the working keys.
For the first (premaster-to-master) derivation you need 48 octets, so you only need the first two chunks of 32 (a0->a1->a2 a1+a0->k1 a2+a0->k2) then concatenate k1+k2 and take the first 48 octets.
For the second (master-to-working) derivation the length you need depends on the ciphersuite that was negotiated. You said you are using RSA-with-AES256CBC-SHA which (in TLS1.2, or 1.1 but not 1.0) needs 40 octets of HMAC keys and 64 octets of encryption keys totalling 104 octets. For 104 ocets you need to compute 4 chunks of 32, concatenate k1+k2+k3+k4, and parcel it out to client-MAC, server-MAC, client-encryption, server-encryption in that order. See 6.3 of the RFC. Also note that the label is different, and for this derivation the seed is (label+)server_random+client_random, not (label+)client_random+server_random as in the first one.
Best Answer
I'm not entirely sure but I think the answer is no. The
opensslcommand line client is a heterogeneous collection of tools. The X.509 commands can be useful to manipulate certificates, but the cryptography commands are rarely useful for anything other than testing OpenSSL itself.If you need to do cryptographic calculations with common algorithms, I recommend the Python interactive command line with the Cryptodome library.
But to decrypt SSL connections, the easiest way is usually to use Wireshark. Tell Wireshark where to find the private key and it will decrypt a TLS connection that uses RSA encryption. For connections using ephemeral Diffie-Hellman, you can't decrypt the traffic with the key alone, you need additional information from either the client or the server.
Note that using the
TLS_RSA_WITH_AES_256_CBC_SHAciphersuite is a bad idea for several reasons: