OpenSSL converting chars to UTF-8 literals

My answer is essentially the same as in your other question on this topic:

$ iconv -f UTF-16LE -t UTF-8 myfile.txt | grep pattern

As in the other question, you might need line ending conversion as well, but the point is that you should convert the file to the local encoding so you can use native tools directly.

We have more than 700 certs... more than half of generated certificates has identical serial number... original index.txt contains only half of all certs (last half), not including previous.

I'm going to try to get you pointed in the right direction, but I probably cannot take it to the end because I did not setup a test rig and duplicate the problem. My apologies in advance.

If you want the higher level overview of issuing a certificate in a private PKI, then see How do you sign Certificate Signing Request with your Certification Authority? It explains how you would do things manually if Easy-RSA was not doing it for you.

Another important document is RFC 4158, Internet X.509 Public Key Infrastructure: Certification Path Building. This is __the__ document which explains what matches are, and how you can use tuples like {Issuer Distinguished Name,Serial Number} or {Subject Distinguished Name,Public Key Identifier} to compare two certificates for equivalency. OpenSSL uses this document for matching. Also see Section 3.5.15, "Endpoint Distinguished Name (DN) Matching" and Section 3.5.12, "Matching Key Identifiers (KIDs)".

The serial numbers are supposed to be unique. This is the problem to overcome. Subject Distinguished Names (DN) are a different story. If your openssl.cnf has unique_subject=yes, then they cannot be duplicated. If unique_subject=no, then the DNs can be duplicated.

I think you need to do a few things. First, use a modern or updated version of the OpenSSL utilities. Here, "modern" means one of the later 1.0.2's or 1.1.0's. Previous versions of the utility had subtle problems in matching names and serial numbers.

Second, examine your config file (normally openssl.cnf but you can use a different, perhaps copied, file with -config filename) and write down the relevant settings, like serial.txt and unique_subject=no. I believe these are the relevant ones from [CA_Default] from openssl.cnf:

base_dir       = .
certificate    = $base_dir/cacert.pem  # The CA certifcate
private_key    = $base_dir/cakey.pem   # The CA private key
new_certs_dir  = $base_dir             # Location for new certs after signing
database       = $base_dir/index.txt   # Database index file
serial         = $base_dir/serial.txt  # The current serial number
unique_subject = no                    # Allow reuse of subjects

Third, backup everything, especially the important stuff like index.txt and serial.txt.

Fourth, create a list of the certificates you want to revoke. The list might have entries like filenames - john-doe-vpn.pem. Move them into their own folder if you like. Preferably each should have a unique serial and they MUST all have the same Issuer name; the openssl ca and ocsp functions can't handle more than one Issuer at a time, although for OCSP the protocol could.

Fifth, create a new index.txt containing a line for each serial. One approach is to extract the subject, serial and expiration from each cert file as in the script you posted, although you can fold most of the shell work into one openssl and one awk per cert:

for f in *files*; do 
  openssl x509 -noout -enddate -serial -subject -in $f \
  | awk 'BEGIN{FS="=";OFS="\t"} /^serial/{num=$2} /^subject/{sub=$2} 
      /^notAfter/{split($2,a,/ /);mon=index(months,a[1])/3+1;day=a[2]...exp=sprintf(...)}
      END{print "V",exp,"",num,sub}' >>index.txt
done

If it's difficult to (reliably) remove duplicate serials in advance, you can put everything in and then discard duplicates with awk -F'\t' '!already[$4]++' or sort -t$'\t' -k4,4 -u or similar.

Another approach available in 1.0.2 up, but only documented in 1.1.0, is to use openssl ca [-config conffile] -valid certfile to do this extraction automatically. But -valid unnecessarily loads the CA privatekey each time so if your privatekey is password-encrypted, as is good practice, this will mean typing your password over and over; to save time temporarily replace the real CA key and cert with a scratch unencrypted key and matching but otherwise bogus (probably selfsigned) cert. -valid won't write a duplicate serial entry, so you needn't worry about excluding or removing them.

Put in the serial file a value which is at least the highest value of any previously issued cert; if you want to jump up to the next 10000 or 1000000 or whatever to be safe and perhaps also more clear, that's fine. You may need to set unique_subject=no at this point.

Sixth, mark each certificate (serial) in the index file as revoked. You can loop through the cert files using openssl ca -revoke on each, but it's easier to just use awk like:

awk -F'\t' -vOFS='\t' '{$1="R"; $3="161101000000Z"}' <index.txt >temp && mv temp index.txt
# if you want, you can add a comma and a reason, see man ca or 
# online at https://www.openssl.org/docs/manmaster/man1/ca.html
# under -crl_reason. But there isn't a code for 'CA stupid', and 
# in practice the reason doesn't really matter to reliers except 
# you should't use hold or remove (latter noted in the man) 

Seventh, generate a CRL from this index with openssl ca -gencrl [-crldays n] [-out file] and/or set up an OCSP responder using it if (any of) the old certs specified the OCSP extension.

Eighth, once you distribute the CRL and/or start running the (new) OCSP responder, all certificates with the affected serials are revoked and will cause communication to fail if used (and properly checked). If any of the duplicated serials are in certificates that your systems are still using, they must be replaced first. If you still have the request files (CSRs) from the systems using the affected certs, you can just re-issue with openssl ca [-config conffile] [-in reqfile | -infiles reqfile...] and send the new certs to the subject systems and have the operators of those systems install them. Otherwise you need to first have the operator of each system send you a CSR, which can be one they previously used (and saved) or a new one they generate.

Finally, restore any 'good' entries (serials you didn't revoke) from the old index file, combining with any new entries for replacement certs issued in #8 just above. If you're running an OCSP responder (see above) you must also keep the revoked entries; it not it doesn't matter but is probably easier. Do not restore the old value to serial if that is lower than the highest old cert or the highest new replacement cert, instead let it continue to increment from the new value.

Regarding the for-loop and printing dates:

hour=substr($3,1,2) ;
minutes=substr($3,4,2);
seconds=substr($3,7,2);
printf "%02d%02d%02d%02d%02d%02dZ", year, month, day, hour, minutes, seconds}'`

Don't even worry about the dates. If they are expired they cannot be used if your PKI is functioning properly. If it makes you feel better, then delete the private key associated with certificate and public key.

All you care about is the list of certs to revoke, and their serial number and distinguished name. Here, your list would be composed of certs like (1) exiting employee who holds a non-expired certificate and private key (i.e., employee is retiring or terminated); (2) an employee who lost a device (i.e., the private key is in the wild); etc.

Another option you have... Burn the existing PKI to the ground and start over. In this case, step (1) is revoke the Root CA and all Intermediate/Subordinate CAs. Then, throw away the private key. Step (2) is create a new Root CA, issue new Intermediate/Subordinate CAs, and finally issue new end-entity certificates. For step (2), you can even do the signing party dance.

Believe it or not, OpenStack uses this strategy (or was looking into using it). Its sort of an "ephemeral PKI" that's meant to stick around long enough to meet your needs; and then be discarded when things go wrong.

For a laugh, you might want to check out Peter Gutmann's Engineering Security. He is ruthless when it comes to PKIs and Public CAs.