On a Centos 7 test system via

$ sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm
$ sudo yum install dotnet-sdk-2.1

which results in dotnet-sdk-2.1-2.1.400-1.x86_64 being installed then with the test code

using System;
using System.Diagnostics;
using System.ComponentModel;
namespace myApp {
    class Program {
        static void Main(string[] args) {
            var process = new Process();
            process.EnableRaisingEvents = true; // to avoid [defunct] sh processes
            process.StartInfo.FileName = "/var/tmp/foo";
            process.StartInfo.Arguments = "";
            process.StartInfo.UseShellExecute = true;
            process.StartInfo.CreateNoWindow = true;
            process.Start();
            process.WaitForExit(10000);
            if (process.HasExited) {
                Console.WriteLine("Exit code: " + process.ExitCode);
            } else {
                Console.WriteLine("Child process still running after 10 seconds");
            }
        }
    }
}

and a shell script as /var/tmp/foo a strace stalls out and shows that /var/tmp/foo is run through xdg-open which on my system does...I'm not sure what, it seems a needless complication.

$ strace -o foo -f dotnet run
Child process still running after 10 seconds
^C
$ grep /var/tmp/foo foo
25907 execve("/usr/bin/xdg-open", ["/usr/bin/xdg-open", "/var/tmp/foo"], [/* 37 vars */] <unfinished ...>
...

a simpler solution is to simply exec a program that in turn can be a shell script that does what you want, which for .NET requires not using the shell:

            process.StartInfo.UseShellExecute = false;

with this set the strace shows that /var/tmp/foo is being run via a (much simpler) execve(2) call:

26268 stat("/var/tmp/foo", {st_mode=S_IFREG|0755, st_size=37, ...}) = 0
26268 access("/var/tmp/foo", X_OK)      = 0
26275 execve("/var/tmp/foo", ["/var/tmp/foo"], [/* 37 vars */] <unfinished ...>

and that .NET refuses to exit:

$ strace -o foo -f dotnet run
Child process still running after 10 seconds
^C^C^C^C^C^C^C^C

because foo replaces itself with something that ignores most signals (notably not USR2, or there is always KILL (but avoid using that!)):

$ cat /var/tmp/foo
#!/bin/sh
exec /var/tmp/stayin-alive
$ cat /var/tmp/stayin-alive
#!/usr/bin/perl
use Sys::Syslog;
for my $s (qw(HUP INT QUIT PIPE ALRM TERM CHLD USR1)) {
   $SIG{$s} = \&shandle;
}
openlog( 'stayin-alive', 'ndelay,pid', LOG_USER );
while (1) {
    syslog LOG_NOTICE, "oh oh oh oh oh stayin alive";
    sleep 7;
}
sub shandle {
    syslog LOG_NOTICE, "nice try - @_";
}

daemonize

With a process that disassociates itself from the parent and a shell script that runs a few commands (hopefully equivalent to your intended apt-get update; apt-get upgrade)

$ cat /var/tmp/a-few-things
#!/bin/sh
sleep 17 ; echo a >/var/tmp/output ; echo b >/var/tmp/output

we can modify the .NET program to run /var/tmp/solitary /var/tmp/a-few-things

            process.StartInfo.FileName = "/var/tmp/solitary";
            process.StartInfo.Arguments = "/var/tmp/a-few-things";
            process.StartInfo.UseShellExecute = false;

which when run causes the .NET program to exit fairly quickly

$ dotnet run
Exit code: 0

and, eventually, the /var/tmp/output file does contain two lines written by a process that was not killed when the .NET program when away.

You probably should save the output from the APT commands somewhere, and may also need something so that two (or more!) updates are not trying to be run at the same time, etc. This version does not stop for questions and ignores any TERM signals (INT may also need to be ignored).

#!/bin/sh
trap '' TERM
set -e
apt-get --yes update
apt-get --yes upgrade

...in shell (I am only interested to Bash) scripts I cannot be 100% sure that a PID I stored in a variable still refers to the background process I started, since it might be reused by the kernel for any other process...

Correct.

The way shells are programmed, as soon as a child process dies, the shell will call wait() on it right away (storing the termination status as part of its internal state), which will free the PID for reuse by another process.

is it true in shell scripts that "If you're still in the parent process that started the child process [...] You're guaranteed the PID is your child process (dead or alive)"?

No, it's not true.

Because, as mentioned earlier (and in the quote), the shell itself will reap children processes right away, which basically destroys that guarantee.

the naive way of starting in background a process in a script, storing its PID in a variable, do some stuff and then use the PID in combination with wait to get its exit code or with kill to send signals might fail due to reusage by the kernel of the PID.

Using a shell, that's the best you can do.

Note that using wait is not really a problem, only using kill, since it's possible your child process has already died, the PID has been reused and you're killing a different process.

wait itself is implemented in the shell. When it reaps children processes, it will store that termination status in memory, so it can implement its wait built-in using that information (as well as waiting for children processes which are still running.)

Also note that the kernel will typically try hard to avoid reusing PIDs, at least try to delay reusing a PID, exactly because in some cases there are no guarantees that the PID hasn't been reused, so the kernel tries to minimize this situation where a signal will be delivered to the wrong process.

Is there a general recipe?

For reliability?

Yes, implement the code launching a background process in C. Or Python, Perl, Ruby, etc. Not in shell.

Those will not have this problem, since they won't reap children by default, like the shell does, you'll have to do it explicitly there.

Or consider launching background processes using a system manager (such as systemd.)

"to have background (process) store the return code in a file and have parent fetch it from file". Is this the reliable way to go?

Maybe.

You have fewer guarantees that there hasn't been interference there. It's hard to have a location where only that single process can write and no others.

The same isn't true with the wait call, the kernel ensures it can't be faked by a different process.

Furthermore, the wait call can also tell you about the process being killed or even crashing, in which case you would probably get incomplete information if you depended on the process itself to record its return status in a file...

Also, the main issue with PID reuse is killing that PID, there's really no issue with getting the return code through wait, and the problem with kill isn't really addressed by using a file to store the return code.

Are there caveats about the usage of wait -n?

Not really. wait is reliable and AFAICT isn't affected by PID reuse, since when the shell reaps a child it will keep that information, including the PID that was being used and the return code, as part of its internal state.

When you call wait, you'll get information from that table.

I think there might be one potential issue if the PID is reused by a new background child of that same shell, before wait has been called on the first instance, since then there will be a clash in that table and you'll end up with two separate background processes with the same PID. That's a corner case and I imagine it's very very rare, but potentially real. Not really sure what the shell would do in those cases... It also probably depends on the implementation of the shell and might vary between versions.

As said, though, the real solution for this issue is to maintain the guarantees about PIDs being around, by using something other than a shell when those guarantees are important to you.

What is a direct child? Is it different from a child?

It's the same as a child.

It's a child you forked yourself.

For instance, if your child process forks a process and passes you the PID of that process, you no longer have the guarantee that it will stay around.

Since it's your child's job to reap that process, then it's your child who can have the guarantee that PID won't be reused until they've reaped it. Not yours.

Of course, a parent process could coordinate with a child to extend that guarantee, for instance by preventing it from reaping any children while it queries the child whether that PID is stilll the one it expects and then sending it a signal, or perhaps by asking the children (who has the guarantee) to send the signal on the parent's behalf.

Hopefully this will help clear it up.