Ubuntu – Install CUDA to different directory in 16.04

The below assumes you start with an Ubuntu system with current Nvidia drivers.

CUDA is just an application, but one needing a specific (frequently old) compiler version, and supplying its own (several versions obsolete) video drivers.
A CUDA release will have its own directories for:

• bin -- for executables
• lib64 -- for link libraries
• include -- compiler files

Adding the CUDA bin directory to the beginning of the PATH environment variable allows those programs to be accessed and to override the system default ones (as in the case of older versions for g++, gcc, ar, nm and ranlib).

Adding the CUDA lib64 directory to the beginning of the LD_LIBRARY_PATH will allow those (shared) libraries to be picked up in standard loads of a program.

The CUDA include directory is typically added as a variable in the makefile compiler flags, so those files may be picked up by the compiler. There is no need to sprinkle these CUDA files into the system bin, lib and include directories. Applications typically don't do that. Keeping the CUDA bin, lib64, and include files under a CUDA release directory allows multiple CUDA releases to be supported simultaneously. Just have the PATH and LD_LIBRARY_PATH modifications in a script associated for each release.

The one exception to program separation might be to install a system supplied older compiler -- gcc-8 and tools (ar-8, nm-8, ranlib-) are required by CUDA 10.2. Access the old compiler through links (gcc, ar, nm, ranlib) added to the CUDA bin directory. Avoid changing the system default to the old compiler (e.g. through links or update-alternatives).

Other parts of the system may break if forced to use an older compiler (think video drivers). A sufficiently old CUDA release may require an old compiler not supplied by the standard repositories (think gcc-5 for CUDA 8.0). You would need to provide your own copies of such a compiler, and any shared libraries it needs -- put into the CUDA release bin and lib64 directories.

Setting up CUDA release in such a way is easy, just unpack the supplied deb file into your selected location:

dpkg-deb --extract CUDAarchive YourDirectory

That used to be all that was needed, but CUDA releases like 10.2 have just made the contents of the supplied deb file for the release more deb files. Note the unwanted Nvidia deb files for older releases, ignore or delete them, and unpack all the individual deb files. You now have a CUDA release directory, so set up a script to alter your PATH and LD_LIBRARY_PATH (or if you don't care about multiple CUDA releases, just put these alterations into your .profile (assuming you use a bash type shell).

This approach keeps all the CUDA files outside the package manager, so no updates will be made -- which may be good or bad depending upon your viewpoint.

Unfortunately, there will be problems if you just run the "run file" script or install the supplied CUDA release deb.

1. Old Nvidia drivers will replace your current video drivers. Hopefully, you will still be able to login.
2. Reinstalling the current Nvidia drivers may cause the old ones AND ALL THEIR DEPENDENCIES (thats' all the CUDA files) to be deleted. Well, never prevent the package manager from doing its thing, but you can make a copy of your whole CUDA release, let the package manager delete it, and restore it.

All these workarounds with the package manager will finally result in a package-manager-free CUDA release like described above.