Long ago, back in the OS 7 and earlier days, it was a snap to perform what was formally called a "Clean Install" of the OS. It would utterly remove the OS and place a brand-spanking new one in its place. Applications and data (other than system preferences and such) would be untouched, and you rarely even had to reinstall a single application. Adjust a few system prefs and you were good as new in no time at all.
I believe this feature went away around OS 8, and since then I've never known of a good way to perform a clean install. Migration Assistant is useless and ranting about it is unhelpful here.
I have a mid-2011 iMac, running "low" Sierra from an external SSD (Thunderbolt). I've got vestiges of Mavericks and all kinds of applications I no longer use floating around, and I feel that it's time to clean up. But every time I've done this in every flavor of OS X, it's ended up being an utter nightmare of lost application files (prefs, extensions, etc.), lost system settings, lost chat logs (Messages, Skype, etc.) and so much more. It's as bad to clean my Mac these days as it always had been on Windows!
My approach to date has been to install a fresh OS on a newly created partition of my drive, and go through countless reboots back & forth between the two systems while I copy applications and their preferences and settings over. Still it's inevitable that something will get lost or not be able to transfer over like mail rules, chat histories, or all manner of other stuff. Eventually I can delete the original partition once I'm confident in the migration (though there are often problems doing that, too!)
So is there a "best practice" or reliable way to completely replace my current macOS Sierra installation with a fresh copy, without losing any application data including preferences, mail & chat histories, app settings, etc.? I'm OK with having to reset basic system settings as I expect that, but losing anything related to my user-installed applications, is really quite unacceptable IMO.
Best Answer
The functionality you're looking for that is closest to the Clean Installation option in the old Mac OS 'System' days is the Reinstall macOS option available from Recovery mode.
See my A bit of history info at end on what the old Clean Installation option was and how it differs slightly from the Reinstall macOS option today.
Reinstalling macOS without deleting applications, data, etc
Prior to following this process, you need to ensure:
Assuming you have the above sorted, follow these steps to reinstall macOS without deleting your apps, data, etc.
Once completed, your iMac will restart as normal. This process does not erase your drive and therefore all your apps and data should still be in place. In other words, this process does nothing but reinstall macOS over the previous version on your drive.
A bit of history
In the old Mac OS System days users were presented with three options when installing system software:
The old clean installation option was actually designed as a troubleshooting option. This allowed users to discover which item(s) in their original System Folder was causing problems. A clean installation created a brand new System Folder and saved everything in the original System Folder to a different location. It also left user data (and user installed apps) alone. Users could then reinstall system extensions, control panels, etc one at a time from the old System Folder to the new System Folder. This procedure allowed users to determine which item in the old System Folder was the source of a problem.
In Apple's own words:
Source: Mac OS 7.1 Installation manual
Today a clean installation has a totally different meaning (even to Apple) to what it did back then. Today a clean installation involves erasing the Mac's startup drive and starting afresh with a totally new installation of macOS, meaning users have to rely on a backup and/or installers to transfer any data or software over. Today's Reinstall macOS option accessible via Recovery Mode is the closest option to replicating the old clean installation option.
Where this differs is that the old option actually kept everything in the original System Folder by copying it to another location so users could move system items (e.g. control panels, system extensions, etc) back one by one to try and determine what was causing problems. This was a troubleshooting process of elimination option.
Today's Reinstall macOS option is not designed so much for troubleshooting as it is simply to reinstall macOS over an existing installation without having to install/migrate apps, data etc.
However, if you're installing a new drive or migrating to a new Mac, then users will need to use Time Machine / Migration Assistant instead (or another backup option) in order to migrate apps and data.