The most likely reason you are seeing packages with pacman -Qe that you don't remember installing is that they were part of a "group" (like base-devel, etc) that you installed.
Side Note: I have personally also been looking for a while to switch a package from "explicit" to "implicit" (and even vice-versa) without reinstalling it, it even taking a package I installed explicitly to get another package working and turn it into a dependency of that package (again without reinstalling).
You could use something like Kickstart, which is a feature of the anaconda installer of Fedora, CentOS, RHEL, and derived distributions that lets you completely customize an installation to your liking. You can choose which packages you want to install, the partition layout, network configuration, package repositories, root password, and much more. You can also create pre and post-installation scripts that do whatever else you might want. You can run the install fully automated, or pick and choose what you'd like it to prompt for. All settings are defined in one simple (not XML) text file, which you can make by hand or with the graphical system-config-kickstart tool.
Once you've got that put together, you can roll your own custom CD if you like, or set a network location in your kickstart file, which can be a local or Internet mirror, via FTP, HTTP, or NFS. Your kickstart file can also be on the network, so you can just burn one stub install disc to bootstrap the installation (or use vanilla install media) and have it use as many different configurations as you can stand. You can even network boot it, eliminating removable media altogether.
The network options or custom spun media eliminate you from needing the "huge package repository", yet it will still be there if you decide you want to add something. But, if you want to go whole hog, you can build your own repository, cherry-picking RPMS from upstream, customizing SRPMS to your liking, rolling your own, or even hacking a make && make install or tar -jxf into the post-install script.
Many distributions have similar functionality, some of which also understand Kickstart files even if they don't implement their entire feature set (like Debian).
Best Answer
I've used Debian, Gentoo and Arch for a couple of years each. The more customizable by far is Gentoo. But it takes thought each time you want a given package. Debian is, well Debian: a mainstream distro, that can feel bloated to some. Given your requirements, I think you might like Arch. It's pretty lightweight and there are tons of bleeding-edge packages.