(I'm not certain of this answer -- it's a somewhat educated hunch, based on using iPods for 10 years and Apple products for 25, and looking at wikipedia. If anyone knows from actual experience, use that answer instead.)
In general, I would guess that you can downgrade an iPod to the version of the OS that it was released with, but not earlier. So, for example, an iPod Touch 2g could probably be downgraded to iPhone OS 2.1, but not 1.1 (which wouldn't really matter since there was no App Store in 1.x!). Each iPod Touch gen-N ships with the N.1 release of iOS, about a month after the corresponding iPhone ships with the N.0 release.
An iPod Touch 2g should run all versions of iOS for the iPod Touch that have an App Store, though it doesn't support all the features of iOS 4. The hardware differences between iPod models might be a bigger issue for testing, e.g., older iPods don't have a built-in microphone.
It is indeed more of an opinion question, but here's my 2ยข anyways. I've heard a lot of talk about it, and it's an important question. Before I continue, let me note, that I think after iOS 5, people will be even more likely to upgrade quickly because of OTA updates, and my guess is the absorption rate of iOS 6, for example, will be far greater then previous major releases. This said, chances are, by the time you release your app and iOS 6 comes out, iOS 5 will likely quickly take the back seat, possibly making iOS 4 even less popular (except for older devices).
One key area is the client base that you intend to go for. If, for example, you are building an app that would most appeal to kids in school, or other people that have a higher possibility of being given devices, it would be important to build for iOS 4. This is because my understanding is that many schools are picking up 1st gen iPad, and older iPod Touches, as well as older iPhones. These obviously will only run on iOS 4. I think that is the main thing that will keep iOS 4 on the pie chart - older models.
If you are building a game or something that may appeal more to people that are more technically savvy, it would probably be a much better choice to go with iOS 5. As for other "normal" apps for the general user, valuable resources such as ARC likely will make iOS 5 the better option. I haven't written an app (yet, hopefully), but from what I hear, ARC is hard to pass up, not to mention all the other APIs that you get.
Having talked to developers that still work with iOS 3 and 4, I hear that it is a royal pain, and you will have to expect to spend more time (and consequently money) in the developing stages of the app.
Personally, if I was to program an app, I'm confident I would do iOS 5. From what I hear, iOS 4 is left to people building apps for schools, for example, as well as apps that larger companies that have more cash to work with put out as they obviously want everyone to be able to use their app.
Here is another interesting thought while on those lines - instead of build for iOS 4, why not localize your application, and make it available in some other countries? This would be a way to reach a significant greater number of users.
Again, just my thoughts, but hopefully somewhat helpful.
Best Answer
I'm afraid not. You can keep running old versions of apps, but you can not download them. For example, Check the Weather was recently updated to require iOS 6. If you have an iOS 5 device you can no longer purchase it on the device, but I can keep running the version I have as long as I don't delete it.
I understand and share your frustration.