What reasons are there for applications in Ubuntu not being the most up-to-date, and why are some applications not available, in its repositories?
An Ubuntu release goes through several stages before it actually makes it to the public as a finished product:
Some time before Ubuntu launches a release it freezes its packages at a certain point.
Before a release is out but after the package freezing, work is done mostly to fix all the bugs and issues that there might be in those packages. New package versions are not imported into the repositories anymore after package or feature freezing.
Once the release happens additional changes to those packages only happen for bug fixing and security issues. There are no more upgrades done to the packages in the official repository even if new versions of the packages are released.
Some packages have an exception to this, such as web browsers (which need to be kept up to date all the time) or certain cases.
New version of packages are consistently being imported (from Debian) for the next release of Ubuntu, until the next freeze happens and the same process repeats itself.
As an example, you can have a look at the release schedule of 12.04.
You can see that even though 12.04 was released in April, in January 12 something called _Debian Import Freeze_ happened.
This is just the first of many freeze stages happening before the actual release and means that at that point the import of packages from Debian testing or unstable stops and work starts on them to customize and fix issues with them.
No upgrades are done after that point in a lot of packages and the version that package had at that point is the version present and maintained through a release's lifetime.
So even though there are higher versions of the same package in developers' PPAs or in Ubuntu+1 repositories those will only be included in the next release of Ubuntu.
This is done for stability, security and functionality. New bleeding packages being imported all the time to the main repository would mean issues and a lot more problems to be solved. A freeze in the packages version helps to sort that out and make Ubuntu safer and more stable for the end user.
A new version of Ubuntu is released every 6 months, so every 6 months new packages are prepared, tested, customized and released with a new version. Future versions of a packages can be installed in your system via a PPA or just by downloading it from a web site, but the version of the package in the official repository remains the same.
For more understanding and an interesting overview of what happened to Ubuntu from 10.04 until the launch of 12.04 have a look at ReleaseSchedule - LTS to LTS and Stable Release Updates page for a complete overview and explanation of an Ubuntu stable release.
main is supported by Canonical. Obviously they're not going to support every arbitrary piece of software on the internet. Software in
universe is inherited from Debian, where a select group of people, designated as Debian Developers, take responsibility for the maintenance of the packages they upload (even if the actual packaging is done by others). Obviously, they don't have the capability to maintain packages for every piece of software on the internet.
What do people do who can't get Canonical or Debian Developers to support their software? They must find other ways, which include creating their own repository. The introduction of the PPA by Ubuntu relieved a lot of pressure on community volunteers to maintain packages, and at the same time, freed developers from a lot of the constraints imposed by the release policies of distros.
Remember that what you want may not be what somebody else wants. By sticking to a stable set of software and allowing PPAs, while the users have a slight overhead of maintaining the list of repositories, they get a lot of flexibility in getting what they need. Developers get flexibility in providing updates. Maintainers are relieved of the pressure of having to keep up to date with every arbitrary piece of software on the internet.