SQLite function to specify exactly how the data is stored internally

Does MySQL have any support for custom data types?

Simple answer: no

Is there anything remotely like this on any database engine? I mostly use MySQL, but I am curious if this has ever been implemented, short of making the application call a function like the INET_ATON function.

Oracle has CREATE TYPE which is analogous to some degree to a OO class, including features like member functions and inheritance

Postgres has CREATE TYPE which is a bit less like OO classes (no member functions or inheritance) but are incredibly flexible and useful, even allowing you to create new base types. There is also CREATE DOMAIN which allows a form of inheritance or sub-typing and basically extends a base type with some constraints. Postgres also has quite a few interesting base types by default, eg inet and geometric types. In Postgres one can write an extension in C for a custom datatype, such as in this example here with base36 data type.

SQL Server has CREATE TYPE which allows you to create a custom data type based on an existing system data type. For example I could create a type called SSN which is basically defined as VARCHAR(11) but this way I don't have to remember how big a field it is.

Maybe I'm missing something but why wouldn't you just use a lookup table with a foreign key relationship for this sort of thing potentially with some views to provide a layer of abstraction? That's a very common approach, it's supported by all databases, it's pretty easy to implement, and it solves other issues such as data consistency (i.e. what happens if you want to update an existing string without updating every row in the table that has that string). If you know the maximum number of entries in the lookup table, you can pick the data type that minimizes data volumes for the primary key. That's also much easier to handle in the future when someone decides that the number of distinct values needs to exceed whatever limit you specified initially-- you just need to increase the size of the column in both tables.