The characters included in the popovers are defined in property lists inside the /System/Library/Input\ Methods/PressAndHold.app/ bundle.
I was able to change à to y by editing /System/Library/Input\ Methods/PressAndHold.app/Contents/Resources/Keyboard-en.plist and logging out and back in.
(note the escape in "Input Method" if you're copy pasting)
<key>Roman-Accent-a</key>
<dict>
<key>Direction</key>
<string>right</string>
<key>Keycaps</key>
<string>a y á â ä æ ã å ā</string>
<key>Strings</key>
<string>a y á â ä æ ã å ā</string>
</dict>
What IPA layout do you use? SIL's website has one for Dvorak. You could edit other keyboard layouts with Ukelele or directly as XML.
You could also try adding something like this to private.xml in KeyRemap4MacBook.
<item>
<name>test</name>
<identifier>private.test</identifier>
<inputmode_not>ROMAN</inputmode_not>
<autogen>--KeyToKey-- KeyCode::Q, KeyCode::QUOTE</autogen>
<autogen>--KeyToKey-- KeyCode::W, KeyCode::COMMA</autogen>
<autogen>--KeyToKey-- KeyCode::E, KeyCode::DOT</autogen>
<autogen>--KeyToKey-- KeyCode::R, KeyCode::P</autogen>
<autogen>--KeyToKey-- KeyCode::T, KeyCode::Y</autogen>
<autogen>--KeyToKey-- KeyCode::Y, KeyCode::F</autogen>
<autogen>--KeyToKey-- KeyCode::U, KeyCode::G</autogen>
<autogen>--KeyToKey-- KeyCode::I, KeyCode::C</autogen>
<autogen>--KeyToKey-- KeyCode::O, KeyCode::R</autogen>
<autogen>--KeyToKey-- KeyCode::P, KeyCode::L</autogen>
<autogen>--KeyToKey-- KeyCode::BRACKET_LEFT, KeyCode::SLASH</autogen>
<autogen>--KeyToKey-- KeyCode::BRACKET_RIGHT, KeyCode::EQUAL</autogen>
<autogen>--KeyToKey-- KeyCode::S, KeyCode::O</autogen>
<autogen>--KeyToKey-- KeyCode::D, KeyCode::E</autogen>
<autogen>--KeyToKey-- KeyCode::F, KeyCode::U</autogen>
<autogen>--KeyToKey-- KeyCode::G, KeyCode::I</autogen>
<autogen>--KeyToKey-- KeyCode::H, KeyCode::D</autogen>
<autogen>--KeyToKey-- KeyCode::J, KeyCode::H</autogen>
<autogen>--KeyToKey-- KeyCode::K, KeyCode::T</autogen>
<autogen>--KeyToKey-- KeyCode::L, KeyCode::N</autogen>
<autogen>--KeyToKey-- KeyCode::SEMICOLON, KeyCode::S</autogen>
<autogen>--KeyToKey-- KeyCode::QUOTE, KeyCode::MINUS</autogen>
<autogen>--KeyToKey-- KeyCode::Z, KeyCode::SEMICOLON</autogen>
<autogen>--KeyToKey-- KeyCode::X, KeyCode::Q</autogen>
<autogen>--KeyToKey-- KeyCode::C, KeyCode::J</autogen>
<autogen>--KeyToKey-- KeyCode::V, KeyCode::K</autogen>
<autogen>--KeyToKey-- KeyCode::B, KeyCode::X</autogen>
<autogen>--KeyToKey-- KeyCode::N, KeyCode::B</autogen>
<autogen>--KeyToKey-- KeyCode::COMMA, KeyCode::W</autogen>
<autogen>--KeyToKey-- KeyCode::DOT, KeyCode::V</autogen>
<autogen>--KeyToKey-- KeyCode::SLASH, KeyCode::Z</autogen>
<autogen>--KeyToKey-- KeyCode::MINUS, KeyCode::BRACKET_LEFT</autogen>
<autogen>--KeyToKey-- KeyCode::EQUAL, KeyCode::BRACKET_RIGHT</autogen>
</item>
It's based on remap.qwerty2dvorak in the checkbox.xml in the test folder.
If someone else was looking for a way to change the input method when holding a key, see this question.
Best Answer
Option/alt + ñ on the Spanish ISO layout, followed by a or o gives you ã and õ.
To make a custom layout, use Ukelele
http://scripts.sil.org/ukelele
PS If you are running 10.7, you can also get those characters by holding down the a and o keys until the Character Picker popup menu appears.