What file sharing protocol should I use with an older Mac

I'm not sure what to tell you about the actual problem but I can share some insight that might help you track it down. The iMac connecting to the lower case "flac" share but mounting it as "flac-1" tells me it's all ready connected to a volume named some form of flac. It could just be that it's all ready connected to the SMB NAS you mentioned.

Regardless I've seen an issue where software expects a certain path and instead of throwing an error about not finding the path it just tried to create it which then leaves a sort of null path that you have to manually remove. If you don't remove it then names like "flac-1" start appearing. Re-reading that it may not make sense but regardless the steps I've used to see what's what are firing up Terminal (found in /Applications/Utilities) and typing:

ls -la /Volumes

If you see more listed here than you expect or can find through normal finder navigation then you can remove them (Only do this if you're 100% sure it's a path that contains nothing you ever want to see again!) by typing:

rm -rf /Volumes/NULLPATH

where NULLPATH is what you wan to get rid of.

Along these lines if the mini is all ready showing a "FLAC" or some such in /Volumes then you're not goign to be able to share a folder by the same name. As in if the mini is connected to the SMB NAS then you're going to have to use a different share name.

Also you can turn ownership on, on your external HD by opening the get info window on the drive and checking the box at the bottom of the window to "Enable Ownership on this volume". I'm not sue it will in any way effect your issue but might be something to try.

regarding getting your SMB sharing connection from linux working...

Samba no more, mount.cifs needs extra options, "nounix,sec=ntlmssp"

Don't use the linux gui to connect, have bro open a terminal and try these commands

(and dig my ascii art!)

          =^..^=       `·.¸¸ ><((((º>.·´¯`·><((((º>   

amitsbrother@linux:~$ 
amitsbrother@linux:~$ sudo apt-get install cifs-utils
  ...
amitsbrother@linux:~$ mkdir /mnt/mavericks_smb
amitsbrother@linux:~$ mount.cifs //172.17.8.212/smb_share /mnt/mavericks_smb/ -o user=amitsbrother,password=trustno1,nounix,sec=ntlmssp
amitsbrother@linux:~$ 
amitsbrother@linux:~$ mkdir /mnt/mavericks_smb_dup
amitsbrother@linux:~$ mount -t cifs //172.17.8.212/smb_share /mnt/mavericks_smb-dup -o username=amitsbrother,password=trustno1,nounix,sec=ntlmssp

Once this is working, you can create a script for your brother to automatically mount when it is executed from the gui. Basically, the script is just the mount point creation, and the cifs connection to the smb server... so 2 or 3 lines including the shebang.

Make sharing work now with no passwords

To make it super simple, I'd enable web sharing on the Mac, and put the files you want to share to the Linux box in a folder in ~/Sites/a_folder/ . Then give your brother the address that it tells is your personal web sharing address in the Sharing Preferences pane when you enabled Web Sharing. Tell your brother to open a browser and put in that address; it will give him a directory listing as long as there is no index.html file in there. He can download files with his browser. This is one way sharing, from the Mac to the linux box, and will work fine as long as there are no files over 4GB (unless apache fixed that issue and didn't tell me about it). I believe directory listing is enabled by default on the Mac apache2 server.

To share in the other direction, from linux to Mac, you could do the same from the Linux box:

 sudo apt-get install apache2

You can enable directory listings on the Linux apache2 server with instructions here. Those instructions inadvertantly also cover how to get the apache2 server up and running. Then you need the ip address of the Linux box, and the relative location from the apache root to see the files in your Mac's browser.

This shouldn't take 5 minutes to set up 2 x 1-way sharing through browsers on both boxes, and relieves you from hacing to trouble-shoot the slightly more complex task of installing and configuring netatalk or running SMB sharing from the mac and getting the linux client to mount it, which isn't always a "it just works" situation, like running 2 apache2 servers is.