In general, this is probably because one of, or a combination of the following reasons:
1. Outgoing mail server requires authentication
To avoid abuse by spammers, many outgoing mail servers require authentication for relaying messages across the the internet. You should check if the outgoing mail server you are using for this account requires authentication, and if so properly set it up in your account settings. These typically use the same credentials as the ones for checking your mail with the same provider.
2. Outgoing mail server only accepts connections from the same network
Many ISPs only allow connections from hosts which are connected through their own network. If sending mails through your ISP's outgoing mail server works fine from your home Wifi, but stops working over a cellular data connection, or an external Wifi hotspot, this probably is the case.
In this case you can use another outgoing mail server. If you have a Gmail or an iCloud account you can use those outgoing servers for sending mails.
3. Incorrect settings
Double-check the settings for your outgoing mail server as they are specified by your email provider, especially server name, encryption and the port to connect to. If everything looks ok, try to recreate the account. For AOL and other popular services, there are presets for creating a new account which will fill out all the correct server details.
The specific settings for AOL's outgoing mail server are specified here:
SMTP Outgoing Server Address: smtp.aol.com (Use port 587 for standard or 465 for SSL connections)
SMTP Username: YourUsername@aol.com (or @love.com, @games.com, etc.)
SMTP Password: password you use to login to Mail
For a secure connection, check the SSL option for IMAP/POP and TLS for
SMTP in your mail program.
(I apologize in advance for the wishy-washiness of this answer, but while I hope I'm on the right track, I do have some gaps in my knowledge.)
The SMTP HELO message, while something of a relic, is supposed to allow the client to identify itself to a server. It is still required for the protocol and is generally expected to at least be a syntactically valid hostname (which yours is not.) Your SMTP server appears to be complaining because it is indeed invalid.
Mail uses the name of your Mac, or the hostname assigned by the DHCP server on your network. DHCP servers don't always assign a hostname. OS X prefers a DHCP assigned hostname over using the computer's name as the hostname.
Based on this, the first place I'd look is my network configuration. I wonder, if you opened a terminal and typed hostname -f, what would you get? Possibly
$ hostname -f
mymac.mydomain\.co\.uk
If you do get this, the next step is to identify where it's coming from. A likely source could be a misconfigured DHCP server. You can try
$ sudo hostname mymac.mydomain.co.uk
to temporarily change the hostname for testing, or
$ sudo scutil --set HostName mymac.mydomain.co.uk
to set it permanently.
Best Answer
Settings → General → Reset → Reset Network Settings.