Mac – How to one re-create Apple’s defunct ‘Back To My Mac’ service with specific open-source tools

back-to-my-macbonjourNetworkremote desktopvpn

Question: how can one re-create Apple’s defunct ‘Back To My Mac’ service with specific open-source tools or, if necessary, with limited custom (non-commercial) software?

(Ideally, this would be done in a way that would preserve integration with Apple applications such as Finder or Terminal; that might, however, happen automatically as soon as a VPN allowing network discovery is enabled on a particular computer.)

Background: Apple discontinued the 'Back to My Mac' service in 2019 after twelve years of operation.

As I understand it, this used a combination of Wide-Area Bonjour, a VPN, and probably some sort of iCloud database/directory to automatically keep all of the Macs signed in under a particular AppleID on a VPN so that they could communicate and use built-in services such as file sharing (AFP/SMB), screen sharing, and SSH through NATs and with dynamic IP addresses. No network configuration was required, once one had signed into a computer with their AppleID and enabling the checkbox for 'Back to My Mac' in Settings.

The other computers appeared in the Finder (and in other applications, such as Terminal) just as they currently do if they were on the same network.

I believe the AppleTV (and possibly other devices) were also used to provide Wake-on-LAN functionality.

Conjecture: It is possible that Apple discontinued this for any number of these (or other) reasons:

  1. to minimize their engineering / maintenance costs
  2. to drive iCloud Drive storage revenue
  3. they had been licensing some of the underlying technology since the beginning, and wanted to avoid those costs
  4. a key component was found to be covered by someone else's patent, patent(s) they were unwilling or unable to license

It is of note, however, that they disabled the feature for all existing installations of older operating systems, instead of just making it an unsupported / depreciated feature.

Possible starting points:

  • if either of the last two reasons are true, does anyone know the patent numbers (or patent holders) and technologies that were licensed?

  • are there any open-source replacements for this functionality (beyond what Apple mentions in the above link)?

  • what technical references are available documenting Apple's implementation and what is still missing to create an open-source replacement?

  • do you know of any other online discussions of this topic that can be linked to this question?

Technologies used by some iteration of BTMM:

  • DNS-based Service Discovery (DNS-SD; RFC 6763, RFC 8553, RFC 8552) to announce host reachability information

  • Dynamic DNS update (RFC 2136) to refresh the DNS resource records (RRs) when a host detects network changes

  • DNS Long-lived Queries (LLQ / DNS-LLQ) to notify hosts immediately when the answers to their earlier DNS queries have changed (Wikipedia and IETF)

  • IPv6 Unique Local Address (ULA; RFC 4193) as the host identifier

  • NAT Port Mapping Protocol (PMP / NAT-PMP; RFC 6886) to assist NAT traversal

  • Kerberos (RFC 4120) for end-to-end authentication

  • IPsec (RFC 4301) to secure data communications between two end hosts

References:

Thank you!! (and thanks to jksoegaard for comments to the previous question)

Note to moderator(s): this is a edited restatement of a previous question, “What technical and/or patent information exists for Apple's discontinued 'Back to My Mac' (BTMM) service, and/or what are possible replacements”, that was recently auto-deleted; if you close this current question in the future, please consider doing so in a way that allows it to remain as a searchable reference for future users.

Best Answer

Your question is broad and contains numerous elements, therefore I'll need to enumerate my answer accordingly:

Starting point 1:

I cannot see how any of the involved technology could be patented, nor that any technology as such is licensed here. Apple could ofcourse be licensing an implementation, but they would be able to reimplement themselves. The BTTM system can be re-implemented in a variety of ways using various protocols and technologies and still end up with similar functionality - so patents is not a problem in recreating similar functionality.

Note also that modern versions of similar systems exist in the market place now. For example you could just install Tailscale and have similar access to all of your devices. As long as you're a single user and have less than 100 devices, it is free of charge.

Starting point 2:

There are open-source versions of almost everything included in Apple's BTTM system. You would however need to integrate those yourself and make it scalable. This would involve things like automatizing a CA, having a specialized DNS-server, etc. It is not rocket science, but it's not something you do in a Sunday afternoon either.

Note that there's a big difference between making a service like Apple's BTTM that is supposed to support thousands (if not millions) of users and even more devices - and making something that will just work for your own purposes. If you're just looking for some way to create a BTTM-like experience for yourself, it can be done much simpler than trying to replicate Apple's BTTM system.

Starting point 3:

For a technical reference you can just look up RFC6281. It holds information enough for a professional (or motivated hobbyist) to create a similar system. However if the goal is to make it interoperable with Apple's BTTM service (i.e. used with the clients included in older versions of macOS) - that is much harder (bordering on impossible) and not worth doing. Some of the issues here is that you wouldn't have access to Apple's secret keys, their me.com domain and the custom protocols for AppleID authentication.

Related Question