For AlwaysOn AGs, can a secondary replica cause contention on the primary when the resources of the secondary are maxed out

If the secondary is up and running, when the log block is flushed to disk (either because it is full or a commit), the record gets pushed to the log writer on the primary and to the log scanner (log reader) process on the primary simultaneously. Then the log scanner communicates with the secondary and the secondary then pulls the transaction from the log scanner on the primary to the secondary and processes the log record. The primary log writer doesn't push transactions across, it just communicates with the secondary, it only does that to see if it is up so that it knows it doesn't have to mark the replica as NOT SYNCRONIZED.

When the secondary is not up, then the log writer cant communicate with the secondary so it marks it as NOT SYNCHRONIZED and stores the records in the tran log on the primary. If you look at sys.databases.log_reuse_wait_desc column it should show AVAILABILITY_REPLICA which means the primary is hanging on to all the records.

Once the secondary is up, it will communicate with the primary to request a log scan, it then processes the transactions and communicates with the primary using progress messages to indicate the hardened LSN, presumably the primary is then adjusting its MinLSN, which in turn means the records prior to MinLSN will get deleted as checkpoints happen and hence VLFs will get truncated releasing space when you do a log backup.

But yes short answer is, if your secondary is down you need as big a log file as you need for as long as it is down. Once it is backup and synched at some time you may need to remove the db from the always on group to shrink the log if it is humungous and you dont want it that big.

How far will/can the secondary replica be behind?

That depends on a number of factors. Network speed between the 2 nodes, disk speed on the secondary, volume of data being transmitted. Many of these KPIs are in the AG Overview dashboard (Right-lick the AG in SSMS and select 'show dashboard'). You can select a slew of different metrics including log send queue, log redo queue, estimated recovery time, estimated data loss, etc. As stated in the comments below from scsimon, when in asynchronous, the secondary isn't every truly 'caught up'. It will always show a 'synchronizing..' status and not 'synchronized'.

If the secondary replica goes offline for any reason will it catch completely up once it comes online?

Yes. Also worth noting that during the time where your secondary replica is unavailable/disconnected, you will not be able to back up any transactions from the log that haven't been sent to the secondary. In in sys.databases the column log_reuse_Wait_desc will be populated with 'AVAILABILITY REPLICA' as transactions cannot be flushed out unless they are committed on the secondary.

How long can the secondary be down before it will not catch up?

The secondary won't just 'not catch up'. If you take the secondary down (without removing it from the AG), log backups won't flush transactions out, log files will fill up and then you'll run out of room, causing any future transactions to fail. The secondary server's amount of time that it can be down is largely dependent on how much space you have and what your transactional volume is. Once the secondary comes back up, it will begin applying transactions that occurred while it was down. I've seen this take from a few minutes to several hours.

For brief outages/patching, leaving the secondary connected is fine, but if you're looking at a long duration outage, it may be easier to just remove the secondary replica and not worry about the hassle of log files filling up. Then you just re-initialize the DBs into the AG once the outage completes.