Vim: use string from search in replace command

\0 is the whole match. To use only part of it you need to set it like this and use \1

.s/(\([0-9]*\))/{\1}/

More detailed instruction you can find here or in vim help.

I'm not sure why Perl isn't acceptable here. On the inputs you provided, this line gives the output you asked for:

perl -0777p -e 's/.* > (.*) joined the channel\.\n(((?!.* \1 (was kicked from channel\.|was banned from channel\.)\n).*\n)+?.*\1 disconnected)/\2/mg' irc.txt

The -e argument is exactly the first argument to your magicregextool except that I added the /mg regex modifier. This may not be "unmodified" but it doesn't seem unreasonable either. If you don't want to type in the whole line, how about this script as magicregextool:

#!/usr/bin/perl -0777p
BEGIN { $::arg = shift @ARGV; }
eval $arg;

Or even:

#!/bin/sh
perl -0777pe $*

Then you just type:

magicregextool 's/.* > (.*) joined the channel\.\n(((?!.* \1 (was kicked from channel\.|was banned from channel\.)\n).*\n)+?.*\1 disconnected)/\2/mg' irc.txt

Which is the same as your sample (again other than adding the /mg modifier).

An additional benefit to this is that if you are running multiple related search/replace operations on each file, you can put them together in the same script:

#!/usr/bin/perl -0777p
s/.* > (.*) joined the channel\.\n(((?!.* \1 (was kicked from channel\.|was banned from channel\.)\n).*\n)+?.*\1 disconnected)/\2/mg;
s/(some other\n)matched text/\1/mg;