USB 3.1 Gen 2 (SuperSpeed+, 10 Gbps) was designed to work over both existing USB 3.0 cables (the ones with the 5 extra contacts), as well as USB Type C cables.
Since existing USB 3.0 cables (the ones with Type A and B connectors, as well as the micro A and B variants) only contain one super-speed pair-of-pairs (Tx pair and Rx pair), USB 3.1 Gen 2 could only use that one pair-of-pairs and still work over existing USB 3.0 cables. So even when you run USB 3.1 Gen 2 over a cable with Type C connectors, it only uses the one super-speed pair-of-pairs. This also makes it possible to have USB 3.0/3.1-capable cables with a Type C connector on one end, and the earlier USB 3.0-style Type A, B, micro A, or micro B connectors on the other end.
Now you might ask a follow-up question, "Why didn't the USB Implementers Forum (USB-IF, the USB standards consortium) define an even-faster-than-10Gbps flavor of the USB protocol, that uses both super-speed pairs-of-pairs in the Type C connector?" That's a valid question, but I'm unwilling to speculate. It would certainly have been a bigger departure from previous USB PHY designs, in that it would have two separate send and receive data streams that would have to be coordinated. In effect, it would be a kind of parallel interface whereas USB has traditionally been nominally serial.
The way you asked your question exposed a few potential misunderstandings that I'd like to address here:
I know that it is possible to run Thunderbolt 3 over any high-quality USB-C cable
That's not quite true. There are many high quality USB Type C compliant cables that are not suitable for Thunderbolt 3. Thunderbolt 3 is limited to ≤ 0.5m cable lengths if you have a passive cable. To go longer than that (like 2m), you need a more expensive active cable (a cable with special IC chips in it to assist in signal handling).
Why doesn't standard USB-C use both of the pairs
USB-C is not a protocol. USB Type C is the name of a connector and cabling specification; it's not the name of the protocols that are used over those connectors and cables. When doing USB protocols over Type C cables, you're doing USB 3.1 Gen 2 (10Gbps "SuperSpeed+"), or earlier flavors of USB.
Best Answer
TB is faster because it uses TWO pairs of Rx/Tx lanes, while USB 3.0/3.1 uses only one pair. USB 3.1 cables contain only one set of differential Rx/Tx pairs, while TB uses two pairs, and therefore is thicker and more expensive.
In Type-C connector, USB 3.1 uses only one set of Rx/Tx contacts, or another set, depending on which way a cable is plugged in. And a USB 3.1 host uses data multiplexer to switch to corresponding pin set. The TB uses BOTH sets of high-speed data pairs.
For the same (comparable generation) transceiver technology, 5Gb, 10Gb, whatever, the TB will be always ~2X faster, since its data path is 2x wider.