Bash – rm command in bash script does not work with variable

I am doing a bash script to create a back-up of my mysql databases.
Actually, I would like to delete the old ones.

So, I create my script with variable for maintability.

The whole script is actually working, the only one thing who does not work, is the delete of the old ones. I was thinking it would be the easiest part.

Anyway, here is my code, can someone telle me what's wrong ?
And the rm command returns me : rm: impossible de supprimer « /path/to/backup/files/*.gz »: Aucun fichier ou dossier de ce type Which means rm: impossible to delete « /path/to/backup/files/*.gz »: no files or directory of this type

But what is really strange, first is that I found the script in a tutorial

Two, that if I launch myself the "rm /path/to/backup/files/*.gz" in a shell command, this is working and deleting all the .gz files (as excpected)

#!/bin/bash

USER="***"
PASSWORD="****"
OUTPUT="/path/to/backup/files"

rm "$OUTPUT/*.gz"

databases=`mysql --user=$USER --password=$PASSWORD -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

for db in $databases; do
    if [[ "$db" != "information_schema" ]] && [[ "$db" != _* ]] && [[ "$db" != "performance_schema" ]] ; then
        echo "Dumping database: $db"
        mysqldump --force --opt --user=$USER --password=$PASSWORD --databases $db > $OUTPUT/`date +%Y%m%d`.$db.sql
        gzip $OUTPUT/`date +%Y%m%d`.$db.sql
    fi
done

Thank you,