Ssh – Specify Specific Identity file when ssh’ing as certain user in ~/.ssh/config

I would like to specify a specific identity file based on the user I am ssh'ing as to a server.

For example when ssh as user1 from host 1 to host 2 as user1

[user1@host1 ~]$ ssh user1@host2

I would like to use a certain identity file. However when I ssh as user1 from host1 to host2 as user2, I would like to use a different identity file

[user1@host1 ~]$ ssh user2@host2

Now, I can do this by specifying the identity file in the command,

[user1@host1 ~]$ ssh -i ~/.ssh/id_user1 user1@host2

[user1@host1 ~]$ ssh -i ~/.ssh/id_user2 user2@host2

but I would love to do it in my ~/.ssh/config file. I tried the following, but it does not seem to work

Host user2@*
    IdentityFile ~/.ssh/id_user2

Host user1@*
    IdentityFile ~/.ssh/id_user1

Any and all help is appreciated. If this has to be configured somewhere else, that is fine as well. I would just like to avoid specifying it on the command line. Would really love to figure this out as it would be a cool solution to my problem!