availability-groupshigh-availabilitysql server
I have setup an AlwaysOn High Availability in Sql Server 2014.
my IP addresses are according to this :
192.168.1.189 -- Listener
192.168.1.180 -- Replica
192.168.1.181 -- Primary
Right now users can login to my Primary directly and I want to avoid this.
Is it possible ?
Always On is not synonymous with (Always On) Availability Groups and instead is more a marketing term to describe a suite of Microsoft High Availability technologies (which I argue is at best ambiguous). The term actually started out as "Always On" (note the space) a long time ago 
as a pure marketing term (see my graphic as an example), but with the release of SQL Server 2012 was adopted as a wider catch all replacement marketing name for what they called "Hadron" during development (which was specifically Availability Groups) and several other HADR technologies (such as SQL Failover Clustering). The only difference was that when the term was first used, it did not have the trailing space (i.e. "AlwaysOn"). Marketing in their infinite wisdom have now added a space back into the term to add even more confusion :).
The bottom line is that if you completely ignore the use of the term AlwaysOn/ Always On and explicitly refer to the technology Availability Groups/ Failover Clustering/ Database Mirroring etc (or ask people to clarify which technology they are referring to) then you cannot go far wrong.
It is not possible to delete a database that currently resides in an availability group. It is also not possible to restore over top of it while it remains in the availability group. Thus, this database was not in an availability group to begin with.
What you are describing is not possible as you will get an error if the database is involved in Always On. You'll receive the following error:
Msg 3752, Level 16, State 1, Line 1 The database '<Database name>' is currently joined to an availability group. Before you can drop the database, you need to remove it from the availability group. You have to SET HADR OFF before issuing a drop database.
Best Answer
There are a few questions that are similar to what you are asking, may be board-line duplicate but either way:
That second question listed is probably the duplicate of this one.
Overall if you are looking at ensuring logins can only access via the listener you are going to have problems trying to use a login trigger. I am not sure you can pick up DNS or IP address it used to actually connect to SQL Server. If it can then it would be a viable option.
I would opt to use your firewall or network infrastructure. A firewall can be much more efficient and easier to manage with blocking access to either the IP or access from an IP range to a specific IP.