Thanks to Andreas Koll, who noticed that Google Plus gives you more control or privacy options over who can chat with you i.e. You can choose with which circle or circles you want to chat and rest can be disabled. This will keep the unwanted guests away for a while until you select them back
- Sign in to chat in Google Plus.
- Click on the Drop Down arrow right next to CHAT label.
- Click on Privacy Settings
- Select which circle you want to chat with. Good thing is you can have multiple circle enabled and not just one but you do not have select all option here which means if you really have a lot of circles, its gonna take sometime to deselect them all.
One thing you should understand that this will not filter out users, you can still see all of them and if you wish you can talk to anybody but if somebody who is out of circle cannot ping you up.
Now this is only possible when you are in Google Plus but if you are using Gtalk Client or Chat in Gmail, you will not see this which is restricted to contact list in your Gmail.