Receive Connectors can be configured using Exchange Management Shell (ADD-ADPermission cmdlet) or using ADSIEdit.msc. Using the second option we can manage permission following these steps:
Expand Configuration, Services, Microsoft Exchange, Administrative Groups, Exchange Administrative Group (FYDIBOHF23SPDLT), Servers, <Server Name>, Protocols and finally SMTP Receive Connectors
In the right side we will see all Connectors available to the server and we can right-click on it and click on Properties
Here we go.. now we can configure specific user/groups permissions and also manage users in the Receive Connnector.