1. you could use a firewall rule
2. you could require a password for your own clients on the server
3. you could give your server a different IP and a different DNS name and adjust the config of all your clients
4. you could deactivate the service of the ocs-inventory client via GPO if the computer is in an AD which can be managed by you