You can forbid access to categories. Switch the category access type to "private" if you want to manage permissions.
You can set a category to private by editing a single category (Administration » Categories » Manage » edit) or by setting options to your whole category tree (Administration » Categories » Properties » Public / Private).
Once the category is private, you can manage permissions for groups and users with 3 screens: