Each photo has a privacy level (a lock). Each user also has a privacy level (a key). There are 5 privacy levels:
Higher levels have permissions on lower levels. For a given user, the higher the privacy level, the more photos he can see.
For example, if photo "peter_wedding-0024.jpg" has "Family" privacy level, then:
A user that has no permission to see the content of a category won't see the category itself, not even its title. The same principle applies to a tag.
You can set the privacy level of a user on Administration » Users » Manage screen.
If privacy levels do not fit your needs, you can also manage permissions on categories for a user or a group. You can manage permissions on photos and categories at the same time, without conflict.
You can forbid access to categories. Switch the category access type to "private" if you want to manage permissions.
You can set a category to private by editing a single category (Administration » Categories » Manage » edit) or by setting options to your whole category tree (Administration » Categories » Properties » Public / Private).
Once the category is private, you can manage permissions for groups and users with 3 screens: