MacOS – Apple network share – how to restrict access for user or staff group

I have a Mac mini acting as a file share server with external hard drive as a 'shared drive'.

I have separate accounts for each user. Previously they were all in the same 'Team' group and could read and write every subfolder in the 'shared drive'.

However I have a new user that should only be able to access one subfolder, so I'd like to restrict permission for that user. However when I look at the info of the folder on the shared drive, there is no option to remove 'staff' group from permissions list.

If I create a folder on the system hard drive I can see the 'staff' group permission and can remove it. However on the external drive there is no option.

Edit: I figured out that I had 'Ignore ownership' enabled on the external drive which allowed any registered user to connect. I disabled this and re-enabled permissions on each subfolder which allowed me to finely control access.