By default, all project-related tasks are visible to all project members.
If you want tasks to be visible only to the staff members who are assigned or added as followers, you need to disable the "Allow all staff to see all tasks related to projects" option.
To do this, go to Setup → Settings → Tasks and set "Allow all staff to see all tasks related to projects" to NO.
Please note: if a staff member has the VIEW permission for tasks or is an admin, they will still be able to see all tasks regardless of this setting.