no... in practice all contributors could (but should not) update anyones objects/patches etc
BUT entering a contributor prefix tells the UI not to commit changes outside your area.
(github does not have any fine level permission, well not at least on public/non org repos)
I repeat though, you should not update others objects/patches.
the reason is very simple...
its possible you could create a conflict, this would put others users local repos into conflict, which we provide no tools to resolve, and we cannot expect users to have git experience or tools to resolve. (unlike in a true dev environment)
so this could be a support nightmare (when we get to a position where all users are using the libraries, not just in testing)
the only resolution for them would be to init() the community library again, but that could mean they loose changes.
SOOO... I dont want to go down this route.
Now... I do know, that if you are careful this won't happen... (and i do it occasionally) but we really cant take this risk, as I cant know that any particular user knows what they are doing. so in practice i take a very dim view of it, and they do cause issue then I will get their access revoked 
(note: if I do it e.g. file renaming/deleting/editing - I do it in conjunction with the contributor so that I know they are not updating, hence I know there will not be a chance of conflict)
