Short Answer: It varies too much on RP type and the Players to say.

Long Answer: Note these are generalizations and on average rules and observations being made.

Closed RP's usually only work if people already know each other. If you don't, there's just not enough communication, closeness to collaborate well or motivation to keep going to kepe most RP's alive. Restricting it to select people without allowing a stream of new people spells failure if you can't keep your current group involved, motivated and loyal. Plus, why the hell are you having closed if you're just adding random people you don't know to start with?

So with that being said.

Open RP's are generally the better way, specific people can still be invited but it also brings in new faces and talent you didn't know of before. And these new faces could end up becoming good friends that you'd want to invite to those closed RPs. It also makes it easier to add in new people if old ones leave for some reason.

Closed RP's generally I find only work to be a benefit if you're doing a long/continuous plot line with the same characters, mainly those that have their own separate RP sites dedicated to that group. But even then having new faces can help expand characters, plot lines etc. Hell my first 2 years of forum RP's were on such specialized sites, but in the end over half of our players were new people who wandered in over time. The only other case I can potentially see Closed RP's being a boon is honestly not even a boon but a preference. If you have a specific plot/story in mind and you want to specifically experience it with your friends, kind of like say if you were to set up a D&D group but you only want friends at the table rather than setting it up it strangers.