Both, leaning more towards Closed, personally.

When I first started my rp, I invited a ton of friends and players I thought would be interested in my rp, and I also kept it open for new players to join as well. I found that out of all the people who were in the rp, those who tended to drop first and vanish without a trace were those whom I had never met before, whereas those who stuck with me are the players I invited. Even later on, when we did a recruitment drive for new players, it was open for all to apply as well. Those who became accepted either had to drop, generally didn't understand the plot or concept, or tended to be pretty horrible personality wise that led to conflicts. So there was little joy there. 

This isn't to say all the players whom I've invited over the course of the rp have stayed, however they tended to be the ones who've stuck in it the longest, contributed the most, and caused the least amount of drama when choosing to leave. Since my rp is long and plot heavy, I only accept new players on a personal invitational basis, simply because at the stage it's at now, it's so very easy for new players who I'm unfamiliar with to get it wrong, and possibly have a bad attitude with regards to that (and I don't have the patience to experience possible drama like that). 

The players who I go out of my way to invite now don't neccessarily have to understand the tone and concept of my rp when joining, or even have the best writing/rping skill, etc. I browse the forum and have invited people who I think look to have a reasonably decent personality about them, are people who I think will have fun in my rp, offer a lot to it, and are people who I think would make great friends with myself and the other players in the rp. It's always a risk, but a closed, invitational only rp is a lot less risky then a generally open rp. It's also a lot more limiting, and blocks you from meeting other possible new players, but with the stage my rp is at atm, and the fact that I have a solid group of players already, I don't really want to add an x factor I haven't properly researched that could end up ruining things.