I think that you should be able to remove people that are co-oping on your island and then leave because there are other people that might want to join and your island is full. I keep on getting this message too, "You can't invite this player" What does it mean?

 

Use /coop kick (username)

 

This is the co-op commands

/coop invite (player name)
/coop kick (player name)
/coop leave
/coop accept

This is the island command

/newisland
/newisland replace