Perhaps have it where if the person is on the access list it will remove them when it kick/bans them.