dont set it every join, check if its set and if not set it.
also i would combine the mode $chan +v $nick into the first on join