The word “sin” as in “Let he who is without sin cast the first stone” is a translation of a Hebrew word, het, that means “to miss the mark.”
So I wonder if perhaps what Jesus had in mind was that whoever had the best aim should take the first shot. Because if we just let anyone lob rocks we could be here all day. It’s a stoning, folks, not a party. Let’s get it over with.