Summary: A set of words is factorially balanced if the set of all the factors of its words is balanced. We prove that if all words of a factorially balanced set have a finite index, then this set is a subset of the set of factors of a Sturmian word. Moreover, characterizing the set of factors of a given length $n$ of a Sturmian word by the left special factor of length $n - 1$ of this Sturmian word, we provide an enumeration formula for the number of sets of words that correspond to some set of factors of length $n$ of a Sturmian word.