Summary: Let $S= {a,b}$ be a two-letter alphabet and $s$ a sequence over $S$. An infinite matrix $(t_{i,j)i,j\geqslant 0}$ is associated with $s$ in the following way: $t_{i,j}=1$ if $s$ has a factor which contains $i$ times $a$ and $j$ times $b$; otherwise $t_{i,j}=0$. This matrix will be called the factor composition matrix (FCM) of the sequence $s$. In this paper, combinatorial properties of certain sequences are studied via their FCMs. In particular (i) the FCM of the Thue-Morse sequence is shown to be pentadiagonal, and substitutions whose fixed points have the same FCM as the Thue-Morse sequence are determined; (ii) an algorithm for computing the FCM of a Sturmian sequence is presented; (iii) the FCMs of fixed points of invertible substitutions are characterized in terms of their singular decompositions.