Klinefelter syndrome (KS), also known as 47, XXY, is characterized by a distinct set of physiological abnormalities, commonly including infertility. The molecular basis for Klinefelter-related infertility is still unclear, largely because of the cellular complexity of the testis and the intricate endocrine and paracrine signaling that regulates spermatogenesis. Here, we demonstrate an analysis framework for dissecting human testis pathology that uses comparative analysis of single-cell RNA-sequencing data from the biopsies of 12 human donors. By comparing donors from a range of ages and forms of infertility, we generate gene expression signatures that characterize normal testicular function and distinguish clinically distinct forms of male infertility. Unexpectedly, we identified a subpopulation of Sertoli cells within multiple individuals with KS that lack transcription from the XIST locus, and the consequence of this is increased X-linked gene expression compared to all other KS cell populations. By systematic assessment of known cell signaling pathways, we identify 72 pathways potentially active in testis, dozens of which appear upregulated in KS. Altogether our data support a model of pathogenic changes in interstitial cells cascading from loss of X inactivation in pubertal Sertoli cells and nominate dosage-sensitive factors secreted by Sertoli cells that may contribute to the process. Our findings demonstrate the value of comparative patient analysis in mapping genetic mechanisms of disease and identify an epigenetic phenomenon in KS Sertoli cells that may prove important for understanding causes of infertility and sex chromosome evolution.