Infertility in men and women is a complex genetic trait with shared biological bases between the sexes. Here, we perform a series of rare variant analyses across 73,185 women and men to identify genes that contribute to primary gonadal dysfunction. We report CSMD1, a complement regulatory protein on chromosome 8p23, as a strong candidate locus in both sexes. We show that CSMD1 is enriched at the germ-cell/somatic-cell interface in both male and female gonads. Csmd1-knockout males show increased rates of infertility with significantly increased complement C3 protein deposition in the testes, accompanied by severe histological degeneration. Knockout females show significant reduction in ovarian quality and breeding success, as well as mammary branching impairment. Double knockout of Csmd1 and C3 causes non-additive reduction in breeding success, suggesting that CSMD1 and the complement pathway play an important role in the normal postnatal development of the gonads in both sexes.