Semiconductor quantum dots (QDs) have recently emerged as a leading platform to generate highly indistinguishable photons efficiently, and this work addresses the timely question of how good these solid-state sources can ultimately be. We establish the crucial role of lattice relaxation in these systems in giving rise to trade-offs between indistinguishability and efficiency. We analyse the two source architectures most commonly employed: a QD embedded in a waveguide and a QD coupled to an optical cavity. For waveguides, we demonstrate that the broadband Purcell effect results in a simple inverse relationship, in which indistinguishability and efficiency cannot be simultaneously increased. For cavities, the frequency selectivity of the Purcell enhancement results in a more subtle trade-off, in which indistinguishability and efficiency can be increased simultaneously, although not arbitrarily, which limits a source with near-unity indistinguishability (>99%) to an efficiency of approximately 96% for realistic parameters.