The evolution of the mammalian jaw is one of the most important innovations in vertebrate history, and underpins the exceptional radiation and diversification of mammals over the last 220 million years1,2. In particular, the transformation of the mandible into a single tooth-bearing bone and the emergence of a novel jaw joint—while incorporating some of the ancestral jaw bones into the mammalian middle ear—is often cited as a classic example of the repurposing of morphological structures3,4. Although it is remarkably well-documented in the fossil record, the evolution of the mammalian jaw still poses the paradox of how the bones of the ancestral jaw joint could function both as a joint hinge for powerful load-bearing mastication and as a mandibular middle ear that was delicate enough for hearing. Here we use digital reconstructions, computational modelling and biomechanical analyses to demonstrate that the miniaturization of the early mammalian jaw was the primary driver for the transformation of the jaw joint. We show that there is no evidence for a concurrent reduction in jaw-joint stress and increase in bite force in key non-mammaliaform taxa in the cynodont–mammaliaform transition, as previously thought5,6,7,8. Although a shift in the recruitment of the jaw musculature occurred during the evolution of modern mammals, the optimization of mandibular function to increase bite force while reducing joint loads did not occur until after the emergence of the neomorphic mammalian jaw joint. This suggests that miniaturization provided a selective regime for the evolution of the mammalian jaw joint, followed by the integration of the postdentary bones into the mammalian middle ear.