Fluvial bedrock erosion rates are thought to be proportional to the energy delivered to the channel bed by impacting bedload particles. Combining methods for field measurements of impact energy and for automatic sensing of grain size distributions in the instrumented channel of the Erlenbach, we show that the larger a bedload grain is, the more efficient it is in transferring energy to the stream bed. On average, the largest grain size class contributes only 9% to the total bedload mass, but is responsible for more than 40% of the energy delivery. This effect may help to explain the predominance of extreme floods, when large grains are mobile, in driving long-term erosion. While the mechanistic saltation-abrasion model accounts for hydraulic forcing of energy delivery, the observed grain size effect is not captured. We suggest that the discrepancy arises from differing transport modes (rolling, sliding, saltation) not currently captured in modeling.