The scrambling of quantum information in closed many-body systems, as measured by out-of-time-ordered correlation functions (OTOCs), has received considerable attention lately. Recently, a hydrodynamical description of OTOCs has emerged from considering random local circuits. Numerical work suggests that aspects of this description are universal to ergodic many-body systems, even without randomness; a conjectured explanation for this is that while the random circuits have noise built into them, deterministic quantum systems, much like classically chaotic ones, “generate their own noise” and look effectively random on sufficient length scales and timescales. In this paper, we extend this approach to systems with locally conserved quantities (e.g., energy). We do this by considering local random unitary circuits with a conserved U(1) charge and argue, with numerical and analytical evidence, that the presence of a conservation law slows relaxation in both time-ordered and out-of-time-ordered correlation functions; both can have a diffusively relaxing component or “hydrodynamic tail” at late times. We verify the presence of such tails also in a deterministic, periodically driven system. We show that for OTOCs, the combination of diffusive and ballistic components leads to a wave front with a specific asymmetric shape, decaying as a power law behind the front. These results also explain existing numerical investigations in non-noisy ergodic systems with energy conservation. Moreover, we consider OTOCs in Gibbs states, parametrized by a chemical potential μ, and apply perturbative arguments to show that for μ≫1 the ballistic front of information spreading can only develop at times exponentially large in μ—with the information traveling diffusively at earlier times. We also develop a new formalism for describing OTOCs and operator spreading, which allows us to interpret the saturation of OTOCs as a form of thermalization on the Hilbert space of operators.