We study the Anderson orthogonality catastrophe (AOC) in finite conductors with diffusive disorder. The. disorder averaged logarithm of chi, the overlap between the ground states before and after adding a static impurity, is found to depend nonmonotonically on the disorder. In two dimensions (In chi(-1))proportional toln(2)N in the weak disorder limit, thus showing a stronger dependence on the number of electrons N than in the canonical AOC. A very broad tail of the distribution of chi, found numerically, is a signature of the importance of a few-level statistics at the Fermi energy.