Selection on the gametophyte can be a major force shaping plant genomes as 7-11% of genes are expressed only in that phase and 60% of genes are expressed in both the gametophytic and sporophytic phases. The efficacy of selection on gametophytic tissues is likely to be influenced by sexual selection acting on male and female functions of hermaphroditic plants. Moreover, the haploid nature of the gametophytic phase allows selection to be efficient in removing recessive deleterious mutations and fixing recessive beneficial mutations. To assess the importance of gametophytic selection, we compared the strength of purifying selection and extent of positive selection on gametophyte- and sporophyte-specific genes in the highly outcrossing plant Capsella grandiflora. We found that pollen-exclusive genes had a larger fraction of sites under strong purifying selection, a greater proportion of adaptive substitutions, and faster protein evolution compared with seedling-exclusive genes. In contrast, sperm cell-exclusive genes had a smaller fraction of sites under strong purifying selection, a lower proportion of adaptive substitutions, and slower protein evolution compared with seedling-exclusive genes. Observations of strong selection acting on pollen-expressed genes are likely explained by sexual selection resulting from pollen competition aided by the haploid nature of that tissue. The relaxation of selection in sperm might be due to the reduced influence of intrasexual competition, but reduced gene expression may also be playing an important role.