Scaling whole exome data processing using workflows on the cloud

Whole exome / genome sequencing (WES/WGS) is poised to become a cornerstone of genetic testing for diagnosis in clinical practice, at population scale. The Cloud-e-Genome project, started in late 2013, addresses three architectural requirements in support of WES-based diagnosis, namely (i) scalability of the storage and computing resources required to extract variants from sequences, (ii) flexibility in the design and evolution of WES processing pipelines, and (iii) reproducibility of the results. Our approach involves using a scientific workflow model to program the pipelines for flexibility, deploying the workflows on the Azure cloud for scalability, and recording the provenance of workflow execution, for reproduciblity. In this discussion paper we elaborate on our design choices, the associated challenges, and the expected benefits.