With the increasing capabilities of wearable sensors and implantable medical devices, new opportunities arise to diagnose, control and treat several chronic conditions. Unfortunately, these advancements also open new attack vectors, making security an essential requirement for the further adoption of these devices. Researchers have already developed security solutions tailored to their unique requirements and constraints. However, a fundamental yet unsolved problem is how to securely and efficiently establish and manage cryptographic keys. One of the most promising approaches is the use of patient's physiological signals for key establishment. This paper aims at identifying common pitfalls in physiological-signal-based cryptographic protocols. These solutions are very fragile because errors can be introduced at different stages, including the choice of the physiological signal, the design of the protocol or its implementation. We start by reviewing previous work that has succeeded in measuring various physiological signals remotely. Subsequently, we conduct a thorough security analysis of two cryptographic solutions well-accepted by the security community, namely the H2H protocol (Rostami et al. - CCS 2013) and the Biosec protocol (Cherukuri et al. - ICISIP 2006). Our evaluation reveals that these protocols have serious design and implementation security weaknesses. Driven by our findings, we then describe how to use fuzzy extractors for designing secure and efficient cryptographic solutions based on the patients' physiological signals. Finally, we discuss research directions for future work.