Run-time evaluation of architectures: a case study of diversification in IoT

Run-time properties of modern software system environments, such as Internet of Things (IoT), are a challenge for existing software architecture evaluation methods. Such systems are largely data-driven, characterized by their dynamism, unpredictability in operation, hyper-connectivity, and scale. Properties, such as performance, delayed delivery, and scalability, are acknowledged to pose great risk and are difficult to evaluate at design-time. Run-time evaluation could potentially be used to complement design-time evaluation, enabling significant deviations from the expected performance values to be captured. However, there are no systematic software architecture evaluation methods that intertwine and interleave design-time and run-time evaluation. This paper addresses this gap by proposing a novel run-time architecture evaluation method suited for systems that exhibit uncertainty and dynamism in their operation. Our method uses machine learning and cost-benefit analysis at run-time to continuously profile the architecture decisions made, to assess their added value. We demonstrate the applicability and effectiveness of this approach in the context of an IoT system architecture, where some architecture design decisions were diversified to meet Quality of Service (QoS) requirements. Our approach provides run-time assessment for these decisions which can inform deployment, refinement, and/or phasing-out decisions.