With a significant deployment of smart meters across end-user platforms, the dynamic visibility of energy flow among the end-users has been increased significantly. The granular information of smart meters can be used to improve the load forecast accuracy and to influence energy consumption patterns with demand side management (DSM) schemes. This paper addresses the challenges of smart meter data size, complexity, variability and volatility for efficient use in load forecast and DSM. A novel clustering-based approach for analysis of smart meter data, aimed at more accurate and detailed load profiling, reduced profile complexity, improved load forecast accuracy and providing optimal DSM solutions is proposed. The proposed approach utilizes an advanced clustering algorithm to reduce the data size. The approach addresses data complexity, variability and volatility by linearizing the load profiles and minimizing the errors. The validity of the approach is demonstrated on an Irish smart meter dataset and on a simulated solar photovoltaic (PV) data and showed an improved load forecast accuracy, improved DSM solutions, and reduced computational burden. The improvements in the DSM solution are evidenced by a higher cost saving with a higher peak load reduction at the lower level of demand flexibility.