The skin is the largest organ in human body. Around 30%-70% of individuals worldwide have skin related health problems, for whom effective and efficient diagnosis is nec-essary. Recently, computer aided diagnosis (CAD) systems have been successfully applied to the recognition of skin cancers in dermatoscopic images. However, little work has concentrated on the commonly encountered skin diseases
in clinical images captured by easily-accessed cameras or mobile phones. Meanwhile, for a CAD system, the repre-sentations of skin lesions are required to be understandable for dermatologists so that the predictions are convincing. To address this problem, we present effective representa-tions inspired by the accepted dermatological criteria for diagnosing clinical skin lesions. We demonstrate that the
dermatological criteria are highly correlated with measur-able visual components. Accordingly, we design six medical representations considering different criteria for the recog-nition of skin lesions, and construct a diagnosis system for clinical skin disease images. Experimental results show that
the proposed medical representations can not only capture the manifestations of skin lesions effectively, and consis-tently with the dermatological criteria, but also improve the prediction performance with respect to the state-of-the-art methods based on uninterpretable features.