Classification tasks today, especially for the medical domain, use datasets which are often hierarchical. These tasks are approached using methods that consider the class taxonomy for predicting a label. The classifiers are gradually becoming increasingly accurate over the complex datasets. While increasing accuracy is a good way to judge a model, in high-risk applications, it needs to be ensured that even if the model makes a mistake, it does not bear a severe consequence. This work explores the concept of severity of an error and extends it to the medical domain. Further, it aims to point out that accuracy or AUROC alone are not sufficient metrics to decide the performance of a model in a setting where a misclassification will incur a severe cost. Various approaches to reduce severity for classification models are compared and evaluated in this work, which indicate that while many of them might be suited for a traditional image classification setting, there is a need for techniques tailored toward tasks and settings of medical domain to push artificial intelligence in healthcare to a deployable state.