...why should I get surgery?
Black-box AI is particularly problematic in healthcare, where software could make suggestions that impact the well-being of patients.
Starting from user experience on real-life examples, we single out the most asked questions and discuss desirable types of explanations for users without deep knowledge of machine learning. We showcase how different methods and packages can come together in a Python interpretability toolbox and describe at which stages of development they should be used. These considerations generalize to the development of any AI-powered product.