 Influence functions can unravel the black box of neural networks, NNs, when trained to predict quantum phases in the one-dimensional extended spin-list Femi-Hubbard model at half-filling, providing strong evidence that NNs correctly learn an order parameter describing the quantum transition. This method is applicable to a broad class of physical models or experimental data and requires no appriory knowledge on the order parameter, has no dependence on the NNs architecture or the underlying physical model. This article was offered by Anna David, Patrick Wembley, Vika Altonza, and others.