This video shows how to get deterministic outputs when using TensorFlow, so that the outputs are reproducible. Everything should be perfectly repeatable.
I created a Jupyter notebook to demonstrate this at: https://github.com/ageron/handson-ml/...
Researchers at Two Sigma managed to get reproducible outputs on the GPU using TensorFlow, but it wasn't a walk in the park, check it out: https://www.twosigma.com/insights/art...
If you have a scenario I didn't mention where you need perfect reproducibility, I'd love to hear about it, please post it in the comments below. If you still have non-deterministic results after implementing the recommendations in this video, I'd also love to hear about it. Please specify your Python, Numpy and TensorFlow versions, as well as your O.S. version.
ERRATA:
* At 4:19, it should be CUDA_VISIBLE_DEVICES with an S.
Hope this is useful!
Aurélien Géron
August 11th 2018