Don't Forget to Explore!