Integrated Offline and Online Decision Making under Uncertainty