In this talk, I will share some of my recent results on offline reinforcement learning (RL), an RL paradigm for domains where exploration is prohibitively expensive or even implausible, but a fixed dataset of previous experiences is available a priori. Specifically, I will focus on discussing how deep neural networks (trained by (stochastic) gradient descents) and randomization lead to a computationally efficient algorithm that has a strong theoretical guarantee for generalization across large state spaces under mild assumptions of distributional shifts while obtaining a favorable empirical performance. I will conclude with a discussion on future directions to make RL more data-efficient, deployment-efficient, and robust.