X-ray ptychography is becoming the standard method for sub-30 nm imaging of thick extended samples. Available algorithms and computing power have traditionally restricted sample reconstruction to 2D slices. We build on recent progress in optimization algorithms and high-performance computing to solve the ptychographic phase retrieval problem directly in 3D. Our approach addresses samples that do not fit entirely within the depth of focus of the imaging system. Such samples pose additional challenges because of internal diffraction effects within the sample. We demonstrate our approach on a computational sample modeled with 17 million complex variables.