The emergence of possible instability modes under undrained biaxial shear of sand was investigated to develop better understanding about the onset of localized and liquefaction-type solid-fluid instability modes using a generalized three-dimensional (3D) material model. These two modes are the primary instability modes that can exist in most cases while conducting undrained biaxial testing of sand at different densities, confining pressures, and boundary conditions. Using a 3D rate-independent and nonassociative constitutive model, the instability analysis was performed as a plane-strain bifurcation problem from a uniform stress-strain state. Large deformation formulation was used to simulate the biaxial test configuration with both rigid and flexible boundaries in the lateral direction. The existing theoretical framework of solid-fluid instability analysis under rigid boundaries was extended to the flexible lateral boundary condition. Interestingly, the onset of solid-fluid instability modes is significantly influenced by the choice of boundary conditions. The trends for the onset of various undrained instability modes were assessed as a function of material state variables.