Physical activity is associated with reduced risk of a number of health outcomes, yet fewer than half of adults in the United States report recommended levels of physical activity. Analyses of structural characteristics of the built environment as correlates of physical activity have yielded mixed results. We examine associations between multiple aspects of urban neighborhood environments and physical activity in order to understand their independent and joint effects, with a focus on the extent to which the condition of the built environment and indicators of the social environment modify associations between structural characteristics and physical activity. We use data from a stratified, multi-stage proportional probability sample of 919 non-Hispanic Black, non-Hispanic White, and Hispanic adults in an urban community, observational data from their residential neighborhoods, and census data to examine independent and joint associations of structural characteristics (e.g., street network connectivity), their condition (e.g., sidewalk condition), and social environments (e.g., territoriality) with physical activity. Our findings suggest that sidewalk condition is associated with physical activity, above and beyond structural characteristics of the built environment. Associations between some structural characteristics of the built environment and physical activity were conditional upon street condition, physical deterioration, and the proportion of parks and playgrounds in good condition. We found modest support for the hypothesis that associations between structural characteristics and physical activity are modified by aspects of the social environment. Results presented here point to the value of and need for understanding and addressing the complexity of factors that contribute to the relationships between the built and social environments and physical activity, and in turn, obesity and co-morbidities. Bringing together urban planners, public health practitioners and policy makers to understand and address aspects of urban environment associated with health outcomes is critical to promoting health and health equity.