A review of the literature on oil toxicity tests showed a high diversity of reported test methods that may affect the composition, stability, and toxicity of oil solutions. Concentrations of oil in test solutions are dynamic because hydrocarbons evaporate, partition to test containers, bioaccumulate, biodegrade, and photo‐oxidize. As a result, the composition and toxicity of test solutions may vary widely and create significant obstacles to comparing toxicity among studies and to applying existing data to new risk assessments. Some differences in toxicity can be resolved if benchmarks are based on measured concentrations of hydrocarbons in test solutions, highlighting the key role of chemical analyses. However, analyses have often been too infrequent to characterize rapid and profound changes in oil concentrations and composition during tests. The lack of practical methods to discriminate particulate from dissolved oil may also contribute to underestimating toxicity. Overall, current test protocols create uncertainty in toxicity benchmarks, with a high risk of errors in measured toxicity. Standard oil toxicity tests conducted in parallel with tests under site‐specific conditions would provide an understanding of how test methods and conditions affect measured oil toxicity. Development of standard test methods could be achieved by collaborations among university, industry, and government scientists to define methods acceptable to all 3 sectors. Environ Toxicol Chem 2019;38:302–311. © 2018 The Authors. Environmental Toxicology and Chemistry published by Wiley Periodicals, Inc. on behalf of SETAC.