We develop a general classification of the nature of the instabilities yielding spatial organization in open nonideal reaction-diffusion systems, based on linear stability analysis. This encompasses dynamics where chemical species diffuse, interact with each other, and undergo chemical reactions driven out of equilibrium by external chemostats. We find analytically that these instabilities can be of two types: instabilities caused by intermolecular energetic interactions (E type), and instabilities caused by multimolecular out-of-equilibrium chemical reactions (R type). Furthermore, we identify a class of chemical reaction networks, containing unimolecular networks but also extending beyond them, that can only undergo E-type instabilities. We illustrate our analytical findings with numerical simulations on two reaction-diffusion models, each displaying one of the two types of instability and generating stable patterns.