This paper presents the concepts of an open software tool (CaRMeN) that can be used to rapidly analyse and derive models, in particular chemical kinetics. The software automates the workflow of comparing model vs. experiment, which must currently be done manually and is thus a time-consuming and error-prone task. The capabilities of the software are illustrated through a case study. Experimental data for the conversion of methane over rhodium catalysts in a wide range of conditions and experimental setups are numerically simulated using five different mechanisms from the literature. The applicability of the mechanisms as well as differences between flow and diffusion models are evaluated. The results show that no single mechanism reliably predicts the chemical conversions of all of the experiments. Although the software was initially developed for chemical kinetics applications, it can also be extended to run any simulation code, and can therefore be applied in other scenarios.