Conversational Systems (CS) are increasingly present in people’s daily lives. CS must provide a good experience and meet the needs of its users. Therefore, the Usability and User Experience (UX) evaluation is an appropriate step before making CSs available to society. To guide developers to identify problems, improvement suggestions, and user perceptions during CSs development, we developed a technology named Usability and User Experience Evaluation of Conversational Systems (U2XECS). U2XECS is a questionnaire-based technology that provides Usability and UX statement specifics to evaluate CSs. We conducted an exploratory study performed to evaluate and evolve U2XECS. Our results evidenced positive points of U2XECS related to ease of use, usefulness, and intentions to use. Moreover, we identified opportunities for improvement in U2XECS, such as ambiguous statements that generated misinterpretations in subjects.