The urbanization problems we face may be alleviated using innovative digital technology. However, employing these technologies entails the risk of creating new urban problems and/or intensifying the old ones instead of alleviating them. Hence, in a world with immense technological opportunities and at the same time enormous urbanization challenges, it is critical to adopt the principles of responsible urban innovation. These principles assure the delivery of the desired urban outcomes and futures. We contribute to the existing responsible urban innovation discourse by focusing on local government artificial intelligence (AI) systems, providing a literature and practice overview, and a conceptual framework. In this perspective paper, we advocate for the need for balancing the costs, benefits, risks and impacts of developing, adopting, deploying and managing local government AI systems in order to achieve responsible urban innovation. The statements made in this perspective paper are based on a thorough review of the literature, research, developments, trends and applications carefully selected and analyzed by an expert team of investigators. This study provides new insights, develops a conceptual framework and identifies prospective research questions by placing local government AI systems under the microscope through the lens of responsible urban innovation. The presented overview and framework, along with the identified issues and research agenda, offer scholars prospective lines of research and development; where the outcomes of these future studies will help urban policymakers, managers and planners to better understand the crucial role played by local government AI systems in ensuring the achievement of responsible outcomes.