Abstract Graphical User Interface (GUI) automation holds significant promise for assisting
users with complex tasks thereby boosting human productivity. Existing works leveraging …

Graphical User Interface (GUI) automation holds significant promise for assisting users with
complex tasks, thereby boosting human productivity. Existing works leveraging Large …

This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models
(LLMs) to automate the interactions with Android mobile applications. Given a natural …

Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone,
which completes a task triggered by natural language through predicting a sequence of …

Graphical User Interface (GUI) agents are designed to automate complex tasks on digital
devices, such as smartphones and desktops. Most existing GUI agents interact with the …

Smartphone users often navigate across multiple applications (apps) to complete tasks such
as sharing content between social media platforms. Autonomous Graphical User Interface …

The emergent large language/multimodal models facilitate the evolution of mobile agents,
especially in the task of mobile UI automation. However, existing evaluation approaches …

Autonomous agents that execute human tasks by controlling computers can enhance
human productivity and application accessibility. Yet, progress in this field will be driven by …

An automation system that can execute natural language instructions by driving the user
interface (UI) of an application can benefit users, especially when situationally or …

We present VASTA, a novel vision and language-assisted Programming By Demonstration
(PBD) system for smartphone task automation. Development of a robust PBD automation …