Computer use

The model sees screenshots, recognises buttons and input fields and performs clicks and keystrokes. This lets it operate programs without an interface, such as filling in forms or gathering information from websites.

The technique is slower and more error-prone than direct interfaces and carries security risks, for example when websites contain hidden instructions. It should run in isolated environments with restricted permissions.