Acts on your actual screen

Nia can take a screenshot, read what's actually on your screen, and control your mouse and keyboard to carry out a real task in a real application — opening something, filling in a form, checking a setting — instead of just telling you which buttons to click.

Giving software direct control of your mouse and keyboard is a harder problem than it sounds, and the first working version of this had real failure modes — including, at one early point, taking over your actual system pointer while you were trying to use it yourself. The current approach avoids that by invoking on-screen elements directly through the accessibility layer instead of simulating physical clicks wherever possible, which is also more reliable than guessing at coordinates.

Computer control doesn't run unsupervised. Reads and searches happen freely, but anything genuinely sensitive stops and waits for your explicit approval first — the same rule as everywhere else in Nia.

The full account of what went wrong before this worked — including the version that hijacked your own pointer — is on the blog: "The AI Needed Its Own Hands."