MacOS automation that can detect pixels on screen

applescriptautomationautomatorkeyboardscreen capture

I need to automate some keyboard and mouses actions depending on what is shown on the screen.

But because the applications don't have applescript hooks, there is no way to know of certain events except to 'read' the pixel values of certain position of the screen.

Is there a way to do this? I've looked into tools like keyboard masestro but was unable to do this.

Best Answer

Consider using a visual scripting tool like Eggplant or SikuliX.

SikuliX

SikuliX is an open source scripting tool that detects images and performs actions:

SikuliX automates anything you see on the screen of your desktop computer running Windows, Mac or some Linux/Unix. It uses image recognition powered by OpenCV to identify GUI components and can act on them with mouse and keyboard actions. This is handy in cases when there is no easy access to a GUI's internals or the source code of the application or web page you want to act on.