AppleScript click Ui Element with modifier keys

applescriptui

I want to click an Ui Element button with modifier keys pressed.

Here is my code:

tell application "System Events"
    tell process "Pro Tools"
        tell (1st window whose title contains "Edit: ")
            tell table "Group List"
                tell row 2
                    tell UI element 2

                        click       

                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

I tried adding key down (option) and key up (option) between the click lines, but it doesn't work.

Best Answer

I do not have Pro Tools, so I can only offer some information that may help you.

First I have to say that UI Scripting can be kludgy and may not always work consistently depending on how its coded and the UI element being targeted. It's always better if one can find a way to do something without resorting to UI Scripting first.

As I originally said to you in a comment... Basic vanilla AppleScript does not have a provision to preform a key down and mouse click simultaneously. You could try using a third-party utility like cliclick wrapped in a do shell script command. You'd have to programmatically determine the position of UI element 2 to then click on it with cliclick.

Without Pro Tools, I'm going to use Safari as an example, and the Show sidebar UI element on its Toolbar in conjunction with cliclick in a do shell script command.

The purpose of following example AppleScript code is to demonstrate how to programmatically get the position and size of the target UI element, in order to calculate where to click. It also assumes that Safari is already running with a standard window opened to, e.g., this question you posted.

activate application "Safari"
delay 1

tell application "System Events" to ¬
    set ps to {position, size} ¬
        of UI element 2 ¬
        of toolbar 1 ¬
        of window 1 ¬
        of application process "Safari"

set p to item 1 of ps
set s to item 2 of ps

set x to (item 1 of p) + (item 1 of s) / 2 as integer
set y to (item 2 of p) + (item 2 of s) / 2 as integer

do shell script "$HOME/bin/cliclick c:" & x & "," & y

      • Change $HOME/bin/cliclick as appropriate to: /path/to/cliclick

When run in Script Editor, this clicked the Show sidebar UI element on the Toolbar in Safari.

Now I know you said you want to click with the option key down, however for a practical example I'm going to modify the example AppleScript code, shown above, to control-click to the right of the Show sidebar UI element on the Toolbar in Safari, which will pop-up the Customize Toolbar… menu.

Change:

set x to (item 1 of p) + (item 1 of s) / 2 as integer

To:

set x to (item 1 of p) + (item 1 of s) * 2 as integer
  • Instead of dividing the x axis size by two to get the center of the target UI element on its x axis , we'll multiply its x axis size by two so it will click on the Toolbar itself.

Also Change:

do shell script "$HOME/bin/cliclick c:" & x & "," & y

To:

do shell script "$HOME/bin/cliclick kd:ctrl c:" & x & "," & y & " ku:ctrl"

Now when run in Script Editor, this control-clicked to the right of the Show sidebar UI element on the Toolbar in Safari and popped-up the Customize Toolbar… menu.

With all of this in mind, maybe the following example AppleScript code will work for you, however remember it's untested as I do not have Pro Tools and may require you to rework the tell application "System Events" ... statement:

activate application "Pro Tools"
delay 1

tell application "System Events" to ¬
    set ps to {position, size} ¬
        of UI element 2 ¬
        of row 2 ¬
        of table "Group List" ¬
        of (first window whose title contains "Edit: ") ¬
        of application process "Pro Tools"

set p to item 1 of ps
set s to item 2 of ps

set x to (item 1 of p) + (item 1 of s) / 2 as integer
set y to (item 2 of p) + (item 2 of s) / 2 as integer

do shell script "$HOME/bin/cliclick -r kd:alt c:" & x & "," & y & " ku:alt"

      • Change $HOME/bin/cliclick as appropriate to: /path/to/cliclick

Note: The example AppleScript code is just that and does not contain any error handling as may be appropriate. The onus is upon the user to add any error handling as may be appropriate, needed or wanted. Have a look at the try statement and error statement in the AppleScript Language Guide. See also, Working with Errors.

Note: The example AppleScript code for use with Safari and cliclick was tested and worked as presented in macOS Catalina.

Note: I am not affiliated with the developer of cliclick, just a satisfied user of the product.

Related Question