Finding UI Elements

Aug 9, 2012 at 8:40 PM

Do you have an example to illustrate how you can find controls? I was looking for a way to search for all controls in a recursive way, starting with Get-UIAWindow. However, all cmdlets only return one element, and the properties of the returned element don't help much in identifying it.

Ultimately, I would like to read and write text to a text control. However, I don't even know the type and name of the control (which is why I would like to use cmdlets to investigate what's present).

Any chance? Your project BTW rocks!


Aug 9, 2012 at 10:53 PM

Hello TobW,

there are several cmdlets that do exactly what MS UI Automation FindAll and TreeWalker do.

Below the names, please find out samples in the blog (I'm on an iPad now and can't test any samples):

Get-UIAControlChildren - FindAll, TreeScope.Children

Get-UIAControlDescendants - FindAll, TreeScope.Descendants

Get-UIAControlParent/Ancestors/NextSibling/firstChild/... - TreeWalker

Search-UIAControl - TreeWalker (in the samples package, the Search- cmdlet is used here and there)

These cmdlets may help you in automatic gathering of controls. If you are interested in manual investigation, I'd recommend using UIAutomationSpy that suggests code for a control you are hovering over.
What's more, the UIAutomation framework was created with unification in mind (the good object model of MS UI Automation marvellously helped in that) and this means that you have the Get-UIAControl cmdlet. Now, I open the terrible secret of the framework: tons of cmdlets are simply hard-coded aliases. :) 

Get-UIAButton == Get-UIAControl -ControlType Button

Get-UIAEdit == Get-UIAControl -ControlType Edit

In other words, the framework can be used similarly to the WASP framework:

Start-Process calc -PassThru | Get-UIAWindow | Get-UIAControl -Name 1 | Invoke-UIAInvokePattern;As can be seen, the 'pattern' cmdlets are aliases:Invoke-UIAButtonClick == Invoke-UIAInvokePatternjust run the following:Get-Command -module uia* *pattern*and you'll obtain the list of supported 'typeless' pattern cmdlets.
Regarding reading and writing text into a control, there are two ways:

0) (the traditional way) Get-UIATextBoxText, Set-UIATextBoxText

1) 'bare' patterns



There are also cmdlets for TextPattern, but, as I remember, they have never been asked by people and left written in half (I even found a second ago a typo: Get-UIAPatternSet)

2) the universal Win32 cmdlet Set-UIAControlText. It works only with control that have a handle.

There even is a SendKeys cmdlet, Invoke-UIAControlKeys, for testers who found no better ways to put text in a control...

Finally, you are welcome to suggest cmdlets if three hundred now-existing don't cover your needs (for example, not all possible patterns are supported for now).

Aug 17, 2012 at 10:17 AM

Even though the best way to investigate into control placement in the UI Automaitn tree is by using UIAutomaitonSpy or the Start-UIARecorder (Start-UIATranscript) cmdlet, there are ways to select necessary controls on the fly.

The following example demonstrates this.

Task: we have a Sharepoint 2007 portal of a project. There is used SCRUM, so that sprints are posted on the portal. Each sprint has its own Sharepoint page and contains a list of stories with proposed man-hours for each.

Every story is a Sharepoint page, and the number at the end of its URL is the unique story Id.

Provide a list of stories for a selected sprint in the format: story_id [tab] story_name

The script:

[UIAutomation.Preferences]::Timeout = 20000;
Get-UIAWindow -n Sprints* | Get-UIAPane -n Sprints* | Get-UIAPane -n Sprints* | Get-UIAControlDescendants -ControlType Hyperlink | ?{($_ | Get-UIAHyperlinkText) -match "(?<=Stories[/]DispForm.aspx[?]ID[=])[\d]+(?=[^\d]?)";} | %{ "$(($_ | Get-UIAHyperlinkText).Substring(($_ | Get-UIAHyperlinkText).IndexOf("=") + 1))`t$($_.Current.Name)";}

Now, I'll explain the code in detail:

1) we get a window (IE10 page in my case) by its title Sprints.....

2) we get a pane

3) we get one more pane. Why? Controls in the UIAutomation tree are placed like branches on a wooden tree, and nobody can predict the order UIAutomaiton will use to go through all the branches before outputting the result.

I investigated into controls with the help of UIAutomaitonSpy not to force the script to search every time. It's like a short cut path in the UI Automation tree. I force the UI Automaiton engine to go this way by ordering what to search.

The second pane is the pane that contains a lot of links.

4) Investigation on the fly. We get all the links from the pane. There are a lot of them. Besides stories' links, there are

- the link to the project

- links to each Sharepoint page that are higher then the current page in the hierarchy of Sharepoint pages

- the link to the page author's page

and so on

5) We check all the links' texts (i.e., URLs) whether they match the regular expression (all stories' links contain the word Stories and a couple of distinguishing things).

6) If we've gotten the match(es), we further concatenate $link.Current.Name (i.e., visible text) and the numeric part of link's URL to produce the result.

As can be seen, this example is a combination of pre-investigation made by hands (and investigation tools, I'd recommend UIAutomaitonSpy and, if you need  richer code generation, the Start-UIARecorder cmdlet) and on-the-fly calculations.

Aug 28, 2012 at 5:47 PM
TobW wrote:

I was looking for a way to search for all controls in a recursive way, starting with Get-UIAWindow. However, all cmdlets only return one element, and the properties of the returned element don't help much in identifying it.

HI Tob, 

the idea about converting cmdlets' output to a collection has been taking me all the time after you've asked for and finally I did that. Now, cmdlets return all objects that match, even through Win32 API. There is a bit about the topic.