Set Focus on full screen windows

Jan 28, 2013 at 2:08 PM

Hello.

I needed to mess around with the UI module on full screen windows, specifically Google Chrome, and hit some issues. I want to let you know the work-arounds I found and ask if you happen to know an easier way.

The situation is this - we have an application that uses a browser (Google Chrome) that must run as full screen (at least most of the time). And we need to programatically interact with the main window, at times.

We used to start Chrome with the --kiosk switch. This puts it in full screen. It's a little different than going full screen by hitting F11, in the sense that you cannot "go out" of full screen if you are in kiosk mode - I guess it was designed specifically for kiosk applications.

But when the window is full screen (either through kiosk switch, or through F11, does not matter), it cannot receive focus for some reason.

$Window = Get-UIAWindow -Name '... - Google Chrome'
$Window | Set-UIAFocus

set-uiafocus : Target element cannot receive focus.
At line:1 char:11
+ $window | set-uiafocus
+           ~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-UIAFocus], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,UIAutomation.Commands.SetUIAFocusCommand

The error is very explicit.

Also, if I'm trying to click on it nothing happens (it is not brought in foreground as I would expect). However, no error is given.

$Window | Invoke-UIAControlClick

 What I noticed is that, if I get the windows while it is in normal mode, and afterwards put it full screen it will work, as long as I don't recreate the object. So we did not start Chrome in kiosk mode anymore.

#start Chrome normally
...
# Get the window 
$Window = Get-UIAWindow -Name '... - Google Chrome'
# Set it full screen
$Window | Set-UIAFocus | Set-UIAControlKeys -Text '{F11}'
# Retain the window object (do not get it every time) to perform operations
$Window | Set-UIAFocus
$Window | Invoke-UIAControlClick

And now it works! But I'm not very sure why it did not work the first time. Is it something specific to full screen windows, or something specific to how Chrome implements it?

Now just to be clear, we have a work-around which works nicely (for most part). I don't want this post to cause you to waste time for investigations, just wanted to let you know and maybe you hit this problem in the past and know of another (simpler) way to do it.

One incovenience of the work-around is that someone could accidentally press F11 and put the windows out of full screen, which is not at all desirable. Starting Chrome in kiosk mode would have prevented this to happen. Which leads me to my second question :)

Is there a way to determine the state of the window with ui module? I mean maximized, minimized, etc.

Thanks in advance,

Johnny.

Coordinator
Jan 28, 2013 at 2:56 PM
Edited Jan 28, 2013 at 4:08 PM

Hello JohnQuest,

I first address what I can answer promptly.

1) window state - it's not implemented yet. However, I'm planning to include this in 0.8.4 (I'll include it in Beta 4 I think).

Now you can perform this manually:

 

# obtain WindowPattern:
$wp = (Start-Process calc -PassThru | Get-UIAWindow).GetCurrentPattern([System.Windows.Automation.WindowPattern]::Pattern);
# use what the pattern offers:

# it can't be maximized via WindowPattern
# we can maximize Calculator by clicking on the Maximize button :)
$wp.Current.CanMaximize
$wp.Current.CanMinimize
$wp.Current.IsModal
$wp.Current.IsTopmost
$wp.Current.WindowInteractionState
$wp.Current.WindowVisualState
$wp.SetWindowVisualState([System.Windows.Automation.WindowVisualState]::Minimized); 
# etc

2) Regarding the question 'how to click the window', there's no problem.

2.1) setting the focus won't work (I suspect that the underlaying level of element.SetFocus() the the SetForegroundWindow API call.

Remarks

The system restricts which processes can set the foreground window. A process can set the foreground window only if one of the following conditions is true:

  • The process is the foreground process. (is your powershell not the foreground process at the time of click?)
  • The process was started by the foreground process.
  • The process received the last input event.
  • There is no foreground process.
  • The foreground process is being debugged.
  • The foreground is not locked (see LockSetForegroundWindow). (this may be true, I don't know)
  • The foreground lock time-out has expired (see SPI_GETFOREGROUNDLOCKTIMEOUT in SystemParametersInfo).
  • No menus are active. 

2.2) You need to run your powershell script from a host that is floating over all windows: UIARunner, UIAutomaitonSpy (yes, it could also run code though I'd suggest yu to use UIARunner), BGShell (0.8.0P5 for Metro testing).

I noticed today that BGShell is no longer in my binaries - a couple of months ago a moved its folder and since then it is never wrapped in the release binaries. I'll fix it.

After you run your script in a host that is above all windows, ir runs the following without any obstacles:

Get-UIAWindow -n *chrome | Invoke-UIAControlClick;

 

Finally, I would say that Chrome could not be automated with MS UI Automation (unlike IE or FF), it's like a container (Pane or Document). All you can automate (as far I can predict) is toolbars, the navigation bar and similar things that at the top of Chrome window.

Thanks for sharing the info, I've never heard about the --kiosk mode (though it's slightly similar to Metro UI apps).

Jan 29, 2013 at 3:44 PM

The tricky part is that there is no difference between a normal (non-maximized) window and a full-screen window. The WindowVisualState property is Normal in both cases. So the WindowPattern is not helpful in this case.

I searched some examples online and they all involved pinvoke, and comparing the screen size with the window size to asses if it runs full screen or not. Not very elegant, but I guess that is the only way.

Coordinator
Jan 29, 2013 at 5:29 PM

If your show-stopper is inability to recognize the full-screen window, what's about such an one-liner:

if ((Get-UIADesktop).Current.BoundingRectangle -eq (Get-UIAWindow -Name *chrome).Current.BoundingRectangle) { "full screen"; } else { "else"; }
It works even in two-display environment.

Jan 29, 2013 at 7:37 PM

Excellent! You beat me up to it, I was wandering in the same area. This will work out nicely, after all.

Thank you.