Problem with "Internet Explorer_Server" objects

Feb 22, 2013 at 11:23 AM
Edited Feb 22, 2013 at 12:49 PM
Hi,
first of all I want to say that these cmdlets are absolutely awesome and they relay makes my life easier.

Secondly I have found issue when I'm trying to get hyperlinks and other control from "Internet Explorer_Server" class object from our application. UIVerify and UIAutomationSpy are not able to return any childs of that object (hyperlinks, texts etc.). I've tested it in more apps and it seems that it only occurs in cases where "Internet Explorer_Server" class based object doesn't have Name property set. Any ideas where the problem might be?

Some screens from UIVerify:
https://skydrive.live.com/redir?resid=CD7E715FE2A44A15!5828&authkey=!AN20z3oeCgFgzxw
https://skydrive.live.com/redir?resid=CD7E715FE2A44A15!5829&authkey=!ADx3c4DeDVZDWIM

Also is there any way how to utilize AutomationID property when accessing objects under "Internet Explorer_Server"? I thought that it would be ID of the HTML element but I don't see that in UIVerify.
Coordinator
Feb 22, 2013 at 1:28 PM
Edited Feb 22, 2013 at 1:41 PM
HI,
first of all thanks for your warm words about the module.
Seconds, I need to say that UIAutomation is not the best tool for browser automation. Why? A simple google first page consists of approx. 243 elements that could be caught as ~93 AutomationElements (by converting from Selenium's web element). This means that UIautomation (that builds the Automation tree in memory) works slowly. Selenium requests elements on-the-fly and this works faster in such situations.

Closer to your question:
1) I hardly could say much about your application (when it exposes hyperlinks and when don't). UI Automation .NET sees less than UI Automation from uiacomwrapper.codeplex.com (the module does not empoley it yet), whereas uiacomwrapper rarer supports patterns.
Sometimes, there are bugs in MS UI Automation .NET (i.e., an element could be got via hovering over but fails be got via search).
MS UI Automation performs some calculations basing on properties that are available via MSAA. And who knows what is the algorithm.
Probably, some developers use accessibility properties in controls, some don't. Or just a matter of luck.
2) Unfortunately, there is no way to use anything to search for except what AutomationElement brings to us: Name, Class, AutomationId, Value. Selenium could but fails to work with embedded browsers.

If a pane is of fixed size of link on fixed positions from left and top, you could use coordinated clicks.

I'm planning to add support for uiacomwrapper later, maybe it'll help.

Upd: if you are close to developers or a developer itself, the following links might (or might not) help in development accessible controls
http://msdn.microsoft.com/en-us/library/windows/desktop/cc307844.aspx
http://msdn.microsoft.com/en-us/library/system.windows.forms.accessibleobject.aspx
Feb 22, 2013 at 3:47 PM
Thank you very much for detailed response. I've found that it's probably caused by self protection feature of process I'm testing, it limits access to process in some way which causes that UIAutomation can't access some controls :s It worked so far with TestComplete (which I'm trying to get rid of) so I haven't thought that that could cause it. So I'm basically stuck here because the only way how to access the process is to sign PowerShell with the same certificate as tested app (certificate I have but can't change it in PS executable), and I can't compile my scripts into executables and sign those because of some complications in our framework :(

I also thought about using Selenium but as you said there is no way how to "hook" it to embedded browser which is basically what I'm trying to achieve.

Anyway thank you for your time when analyzing this.
Coordinator
Feb 22, 2013 at 5:54 PM
Edited Feb 22, 2013 at 5:55 PM
If the problem is in certificate, and it's not a *.snk certificate
1) you could (try to) sign powershell.exe (I'm sure that this will work)
I sign my software creatures is the following way:
"C:\Program Files (x86)\Windows Kits\8.0\bin\x64\signtool.exe" sign /f C:\Projects\PS\STUPS\certificate\my\SoftwareTestingUsingPowerShell.pfx /t http://timestamp.verisign.com/scripts/timestamp.dll /p %1 C:\Projects\PS\STUPS\UIA\UIAutomationSpy\bin\Release35\UIAutomationSpy.exe
where
SoftwareTestingUsingPowerShell.pfx is a local certificate I generated months ago
%1 is certificate's password
signtool.exe in is Windows SDK 8.0, 7.1, should be in 7.0 also
the last is the executable I sign

2) you could sign my exes:
the UIAutomation module is shipped with three executables: UIAutomationSpy (it can run PS code), UIARunner (it is intended to run PS code in the GUI or from command-line) and BGShell (for running code over Metro UI)
Both types of signing are possible.
Open source rules!

3) you could follow steps how to install an accessible app http://uiautomation.codeplex.com/wikipage?title=Installation%20for%20Metro%20UI%20testing&referringTitle=Documentation after the phrase "Putting module files into a secure location".
(put an app in the secure location, add my certificate to the Trusted Root Certification Authorities store (a Microsoft's requirement), etc)
Accessible apps have more privileges.
You also could try to combine 2) and 3).

P.S. Are you working on an AV? A fun fact about avast: when I build another release/beta and put my executables in a new location to publish them, avast on the first run embraces my app with sandbox border and label. If I start spying with UIAutomationSpy, it also sandboxes my squares over windows or controls with the Avast label...
Feb 25, 2013 at 3:54 PM
I signed everything with certificate, which tested app accepts as trusted, and it works. However I had to also sign powershell.exe in order to get it working. Do you know if it possible that something could stop working since it's not signed by Microsoft anymore (I mean like access to WMI etc.)?

P.S. Yes it's an AV product, it's a bitch to automate sometimes due to all protection stuff.

Thank you very much for all the help.
Coordinator
Feb 25, 2013 at 5:07 PM
Hi,
it's hard to say whether MS checks or doesn't when running powershell.exe. The vast majority of users (even those German people who created 'portable powershell') hardly change signature of powershell.exe.

Alternatively, you could you UIARunner.exe (a very simple PowerShell host) that could be used with GUI or from command line:
UIARunner.exe [path]\script.ps1
UIARunner requires some files (UIAutomaiton.dll and TMX.dll, for example), so that it should be run from the folder it has been unpacked to.

One more option is BGShell (by Lee Holmes and several other guys) that is also PowerShell host.
Nov 13, 2014 at 4:52 PM
Guys, it seems that I have similar problem with IE that for some reason "Internet Explorer_Server" class based object doesn't have Name property set when app tries to get it using MSAA accessibility interface.

I am working on project which helps end user to create daily reports by tracking it's activity.
To track activity in IE browser application uses accessibility interface (MSAA).

In brief, tracking works the following way:
  1. App searches for all child windows of IE browser which have "Internet Explorer_Server" window class (tab documents windows)
  2. Gets IAccessible object by window handle.
  3. Gets name of the IAccessible object which represents URL of the document loaded for this tab
Everything works fine for most of users. But several users reported that IE tracking doesn't work for them.
Nothing fails, but empty string is always returned as name of the IAccessible object (document).

I checked that URL can be obtained using UIAutomation API (I used Inspect.exe, AccChecker.exe tools), but for some reason the name is always empty for this element when MSAA api is used on problem machine.

The issue is reproduced for these users even when AntiVirus software is turned off and with all IE Add-ons disabled.

Note: Application has valid signature. I also tried to set UIAccess=true in application manifest. It didn't help.

Is anybody aware of the settings or anything else which influence on this behavior?