[ CLOSE ] Parallelizing tests

Oct 17, 2013 at 3:10 PM
Hello,

I am wondering if we could parallelize some tests with UIAutomation?
It's for testing an app available in 40 countries and I have to check if every version if OK.

Thanks.
Coordinator
Oct 17, 2013 at 4:05 PM
Hi mayela,
what type of parallelization do you mean?

There are two types known to me:
1) FitNesse by Uncle Bob + powerslim by K. Vlasenko
The classic way to run tests is to type PowerShell code in FitNesse tables and the PowerShell slim runner runs tests in parallel, on several hosts and/or several instances per host, even under various credentials.

Personally, I don't like this way as there is the need to write a lot of pipe characters on the FitNesse pages and there are two sets of global variables, FitNesse variables and powerslim global variables, with the need to set values from those to these, with a number of curly braces.

This is parallelization on the command level, i.e. every command/one-liner is running on all instances simultaneously.
You have the result of each line/bunch of code only after all instances finished or failed.

2) I use test step parallelization. I wrote a PowerShell framework (later, I'm planning to rewrite it in C# cmdlets, maybe) that does these actions:
  • reads XML with info about hosts (hostname/IP, credentials if needed, host type in test, etc)
  • reads XML with test steps (name of step, step order, type of host where the step should be run, numbers of 'user stories' to run, etc
  • generates RDP files (the New-UIARemoteDesktopProtocolFile cmdlet available in 0.8.6Beta 5)
    -- hosts already have the Starter RemoteApp loaded in the Registry
  • distributes tests to test hosts, generates Starter.cmd for the step to be run on test hosts
  • start RDP sessions to hosts
    After that, RemoteApp start on hosts and run tests (for example, my tests are Starter.cmd that run powershell.exe with a lot parameters the test runner generated from XML). I'm sitting on the central test console and could see if I would see several GUI tests, even the same tests, from test hosts sent to the central test console.
    After step finished, failed or not started, the central test runner gets test results, logs, anything alse useful and writes to the folder structure on the central test station.
Test hosts should have ~2GB RAM for my tests, the central test console has only 768MB as RemoteApps work on remote hosts almost not consuming resources of the central host.

3) There is also a simple way to run tests in parallel (the user should check twice the code as it's easy to try to get controls of the second app from the window of the first app :)):
calc; calc; calc; sleep -Seconds 3; Get-UIAWindow -n *calc* | Get-UIAButton 1 | Invoke-UIAButtonClick;
Coordinator
Oct 17, 2013 at 4:09 PM
If you are planning to use the way 3, I'd recommend to test thoroughly your tests on one instance, test tests on two instances, turn off highlighters and minimize all that possible, and finally run tests on several instances.
Oct 18, 2013 at 7:58 AM
Hi

First, thank you for replying so fast!

Two, I'm sorry if I make mystake (French guy).

Three, I should have give you more details on this project.

Our client wants to have availability rate of our silverlight app by country. So my coworkers are developping a main program, which start my PS scripts to check availabity of the app for a country. If an app fail, the main program will restart it, regardless of the other scripts potentially started at the same time. My job is to write the PS scripts, which can be parallelized.

So I've tried the third way with two instances, it work well. I propose a script which start 4 or 5 app at the same time, but it won't fit because if an app fail, we can't restart it alone...

So I will try the second way. Can you give me the name of your framework?

Thank you again for your time and ideas.
Coordinator
Oct 18, 2013 at 8:35 AM
Edited Oct 18, 2013 at 8:44 AM
Hi,
I'm not keen on using Silverlight apps and many other modern things as the company I'm working now and the previous one both work in the area of system administration, their UIs are often Win32, WinForms, rarely WPF (system administrators are considered to be austere in terms of usability:)).

1) First of all, what is your test app from the test scripts' perspective? An independent app in the container your coworkers made or a browser add-in?
As you could use the way #3, your application under test is something like an app (.exe, .dll)?

2) What does the phrase about availability for a country mean? Testing with separate Urls, testing with separate language settings?

3) Why do you want to run tests for localization/availability in parallel? It's not a performance test. A big time to test? TMX (an underlaying library to UIA and a separate framework at the same time) now supports platforms, so that you could run tests consequently with results grouped by platform (platform is a string like 'Windows 2012' or 'country #9').

4) If you suffer from failures in tests, there are 'mild' cmdlets that don't stop tests. (If your app fails and test is unable to restart it, test stops of course).
Wait-UIAWindow, Test-UIAControlState, Wait-UIAControlState, Wait-UIA[ControlType]IsEnabled, Wait-UIA[ControlType]IsVisible
in latest betas of 0.8.6, they are of a good quality and some bugs are fixed

Regarding the framework, it's not public yet. My employer does not apply restriction to test engines (only to product-oriented scripts) in terms of publicity, so that I could publish it today, after converting testers' tips to an instruction and some testing of the framework.
I just wanted to say that there might be a way to simplify the task vs brute-force testing of many instances in parallel?
upd: the framework is oriented on domain infrastructure (though could be adapted to workgroups). Also, I need to change a couple of XMLs to separate infrastructure data from company products' related data.
Oct 18, 2013 at 9:48 AM
Edited Oct 18, 2013 at 9:49 AM
Hi

1) It's a silverlight web app, so a browser add-in. I use Firefox to access it, which allow to run profils on different processes.

2) Each version of the app is accessed by an url of this type : https://app.company.fr/COUNTRY . We want to test each urls to calculate availability rate (99,99% for example).

3) It's one of the client's specifications. We have to check 4 scenarios for the 40 countries in 15 minutes. So we have no choice to parallelize the tests. I'm interessed by TMX, I will look to it.

4) I already use some of this cmdlets in my scripts to check if the datas are fully charged (charging the list of all clients for example).
Two things that stopped us to use Way 3.
  1. The first I've tried to explain previously. If we parallelize 4 countries at a time (countries : SENEGAL, GHANA, GABON, CAMEROUN) and GHANA is down but others are up. The client would like to restart the execution of GHANA and still continue the execution of the 3 other ones. I don't see a way to do this, but maybe there is one.
  2. One of the scenarios is to search a client int the database, but each country have his own client's database, unless we put a test client in all database, we have some unique data in each instance.
Hope you understand what I try to explain.
For the framework, I think it was like this one on codeplex so I understand if your employer won't let you make it public.

Thanks again for your time.
Coordinator
Oct 18, 2013 at 10:41 AM
Edited Oct 18, 2013 at 11:04 AM
Well, I got the idea what you need.
The framework could not be published at least until the time after this weekend as I remembered that I need to change here and there, and supply with some samples (and remove product-oriented modules, and put samples instead of them).

However, right now I'll explain the idea you probably could exploit.
In the framework there are an XML file for hosts (let's call it hosts.xml)
Every host is an XML section and have a parameter 'role'. It's a string like 'console', 'agent', 'solo' (the last is for GUI-only tests)
<host><hostname>h1</hostname><role>console</role></host>
<host><hostname>h2</hostname><role>console,agent</role></host>
<host><hostname>h3</hostname><role>agent</role></host>

There are also an XML file with steps, it's called workflow.xml
Every step is a section, one parameter is role.
<step><name>super test 1</name><role>console</role></step>
<step><name>super test 2</name><role>agent,console</role></step>
<step><name>super test 3</name><role>agent</role></step>

These two files are used to separate tests automatically between hosts of various roles. I mean that all hosts of role 'console' run the step the requires hosts of the 'console' role. No precedence, just all of that role.
Hosts h1, h2 run steps 'super test 1', 'super test 2'
hosts h2, h3 run step 'super test 2'
host h3 runs also step 'super test 3'

On the central test console, host lists are just ArrayLists: $ConsoleCollection = New-Object System.Collections.ArrayList;

Now, let's see which fate suffer hosts and test steps.
Test step could be passed - OK, loading test report to the test console
Test step could be interrupted by timeout - FAIL, killing processes
Test step could be hung - FAIL, killing processes
Test step could be not even run (the user visited that host and forgot to log off. Policies don't permit more than one session) - FAIL, nothing to kill, just to close an RDP window
and several other variants with FAIL, loading logs and preliminary data onto the central test console

What the test console does about hosts that failed? It removes such hosts from a collection the step requires (console collection or agent collection, etc).
for example, the the host h1 is removed from the console collection

It was just an intro, the recommendation is much shorter:
use array/collection/queue/list to hold your country pointers (Urls, for example). It is a TODO collection.
  • If a country passes, you need to remove successful link from the TODO collection (maybe, with adding it to OK collection or writing in gold letters to somewhere:)) and load new countries
  • If a country fails, you need to left the failed country for the next test run.
  • If all countries passed, just remove all of them and load next countries.
    If I've got it right, you could set a list of countries/links, and your tests will load countries from the list until the collection is empty.
    Just a big do/while in simple words.
Could it help?
Oct 18, 2013 at 2:44 PM
Hi,

I had a meeting with my coworker to see if your framework could fit our needs.
We have to change some rules, but it definitly worth a try.
If it works, it could simplify the main program so less work to do for my coworkers.

As soon as could work with your framework, I will have to make a proof of concept.
I will let you know about my progress.

Again, thank you! :)
Coordinator
Oct 18, 2013 at 3:57 PM
Edited Oct 21, 2013 at 10:04 AM
Okay, here is the part that is now requires manual intervention:
1) on test hosts (hosts your central test console will be connecting to) you need to load these three regs:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Terminal Server Client]
"RemoteDesktop_SuppressWhenMinimized"=dword:00000002
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Terminal Server Client]
"RemoteDesktop_SuppressWhenMinimized"=dword:00000002
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList]
"fDisabledAllowList"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList\Applications]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList\Applications\Starter]
"Name"="Starter"
"Path"="C:\\TestHome\\Starter.cmd"
As can be seen, you need to create C:\testhome on test hosts (the framework creates) and put there some Starter.cmd (the frameworks generates cmds). The folder path and the command file names are default and now there are no ways to change them (just hard-coded in PowerShell code in a couple of places).
I'm not sure whether restart of hosts is needed.

Of course, Remote Desktop connections should be enabled (Control panel -> system -> remote) and the user your tests will be running under is allowed to access remotely.

2) you need to test the New-UIARemoteDesktopProtocolFile cmdlet:
I use the following code:
                New-UIARemoteDesktopProtocolFile -Path "$($TestHome)\$($hostObject.Hostname).RDP" `
                    -Domain $hostObject.Domain -Username $hostObject.Username -Password $hostObject.Password -Hostname $hostObject.Hostname `
                    -RemoteAppMode -RemoteAppProgram Starter `
                    -DesktopHeight 768 -DesktopWidth 1024 -DisableWallpaper $true -DisableThemes $true -Autoreconnection $true `
                    -ShellWorkingDir "$($TestHome)" `
                    -SmartSizing;
The -Path param is to where output the RDP file. Credentials (the three following parameters) are strings like mydomain testadmin pas$w0rd. Starter is the name of RemoteApp. -ShellWorkingDir is c:\testhome.
Other parameters are used by default, you may play with them if you want. The above is what works for us.

The result is an *.rdp file, inside you should see all these settings and many others. Password is encrypted. You need to test the file manually to ensure that it connects to a test host.
The framework produces such files in the cycle of loading main config data, one file per host.
The framework clicks out about five types of dialogs that could appear (you couldn't save passwords, no certificate, etc, etc) so don't worry about dialogs. You can also click them out manually with check box 'remember the settings' ON where applicable.

3) You can already test some scripts.
A typical Starter.cmd is:
chcp 65001
C:
cd C:\TESTHOME
echo "deleting the finished flag" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
del C:\TESTHOME\finished /f /q > C:\TESTHOME\finished_report.txt 2> C:\TESTHOME\finished_errors.txt
powershell.exe -sta -noninteractive -executionpolicy bypass -command "& { $global:TestHome = 'C:\TESTHOME';  $previousResults = @{}; $previousResults.Add('a','http://vl-at-sp:7777/');  $params = @{}; $params.Add('a','outcast\sp_user_004'); C:\TESTHOME\tests\Common\StartTests.ps1  -ProductCode abcd -Requirement @('1234') -ProductUninstallName ''  -ClearProductData $false  -InstallProduct $false  -BuildPath ''  -MsiName our_setup.msi  -PreviousResults $previousResults  -Parameters $params  -TestSuiteName 'Iteration xyz'  -TestScenarioName 'my test scenario 333'  -TestScenarioId '1234'  -ProductAcronym ABCD  -TestPlatform 'Windows 2008 R2 with SP 2010 Foundation'  -Timeout 1200  -BannerTop 100 ; }" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
echo "test finished in some way" > C:\TESTHOME\finished
logoff
4) Now it's time to test
put binaries in c:\testhome\modules (the default path the frameworks expects, it puts all during deployment of tests), for example c:\testhome\modules\uiautomation*.*
try to run via RDP something like (let's your test runner is called my_runner.ps1):
chcp 65001
C:
cd C:\TESTHOME
echo "deleting the finished flag" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
del C:\TESTHOME\finished /f /q > C:\TESTHOME\finished_report.txt 2> C:\TESTHOME\finished_errors.txt
powershell.exe -sta -noninteractive -executionpolicy bypass -command "& { $global:TestHome = 'C:\TESTHOME';  C:\TESTHOME\my_runner.ps1; }" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
echo "test finished in some way" > C:\TESTHOME\finished
logoff
my_runner.ps1 could be something like
ipmo "$($TestHome)\Modules\UIAutomation";
ipmo "$($TestHome)\Modules\UIAutomation\TMX.dll";
[UIAutomation.Preferences]::BannerLeft = 100;
[UIAutomation.Preferences]::BannerTop = 100;
Show-UIABanner "A Calculator test on $(hostname)";
Start-Process calc -PassThru | Get-UIAWindow | Get-UIAButton 1 | Invoke-UIAButtonClick;
sleep -Seconds 10;
all I wrote in the script you should see from the central test console, the banner and calc.exe.
You easily could run several RDP files to various hosts at the same time, i.e. in parallel.

We used this from Windows 7 and Windows 2008 R2 (the central test console) and to Windows 8 (x64, x86), 2008R2, 2012, 2012R2 Preview (test hosts).
Oct 21, 2013 at 1:43 PM
Hi,

So, this morning I have made a virtual server to run some test on.
The registry key have been loaded, the testhome directory, the Starter.cmd and my_runner.ps1 have been created, and UIAutomation has been loaded.

I have generated some rdp files on my computer to test and it's working!
I can see the calculator and the banner on my sceen.

Now, I'm trying to run several test at the same time. I will come back later with some results.
Coordinator
Oct 21, 2013 at 2:09 PM
Hi,
I'm glad to hear that it works.
In addition, below are fragments of the framework.
How to tell the test on a host which platform to use, for example, my platform are hosts' platforms:
# if host name contains information like w2k12-en-sp2010
New-TMXTestPlatform -Name (hostname) -Id (hostname);
# any string is also possible:
New-TMXTestPlatform -Name $TestPlatform -Id $TestPlatform;
# after that you can use any TMX cmdlets, their test results will be attached to the platform
$null = New-TMXTestSuite -Name "$($suiteName)";
$null = Add-TMXTestScenario -Name "$($scenarioName)" -Id "001";
How to export test results ('automatic' test results are those that generated by UIAutomation automatically via [UIAutomation.Preferences]::EveryCmdletAsTestResult):
Export-TMXTestResults -As XML -Path "$($TestHome)\Reports\rdp.xml" -ExcludeAutomatic;
The central console loads test results:
Import-TMXTestResults -As XML -Path "\\$($hostname)\C$\TestHome\Reports\rdp.xml";
TMX is the same that is wrapped in the corresponding UIAutomation release package.
Oct 21, 2013 at 3:39 PM
Hi,

I succesfully launched 2 instances of your examples with calc, but I can't do it with Firefox...
I work on my computer and one virtual 2008 R2 server with TSE role installed.
It's basicly the same script, except it launch Firefox instead of calc.
ipmo "C:\TestHome\Modules\UIAutomation";
ipmo "C:\TestHome\Modules\UIAutomation\TMX.dll";
$env:MOZ_NO_REMOTE = 1; # To start Firefox in an other process if already exist
[UIAutomation.Preferences]::BannerLeft = 100;
[UIAutomation.Preferences]::BannerTop = 100;
Show-UIABanner "A Firefox test on $(hostname)";
$ProcId = Start-Process 'C:\Program Files (x86)\Mozilla Firefox\firefox.exe' -PassThru | Get-Process | % { echo $_.id };
Get-UIAWindow -pid $ProcId;
sleep -Seconds 5;
Get-UIAWindow -pid $ProcId | Get-UIAButton -AutomationId "Close" | Invoke-UIAButtonClick;
I will see tomorow as soon as I finish my other tasks.

Thanks
Coordinator
Oct 21, 2013 at 4:24 PM
Edited Oct 21, 2013 at 4:25 PM
Hmm, this way to parallelize was initially considered as "several hosts, one instance per host".

The problem that it's not so easy to run two separate instances of Firefox, I tried this https://support.mozilla.org/en-US/questions/941353
but failed to get two processes (the second instance knows about the first):
$firefoxPath = (Get-ChildItem ${env:ProgramFiles(x86)}\Mozilla*Firefox\firefox.exe).FullName
Start-Process $firefoxPath -ArgumentList "-profile","%temp%\ffprof","-no-remote" -PassThru | Get-UIAWindow -Seconds 10;
Also I noticed, that setting the variable in powershell.exe is not enough. Firefox runs outside powershell.exe, so that you need to set it in Control panel/System (i.e., per system, not per session).

You could learn how to run Firefox as a separate process (i.e., several instances) or I'd suggest using Selenium here to run several processes.
There are old cmdlets sepsx.codeplex.com 0.4.7 (I'm not sure that they would work with the latest Firefox). The code is something like:
Start-SeFirefox -Count 3  | ConvertTo-SeAutomationElement | %{ Get-UIAWindowFromHandle $_.Current.NativeWindowHandle; }
If the old version failed to run with the contemporary Firefox, you could try manual call from PowerShell: http://huddledmasses.org/did-you-know-powershell-can-use-selenium/
Oct 22, 2013 at 12:06 PM
Edited Oct 22, 2013 at 12:16 PM
Hi,

I have submited my results to the project manager and he will try to get an esx server to run some test vm.
So, I installed a Windows 7 and a Windows 8 vm to test.
I've created the registry keys, copied the scripts in C:\TestHome, configure rdp but it won't work like yesterday on 2008 R2 Server...

The RemoteApp window open and close a few seconds later without launch starter.cmd.

There is my starter.cmd :
chcp 65001
C:
cd C:\TESTHOME
echo "deleting the finished flag" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
del C:\TESTHOME\finished /f /q > C:\TESTHOME\finished_report.txt 2> C:\TESTHOME\finished_errors.txt
powershell.exe -sta -noninteractive -executionpolicy bypass -command "& { $global:TestHome = 'C:\TESTHOME';  C:\TESTHOME\my_runner.ps1; }" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
#powershell.exe -sta -noninteractive -executionpolicy bypass -command "& { $global:TestHome = 'C:\TESTHOME';  C:\TESTHOME\Total_scenarii_v1.ps1; }" >> C:\TESTHOME\sut_started 2>> C:\TESTHOME\sut_started_errors
echo "test finished in some way" > C:\TESTHOME\finished
logoff
My my_runner.ps1 :
ipmo "C:\TestHome\Modules\UIAutomation";
ipmo "C:\TestHome\Modules\UIAutomation\TMX.dll";
[UIAutomation.Preferences]::BannerLeft = 100;
[UIAutomation.Preferences]::BannerTop = 100;
Show-UIABanner "A Firefox test on $(hostname)";
$FFPath = "C:\Program Files (x86)\Mozilla Firefox\firefox.exe"
Start-Process $FFPath -PassThru | Get-UIAWindow
sleep -Seconds 5;
Get-UIAWindow -pn Firefox | Get-UIAButton -AutomationId "Close" | Invoke-UIAButtonClick;
And the script to create rdp files :
ipmo UIAutomation

$testhome = "C:\TestHome"
$hostObject = New-Object -TypeName PSObject
$hostObject | Add-Member -membertype NoteProperty -Name Hostname -value "XX.XX.XX.XX"
$hostObject | Add-Member -membertype NoteProperty -Name Domain -value "Win8-Total"
$hostObject | Add-Member -membertype NoteProperty -Name Username -value "User"
$hostObject | Add-Member -membertype NoteProperty -Name Password -Value "*******"

New-UIARemoteDesktopProtocolFile `
   -Path "$($TestHome)\$($hostObject.Domain).RDP" `
   -Domain $hostObject.Domain `
   -Username $hostObject.Username `
   -Password $hostObject.Password `
   -Hostname $hostObject.Hostname `
   -RemoteAppMode `
   -RemoteAppProgram "Starter" `
   -DesktopHeight 768 `
   -DesktopWidth 1024 `
   -DisableWallpaper $true `
   -DisableThemes $true `
   -Autoreconnection $true `
   -ShellWorkingDir "$($TestHome)" `
   -SmartSizing;
Do you have some tips to enable Remote App on Win7 Pro and Win8 Pro?

Thanks
Coordinator
Oct 22, 2013 at 12:51 PM
Edited Oct 22, 2013 at 3:22 PM
Hi,
there's nothing special in connection to hosts Windows 7 and 8.
Practically, RDP files are the same with exclusion to hostname/IP.
A typical RDP file is as follows:
screen mode id:i:1
use multimon:i:0
desktopwidth:i:1024
desktopheight:i:768
smart sizing:i:1
session bpp:i:32
winposstr:s:0,1,63,37,1200,900
compression:i:1
keyboardhook:i:2
audiocapturemode:i:0
videoplaybackmode:i:1
connection type:i:7
networkautodetect:i:1
bandwidthautodetect:i:1
displayconnectionbar:i:1
enableworkspacereconnect:i:0
disable wallpaper:i:1
allow font smoothing:i:0
allow desktop composition:i:0
disable full window drag:i:1
disable menu anims:i:1
disable themes:i:1
disable cursor setting:i:0
bitmapcachepersistenable:i:1
audiomode:i:2
redirectprinters:i:0
redirectcomports:i:0
redirectsmartcards:i:0
redirectclipboard:i:0
redirectposdevices:i:0
autoreconnection enabled:i:1
authentication level:i:2
prompt for credentials:i:0
negotiate security layer:i:1
remoteapplicationmode:i:1
remoteapplicationprogram:s:Starter
remoteapplicationcmdline:s:
alternate shell:s:
shell working directory:s:C:\TESTHOME
gatewayhostname:s:
gatewayusagemethod:i:4
gatewaycredentialssource:i:4
gatewayprofileusagemethod:i:0
promptcredentialonce:i:0
use redirection server name:i:0
rdgiskdcproxy:i:0
kdcproxyname:s:
drivestoredirect:s:
full address:s:vl-at-sp.castaway.at.local
username:s:spmgmtaccount
domain:s:outcast
password 51:b:01000000D08C9DDF0115D1118C7A00C04FC297EB010000007316A69DA920C849B349AC509F984D060400000008000000700073007700000003660000C000000010000000E39D08B7665AEA823C94F185F2ED9A3A0000000004800000A000000010000000B190DEA0A4134DA4DAF3947BBB5DFA0418000000456FE4F550C26353780C5EB361018788DF321F2986CD5A67140000003EAEA9B40A02ECA2BB94B92081E89C5E15CDC036
Steps to troubleshoot:
1) log on on a workstation (a test host with Windows 7/8)
2) open cmd.exe, run goto c:\testhome, run starter.cmd
3) if it failed, make it workable. If it works, go to the next item
3.1) If Starter.cmd doesn't work: initially I used a tip from stackoverflow to support Unicode in cmd (an empty string before chcp) as people here used to use cyrillic for name of user stories :) Sometimes test didn't run (starter.cmd) so I removed the empty line. I think you could also remove the string with chcp.
3.2) all that works with the finished file could be removed: this is just a flag for the framework. Try without these lines.
4) create an RDP file, compare it with mine. If there is difference except for domain/user/password/hostname, copy/paste your data in my file and run it
5) Probably, I forgot to mention the following setting on test hosts: fDisabledAllowList. The reg file below is from the Registry of my test machine Windows 8 x86:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList]
"LicenseServers"=hex(7):00,00
"CertificateIssuedBy"=""
"LicensingType"=dword:00000005
"fHasCertificate"=dword:00000000
"CertificateExpiresOn"=""
"CentralLicensing"=dword:00000000
"fDisabledAllowList"=dword:00000001
"CertificateIssuedTo"=""
"CustomRDPSettings"="authentication level:i:2"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList\Applications]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\TSAppAllowList\Applications\Starter]
"Name"="Starter"
"Path"="C:\\TestHome\\Starter.cmd"
6) I use the following Group Policy settings (along with incoming rdp connections enabled in Windows Firewall settings) in a test domain:
Administrative Templates ->
Policy definitions (ADMX files) retrieved from the local machine.
Windows Components/Remote Desktop Services/Remote Desktop Connection Client
Allow .rdp files from unknown publishers Enabled
Allow .rdp files from valid publishers and user's default .rdp settings Enabled
Configure server authentication for client Enabled
Authentication setting: Always connect, even if authentication fails
Windows Components/Remote Desktop Services/Remote Desktop Session Host/Connections
Allow users to connect remotely using Remote Desktop Services Enabled
Automatic reconnection Enabled
Limit number of connections Enabled
RD Maximum Connections allowed 1
Type 999999 for unlimited connections.
Restrict Remote Desktop Services users to a single Remote Desktop Services session Enabled
Set rules for remote control of Remote Desktop Services user sessions Enabled
Options: Full Control without user's permission
Windows Components/Remote Desktop Services/Remote Desktop Session Host/Remote Session Environment
Always show desktop on connection Disabled
Limit maximum number of monitors Enabled
Maximum Monitors 1
Remove "Disconnect" option from Shut Down dialog Enabled
Start a program on connection Enabled
Program path and file name c:\testhome\starter.cmd
Working Directory c:\testhome
Windows Components/Remote Desktop Services/Remote Desktop Session Host/Session Time Limits
Set time limit for disconnected sessions Enabled
End a disconnected session 1 minute
Windows Components/Windows Remote Shell
Allow Remote Shell Access Enabled

I think that you don't need to set up item 6.

Upd:
7) there could also be some problem like a cmd copied via network is blocked by operating system be default (the Unblock button in the Properties dialog of .cmd or .ps1)
Oct 22, 2013 at 3:00 PM
I checked and double checked all steps and it didn't work...

I made some research and found that only Enterprise or Ultimate version of Windows 7/8 can make RemoteApp, I've installed Pro versions.
I just test with a 2008 R2 server and an Ultimate Windows 7 and it works!
I can run 2 instances of Firefox on 2 different hosts.

I submit the results to the project manager and see if we continue with your framework or go to something else.
I come back as soon as possible.
Oct 24, 2013 at 9:00 AM
Hi,

The projet manager made his decision and decide to not use your framework because it would cost a lot of money in licence for test hosts...
I'm keeping an eye on this module and your framework, it could be usefull for an other project.

Thank you for all your time and ideas.

I edit the subject.