7dadde5dd6
Implement PGO in pipelines for AMD64 architecture; supply training test scenarios ## References - #3075 - Relevant to speed interests there and other linked issues. ## PR Checklist * [x] Closes #6963 * [x] I work here. * [x] New UIA Tests added and passed. Manual build runs also tested. ## Detailed Description of the Pull Request / Additional comments - Creates a new pipeline run for creating instrumented binaries for Profile Guided Optimization (PGO). - Creates a new suite of UIA tests on the full Windows Terminal app to run PGO training scenarios on instrumented binaries (and incidentally can be used to write other UIA tests later for the full Terminal app.) - Creates a new NuGet artifact to store trained PGO databases (PGD files) at `Microsoft.Internal.Windows.Terminal.PGODatabase` - Creates a new NuGet artifact to supply large-scale test content for automated tests at `Microsoft.Internal.Windows.Terminal.TestContent` - Adjusts the release pipeline to run binaries in PGO optimized mode where content from PGO databases is leveraged at link time to optimize the final release build The following binaries are trained: - OpenConsole.exe - WindowsTerminal.exe - TerminalApp.dll - TerminalConnection.dll - Microsoft.Terminal.Control.dll - Microsoft.Terminal.Remoting.dll - Microsoft.Terminal.Settings.Editor.dll - Microsoft.Terminal.Settings.Model.dll In the future, adding `<PgoTarget>true</PgoTarget>` to a new `vcxproj` file will automatically enroll the DLL/EXE for PGO instrumentation and optimization going forward. Two training test scenarios are implemented: - Smoke test the Terminal by just opening it and typing a bit of text then exiting. (Should help focus on the standard launch path.) - Optimize bulk text output by launching terminal, outputting `big.txt`, then exiting. Additional scenarios can be contributed to the `WindowsTerminal_UIATests` project with the `[TestProperty("IsPGO", "true")]` annotation to add them to the suite of scenarios for PGO. **NOTE:** There are currently no weights applied to the various test scenarios. We will revisit that in the future when/if necessary. ## Validation Steps Performed - [x] - Training run completed at https://dev.azure.com/ms/terminal/_build?definitionId=492&_a=summary - [x] - Optimization run completed locally (by forcing `PGOBuildMode` to `Optimize` on my local machine, manually retrieving the databases with NuGet, and building). - [x] - Validated locally that x86 and ARM64 do not get trained and automatically skip optimization as databases are not present for them. - [x] - Smoke tested optimized binary versus latest releases. `big.txt` output through CMD is ~11-12seconds prior to PGO and just over 8 seconds with PGO.
106 lines
4 KiB
Batchfile
106 lines
4 KiB
Batchfile
setlocal ENABLEDELAYEDEXPANSION
|
|
|
|
echo %TIME%
|
|
|
|
robocopy %HELIX_CORRELATION_PAYLOAD% . /s /NP > NUL
|
|
|
|
echo %TIME%
|
|
|
|
reg add HKLM\Software\Policies\Microsoft\Windows\Appx /v AllowAllTrustedApps /t REG_DWORD /d 1 /f
|
|
|
|
rem enable dump collection for our test apps:
|
|
rem note, this script is run from a 32-bit cmd, but we need to set the native reg-key
|
|
FOR %%A IN (TestHostApp.exe,te.exe,te.processhost.exe,conhost.exe,OpenConsole.exe,WindowsTerminal.exe) DO (
|
|
%systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpFolder /t REG_EXPAND_SZ /d %HELIX_DUMP_FOLDER% /f
|
|
%systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpType /t REG_DWORD /d 2 /f
|
|
%systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpCount /t REG_DWORD /d 10 /f
|
|
)
|
|
|
|
echo %TIME%
|
|
|
|
:: kill dhandler, which is a tool designed to handle unexpected windows appearing. But since our tests are
|
|
:: expected to show UI we don't want it running.
|
|
taskkill -f -im dhandler.exe
|
|
|
|
echo %TIME%
|
|
powershell -ExecutionPolicy Bypass .\EnsureMachineState.ps1
|
|
echo %TIME%
|
|
powershell -ExecutionPolicy Bypass .\InstallTestAppDependencies.ps1
|
|
echo %TIME%
|
|
|
|
set testBinaryCandidates=TerminalApp.LocalTests.dll SettingsModel.LocalTests.dll Conhost.UIA.Tests.dll WindowsTerminal.UIA.Tests.dll
|
|
set testBinaries=
|
|
for %%B in (%testBinaryCandidates%) do (
|
|
if exist %%B (
|
|
set "testBinaries=!testBinaries! %%B"
|
|
)
|
|
)
|
|
|
|
echo %TIME%
|
|
te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError %*
|
|
echo %TIME%
|
|
|
|
powershell -ExecutionPolicy Bypass Get-Process
|
|
|
|
move te.wtl te_original.wtl
|
|
|
|
copy /y te_original.wtl %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
copy /y *.pgc %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
|
|
set FailedTestQuery=
|
|
for /F "tokens=* usebackq" %%I IN (`powershell -ExecutionPolicy Bypass .\OutputFailedTestQuery.ps1 te_original.wtl`) DO (
|
|
set FailedTestQuery=%%I
|
|
)
|
|
|
|
rem The first time, we'll just re-run failed tests once. In many cases, tests fail very rarely, such that
|
|
rem a single re-run will be sufficient to detect many unreliable tests.
|
|
if "%FailedTestQuery%" == "" goto :SkipReruns
|
|
|
|
echo %TIME%
|
|
te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError /select:"%FailedTestQuery%"
|
|
echo %TIME%
|
|
|
|
move te.wtl te_rerun.wtl
|
|
|
|
copy /y te_rerun.wtl %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
|
|
rem If there are still failing tests remaining, we'll run them eight more times, so they'll have been run a total of ten times.
|
|
rem If any tests fail all ten times, we can be pretty confident that these are actual test failures rather than unreliable tests.
|
|
if not exist te_rerun.wtl goto :SkipReruns
|
|
|
|
set FailedTestQuery=
|
|
for /F "tokens=* usebackq" %%I IN (`powershell -ExecutionPolicy Bypass .\OutputFailedTestQuery.ps1 te_rerun.wtl`) DO (
|
|
set FailedTestQuery=%%I
|
|
)
|
|
|
|
if "%FailedTestQuery%" == "" goto :SkipReruns
|
|
|
|
echo %TIME%
|
|
te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError /testmode:Loop /LoopTest:8 /select:"%FailedTestQuery%"
|
|
echo %TIME%
|
|
|
|
powershell -ExecutionPolicy Bypass Get-Process
|
|
|
|
move te.wtl te_rerun_multiple.wtl
|
|
|
|
copy /y te_rerun_multiple.wtl %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
powershell -ExecutionPolicy Bypass .\CopyVisualTreeVerificationFiles.ps1
|
|
|
|
:SkipReruns
|
|
|
|
powershell -ExecutionPolicy Bypass Get-Process
|
|
|
|
echo %TIME%
|
|
powershell -ExecutionPolicy Bypass .\OutputSubResultsJsonFiles.ps1 te_original.wtl te_rerun.wtl te_rerun_multiple.wtl %testnameprefix%
|
|
powershell -ExecutionPolicy Bypass .\ConvertWttLogToXUnit.ps1 te_original.wtl te_rerun.wtl te_rerun_multiple.wtl testResults.xml %testnameprefix%
|
|
echo %TIME%
|
|
|
|
copy /y *_subresults.json %HELIX_WORKITEM_UPLOAD_ROOT%
|
|
|
|
type testResults.xml
|
|
|
|
echo %TIME%
|