7dadde5dd6
Implement PGO in pipelines for AMD64 architecture; supply training test scenarios ## References - #3075 - Relevant to speed interests there and other linked issues. ## PR Checklist * [x] Closes #6963 * [x] I work here. * [x] New UIA Tests added and passed. Manual build runs also tested. ## Detailed Description of the Pull Request / Additional comments - Creates a new pipeline run for creating instrumented binaries for Profile Guided Optimization (PGO). - Creates a new suite of UIA tests on the full Windows Terminal app to run PGO training scenarios on instrumented binaries (and incidentally can be used to write other UIA tests later for the full Terminal app.) - Creates a new NuGet artifact to store trained PGO databases (PGD files) at `Microsoft.Internal.Windows.Terminal.PGODatabase` - Creates a new NuGet artifact to supply large-scale test content for automated tests at `Microsoft.Internal.Windows.Terminal.TestContent` - Adjusts the release pipeline to run binaries in PGO optimized mode where content from PGO databases is leveraged at link time to optimize the final release build The following binaries are trained: - OpenConsole.exe - WindowsTerminal.exe - TerminalApp.dll - TerminalConnection.dll - Microsoft.Terminal.Control.dll - Microsoft.Terminal.Remoting.dll - Microsoft.Terminal.Settings.Editor.dll - Microsoft.Terminal.Settings.Model.dll In the future, adding `<PgoTarget>true</PgoTarget>` to a new `vcxproj` file will automatically enroll the DLL/EXE for PGO instrumentation and optimization going forward. Two training test scenarios are implemented: - Smoke test the Terminal by just opening it and typing a bit of text then exiting. (Should help focus on the standard launch path.) - Optimize bulk text output by launching terminal, outputting `big.txt`, then exiting. Additional scenarios can be contributed to the `WindowsTerminal_UIATests` project with the `[TestProperty("IsPGO", "true")]` annotation to add them to the suite of scenarios for PGO. **NOTE:** There are currently no weights applied to the various test scenarios. We will revisit that in the future when/if necessary. ## Validation Steps Performed - [x] - Training run completed at https://dev.azure.com/ms/terminal/_build?definitionId=492&_a=summary - [x] - Optimization run completed locally (by forcing `PGOBuildMode` to `Optimize` on my local machine, manually retrieving the databases with NuGet, and building). - [x] - Validated locally that x86 and ARM64 do not get trained and automatically skip optimization as databases are not present for them. - [x] - Smoke tested optimized binary versus latest releases. `big.txt` output through CMD is ~11-12seconds prior to PGO and just over 8 seconds with PGO. |
||
|---|---|---|
| .. | ||
| AzurePipelinesHelperScripts.ps1 | ||
| ConvertWttLogToXUnit.ps1 | ||
| EnsureMachineState.ps1 | ||
| GenerateTestProjFile.ps1 | ||
| global.json | ||
| HelixTestHelpers.cs | ||
| InstallTestAppDependencies.ps1 | ||
| OutputFailedTestQuery.ps1 | ||
| OutputSubResultsJsonFiles.ps1 | ||
| OutputTestResults.ps1 | ||
| packages.config | ||
| PrepareHelixPayload.ps1 | ||
| ProcessHelixFiles.ps1 | ||
| readme.md | ||
| runtests.cmd | ||
| RunTestsInHelix.proj | ||
| UpdateUnreliableTests.ps1 | ||
This directory contains code and configuration files to run WinUI tests in Helix.
Helix is a cloud hosted test execution environment which is accessed via the Arcade SDK. More details:
WinUI tests are scheduled in Helix by the Azure DevOps Pipeline: RunHelixTests.yml.
The workflow is as follows:
- NuGet Restore is called on the packages.config in this directory. This downloads any runtime dependencies that are needed to run tests.
- PrepareHelixPayload.ps1 is called. This copies the necessary files from various locations into a Helix payload directory. This directory is what will get sent to the Helix machines.
- RunTestsInHelix.proj is executed. This proj has a dependency on Microsoft.DotNet.Helix.Sdk which it uses to publish the Helix payload directory and to schedule the Helix Work Items. The WinUI tests are parallelized into multiple Helix Work Items.
- Each Helix Work Item calls runtests.cmd with a specific query to pass to TAEF which runs the tests.
- If a test is detected to have failed, we run it again, first once, then eight more times if it fails again. If it fails all ten times, we report the test as failed; otherwise, we report it as unreliable, which will show up as a warning, but which will not fail the build. When a test is reported as unreliable, we include the results for each individual run via a JSON string in the original test's errorMessage field.
- TAEF produces logs in WTT format. Helix is able to process logs in XUnit format. We run ConvertWttLogToXUnit.ps1 to convert the logs into the necessary format.
- RunTestsInHelix.proj has EnableAzurePipelinesReporter set to true. This allows the XUnit formatted test results to be reported back to the Azure DevOps Pipeline.
- We process unreliable tests once all tests have been reported by reading the JSON string from the errorMessage field and calling the Azure DevOps REST API to modify the unreliable tests to have sub-results added to the test and to mark the test as "warning", which will enable people to see exactly how the test failed in runs where it did.