Implement PGO in pipelines for AMD64 architecture; supply training test scenarios
## References
- #3075 - Relevant to speed interests there and other linked issues.
## PR Checklist
* [x] Closes#6963
* [x] I work here.
* [x] New UIA Tests added and passed. Manual build runs also tested.
## Detailed Description of the Pull Request / Additional comments
- Creates a new pipeline run for creating instrumented binaries for Profile Guided Optimization (PGO).
- Creates a new suite of UIA tests on the full Windows Terminal app to run PGO training scenarios on instrumented binaries (and incidentally can be used to write other UIA tests later for the full Terminal app.)
- Creates a new NuGet artifact to store trained PGO databases (PGD files) at `Microsoft.Internal.Windows.Terminal.PGODatabase`
- Creates a new NuGet artifact to supply large-scale test content for automated tests at `Microsoft.Internal.Windows.Terminal.TestContent`
- Adjusts the release pipeline to run binaries in PGO optimized mode where content from PGO databases is leveraged at link time to optimize the final release build
The following binaries are trained:
- OpenConsole.exe
- WindowsTerminal.exe
- TerminalApp.dll
- TerminalConnection.dll
- Microsoft.Terminal.Control.dll
- Microsoft.Terminal.Remoting.dll
- Microsoft.Terminal.Settings.Editor.dll
- Microsoft.Terminal.Settings.Model.dll
In the future, adding `<PgoTarget>true</PgoTarget>` to a new `vcxproj` file will automatically enroll the DLL/EXE for PGO instrumentation and optimization going forward.
Two training test scenarios are implemented:
- Smoke test the Terminal by just opening it and typing a bit of text then exiting. (Should help focus on the standard launch path.)
- Optimize bulk text output by launching terminal, outputting `big.txt`, then exiting.
Additional scenarios can be contributed to the `WindowsTerminal_UIATests` project with the `[TestProperty("IsPGO", "true")]` annotation to add them to the suite of scenarios for PGO.
**NOTE:** There are currently no weights applied to the various test scenarios. We will revisit that in the future when/if necessary.
## Validation Steps Performed
- [x] - Training run completed at https://dev.azure.com/ms/terminal/_build?definitionId=492&_a=summary
- [x] - Optimization run completed locally (by forcing `PGOBuildMode` to `Optimize` on my local machine, manually retrieving the databases with NuGet, and building).
- [x] - Validated locally that x86 and ARM64 do not get trained and automatically skip optimization as databases are not present for them.
- [x] - Smoke tested optimized binary versus latest releases. `big.txt` output through CMD is ~11-12seconds prior to PGO and just over 8 seconds with PGO.
## Summary of the Pull Request
ControlCore's _renderer (IRenderTarget) is allocated as std::unique_ptr,
but is given to Terminal::CreateFromSettings as a reference.
ControlCore::Close deallocates the _renderer, but if ThrottledFuncs
are still scheduled to call ControlCore::UpdatePatternLocations
it'll cause Terminal::UpdatePatterns to be called, which in turn ends up
accessing the deallocated IRenderTarget reference and lead to a crash.
A proper solution with shared pointers is nontrivial and should be
attempted at a later point in time. This solution moves the teardown of
the _renderer into ControlCore::~ControlCore, where we can be certain
that no further strong references are held by ThrottledFuncs.
## PR Checklist
* [x] Closes#9910
* [x] I work here
* [x] Tests added/passed
## Validation Steps Performed
The crash is a race condition and inherently hard to reproduce.
During validation this PR didn't appear to introduce new crashes.
<!-- Enter a brief description/summary of your PR here. What does it fix/what does it change/how was it tested (even manually, if necessary)? -->
## Summary of the Pull Request
👋 Just a minor change to fix an outdated link.
<!-- Other than the issue solved, is this relevant to any other issues/existing PRs? -->
## References
N/A - subject matter expert
<!-- Please review the items on the PR checklist before submitting-->
## PR Checklist
* [ ] Closes #xxx
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [ ] Tests added/passed
* [ ] Documentation updated. If checked, please file a pull request on [our docs repo](https://github.com/MicrosoftDocs/terminal) and link it here: #xxx
* [ ] Schema updated.
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
<!-- Provide a more detailed description of the PR, other things fixed or any additional comments/features here -->
## Detailed Description of the Pull Request / Additional comments
Original link was demised late 2020. Updated link to be correct.
<!-- Describe how you validated the behavior. Add automated tests wherever possible, but list manual validation steps taken as well -->
## Validation Steps Performed
Implement dropdown menu for choosing a default terminal application from inside the Windows Terminal Settings UI
## PR Checklist
* [x] Closes#9463
* [x] I work here.
* [x] Manual tests passed
* [x] https://github.com/MicrosoftDocs/terminal/issues/314 (and cross reference #9462)
## Detailed Description of the Pull Request / Additional comments
- Adds dropdown menu and a template card for displaying the available default applications (using the same lookup code as the console property sheet `console.dll`)
- Adds model to TSM for adapting the data for display and binding on XAML
- Lookup occurs on every page reload. Persistence only happens on Save Changes.
- Manifest changed for Terminal to add capability to opt-out of registry redirection so we can edit this setting
## Validation Steps Performed
- [x] Flipped the menu and pressed Save Changes and launched cmd from run box... it moved between the two.
- [x] Flipped system theme from light to dark and ensured secondary color looked good
- [x] Flipped the status with a different mechanism (conhost propsheet) and then reopened settings page and confirmed it loaded the updated status
## Summary of the Pull Request
This PR adds some special behavior to the window named "\_quake".
* When creating the quake window, it ignores "initialRows" and "initialCols" and opens on the top half of the monitor.
- It uses `initialPosition` to determine which monitor this is
* It cannot be moved
* It can only be vertically resized on the bottom border.
* It's always in focus mode.
- We should probably have an issue tracking "Allow showing tabs in focus mode"? Maybe?
- This one element is maybe the one I'm least attached to
When renaming a window to "\_quake", it adopts all those behaviors as well. It does not exit focus mode when leaving QM, nor does it resize back. That seemed unnecessary.
## References
* As spec'ed in #9274
* See also #8888
## PR Checklist
* [x] In the pursuit of #653
* [x] I work here
* [ ] Tests added/passed
* [ ] Requires documentation to be updated, but I'm not gonna do any of that till quake mode is totally done.
## Detailed Description of the Pull Request / Additional comments
Note that this doesn't do things like:
* dropdown
* global hotkey summon
* summon to the current monitor
* summon to the current desktop
I'm doing #653 _very_ piecemeal, to try and make the PRs less egregious.
## Validation Steps Performed
* validated that center on launch still works
* validated that QM works on different monitors based on `initialPosition`
* validated entering/exiting QM behaves as expected
## TODO!
* [ ] When snapping the quake window between desktops with <kbd>win+shift+arrow</kbd>, the window doesn't horizontally re-size to the new monitor dimensions. It should.
### ⇒ [doc link](https://github.com/microsoft/terminal/blob/dev/migrie/s/653-quake-mode/doc/specs/%23653%20-%20Quake%20Mode/%23653%20-%20Quake%20Mode.md) ⇐
## Summary of the Pull Request
After reading through 114+ comments in #653 and related issues, I think I've finally wrapped my head around all the possible scenarios for quake mode. <!-- Speak now or forever hold your peace. --> This also includes "minimize to tray", because the two are a powerful combination. With the work already prototyped in [`dev/migrie/f/653-QUAKE-MODE`](https://github.com/microsoft/terminal/tree/dev/migrie/f/653-QUAKE-MODE), [I'm starting to believe](https://j.gifs.com/58vKNx.gif) that we could actually land this in 2.0.
### Abstract
> Many existing terminals support a feature whereby a user can press a keybinding
> anywhere in the OS, and summon their terminal application. Oftentimes the act of
> summoning this window is accompanied by a "dropdown" animation, where the window
> slides in to view from the top of the screen. This global summon action is often
> referred to as "quake mode", a reference to the videogame Quake who's console
> slid in from the top.
>
> This spec addresses both of the following two issues:
> * "Quake Mode" ([#653])
> * "Minimize to tray" ([#5727])
## PR Checklist
* [x] Specs: #653, #5727
* [x] References: #5000, #4472, #2227, #7240, #8135
* [x] I work here
## Detailed Description of the Pull Request / Additional comments
_\*<sup>\*</sup><sub>\*</sub> read the spec <sub>\*</sub><sup>\*</sup>\*_
Using Pickers from an elevated application yields an
ERROR_ACCESS_DENIED. Of course it does: it was designed for the modern
app platform.
Using the common dialog infrastructure has some downsides¹, but it
doesn't crash and is just as flexible.
I've added some fun templated functions that help us with the
complexity.
Fixes#8957
¹You've got to use raw COM, and it runs in-proc instead of out-of-proc.
## Validation Steps Performed
I tested every picker.
Does what it says on the can.
This is a follow up to #9472. Now that we have a control .lib, we can add tests for it.
Unfortunately, the `TermControl` itself is a horrible mess. So this new unittest lib is empty for now. I'm working on actual tests as a part of #6842, but this PR is here to keep the diffs smaller.
Also, apparently `server.vcxproj` had the wrong GUID in it.
* [x] I work here
* [x] Adds tests
This adds [`XamlStyler.Console`] to our solution, and calls it when we
format the code, to also format
our .xaml files.
* `XamlStyler.Console` is a dotnet tool so it needs to be restored with
`dotnet tool restore`
* I've added a set of rules to approximately follow [@cmaneu's XAML guidelines].
Those guidelines also recommend things based on the code-behind, which
this tool can't figure out, but also _don't matter that much_.
* There's an extra step to strip BOMs from the output, since Xaml Styler
adds a BOM by default. Some had them before and others didn't. BOMs
have been nothing but trouble though.
[`XamlStyler.Console`]: https://github.com/Xavalon/XamlStyler
[@cmaneu's XAML guidelines]: https://github.com/cmaneu/xaml-coding-guidelines
This commit introduces a new build configuration, "Fuzzing", which
enables the new address sanitizer (shipped in VS 16.9) and code
coverage over the entire solution. Only a small subset of projects
(those comprising original conhost, right now) are selected to build in
this configuration, and even then only in Fuzzing|x64.
It also adds a fuzzing-adapted build of conhost, which makes no server
connections and handles no client applications. To do this, I've
replicated a bit of the console startup routine into fuzzmain.cpp and
made up some fake data. This is the bare minimum required to boot up
Win32 interactivity (or VT interactivity!) and pretend that a process
has connected.
If we don't pretend that a process has connected, "conhost" will exit
immediately. If we don't forge the process list, conhost will exit. If
we can't provide a server handle, we can't provide a "device comm".
Minor changes were necessary to server/host such that they would accept
a preexisting "device comm". We use this new behavior to provide a
"null" one that only hangs up threads and otherwise responds to requests
successfully.
This fuzzing-adapted build links LLVM's libFuzzer, which is an excellent
coverage-based fuzzer that will produce a corpus of inputs that exercise
unique codepaths. Eventually, we can use this to generate known-"good"
inputs for anything.
I've gone ahead and added a fuzz function that yeets bytes directly into
WriteCharsLegacy, which was the original reason I went down this path.
The implementation of LLVMFuzzerTestOneInput should be replaced with
whatever you want to fuzz.
- Implements the default application behavior and handoff mechanisms
between console and terminal. The inbox portion is done already. This
adds the ability for our OpenConsole.exe to accept the incoming server
connection from the Windows OS, stand up a PTY session, start the
Windows Terminal as a listener for an incoming connection, and then
send it the incoming PTY connection for it to launch a tab.
- The tab is launched with default settings at the moment.
- You must configure the default application using the `conhost.exe`
propsheet or with the registry keys. Finishing the setting inside
Windows Terminal will be a todo after this is complete. The OS
Settings panel work to surface this setting is a dependency delivered
by another team and you will not see it here.
## Validation Steps Performed
- [x] Manual adjust of registry keys to the delegation conhost/terminal
behavior
- [x] Adjustment of the delegation options with the propsheet
- [x] Launching things from the run box manually and watching them show
in Terminal
- [x] Launching things from shortcuts and watching them show in the
Terminal
Documentation on how it works will be a TODO post completion in #9462
References #7414 - Default Terminal spec
Closes#492
I've also taken the opportunity to kill the xxxFullPDB trick. The
intermediate PDB is allowed to remain "vc141.pdb" or whatever it wants
to be. PDBs are now simply named after their projects, as was always
tradition.
This PR adds support for the VT line rendition attributes, which allow
for double-width and double-height line renditions. These renditions are
enabled with the `DECDWL` (double-width line) and `DECDHL`
(double-height line) escape sequences. Both reset to the default
rendition with the `DECSWL` (single-width line) escape sequence. For now
this functionality is only supported by the GDI renderer in conhost.
There are a lot of changes, so this is just a general overview of the
main areas affected.
Previously it was safe to assume that the screen had a fixed width, at
least for a given point in time. But now we need to deal with the
possibility of different lines have different widths, so all the
functions that are constrained by the right border (text wrapping,
cursor movement operations, and sequences like `EL` and `ICH`) now need
to lookup the width of the active line in order to behave correctly.
Similarly it used to be safe to assume that buffer and screen
coordinates were the same thing, but that is no longer true. Lots of
places now need to translate back and forth between coordinate systems
dependent on the line rendition. This includes clipboard handling, the
conhost color selection and search, accessibility location tracking and
screen reading, IME editor positioning, "snapping" the viewport, and of
course all the rendering calculations.
For the rendering itself, I've had to introduce a new
`PrepareLineTransform` method that the render engines can use to setup
the necessary transform matrix for a given line rendition. This is also
now used to handle the horizontal viewport offset, since that could no
longer be achieved just by changing the target coordinates (on a double
width line, the viewport offset may be halfway through a character).
I've also had to change the renderer's existing `InvalidateCursor`
method to take a `SMALL_RECT` rather than a `COORD`, to allow for the
cursor being a variable width. Technically this was already a problem,
because the cursor could occupy two screen cells when over a
double-width character, but now it can be anything between one and four
screen cells (e.g. a double-width character on the double-width line).
In terms of architectural changes, there is now a new `lineRendition`
field in the `ROW` class that keeps track of the line rendition for each
row, and several new methods in the `ROW` and `TextBuffer` classes for
manipulating that state. This includes a few helper methods for handling
the various issues discussed above, e.g. position clamping and
translating between coordinate systems.
## Validation Steps Performed
I've manually confirmed all the double-width and double-height tests in
_Vttest_ are now working as expected, and the _VT100 Torture Test_ now
renders correctly (at least the line rendition aspects). I've also got
my own test scripts that check many of the line rendition boundary cases
and have confirmed that those are now passing.
I've manually tested as many areas of the conhost UI that I could think
of, that might be affected by line rendition, including things like
searching, selection, copying, and color highlighting. For
accessibility, I've confirmed that the _Magnifier_ and _Narrator_
correctly handle double-width lines. And I've also tested the Japanese
IME, which while not perfect, is at least useable.
Closes#7865
Implement the `XTPUSHSGR` and `XTPOPSGR` control sequences (see #1796).
This change adds a new pair of methods to `ITermDispatch`:
`PushGraphicsRendition` and `PopGraphicsRendition`, and then plumbs the
change through `AdaptDispatch`, `TerminalDispatch`, `ITerminalApi` and
`TerminalApi`.
The stack logic is encapsulated in the `SgrStack` class, to allow it to
be reused between the two APIs (`AdaptDispatch` and `TerminalDispatch`).
Like xterm, only ten levels of nesting are supported.
The stack is implemented as a "ring stack": if you push when the stack
is full, the bottom of the stack will be dropped to make room.
Partial pushes (see the description of `XTPUSHSGR` in Issue #1796) are
implemented per xterm spec.
## Validation Steps Performed
Tests added, plus manual verification of the feature.
Closes#1796
### ⇒ [doc link](https://github.com/microsoft/terminal/blob/dev/migrie/s/4472-window-management/doc/specs/%235000%20-%20Process%20Model%202.0/%234472%20-%20Windows%20Terminal%20Session%20Management.md) ⇐
## Summary of the Pull Request
This is a more detailed spec for two parts of the "Process Model 2.0" work that's being tracked in #5000. In particular, this spec focuses on the management of Windows Terminal windows, including opening new tabs in existing windows.
Largely, the reader is expected to have already read the spec in progress in #7240, and already be familiar with the concept of "Monarch" and "Peasant" windows as introduced by that spec. For that reason, ⚠ **THIS PR IS TARGETING THE BRANCH FOR #7240** ⚠.
### Abstract
> This document is intended to serve as an addition to the [Process Model 2.0
> Spec]. That document provides a big-picture overview of changes to the entirety
> of the Windows Terminal process architecture, including both the split of
> window/content processes, as well as the introduction of monarch/peasant
> processes. The focus of that document was to identify solutions to a set of
> scenarios that were closely intertwined, and establish these solutions would
> work together, without preventing any one scenario from working. What that
> document did not do was prescribe specific solutions to the given scenarios.
>
> This document offers a deeper dive on a subset of the issues in [#5000], to
> describe specifics for managing multiple windows with the Windows Terminal. This
> includes features such as:
>
> * Run `wt` in the current window ([#4472])
> * Single Instance Mode ([#2227])
## PR Checklist
* [x] Specs: #4472, Specs #2227
* [x] References: #5000, #4472, #2227, #7240
* [x] I work here
## Detailed Description of the Pull Request / Additional comments
_\*<sup>\*</sup><sub>\*</sub> read the spec <sub>\*</sub><sup>\*</sup>\*_
### Why are these two separate documents?
I felt that the spec that is currently in review in #7240 and this doc should remain separate, yet closely related documents. #7240 is more about showing how this large set of problems discussed in #5000 can all be solved technically, and how those solutions can be used together. It establishes that none of the proposed solutions for components of #5000 will preclude the possibility of other components being solved. What it does _not_ do however is drill too deeply on the user experience that will be built on top of those architectural changes.
This doc on the other hand focuses more closely on a pair of scenarios, and establishes how those scenarios will work technically, and how they'll be exposed to the user.
### TODO:
* [x] A thought - How will we handle arguments like `--fullscreen`, `--initialSize r,c`? They only apply when creating a new window, right?
* [x] When a `wt -s 1 split-pane` command is executed, we'll need to make sure to not _also_ create a new tab
### ⇒ [doc link](https://github.com/microsoft/terminal/blob/dev/migrie/s/5000/doc/specs/%235000%20-%20Process%20Model%202.0/%235000%20-%20Process%20Model%202.0.md) ⇐
## Summary of the Pull Request
This spec is _exceptionally long_, and is currently a work in progress. There are a few more things I'd like to have experimentally verified (though, I'm fairly certain they _will_ work, with the right combination of flags and such). Additionally, a few sections have remaining TODOs before the spec is finished. However, this spec is already fairly long, and I want to give people as much time to get their eyes on it as possible.
### Abstract
>
> The Windows Terminal currently exists as a single process per window, with one
> connection per terminal pane (which could be an additional conpty process and
> associated client processes). This model has proven effective for the simple
> windowing we've done so far. However, in order to support scenarios like
> dragging tabs into other windows, or having one top-level window with different
> elevation levels within it, this single process model will not be sufficient.
>
> This spec outlines changes to the Terminal process model in order to enable the
> following scenarios:
>
> * Tab Tearoff/ Reattach ([#1256])
> * Run `wt` in the current window ([#4472])
> * Single Instance Mode ([#2227])
> * Quake Mode ([#653])
> * Mixed Elevation ([#1032] & [#632])
## PR Checklist
* [x] Specs: #5000
* [x] References: #1256, #4472, #2227, #653, #1032, #632, #492
* [x] I work here
## Detailed Description of the Pull Request / Additional comments
_\*<sup>\*</sup><sub>\*</sub> read the spec <sub>\*</sub><sup>\*</sup>\*_
### Plurals and paste tenses
In the past, plurals `foo`+`s` and past tenses `foo`+`ed` were
automatically tolerated. This turned out to be a bad design choice on my
part.
The basic example is that `potatos` would sometimes be treated as a
mistake and sometimes not (depending on the presence of `potato`).
You can see in this PR, that this logic resulted in `Applys` being
accepted as a word along with `AppContainered` -- there's nothing
intrinsically wrong w/ the latter, but unfortunately in order to screen
out the former, my shortcut just couldn't stick around. This means that
the `dictionary`/`expect` files will grow perhaps by a tiny bit, but as
you can see, not really by much.
This is also why `thereses` (a user) was accepted as a word in the past
(therese is in the base dictionary, so `therese` + `s` was acceptable).
### Pull requests
When GitHub initially introduced GitHub Actions, the event for
`pull_request` was created without enough permission for a tool like
this to work properly. I worked around that by using the `schedule`
event. In 2020, they introduced a replacement event
`pull_request_target` which has enough permission. This means that I can
stop relying on the `schedule` event.
### Miscellaneous
* I've folded together some `expect/` files since now is as good a time
as any.
* I've included a hint about `excludes.txt` (I added a similar one for
our primary repo recently, and it came up this week in
`microsoft/terminal` -- @zadjii-msft)
* I've standardized on a default of `.github/actions/spelling` to make
the out of the box experience easier for new adopters, so I'm applying
that change here -- if you're attached to the old directory name,
specifying it is still supported. -- note the directory rename may
cause a merge conflict for people with open PRs and changes to the
contents, this shouldn't be a big problem.
2021-02-03 11:17:38 -08:00
Renamed from .github/actions/spell-check/expect/expect.txt (Browse further)
