## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.
We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).
It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.
The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.
Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes
## References
Introduced in #10051Closes#11488
## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
## Summary of the Pull Request
This implements `GetAttributeValue` and `FindAttribute` for `UiaTextRangeBase` (the shared `ITextRangeProvider` for Conhost and Windows Terminal). This also updates `UiaTracing` to collect more useful information on these function calls.
## References
#7000 - Epic
[Text Attribute Identifiers](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-textattribute-ids)
[ITextRangeProvider::GetAttributeValue](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-getattributevalue)
[ITextRangeProvider::FindAttribute](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-findattribute)
## PR Checklist
* [X] Closes#2161
* [X] Tests added/passed
## Detailed Description of the Pull Request / Additional comments
- `TextBuffer`:
- Exposes a new `TextBufferCellIterator` that takes in an end position. This simplifies the logic drastically as we can now use this iterator to navigate through the text buffer. The iterator can also expose the position in the buffer.
- `UiaTextRangeBase`:
- Shared logic & helper functions:
- Most of the text attributes are stored as `TextAttribute`s in the text buffer. To extract them, we generate an attribute verification function via `_getAttrVerificationFn()`, then use that to verify if a given cell has the desired attribute.
- A few attributes are special (i.e. font name, font size, and "is read only"), in that they are (1) acquired differently and (2) consistent across the entire text buffer. These are handled separate from the attribute verification function.
- `GetAttributeValue`: Retrieve the attribute verification of the first cell in the range. Then, verify that the entire range has that attribute by iterating through the text range. If a cell does not have that attribute, return the "reserved mixed attribute value".
- `FindAttribute`: Iterate through the text range and leverage the attribute verification function to find the first contiguous range with that attribute. Then, make the end exclusive and output a `UiaTextRangeBase`. This function must be able to perform a search backwards, so we abstract the "start" and "end" into `resultFirstAnchor` and `resultSecondAnchor`, then perform post processing to output a valid `UiaTextRangeBase`.
- `UiaTracing`:
- `GetAttributeValue`: Log uia text range, desired attribute, resulting attribute metadata, and the type of the result.
- `FindAttribute`: Log uia text range, desired attribute and attribute metadata, if we were searching backwards, the type of the result, and the resulting text range.
- `AttributeType` is a nice way to understand/record if the result was either of the reserved UIA values, a normal result, or an error.
- `UiaTextRangeTests`:
- `GetAttributeValue`:
- verify that we know which attributes we support
- test each of the known text attributes (expecting 100% code coverage for `_getAttrVerificationFn()`)
- `FindAttribute`:
- test each of the known _special_ text attributes
- test `IsItalic`. NOTE: I'm explicitly only testing one of the standard text attributes because the logic is largely the same between all of them and they leverage `_getAttrVerificationFn()`.
## Validation Steps Performed
- @codeofdusk has been testing this Conhost build
- Tests added for Conhost and shared implementation
- Windows Terminal changes were manually verified using accessibility insights and NVDA
**BE NOT AFRAID**. I know that there's 107 files in this PR, but almost
all of it is just find/replacing `TerminalControl` with `Control`.
This is the start of the work to move TermControl into multiple pieces,
for #5000. The PR starts this work by:
* Splits `TerminalControl` into separate lib and dll projects. We'll
want control tests in the future, and for that, we'll need a lib.
* Moves `ICoreSettings` back into the `Microsoft.Terminal.Core`
namespace. We'll have other types in there soon too.
* I could not tell you why this works suddenly. New VS versions? New
cppwinrt version? Maybe we're just better at dealing with mdmerge
bugs these days.
* RENAMES `Microsoft.Terminal.TerminalControl` to
`Microsoft.Terminal.Control`. This touches pretty much every file in
the sln. Sorry about that (not sorry).
An upcoming PR will move much of the logic in TermControl into a new
`ControlCore` class that we'll add in `Microsoft.Terminal.Core`.
`ControlCore` will then be unittest-able in the
`UnitTests_TerminalCore`, which will help prevent regressions like #9455
## Detailed Description of the Pull Request / Additional comments
You're really gonna want to clean the sln first, then merge this into
your branch, then rebuild. It's very likely that old winmds will get
left behind. If you see something like
```
Error MDM2007 Cannot create type
Microsoft.Terminal.TerminalControl.KeyModifiers in read-only metadata
file Microsoft.Terminal.TerminalControl.
```
then that's what happened to you.
This PR hooks up the existing UIA implementation to the WPF control. Some existing code that was specific to the UWP terminal control could be shared so that has been refactored to a common location as well.
## Validation Steps Performed
WPF control was brought up in UISpy and the UIA tree was verified. NVDA was then used to check that screen readers were operating properly.
Moved `FindText` to `UiaTextRangeBase`. Now that Search is a shared component (thanks #3279), I can just reuse it basically as-is.
#3279 - Make Search a shared component
#4018 - UiaTextRange Refactor
I removed it from the two different kinds of UiaTextRange and put it in the base class.
I needed a very minor change to ensure we convert from an inclusive end (from Search) to an exclusive end (in UTR).
Worked with `FindText` was globally messed with in windows.h. So we had to do a few weird things there (thanks Michael).
No need for additional tests because it _literally_ just sets up a Searcher and calls it.
## Summary of the Pull Request
This pull request is intended to achieve the following goals...
1) reduce duplicate code
2) remove static functions
3) improve readability
4) improve reliability
5) improve code-coverage for testing
6) establish functioning text buffer navigation in Narrator and NVDA
This also required a change to the wrapper class `XamlUiaTextRange` that has been causing issues with Narrator and NVDA.
See below for additional context.
## References
#3976 - I believe this might have been a result of improperly handling degenerate ranges. Fixed here.
#3895 - reduced the duplicate code. No need to separate into different files
#2160 - same as #3976 above
#1993 - I think just about everything is no longer static
## PR Checklist
* [x] Closes#3895, Closes#1993, Closes#3976, Closes#2160
* [x] CLA signed
* [x] Tests added/passed
## Detailed Description of the Pull Request / Additional comments
### UiaTextRange
- converted endpoints into the COORD system in the TextBuffer coordinate space
- `start` is inclusive, `end` is exclusive. A degenerate range is when start == end.
- all functions are no longer static
- `MoveByUnit()` functions now rely on `MoveEndpointByUnit()` functions
- removed unnecessary typedefs like `Endpoint`, `ScreenInfoRow`, etc..
- relied more heavily on existing functionality from `TextBuffer` and `Viewport`
### XamlUiaTextRange
- `GetAttributeValue()` must return a special HRESULT that signifies that the requested attribute is not supported. This was the cause of a number of inconsistencies between Narrator and NVDA.
- `FindText()` should return `nullptr` if nothing was found. #4373 properly fixes this functionality now that Search is a shared module
### TextBuffer
- Word navigation functionality is entirely in `TextBuffer` for proper abstraction
- a total of 6 functions are now dedicated to word navigation to get a good understanding of the differences between a "word" in Accessibility and a "word" in selection
As an example, consider a buffer with this text in it:
" word other "
In selection, a "word" is defined as the range between two delimiters, so the words in the example include [" ", "word", " ", "other", " "].
In accessibility , a "word" includes the delimiters after a range of readable characters, so the words in the example include ["word ", "other "].
Additionally, accessibility word navigation must be able to detect if it is on the first or last word. This resulted in a slight variant of word navigation functions that return a boolean instead of a COORD.
Ideally, these functions can be consolidated, but that is too risky for a PR of this size as it can have an effect on selection.
### Viewport
- the concept of `EndExclusive` is added. This is used by UiaTextRange's `end` anchor as it is exclusive. To signify that the last character in the buffer is included in this buffer, `end` must be one past the end of the buffer. This is `EndExclusive`
- Since many functions check if the given `COORD` is in bounds, a flag must be set to allow `EndExclusive` as a valid `COORD` that is in bounds.
### Testing
- word navigation testing relies more heavily on TextBuffer tests
- additional testing was created for non-movement focused functions of UiaTextRange
- The results have been compared to Microsoft Word and some have been verified by UiAutomation/Narrator contacts as expected results.
## Validation Steps Performed
Tests pass
Narrator works
NVDA works
This commit cleans up and deduplicates all of the common build
preamble/postamble across exe, dll, lib and c++/winrt projects.
The following specific changes have been made:
* All projects now define their ConfigurationType
* All projects now set all their properties *before* including a common
build file (or any other build files)
* cppwinrt.pre and cppwinrt.post now delegate most of their
configuration to common.pre and common.post
* (becuase of the above,) all build options are conserved between
console and c++/winrt components, including specific warnings and
preprocessor definitions.
* More properties that are configurable per-project are now
conditioned so the common props don't override them.
* The exe, dll, exe.or.dll, and lib postincludes have been merged into
pre or post and switched based on condition as required
* Shared items (-shared, -common) are now explicitly vcxitems instead of
vcxproj files.
* The link line is now manipulated after Microsoft.Cpp sets it, so the
libraries we specify "win". All console things link first against
onecore_apiset.lib.
* Fix all compilation errors caused by build unification
* Move CascadiaPackage's resources into a separate item file
Fixes#922.
Refactors the accessibility providers (ScreenInfoUiaProvider and UiaTextRange) into a better separated model between ConHost and Windows Terminal.
ScreenInfoUiaProviderBase and UiaTextRangeBase are introduced. ConHost and Windows Terminal implement their own versions of ScreenInfoUiaProvider and UiaTextRange that inherit from their respective base classes.
WindowsTerminal's ScreenInfoUiaProvider --> TermControlUiaProvider
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes#1353.
Carlos Zamora
Mike Griese
Zoey Riordan
Dustin L. Howett (MSFT)