Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
// Copyright (c) Microsoft Corporation.
|
|
|
|
// Licensed under the MIT license.
|
|
|
|
|
|
|
|
#include "pch.h"
|
|
|
|
#include <UIAutomationCore.h>
|
2020-02-28 01:37:56 +01:00
|
|
|
#include <LibraryResources.h>
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
#include "TermControlAutomationPeer.h"
|
|
|
|
#include "TermControl.h"
|
|
|
|
#include "TermControlAutomationPeer.g.cpp"
|
|
|
|
|
|
|
|
#include "XamlUiaTextRange.h"
|
2020-11-25 22:02:10 +01:00
|
|
|
#include "../types/UiaTracing.h"
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
|
|
|
|
using namespace Microsoft::Console::Types;
|
|
|
|
using namespace winrt::Windows::UI::Xaml::Automation::Peers;
|
2020-02-25 00:17:55 +01:00
|
|
|
using namespace winrt::Windows::Graphics::Display;
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
|
|
|
|
namespace UIA
|
|
|
|
{
|
|
|
|
using ::ITextRangeProvider;
|
|
|
|
using ::SupportedTextSelection;
|
|
|
|
}
|
|
|
|
|
|
|
|
namespace XamlAutomation
|
|
|
|
{
|
|
|
|
using winrt::Windows::UI::Xaml::Automation::SupportedTextSelection;
|
|
|
|
using winrt::Windows::UI::Xaml::Automation::Provider::IRawElementProviderSimple;
|
|
|
|
using winrt::Windows::UI::Xaml::Automation::Provider::ITextRangeProvider;
|
|
|
|
}
|
|
|
|
|
2021-03-17 21:47:24 +01:00
|
|
|
namespace winrt::Microsoft::Terminal::Control::implementation
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
TermControlAutomationPeer::TermControlAutomationPeer(TermControl* owner,
|
2021-08-11 17:13:38 +02:00
|
|
|
const Core::Padding padding,
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
Control::InteractivityAutomationPeer impl) :
|
2020-02-25 00:17:55 +01:00
|
|
|
TermControlAutomationPeerT<TermControlAutomationPeer>(*owner), // pass owner to FrameworkElementAutomationPeer
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
_termControl{ owner },
|
|
|
|
_contentAutomationPeer{ impl }
|
2019-11-22 01:08:37 +01:00
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
UpdateControlBounds();
|
2021-08-11 17:13:38 +02:00
|
|
|
SetControlPadding(padding);
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
// Listen for UIA signalling events from the implementation. We need to
|
|
|
|
// be the one to actually raise these automation events, so they go
|
|
|
|
// through the UI tree correctly.
|
|
|
|
_contentAutomationPeer.SelectionChanged([this](auto&&, auto&&) { SignalSelectionChanged(); });
|
|
|
|
_contentAutomationPeer.TextChanged([this](auto&&, auto&&) { SignalTextChanged(); });
|
|
|
|
_contentAutomationPeer.CursorChanged([this](auto&&, auto&&) { SignalCursorChanged(); });
|
Properly initialize XamlUiaTextRange with ProviderFromPeer (#11501)
## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.
We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).
It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.
The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.
Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes
## References
Introduced in #10051
Closes #11488
## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
2021-10-14 01:01:43 +02:00
|
|
|
_contentAutomationPeer.ParentProvider(*this);
|
2019-11-22 01:08:37 +01:00
|
|
|
};
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
// Method Description:
|
|
|
|
// - Inform the interactivity layer about the bounds of the control.
|
|
|
|
// IControlAccessibilityInfo needs to know this information, but it cannot
|
|
|
|
// ask us directly.
|
|
|
|
// Arguments:
|
|
|
|
// - <none>
|
|
|
|
// Return Value:
|
|
|
|
// - <none>
|
|
|
|
void TermControlAutomationPeer::UpdateControlBounds()
|
|
|
|
{
|
|
|
|
// FrameworkElementAutomationPeer has this great GetBoundingRectangle
|
|
|
|
// method that's seemingly impossible to recreate just from the
|
|
|
|
// UserControl itself. Weird. But we can use it handily here!
|
|
|
|
_contentAutomationPeer.SetControlBounds(GetBoundingRectangle());
|
|
|
|
}
|
|
|
|
void TermControlAutomationPeer::SetControlPadding(const Core::Padding padding)
|
|
|
|
{
|
|
|
|
_contentAutomationPeer.SetControlPadding(padding);
|
|
|
|
}
|
|
|
|
|
2019-12-11 22:52:49 +01:00
|
|
|
// Method Description:
|
|
|
|
// - Signals the ui automation client that the terminal's selection has changed and should be updated
|
|
|
|
// Arguments:
|
|
|
|
// - <none>
|
|
|
|
// Return Value:
|
|
|
|
// - <none>
|
|
|
|
void TermControlAutomationPeer::SignalSelectionChanged()
|
|
|
|
{
|
2020-03-18 19:01:05 +01:00
|
|
|
UiaTracing::Signal::SelectionChanged();
|
2021-05-19 00:36:40 +02:00
|
|
|
auto dispatcher{ Dispatcher() };
|
|
|
|
if (!dispatcher)
|
|
|
|
{
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
dispatcher.RunAsync(Windows::UI::Core::CoreDispatcherPriority::Normal, [weakThis{ get_weak() }]() {
|
|
|
|
if (auto strongThis{ weakThis.get() })
|
|
|
|
{
|
|
|
|
// The event that is raised when the text selection is modified.
|
|
|
|
strongThis->RaiseAutomationEvent(AutomationEvents::TextPatternOnTextSelectionChanged);
|
|
|
|
}
|
2019-12-11 22:52:49 +01:00
|
|
|
});
|
|
|
|
}
|
|
|
|
|
|
|
|
// Method Description:
|
|
|
|
// - Signals the ui automation client that the terminal's output has changed and should be updated
|
|
|
|
// Arguments:
|
|
|
|
// - <none>
|
|
|
|
// Return Value:
|
|
|
|
// - <none>
|
|
|
|
void TermControlAutomationPeer::SignalTextChanged()
|
|
|
|
{
|
2020-03-18 19:01:05 +01:00
|
|
|
UiaTracing::Signal::TextChanged();
|
2021-05-19 00:36:40 +02:00
|
|
|
auto dispatcher{ Dispatcher() };
|
|
|
|
if (!dispatcher)
|
|
|
|
{
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
dispatcher.RunAsync(Windows::UI::Core::CoreDispatcherPriority::Normal, [weakThis{ get_weak() }]() {
|
|
|
|
if (auto strongThis{ weakThis.get() })
|
|
|
|
{
|
|
|
|
// The event that is raised when textual content is modified.
|
|
|
|
strongThis->RaiseAutomationEvent(AutomationEvents::TextPatternOnTextChanged);
|
|
|
|
}
|
2019-12-11 22:52:49 +01:00
|
|
|
});
|
|
|
|
}
|
|
|
|
|
|
|
|
// Method Description:
|
|
|
|
// - Signals the ui automation client that the cursor's state has changed and should be updated
|
|
|
|
// Arguments:
|
|
|
|
// - <none>
|
|
|
|
// Return Value:
|
|
|
|
// - <none>
|
|
|
|
void TermControlAutomationPeer::SignalCursorChanged()
|
|
|
|
{
|
2020-03-18 19:01:05 +01:00
|
|
|
UiaTracing::Signal::CursorChanged();
|
2021-05-19 00:36:40 +02:00
|
|
|
auto dispatcher{ Dispatcher() };
|
|
|
|
if (!dispatcher)
|
|
|
|
{
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
dispatcher.RunAsync(Windows::UI::Core::CoreDispatcherPriority::Normal, [weakThis{ get_weak() }]() {
|
|
|
|
if (auto strongThis{ weakThis.get() })
|
|
|
|
{
|
|
|
|
// The event that is raised when the text was changed in an edit control.
|
|
|
|
// Do NOT fire a TextEditTextChanged. Generally, an app on the other side
|
|
|
|
// will expect more information. Though you can dispatch that event
|
|
|
|
// on its own, it may result in a nullptr exception on the other side
|
|
|
|
// because no additional information was provided. Crashing the screen
|
|
|
|
// reader.
|
|
|
|
strongThis->RaiseAutomationEvent(AutomationEvents::TextPatternOnTextSelectionChanged);
|
|
|
|
}
|
2019-12-11 22:52:49 +01:00
|
|
|
});
|
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
hstring TermControlAutomationPeer::GetClassNameCore() const
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
|
|
|
return L"TermControl";
|
|
|
|
}
|
|
|
|
|
|
|
|
AutomationControlType TermControlAutomationPeer::GetAutomationControlTypeCore() const
|
|
|
|
{
|
|
|
|
return AutomationControlType::Text;
|
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
hstring TermControlAutomationPeer::GetLocalizedControlTypeCore() const
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
2020-02-28 01:37:56 +01:00
|
|
|
return RS_(L"TerminalControl_ControlType");
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
Windows::Foundation::IInspectable TermControlAutomationPeer::GetPatternCore(PatternInterface patternInterface) const
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
|
|
|
switch (patternInterface)
|
|
|
|
{
|
|
|
|
case PatternInterface::Text:
|
|
|
|
return *this;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return nullptr;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
AutomationOrientation TermControlAutomationPeer::GetOrientationCore() const
|
|
|
|
{
|
|
|
|
return AutomationOrientation::Vertical;
|
|
|
|
}
|
|
|
|
|
|
|
|
hstring TermControlAutomationPeer::GetNameCore() const
|
|
|
|
{
|
|
|
|
// fallback to title if profile name is empty
|
|
|
|
auto profileName = _termControl->GetProfileName();
|
|
|
|
if (profileName.empty())
|
|
|
|
{
|
|
|
|
return _termControl->Title();
|
|
|
|
}
|
|
|
|
return profileName;
|
|
|
|
}
|
|
|
|
|
|
|
|
hstring TermControlAutomationPeer::GetHelpTextCore() const
|
|
|
|
{
|
|
|
|
return _termControl->Title();
|
|
|
|
}
|
|
|
|
|
|
|
|
AutomationLiveSetting TermControlAutomationPeer::GetLiveSettingCore() const
|
|
|
|
{
|
|
|
|
return AutomationLiveSetting::Polite;
|
|
|
|
}
|
|
|
|
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
#pragma region ITextProvider
|
2020-02-28 01:37:56 +01:00
|
|
|
com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetSelection()
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.GetSelection();
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetVisibleRanges()
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.GetVisibleRanges();
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromChild(XamlAutomation::IRawElementProviderSimple childElement)
|
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.RangeFromChild(childElement);
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromPoint(Windows::Foundation::Point screenLocation)
|
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.RangeFromPoint(screenLocation);
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::DocumentRange()
|
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.DocumentRange();
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|
|
|
|
|
2020-02-28 01:37:56 +01:00
|
|
|
XamlAutomation::SupportedTextSelection TermControlAutomationPeer::SupportedTextSelection()
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
{
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
return _contentAutomationPeer.SupportedTextSelection();
|
2020-02-25 00:17:55 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
#pragma endregion
|
Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915
Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
now.
Ok. Now we've got a story to tell here....
### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers
We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.
We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.
But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.
So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)
### Wrapping our shared Automation Providers
Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.
So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.
#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.
Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...
#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea. We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.
### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
classes
- #1915: how we pulled these Accessibility Providers into a shared area
TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.
Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.
#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.
In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.
#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.
This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.
## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.
Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.
## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
these. I want to include them in this PR.
Fixes #1353.
2019-07-31 01:43:10 +02:00
|
|
|
}
|