terminal/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp

157 lines
6 KiB
C++
Raw Normal View History

Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
#include "pch.h"
#include <UIAutomationCore.h>
#include "TermControlAutomationPeer.h"
#include "TermControl.h"
#include "TermControlAutomationPeer.g.cpp"
#include "XamlUiaTextRange.h"
using namespace Microsoft::Console::Types;
using namespace winrt::Windows::UI::Xaml::Automation::Peers;
namespace UIA
{
using ::ITextRangeProvider;
using ::SupportedTextSelection;
}
namespace XamlAutomation
{
using winrt::Windows::UI::Xaml::Automation::SupportedTextSelection;
using winrt::Windows::UI::Xaml::Automation::Provider::IRawElementProviderSimple;
using winrt::Windows::UI::Xaml::Automation::Provider::ITextRangeProvider;
}
namespace winrt::Microsoft::Terminal::TerminalControl::implementation
{
TermControlAutomationPeer::TermControlAutomationPeer(winrt::Microsoft::Terminal::TerminalControl::implementation::TermControl* owner) :
TermControlAutomationPeerT<TermControlAutomationPeer>(*owner) // pass owner to FrameworkElementAutomationPeer
{
THROW_IF_FAILED(::Microsoft::WRL::MakeAndInitialize<::Microsoft::Terminal::TermControlUiaProvider>(&_uiaProvider, owner, std::bind(&TermControlAutomationPeer::GetBoundingRectWrapped, this)));
};
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
winrt::hstring TermControlAutomationPeer::GetClassNameCore() const
{
return L"TermControl";
}
AutomationControlType TermControlAutomationPeer::GetAutomationControlTypeCore() const
{
return AutomationControlType::Text;
}
winrt::hstring TermControlAutomationPeer::GetLocalizedControlTypeCore() const
{
// TODO GitHub #2142: Localize string
return L"TerminalControl";
}
winrt::Windows::Foundation::IInspectable TermControlAutomationPeer::GetPatternCore(PatternInterface patternInterface) const
{
switch (patternInterface)
{
case PatternInterface::Text:
return *this;
break;
default:
return nullptr;
}
}
#pragma region ITextProvider
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetSelection()
{
SAFEARRAY* pReturnVal;
THROW_IF_FAILED(_uiaProvider->GetSelection(&pReturnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
return WrapArrayOfTextRangeProviders(pReturnVal);
}
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::GetVisibleRanges()
{
SAFEARRAY* pReturnVal;
THROW_IF_FAILED(_uiaProvider->GetVisibleRanges(&pReturnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
return WrapArrayOfTextRangeProviders(pReturnVal);
}
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromChild(XamlAutomation::IRawElementProviderSimple childElement)
{
UIA::ITextRangeProvider* returnVal;
// ScreenInfoUiaProvider doesn't actually use parameter, so just pass in nullptr
THROW_IF_FAILED(_uiaProvider->RangeFromChild(/* IRawElementProviderSimple */ nullptr,
&returnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
auto parentProvider = this->ProviderFromPeer(*this);
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
}
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::RangeFromPoint(Windows::Foundation::Point screenLocation)
{
UIA::ITextRangeProvider* returnVal;
THROW_IF_FAILED(_uiaProvider->RangeFromPoint({ screenLocation.X, screenLocation.Y }, &returnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
auto parentProvider = this->ProviderFromPeer(*this);
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
}
XamlAutomation::ITextRangeProvider TermControlAutomationPeer::DocumentRange()
{
UIA::ITextRangeProvider* returnVal;
THROW_IF_FAILED(_uiaProvider->get_DocumentRange(&returnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
auto parentProvider = this->ProviderFromPeer(*this);
auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
}
Windows::UI::Xaml::Automation::SupportedTextSelection TermControlAutomationPeer::SupportedTextSelection()
{
UIA::SupportedTextSelection returnVal;
THROW_IF_FAILED(_uiaProvider->get_SupportedTextSelection(&returnVal));
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
return static_cast<XamlAutomation::SupportedTextSelection>(returnVal);
}
#pragma endregion
RECT TermControlAutomationPeer::GetBoundingRectWrapped()
{
auto rect = GetBoundingRectangle();
return {
gsl::narrow<LONG>(rect.X),
gsl::narrow<LONG>(rect.Y),
gsl::narrow<LONG>(rect.X + rect.Width),
gsl::narrow<LONG>(rect.Y + rect.Height)
};
}
// Method Description:
// - extracts the UiaTextRanges from the SAFEARRAY and converts them to Xaml ITextRangeProviders
// Arguments:
// - SAFEARRAY of UIA::UiaTextRange (ITextRangeProviders)
// Return Value:
// - com_array of Xaml Wrapped UiaTextRange (ITextRangeProviders)
winrt::com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::WrapArrayOfTextRangeProviders(SAFEARRAY* textRanges)
{
// transfer ownership of UiaTextRanges to this new vector
auto providers = SafeArrayToOwningVector<::Microsoft::Terminal::UiaTextRange>(textRanges);
int count = gsl::narrow<int>(providers.size());
Accessibility: TermControl Automation Peer (#2083) Builds on the work of #1691 and #1915 Let's start with the easy change: - `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl` now. Ok. Now we've got a story to tell here.... ### TermControlAP - the Automation Peer Here's an in-depth guide on custom automation peers: https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers We have a custom XAML element (TermControl). So XAML can't really hold our hands and determine an accessible behavior for us. So this automation peer is responsible for enabling that interaction. We made it a FrameworkElementAutomationPeer to get as much accessibility as possible from it just being a XAML element (i.e.: where are we on the screen? what are my dimensions?). This is recommended. Any functions with "Core" at the end, are overwritten here to tweak this automation peer into what we really need. But what kind of interactions can a user expect from this XAML element? Introducing ControlPatterns! There's a ton of interfaces that just define "what can I do". Thankfully, we already know that we're supposed to be `ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the TermControlAP an `ITextProvider` too. So now we have a way to define what accessible actions can be performed on us, but what should those actions do? Well let's just use the automation providers from ConHost that are now in a shared space! (Note: this is a great place to stop and get some coffee. We're about to hop into the .cpp file in the next section) ### Wrapping our shared Automation Providers Unfortunately, we can't just use the automation providers from ConHost. Or, at least not just hook them up as easily as we wish. ConHost's UIA Providers were written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces ITextProvider and ITextRangeProvider are lined up to be exactly the same. So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML ones. We had two providers, so that means we have two wrappers. #### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore) Each of the functions in the pragma region `ITextProvider` for TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and returning an acceptable version of it. Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need to wrap that too. That's this next section... #### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore) Same idea. We're wrapping everything that we could do with `UiaTextRange` and putting it inside of `XamlUiaTextRange`. ### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider` If you don't know what I just said, please read this background: - #1691: how accessibility works and the general responsibility of these two classes - #1915: how we pulled these Accessibility Providers into a shared area TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text. `UiaTextRange` is specific ranges of text in the display and navigate the text. Thankfully, we didn't do many changes here. I feel like some of it is hacked together but now that we have a somewhat working system, making changes shouldn't be too hard...I hope. #### UiaTextRange We don't have access to the window handle. We really only need it to draw the bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I need to figure out how to get around this. In the meantime, I made the window handle optional. And if we don't have one....well, we need to figure that out. But other than that, we have a `UiaTextRange`. #### ScreenInfoUiaProvider At some point, we need to hook up this automation provider to the WindowUiaProvider. This should help with navigation of the UIA Tree and make everything just look waaaay better. For now, let's just do the same approach and make the pUiaParent optional. This one's the one I'm not that proud of, but it works. We need the parent to get a bounding rect of the terminal. While we figure out how to attach the WindowUiaProvider, we should at the very least be able to get a bunch of info from our xaml automation peer. So, I've added a _getBoundingRect optional function. This is what's called when we don't have a WindowUiaProvider as our parent. ## Validation Steps Performed I've been using inspect.exe to see the UIA tree. I was able to interact with the terminal mostly fine. A few known issues below. Unfortunately, I tried running Narrator on this and it didn't seem to like it (by that I mean WT crashed). Then again, I don't really know how to use narrator other than "click on object" --> "listen voice". I feel like there's a way to get the other interactions with narrator, but I'll be looking into more of that soon. I bet if I fix the two issues below, Narrator will be happy. ## Miscellaneous Known Issues - `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through these. I want to include them in this PR. Fixes #1353.
2019-07-31 01:43:10 +02:00
std::vector<XamlAutomation::ITextRangeProvider> vec;
vec.reserve(count);
auto parentProvider = this->ProviderFromPeer(*this);
for (int i = 0; i < count; i++)
{
auto xutr = winrt::make_self<XamlUiaTextRange>(providers[i].detach(), parentProvider);
vec.emplace_back(xutr.as<XamlAutomation::ITextRangeProvider>());
}
winrt::com_array<XamlAutomation::ITextRangeProvider> result{ vec };
return result;
}
}