Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
/*++
|
|
|
|
Copyright (c) Microsoft Corporation
|
|
|
|
Licensed under the MIT license.
|
|
|
|
|
|
|
|
Module Name:
|
|
|
|
- InteractivityAutomationPeer.h
|
|
|
|
|
|
|
|
Abstract:
|
|
|
|
- This module provides UI Automation access to the ControlInteractivity,
|
|
|
|
to support both automation tests and accessibility (screen
|
|
|
|
reading) applications.
|
|
|
|
- See TermControlAutomationPeer for more details on how UIA is implemented.
|
|
|
|
- This is the primary implementation of the ITextProvider interface, for the
|
|
|
|
TermControlAutomationPeer. The TermControlAutomationPeer will be attached to
|
|
|
|
the actual UI tree, via FrameworkElementAutomationPeer. However, the
|
|
|
|
ControlInteractivity is totally oblivious to the UI tree that might be hosting
|
|
|
|
it. So this class implements the actual text pattern for the buffer, because
|
|
|
|
it has access to the buffer. TermControlAutomationPeer can then call the
|
|
|
|
methods on this class to expose the implementation in the actual UI tree.
|
|
|
|
|
|
|
|
Author(s):
|
|
|
|
- Mike Griese (migrie), May 2021
|
|
|
|
|
|
|
|
--*/
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
#include "ControlInteractivity.h"
|
|
|
|
#include "InteractivityAutomationPeer.g.h"
|
|
|
|
#include "../types/TermControlUiaProvider.hpp"
|
|
|
|
#include "../types/IUiaEventDispatcher.h"
|
|
|
|
#include "../types/IControlAccessibilityInfo.h"
|
|
|
|
|
|
|
|
namespace winrt::Microsoft::Terminal::Control::implementation
|
|
|
|
{
|
|
|
|
struct InteractivityAutomationPeer :
|
|
|
|
public InteractivityAutomationPeerT<InteractivityAutomationPeer>,
|
|
|
|
::Microsoft::Console::Types::IUiaEventDispatcher,
|
|
|
|
::Microsoft::Console::Types::IControlAccessibilityInfo
|
|
|
|
{
|
|
|
|
public:
|
|
|
|
InteractivityAutomationPeer(Microsoft::Terminal::Control::implementation::ControlInteractivity* owner);
|
|
|
|
|
|
|
|
void SetControlBounds(const Windows::Foundation::Rect bounds);
|
|
|
|
void SetControlPadding(const Core::Padding padding);
|
Properly initialize XamlUiaTextRange with ProviderFromPeer (#11501)
## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.
We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).
It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.
The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.
Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes
## References
Introduced in #10051
Closes #11488
## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
2021-10-14 01:01:43 +02:00
|
|
|
void ParentProvider(Windows::UI::Xaml::Automation::Peers::AutomationPeer parentProvider);
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
|
|
|
|
#pragma region IUiaEventDispatcher
|
|
|
|
void SignalSelectionChanged() override;
|
|
|
|
void SignalTextChanged() override;
|
|
|
|
void SignalCursorChanged() override;
|
|
|
|
#pragma endregion
|
|
|
|
|
|
|
|
#pragma region ITextProvider Pattern
|
|
|
|
Windows::UI::Xaml::Automation::Provider::ITextRangeProvider RangeFromPoint(Windows::Foundation::Point screenLocation);
|
|
|
|
Windows::UI::Xaml::Automation::Provider::ITextRangeProvider RangeFromChild(Windows::UI::Xaml::Automation::Provider::IRawElementProviderSimple childElement);
|
|
|
|
com_array<Windows::UI::Xaml::Automation::Provider::ITextRangeProvider> GetVisibleRanges();
|
|
|
|
com_array<Windows::UI::Xaml::Automation::Provider::ITextRangeProvider> GetSelection();
|
|
|
|
Windows::UI::Xaml::Automation::SupportedTextSelection SupportedTextSelection();
|
|
|
|
Windows::UI::Xaml::Automation::Provider::ITextRangeProvider DocumentRange();
|
|
|
|
#pragma endregion
|
|
|
|
|
|
|
|
#pragma region IControlAccessibilityInfo Pattern
|
|
|
|
// Inherited via IControlAccessibilityInfo
|
2021-10-29 16:08:41 +02:00
|
|
|
virtual COORD GetFontSize() const noexcept override;
|
|
|
|
virtual RECT GetBounds() const noexcept override;
|
|
|
|
virtual RECT GetPadding() const noexcept override;
|
|
|
|
virtual double GetScaleFactor() const noexcept override;
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
virtual void ChangeViewport(SMALL_RECT NewWindow) override;
|
|
|
|
virtual HRESULT GetHostUiaProvider(IRawElementProviderSimple** provider) override;
|
|
|
|
#pragma endregion
|
|
|
|
|
|
|
|
TYPED_EVENT(SelectionChanged, IInspectable, IInspectable);
|
|
|
|
TYPED_EVENT(TextChanged, IInspectable, IInspectable);
|
|
|
|
TYPED_EVENT(CursorChanged, IInspectable, IInspectable);
|
|
|
|
|
|
|
|
private:
|
Properly initialize XamlUiaTextRange with ProviderFromPeer (#11501)
## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.
We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).
It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.
The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.
Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes
## References
Introduced in #10051
Closes #11488
## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
2021-10-14 01:01:43 +02:00
|
|
|
Windows::UI::Xaml::Automation::Provider::ITextRangeProvider _CreateXamlUiaTextRange(::ITextRangeProvider* returnVal) const;
|
|
|
|
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
::Microsoft::WRL::ComPtr<::Microsoft::Terminal::TermControlUiaProvider> _uiaProvider;
|
|
|
|
winrt::Microsoft::Terminal::Control::implementation::ControlInteractivity* _interactivity;
|
Properly initialize XamlUiaTextRange with ProviderFromPeer (#11501)
## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.
We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).
It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.
The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.
Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes
## References
Introduced in #10051
Closes #11488
## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
2021-10-14 01:01:43 +02:00
|
|
|
weak_ref<Windows::UI::Xaml::Automation::Peers::AutomationPeer> _parentProvider;
|
Only access ControlInteractivity through the projection (#10051)
## Summary of the Pull Request
This forces the `TermControl` to only use `ControlCore` and `ControlInteractivity` via their WinRT projections. We want this, because WinRT projections can be used across process boundaries. In the future, `ControlCore` and `ControlInteractivity` are going to be living in a different process entirely from `TermControl`. By enforcing this boundary now, we can make sure that they will work seamlessly in the future.
## References
* Tear-out: #1256
* Megathread: #5000
* Project: https://github.com/microsoft/terminal/projects/5
## PR Checklist
* [x] Closes https://github.com/microsoft/terminal/projects/5#card-50760270
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Most all this was just converting pure c++ types to winrt types when possible. I've added a couple helper projections with `til` converters, which made most of this really easy.
The "`MouseButtonState` needs to be composed of `Int32`s instead of `bool`s" is MENTAL. I have no idea why this is, but when I had the control OOP in the sample, that would crash when trying to de-marshal the bools. BODGY.
The biggest changes are in the way the UIA stuff is hooked up. The UiaEngine needs to be attached directly to the `Renderer`, and it can't be easily projected, so it needs to live next to the `ControlCore`. But the `TermControlAutomationPeer` needed the `UiaEngine` to help implement some interfaces.
Now, there's a new layer we've introduced. `InteractivityAutomationPeer` does the `ITextProvider`, `IControlAccessibilityInfo` and the `IUiaEventDispatcher` thing. `TermControlAutomationPeer` now has a
`InteractivityAutomationPeer` stashed inside itself, so that it can ask the interactivity layer to do the real work. We still need the `TermControlAutomationPeer` though, to be able to attach to the real UI tree.
## Validation Steps Performed
The terminal behaves basically the same as before.
Most importantly, I whipped out Accessibility Insights, and the Terminal looks the same as before.
2021-07-19 18:59:30 +02:00
|
|
|
|
|
|
|
til::rectangle _controlBounds{};
|
|
|
|
til::rectangle _controlPadding{};
|
|
|
|
|
|
|
|
winrt::com_array<Windows::UI::Xaml::Automation::Provider::ITextRangeProvider> WrapArrayOfTextRangeProviders(SAFEARRAY* textRanges);
|
|
|
|
};
|
|
|
|
}
|