maxmustermann
/
terminal
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Properly initialize XamlUiaTextRange with ProviderFromPeer (#11501) 

## Summary of the Pull Request
As a part of the Interactivity split, `TermControlAutomationPeer` had to be split into `TermControlAutomationPeer` (TCAP) and `InteractivityAutomationPeer` (IAP). Just about all of the functions in `InterativityAutomationPeer` operate by calling the non-XAML UIA Provider then wrapping the resulting `UIATextRange` into a XAML format (a `XamlUiaTextRange` [XUTR]). As a part of that XUTR constructor, we need a reference to the parent provider.

We generally get that via `ProviderFromPeer()`, but IAP's `ProviderFromPeer()` returned null (presumably because IAP isn't in the UI tree, whereas TCAP is directly registered as the automation peer for the `TermControl`).

It looks like some screen readers didn't care (like NVDA, though there may be a chance we just didn't encounter an issue just yet), but Narrator definitely did.

The fix was to provide XUTR constructors the `ProviderFromPeer` from TCAP, _not_ IAP. To accomplish this, IAP now holds a weak reference to TCAP, and provides the `ProviderFromPeer` when needed. We can't cache this result because there is no guarantee that it won't change.

Some miscellaneous changes include:
- `TermControl::OnCreateAutomationPeer` now returns the existing auto peer instead of always creating a new one
- `TCAP::WrapArrayOfTextRangeProviders` was removed as it was unused (normally, this would be directly affected by the main `ProviderFromPeer` change here)
- `XUTR::GetEnclosingElement` is now hooked up to trace logging for debugging purposes

## References
Introduced in #10051
Closes #11488 

## Validation Steps Performed
✅ Narrator scan mode now works (verified with character, word, and line navigation)
✅ NVDA movement still works (verified with word and line navigation)
This commit is contained in:
Carlos Zamora 2021-10-13 16:01:43 -07:00 committed by GitHub
parent a89c3e2f85
commit 02ac246807
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 33 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
src/cascadia/TerminalControl/InteractivityAutomationPeer.cpp
View File

@ -43,6 +43,10 @@ namespace winrt::Microsoft::Terminal::Control::implementation
{
_controlPadding = padding;
}
void InteractivityAutomationPeer::ParentProvider(AutomationPeer parentProvider)
{
_parentProvider = parentProvider;
}
// Method Description:
// - Signals the ui automation client that the terminal's selection has
@ -110,30 +114,21 @@ namespace winrt::Microsoft::Terminal::Control::implementation
// ScreenInfoUiaProvider doesn't actually use parameter, so just pass in nullptr
THROW_IF_FAILED(_uiaProvider->RangeFromChild(/* IRawElementProviderSimple */ nullptr,
&returnVal));
const auto parentProvider = this->ProviderFromPeer(*this);
const auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
return _CreateXamlUiaTextRange(returnVal);
}
XamlAutomation::ITextRangeProvider InteractivityAutomationPeer::RangeFromPoint(Windows::Foundation::Point screenLocation)
{
UIA::ITextRangeProvider* returnVal;
THROW_IF_FAILED(_uiaProvider->RangeFromPoint({ screenLocation.X, screenLocation.Y }, &returnVal));
const auto parentProvider = this->ProviderFromPeer(*this);
const auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
return _CreateXamlUiaTextRange(returnVal);
}
XamlAutomation::ITextRangeProvider InteractivityAutomationPeer::DocumentRange()
{
UIA::ITextRangeProvider* returnVal;
THROW_IF_FAILED(_uiaProvider->get_DocumentRange(&returnVal));
const auto parentProvider = this->ProviderFromPeer(*this);
const auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parentProvider);
return xutr.as<XamlAutomation::ITextRangeProvider>();
return _CreateXamlUiaTextRange(returnVal);
}
XamlAutomation::SupportedTextSelection InteractivityAutomationPeer::SupportedTextSelection()
@ -180,6 +175,20 @@ namespace winrt::Microsoft::Terminal::Control::implementation
}
#pragma endregion
XamlAutomation::ITextRangeProvider InteractivityAutomationPeer::_CreateXamlUiaTextRange(UIA::ITextRangeProvider* returnVal) const
{
// LOAD-BEARING: use _parentProvider->ProviderFromPeer(_parentProvider) instead of this->ProviderFromPeer(*this).
// Since we split the automation peer into TermControlAutomationPeer and InteractivityAutomationPeer,
// using "this" returns null. This can cause issues with some UIA Client scenarios like any navigation in Narrator.
const auto parent{ _parentProvider.get() };
if (!parent)
{
return nullptr;
}
const auto xutr = winrt::make_self<XamlUiaTextRange>(returnVal, parent.ProviderFromPeer(parent));
return xutr.as<XamlAutomation::ITextRangeProvider>();
};
// Method Description:
// - extracts the UiaTextRanges from the SAFEARRAY and converts them to Xaml ITextRangeProviders
// Arguments:
@ -194,11 +203,12 @@ namespace winrt::Microsoft::Terminal::Control::implementation
std::vector<XamlAutomation::ITextRangeProvider> vec;
vec.reserve(count);
auto parentProvider = this->ProviderFromPeer(*this);
for (int i = 0; i < count; i++)
{
auto xutr = make_self<XamlUiaTextRange>(providers[i].detach(), parentProvider);
vec.emplace_back(xutr.as<XamlAutomation::ITextRangeProvider>());
if (auto xutr = _CreateXamlUiaTextRange(providers[i].detach()))
{
vec.emplace_back(std::move(xutr));
}
}
com_array<XamlAutomation::ITextRangeProvider> result{ vec };

4
src/cascadia/TerminalControl/InteractivityAutomationPeer.h
View File

@ -43,6 +43,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
void SetControlBounds(const Windows::Foundation::Rect bounds);
void SetControlPadding(const Core::Padding padding);
void ParentProvider(Windows::UI::Xaml::Automation::Peers::AutomationPeer parentProvider);
#pragma region IUiaEventDispatcher
void SignalSelectionChanged() override;
@ -74,8 +75,11 @@ namespace winrt::Microsoft::Terminal::Control::implementation
TYPED_EVENT(CursorChanged, IInspectable, IInspectable);
private:
Windows::UI::Xaml::Automation::Provider::ITextRangeProvider _CreateXamlUiaTextRange(::ITextRangeProvider* returnVal) const;
::Microsoft::WRL::ComPtr<::Microsoft::Terminal::TermControlUiaProvider> _uiaProvider;
winrt::Microsoft::Terminal::Control::implementation::ControlInteractivity* _interactivity;
weak_ref<Windows::UI::Xaml::Automation::Peers::AutomationPeer> _parentProvider;
til::rectangle _controlBounds{};
til::rectangle _controlPadding{};

2
src/cascadia/TerminalControl/InteractivityAutomationPeer.idl
View File

@ -1,6 +1,5 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
namespace Microsoft.Terminal.Control
{
[default_interface] runtimeclass InteractivityAutomationPeer :
@ -10,6 +9,7 @@ namespace Microsoft.Terminal.Control
void SetControlBounds(Windows.Foundation.Rect bounds);
void SetControlPadding(Microsoft.Terminal.Core.Padding padding);
void ParentProvider(Windows.UI.Xaml.Automation.Peers.AutomationPeer parentProvider);
event Windows.Foundation.TypedEventHandler<Object, Object> SelectionChanged;
event Windows.Foundation.TypedEventHandler<Object, Object> TextChanged;

27
src/cascadia/TerminalControl/TermControlAutomationPeer.cpp
View File

@ -45,6 +45,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
_contentAutomationPeer.SelectionChanged([this](auto&&, auto&&) { SignalSelectionChanged(); });
_contentAutomationPeer.TextChanged([this](auto&&, auto&&) { SignalTextChanged(); });
_contentAutomationPeer.CursorChanged([this](auto&&, auto&&) { SignalCursorChanged(); });
_contentAutomationPeer.ParentProvider(*this);
};
// Method Description:
@ -226,30 +227,4 @@ namespace winrt::Microsoft::Terminal::Control::implementation
}
#pragma endregion
// Method Description:
// - extracts the UiaTextRanges from the SAFEARRAY and converts them to Xaml ITextRangeProviders
// Arguments:
// - SAFEARRAY of UIA::UiaTextRange (ITextRangeProviders)
// Return Value:
// - com_array of Xaml Wrapped UiaTextRange (ITextRangeProviders)
com_array<XamlAutomation::ITextRangeProvider> TermControlAutomationPeer::WrapArrayOfTextRangeProviders(SAFEARRAY* textRanges)
{
// transfer ownership of UiaTextRanges to this new vector
auto providers = SafeArrayToOwningVector<::Microsoft::Terminal::TermControlUiaTextRange>(textRanges);
int count = gsl::narrow<int>(providers.size());
std::vector<XamlAutomation::ITextRangeProvider> vec;
vec.reserve(count);
auto parentProvider = this->ProviderFromPeer(*this);
for (int i = 0; i < count; i++)
{
auto xutr = make_self<XamlUiaTextRange>(providers[i].detach(), parentProvider);
vec.emplace_back(xutr.as<XamlAutomation::ITextRangeProvider>());
}
com_array<XamlAutomation::ITextRangeProvider> result{ vec };
return result;
}
}

2
src/cascadia/TerminalControl/TermControlAutomationPeer.h
View File

@ -78,7 +78,5 @@ namespace winrt::Microsoft::Terminal::Control::implementation
private:
winrt::Microsoft::Terminal::Control::implementation::TermControl* _termControl;
Control::InteractivityAutomationPeer _contentAutomationPeer;
winrt::com_array<Windows::UI::Xaml::Automation::Provider::ITextRangeProvider> WrapArrayOfTextRangeProviders(SAFEARRAY* textRanges);
};
}

2
src/cascadia/TerminalControl/XamlUiaTextRange.cpp
View File

@ -6,6 +6,7 @@
#include "../types/TermControlUiaTextRange.hpp"
#include <UIAutomationClient.h>
#include <UIAutomationCoreApi.h>
#include "../types/UiaTracing.h"
// the same as COR_E_NOTSUPPORTED
// we don't want to import the CLR headers to get it
@ -182,6 +183,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
XamlAutomation::IRawElementProviderSimple XamlUiaTextRange::GetEnclosingElement()
{
::Microsoft::Console::Types::UiaTracing::TextRange::GetEnclosingElement(*static_cast<::Microsoft::Console::Types::UiaTextRangeBase*>(_uiaProvider.get()));
return _parentProvider;
}