terminal/src/types/ScreenInfoUiaProvider.h
Carlos Zamora a08666b58e Accessibility: TermControl Automation Peer (#2083)
Builds on the work of #1691 and #1915 

Let's start with the easy change:
- `TermControl`'s `controlRoot` was removed. `TermControl` is a `UserControl`
  now.

Ok. Now we've got a story to tell here....

### TermControlAP - the Automation Peer
Here's an in-depth guide on custom automation peers:
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/custom-automation-peers

We have a custom XAML element (TermControl). So XAML can't really hold our
hands and determine an accessible behavior for us. So this automation peer is
responsible for enabling that interaction.

We made it a FrameworkElementAutomationPeer to get as much accessibility as
possible from it just being a XAML element (i.e.: where are we on the screen?
what are my dimensions?). This is recommended. Any functions with "Core" at the
end, are overwritten here to tweak this automation peer into what we really
need.

But what kind of interactions can a user expect from this XAML element?
Introducing ControlPatterns! There's a ton of interfaces that just define "what
can I do". Thankfully, we already know that we're supposed to be
`ScreenInfoUiaProvider` and that was an `ITextProvider`, so let's just make the
TermControlAP an `ITextProvider` too.

So now we have a way to define what accessible actions can be performed on us,
but what should those actions do? Well let's just use the automation providers
from ConHost that are now in a shared space! (Note: this is a great place to
stop and get some coffee. We're about to hop into the .cpp file in the next
section)


### Wrapping our shared Automation Providers

Unfortunately, we can't just use the automation providers from ConHost. Or, at
least not just hook them up as easily as we wish. ConHost's UIA Providers were
written using UIAutomationCore and ITextRangeProiuder. XAML's interfaces
ITextProvider and ITextRangeProvider are lined up to be exactly the same.

So we need to wrap our ConHost UIA Providers (UIAutomationCore) with the XAML
ones. We had two providers, so that means we have two wrappers.

#### TermControlAP (XAML) <----> ScreenInfoUiaProvider (UIAutomationCore)
Each of the functions in the pragma region `ITextProvider` for
TermControlAP.cpp is just wrapping what we do in `ScreenInfoUiaProvider`, and
returning an acceptable version of it.

Most of `ScreenInfoUiaProvider`'s functions return `UiaTextRange`s. So we need
to wrap that too. That's this next section...

#### XamlUiaTextRange (XAML) <----> UiaTextRange (UIAutomationCore)
Same idea.  We're wrapping everything that we could do with `UiaTextRange` and
putting it inside of `XamlUiaTextRange`.


### Additional changes to `UiaTextRange` and `ScreenInfoUiaProvider`
If you don't know what I just said, please read this background:
- #1691: how accessibility works and the general responsibility of these two
  classes
- #1915: how we pulled these Accessibility Providers into a shared area

TL;DR: `ScreenInfoUiaProvider` lets you interact with the displayed text.
`UiaTextRange` is specific ranges of text in the display and navigate the text.

Thankfully, we didn't do many changes here. I feel like some of it is hacked
together but now that we have a somewhat working system, making changes
shouldn't be too hard...I hope.

#### UiaTextRange
We don't have access to the window handle. We really only need it to draw the
bounding rects using WinUser's `ScreenToClient()` and `ClientToScreen()`. I
need to figure out how to get around this.

In the meantime, I made the window handle optional. And if we don't have
one....well, we need to figure that out. But other than that, we have a
`UiaTextRange`.

#### ScreenInfoUiaProvider
At some point, we need to hook up this automation provider to the
WindowUiaProvider. This should help with navigation of the UIA Tree and make
everything just look waaaay better. For now, let's just do the same approach
and make the pUiaParent optional.

This one's the one I'm not that proud of, but it works. We need the parent to
get a bounding rect of the terminal. While we figure out how to attach the
WindowUiaProvider, we should at the very least be able to get a bunch of info
from our xaml automation peer. So, I've added a _getBoundingRect optional
function. This is what's called when we don't have a WindowUiaProvider as our
parent.


## Validation Steps Performed
I've been using inspect.exe to see the UIA tree.
I was able to interact with the terminal mostly fine. A few known issues below.

Unfortunately, I tried running Narrator on this and it didn't seem to like it
(by that I mean WT crashed). Then again, I don't really know how to use
narrator other than "click on object" --> "listen voice". I feel like there's a
way to get the other interactions with narrator, but I'll be looking into more
of that soon. I bet if I fix the two issues below, Narrator will be happy.

## Miscellaneous Known Issues
- `GetSelection()` and `GetVisibleRanges()` crashes. I need to debug through
  these. I want to include them in this PR.

Fixes #1353.
2019-07-30 16:43:10 -07:00

169 lines
6.2 KiB
C++

/*++
Copyright (c) Microsoft Corporation
Licensed under the MIT license.
Module Name:
- screenInfoUiaProvider.hpp
Abstract:
- This module provides UI Automation access to the screen buffer to
support both automation tests and accessibility (screen reading)
applications.
- ConHost and Windows Terminal must use IRenderData to have access to the proper information
- Based on examples, sample code, and guidance from
https://msdn.microsoft.com/en-us/library/windows/desktop/ee671596(v=vs.85).aspx
Author(s):
- Michael Niksa (MiNiksa) 2017
- Austin Diviness (AustDi) 2017
- Carlos Zamora (CaZamor) 2019
--*/
#pragma once
#include "precomp.h"
#include "../buffer/out/textBuffer.hpp"
#include "../renderer/inc/IRenderData.hpp"
namespace Microsoft::Console::Types
{
class WindowUiaProviderBase;
class Viewport;
class ScreenInfoUiaProvider :
public IRawElementProviderSimple,
public IRawElementProviderFragment,
public ITextProvider
{
public:
ScreenInfoUiaProvider(_In_ Microsoft::Console::Render::IRenderData* pData,
_In_ WindowUiaProviderBase* const pUiaParent,
_In_ std::function<RECT()> GetBoundingRect);
// TODO GitHub 2120: pUiaParent should not be allowed to be null
ScreenInfoUiaProvider(_In_ Microsoft::Console::Render::IRenderData* pData,
_In_ WindowUiaProviderBase* const pUiaParent);
virtual ~ScreenInfoUiaProvider();
[[nodiscard]] HRESULT Signal(_In_ EVENTID id);
// IUnknown methods
IFACEMETHODIMP_(ULONG)
AddRef();
IFACEMETHODIMP_(ULONG)
Release();
IFACEMETHODIMP QueryInterface(_In_ REFIID riid,
_COM_Outptr_result_maybenull_ void** ppInterface);
// IRawElementProviderSimple methods
IFACEMETHODIMP get_ProviderOptions(_Out_ ProviderOptions* pOptions);
IFACEMETHODIMP GetPatternProvider(_In_ PATTERNID iid,
_COM_Outptr_result_maybenull_ IUnknown** ppInterface);
IFACEMETHODIMP GetPropertyValue(_In_ PROPERTYID idProp,
_Out_ VARIANT* pVariant);
IFACEMETHODIMP get_HostRawElementProvider(_COM_Outptr_result_maybenull_ IRawElementProviderSimple** ppProvider);
// IRawElementProviderFragment methods
IFACEMETHODIMP Navigate(_In_ NavigateDirection direction,
_COM_Outptr_result_maybenull_ IRawElementProviderFragment** ppProvider);
IFACEMETHODIMP GetRuntimeId(_Outptr_result_maybenull_ SAFEARRAY** ppRuntimeId);
IFACEMETHODIMP get_BoundingRectangle(_Out_ UiaRect* pRect);
IFACEMETHODIMP GetEmbeddedFragmentRoots(_Outptr_result_maybenull_ SAFEARRAY** ppRoots);
IFACEMETHODIMP SetFocus();
IFACEMETHODIMP get_FragmentRoot(_COM_Outptr_result_maybenull_ IRawElementProviderFragmentRoot** ppProvider);
// ITextProvider
IFACEMETHODIMP GetSelection(_Outptr_result_maybenull_ SAFEARRAY** ppRetVal);
IFACEMETHODIMP GetVisibleRanges(_Outptr_result_maybenull_ SAFEARRAY** ppRetVal);
IFACEMETHODIMP RangeFromChild(_In_ IRawElementProviderSimple* childElement,
_COM_Outptr_result_maybenull_ ITextRangeProvider** ppRetVal);
IFACEMETHODIMP RangeFromPoint(_In_ UiaPoint point,
_COM_Outptr_result_maybenull_ ITextRangeProvider** ppRetVal);
IFACEMETHODIMP get_DocumentRange(_COM_Outptr_result_maybenull_ ITextRangeProvider** ppRetVal);
IFACEMETHODIMP get_SupportedTextSelection(_Out_ SupportedTextSelection* pRetVal);
HWND GetWindowHandle() const;
void ChangeViewport(const SMALL_RECT NewWindow);
private:
// Ref counter for COM object
ULONG _cRefs;
// weak reference to uia parent
WindowUiaProviderBase* const _pUiaParent;
// weak reference to IRenderData
Microsoft::Console::Render::IRenderData* _pData;
// this is used to prevent the object from
// signaling an event while it is already in the
// process of signalling another event.
// This fixes a problem with JAWS where it would
// call a public method that calls
// UiaRaiseAutomationEvent to signal something
// happened, which JAWS then detects the signal
// and calls the same method in response,
// eventually overflowing the stack.
// We aren't using this as a cheap locking
// mechanism for multi-threaded code.
std::map<EVENTID, bool> _signalFiringMapping;
const COORD _getScreenBufferCoords() const;
const TextBuffer& _getTextBuffer() const;
const Viewport _getViewport() const;
void _LockConsole() noexcept;
void _UnlockConsole() noexcept;
// these functions are reserved for Windows Terminal
std::function<RECT(void)> _getBoundingRect;
};
namespace ScreenInfoUiaProviderTracing
{
enum class ApiCall
{
Constructor,
Signal,
AddRef,
Release,
QueryInterface,
GetProviderOptions,
GetPatternProvider,
GetPropertyValue,
GetHostRawElementProvider,
Navigate,
GetRuntimeId,
GetBoundingRectangle,
GetEmbeddedFragmentRoots,
SetFocus,
GetFragmentRoot,
GetSelection,
GetVisibleRanges,
RangeFromChild,
RangeFromPoint,
GetDocumentRange,
GetSupportedTextSelection
};
struct IApiMsg
{
};
struct ApiMsgSignal : public IApiMsg
{
EVENTID Signal;
};
struct ApiMsgNavigate : public IApiMsg
{
NavigateDirection Direction;
};
struct ApiMsgGetSelection : public IApiMsg
{
bool AreaSelected;
unsigned int SelectionRowCount;
};
}
}