terminal/src/host/ft_uia/WinEventTests.cs
Michael Niksa 5d082ffe67
Helix Testing (#6992)
Use the Helix testing orchestration framework to run our Terminal LocalTests and Console Host UIA tests.

## References
#### Creates the following new issues:
- #7281 - re-enable local tests that were disabled to turn on Helix
- #7282 - re-enable UIA tests that were disabled to turn on Helix
- #7286 - investigate and implement appropriate compromise solution to how Skipped is handled by MUX Helix scripts

#### Consumes from:
- #7164 - The update to TAEF includes wttlog.dll. The WTT logs are what MUX's Helix scripts use to track the run state, convert to XUnit format, and notify both Helix and AzDO of what's going on.

#### Produces for:
- #671 - Making Terminal UIA tests is now possible
- #6963 - MUX's Helix scripts are already ready to capture PGO data on the Helix machines as certain tests run. Presuming we can author some reasonable scenarios, turning on the Helix environment gets us a good way toward automated PGO.

#### Related:
- #4490 - We lost the AzDO integration of our test data when I moved from the TAEF/VSTest adapter directly back to TE. Thanks to the WTTLog + Helix conversion scripts to XUnit + new upload phase, we have it back!

## PR Checklist
* [x] Closes #3838
* [x] I work here.
* [x] Literally adds tests.
* [ ] Should I update a testing doc in this repo?
* [x] Am core contributor. Hear me roar.
* [ ] Correct spell-checking the right way before merge.

## Detailed Description of the Pull Request / Additional comments
We have had two classes of tests that don't work in our usual build-machine testing environment:
1. Tests that require interactive UI automation or input injection (a.k.a. require a logged in user)
2. Tests that require the entire Windows Terminal to stand up (because our Xaml Islands dependency requires 1903 or later and the Windows Server instance for the build is based on 1809.)

The Helix testing environment solves both of these and is brought to us by our friends over in https://github.com/microsoft/microsoft-ui-xaml.

This PR takes a large portion of scripts and pipeline configuration steps from the Microsoft-UI-XAML repository and adjusts them for Terminal needs.
You can see the source of most of the files in either https://github.com/microsoft/microsoft-ui-xaml/tree/master/build/Helix or https://github.com/microsoft/microsoft-ui-xaml/tree/master/build/AzurePipelinesTemplates

Some of the modifications in the files include (but are not limited to) reasons like:
- Our test binaries are named differently than MUX's test binaries
- We don't need certain types of testing that MUX does.
- We use C++ and C# tests while MUX was using only C# tests (so the naming pattern and some of the parsing of those names is different e.g. :: separators in C++ and . separators in C#)
- Our pipeline phases work a bit differently than MUX and/or we need significantly fewer pieces to the testing matrix (like we don't test a wide variety of OS versions).

The build now runs in a few stages:
1. The usual build and run of unit tests/feature tests, packaging verification, and whatnot. This phase now also picks up and packs anything required for running tests in Helix into an artifact. (It also unifies the artifact name between the things Helix needs and the existing build outputs into the single `drop` artifact to make life a little easier.)
2. The Helix preparation build runs that picks up those artifacts, generates all the scripts required for Helix to understand the test modules/functions from our existing TAEF tests, packs it all up, and queues it on the Helix pool.
3. Helix generates a VM for our testing environment and runs all the TAEF tests that require it. The orchestrator at helix.dot.net watches over this and tracks the success/fail and progress of each module and function. The scripts from our MUX friends handle installing dependencies, making the system quiet for better reliability, detecting flaky tests and rerunning them, and coordinating all the log uploads (including for the subruns of tests that are re-run.)
4. A final build phase is run to look through the results with the Helix API and clean up the marking of tests that are flaky, link all the screenshots and console output logs into the AzDO tests panel, and other such niceities.

We are set to run Helix tests on the Feature test policy of only x64 for now. 

Additionally, because the set up of the Helix VMs takes so long, we are *NOT* running these in PR trigger right now as I believe we all very much value our 15ish minute PR turnaround (and the VM takes another 15 minutes to just get going for whatever reason.) For now, they will only run as a rolling build on master after PRs are merged. We should still know when there's an issue within about an hour of something merging and multiple PRs merging fast will be done on the rolling build as a batch run (not one per).

In addition to setting up the entire Helix testing pipeline for the tests that require it, I've preserved our classic way of running unit and feature tests (that don't require an elaborate environment) directly on the build machines. But with one bonus feature... They now use some of the scripts from MUX to transform their log data and report it to AzDO so it shows up beautifully in the build report. (We used to have this before I removed the MStest/VStest wrapper for performance reasons, but now we can have reporting AND performance!) See https://dev.azure.com/ms/terminal/_build/results?buildId=101654&view=ms.vss-test-web.build-test-results-tab for an example. 

I explored running all of the tests on Helix but.... the Helix setup time is long and the resources are more expensive. I felt it was better to preserve the "quick signal" by continuing to run these directly on the build machine (and skipping the more expensive/slow Helix setup if they fail.) It also works well with the split between PR builds not running Helix and the rolling build running Helix. PR builds will get a good chunk of tests for a quick turn around and the rolling build will finish the more thorough job a bit more slowly.

## Validation Steps Performed
- [x] Ran the updated pipelines with Pull Request configuration ensuring that Helix tests don't run in the usual CI
- [x] Ran with simulation of the rolling build to ensure that the tests now running in Helix will pass. All failures marked for follow on in reference issues.
2020-08-18 18:23:24 +00:00

436 lines
19 KiB
C#

//----------------------------------------------------------------------------------------------------------------------
// <copyright file="WinEventTests.cs" company="Microsoft">
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
// </copyright>
// <summary>UI Automation tests for the Accessibility WinEvents feature.</summary>
//----------------------------------------------------------------------------------------------------------------------
namespace Conhost.UIA.Tests
{
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using WEX.Common.Managed;
using WEX.Logging.Interop;
using WEX.TestExecution;
using WEX.TestExecution.Markup;
using Conhost.UIA.Tests.Common;
using Conhost.UIA.Tests.Common.NativeMethods;
using Conhost.UIA.Tests.Elements;
using OpenQA.Selenium;
using System.Threading;
using System.Runtime.InteropServices;
using System.Drawing;
// Test hooks adapted from C++ WinEvent accessibility sample at https://msdn.microsoft.com/en-us/library/ms971319.aspx#atg_consoleaccessibility_topic05
[TestClass]
public class WinEventTests : IWinEventCallbacks
{
public TestContext TestContext { get; set; }
enum EventType
{
CaretSelection,
CaretVisible,
EndApplication,
Layout,
StartApplication,
UpdateRegion,
UpdateScroll,
UpdateSimple
}
class EventData
{
EventType type;
// Of all the WinEvents that we return in the console, the max number of parameters returned is 4.
int[] data = new int[4];
public EventData(EventType type, int a = 0, int b = 0, int c = 0, int d = 0)
{
this.type = type;
data[0] = a;
data[1] = b;
data[2] = c;
data[3] = d;
}
public override bool Equals(object obj)
{
if (typeof(EventData) == obj.GetType())
{
return Equals((EventData)obj);
}
else
{
return base.Equals(obj);
}
}
public bool Equals(EventData other)
{
return type == other.type &&
data[0] == other.data[0] &&
data[1] == other.data[1] &&
data[2] == other.data[2] &&
data[3] == other.data[3];
}
public override string ToString()
{
return $"\r\nType: {type} Data[0]: {data[0]} Data[1]: {data[1]} Data[2]: {data[2]} Data[3]: {data[3]}\r\n";
}
public override int GetHashCode()
{
return type.GetHashCode() ^ data[0].GetHashCode() ^ data[1].GetHashCode() ^ data[2].GetHashCode() ^ data[3].GetHashCode();
}
}
Queue<EventData> received = new Queue<EventData>();
public void CaretSelection(int x, int y)
{
Log.Comment($"Event Console CARET SELECTION! X: {x} Y: {y}");
received.Enqueue(new EventData(EventType.CaretSelection, x, y));
}
public void CaretVisible(int x, int y)
{
Log.Comment($"Event Console CARET VISIBLE! X: {x} Y: {y}");
received.Enqueue(new EventData(EventType.CaretVisible, x, y));
}
public void EndApplication(int processId, int childId)
{
Log.Comment($"Event Console END APPLICATION! Process ID: {processId} - Child ID: {childId}");
received.Enqueue(new EventData(EventType.EndApplication)); // we don't have a great way of detecting pids now so don't store it for comparison
}
public void Layout()
{
Log.Comment("Event Console LAYOUT!");
received.Enqueue(new EventData(EventType.Layout));
}
public void StartApplication(int processId, int childId)
{
Log.Comment($"Event Console START APPLICATION! Process ID: {processId} - Child ID: {childId}");
received.Enqueue(new EventData(EventType.StartApplication)); // we don't have a great way of detecting pids now so don't store it for comparison
}
public void UpdateRegion(int left, int top, int right, int bottom)
{
Log.Comment($"Event Console UPDATE REGION! Left: {left} - Top: {top} - Right: {right} - Bottom: {bottom}");
received.Enqueue(new EventData(EventType.UpdateRegion, left, top, right, bottom));
}
public void UpdateScroll(int deltaX, int deltaY)
{
Log.Comment($"Event Console UPDATE SCROLL! dx: {deltaX} - dy: {deltaY}");
received.Enqueue(new EventData(EventType.UpdateScroll, deltaX, deltaY));
}
public void UpdateSimple(int x, int y, int character, int attribute)
{
Log.Comment($"Event Console UPDATE SIMPLE! X: {x} - Y: {y} - Char: {character} - Attr: {attribute}");
received.Enqueue(new EventData(EventType.UpdateSimple, x, y, character, attribute));
}
void VerifyQueue(Queue<EventData> testQueue)
{
while (testQueue.Count > 0)
{
Verify.AreEqual(testQueue.Dequeue(), received.Dequeue());
}
}
public void TestTypeStringHelper(string str, CmdApp app, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex)
{
Queue<EventData> expected = new Queue<EventData>();
foreach (char c in str)
{
// For each character, we expect a pair of messages.
// 1. A Update Simple where the cursor was to represent the letter inserted where the cursor was.
expected.Enqueue(new EventData(EventType.UpdateSimple, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y, c, (int)sbiex.wAttributes));
// Move cursor right by 1.
sbiex.dwCursorPosition.X++;
// 2. A caret visible to represent where the cursor moved after the letter was inserted.
expected.Enqueue(new EventData(EventType.CaretVisible, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
// Type in the letter
app.UIRoot.SendKeys($"{c}");
Globals.WaitForTimeout();
}
// Verify that we saw the right messages
this.VerifyQueue(expected);
}
delegate void AccessibilityTest(CmdApp app, ViewportArea area, IntPtr hConsole, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex, Queue<EventData> expected, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal);
void RunTest(AccessibilityTest test)
{
using (RegistryHelper reg = new RegistryHelper())
{
reg.BackupRegistry();
using (CmdApp app = new CmdApp(CreateType.ProcessOnly, TestContext))
{
using (WinEventSystem sys = app.AttachWinEventSystem(this))
{
using (ViewportArea area = new ViewportArea(app))
{
Globals.WaitForTimeout(); // wait for everything to settle with winevents
IntPtr hConsole = app.GetStdOutHandle();
// Prep structures to hold cursor and size of buffer.
WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex = new WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX();
sbiex.cbSize = (uint)Marshal.SizeOf(sbiex);
// this is where we will hold our expected messages
Queue<EventData> expected = new Queue<EventData>();
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Get initial console data.");
WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal = sbiex; // keep a copy of the original data for later.
// Run the test
test(app, area, hConsole, sbiex, expected, sbiexOriginal);
}
}
}
}
}
[TestMethod]
[TestProperty("Ignore", "True")] // GH#7282 - investigate and reenable
public void TestSelection()
{
RunTest(TestSelectionImpl);
}
private void TestSelectionImpl(CmdApp app, ViewportArea area, IntPtr hConsole, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex, Queue<EventData> expected, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal)
{
// A. Test single area click
{
// Move mouse pointer to where the cursor is
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
Point pt = new Point(sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y);
area.MouseMove(pt);
// Click on this area.
expected.Enqueue(new EventData(EventType.CaretSelection, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
area.MouseDown();
area.MouseUp();
Globals.WaitForTimeout();
VerifyQueue(expected);
// We may receive more than one caret and that's OK. Clear it out.
this.received.Clear();
// End selection with escape
app.UIRoot.SendKeys(Keys.Escape);
Globals.WaitForTimeout();
// Expect to see the caret again after leaving selection mode
expected.Enqueue(new EventData(EventType.CaretVisible, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
VerifyQueue(expected);
}
// B. Drag area click
{
// Move mouse pointer to where the cursor is
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
Point pt = new Point(sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y);
area.MouseMove(pt);
// Click on this area.
expected.Enqueue(new EventData(EventType.CaretSelection, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
area.MouseDown();
Globals.WaitForTimeout();
Point ptDrag = pt;
// Drag down and right for "some" distance. 10 isn't for a specific reason, it's just "some".
ptDrag.X += 10;
ptDrag.Y += 10;
area.MouseMove(ptDrag);
Globals.WaitForTimeout();
area.MouseUp();
Globals.WaitForTimeout();
// Verify that the first one in the queue starts with where we put the mouse down.
VerifyQueue(expected);
// Now we have to take the final message in the queue and make sure it is where we released the mouse
EventData expectedLast = new EventData(EventType.CaretSelection, ptDrag.X, ptDrag.Y);
EventData actualLast = received.Last();
Verify.AreEqual(expectedLast, actualLast);
// Empty the received queue.
received.Clear();
// End selection with escape
app.UIRoot.SendKeys(Keys.Escape);
Globals.WaitForTimeout();
// Expect to see the caret again after leaving selection mode
expected.Enqueue(new EventData(EventType.CaretVisible, sbiex.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
VerifyQueue(expected);
}
}
[TestMethod]
[TestProperty("Ignore", "True")] // GH#7282 - investigate and reenable
public void TestLaunchAndExitChild()
{
RunTest(TestLaunchAndExitChildImpl);
}
private void TestLaunchAndExitChildImpl(CmdApp app, ViewportArea area, IntPtr hConsole, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex, Queue<EventData> expected, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal)
{
// A. We're going to type "cmd" into the prompt to start a command prompt.
{
Log.Comment("Type 'cmd' to get ready to start nested prompt.");
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
TestTypeStringHelper("cmd", app, sbiex);
}
// B. Now we're going to press enter to launch the CMD application
{
Log.Comment("Press enter to launch and observe launch events.");
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
expected.Enqueue(new EventData(EventType.StartApplication));
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, 0, sbiex.dwSize.X - 1, sbiex.dwSize.Y - 1));
sbiex.dwCursorPosition.Y++;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, sbiex.dwCursorPosition.Y, "Microsoft Windows [Version 10.0.14974.1001]".Length - 1, sbiex.dwCursorPosition.Y));
sbiex.dwCursorPosition.Y++;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, sbiex.dwCursorPosition.Y, "(c) 2016 Microsoft Corporation. All rights reserved.".Length - 1, sbiex.dwCursorPosition.Y));
sbiex.dwCursorPosition.Y++;
sbiex.dwCursorPosition.Y++;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, sbiex.dwCursorPosition.Y, sbiexOriginal.dwCursorPosition.X - 1, sbiex.dwCursorPosition.Y));
expected.Enqueue(new EventData(EventType.CaretVisible, sbiexOriginal.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
app.UIRoot.SendKeys(Keys.Enter);
Globals.WaitForTimeout();
Globals.WaitForTimeout();
VerifyQueue(expected);
}
// C. Now we're going to type exit to leave the nested CMD application
{
Log.Comment("Type 'exit' to get ready to exit nested prompt.");
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
TestTypeStringHelper("exit", app, sbiex);
}
// D. Now we're going to press enter to exit the CMD application
{
Log.Comment("Press enter to launch and observe exit events.");
NativeMethods.Win32BoolHelper(WinCon.GetConsoleScreenBufferInfoEx(hConsole, ref sbiex), "Update console data.");
expected.Enqueue(new EventData(EventType.EndApplication));
sbiex.dwCursorPosition.Y++;
sbiex.dwCursorPosition.Y++;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, sbiex.dwCursorPosition.Y, sbiexOriginal.dwCursorPosition.X - 1, sbiex.dwCursorPosition.Y));
expected.Enqueue(new EventData(EventType.CaretVisible, sbiexOriginal.dwCursorPosition.X, sbiex.dwCursorPosition.Y));
app.UIRoot.SendKeys(Keys.Enter);
Globals.WaitForTimeout();
Globals.WaitForTimeout();
VerifyQueue(expected);
}
}
[TestMethod]
[TestProperty("Ignore", "True")] // GH#7282 - investigate and reenable
public void TestScrollByWheel()
{
RunTest(TestScrollByWheelImpl);
}
private void TestScrollByWheelImpl(CmdApp app, ViewportArea area, IntPtr hConsole, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex, Queue<EventData> expected, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal)
{
int rowsPerScroll = app.GetRowsPerScroll();
int scrollDelta;
// A. Scroll down.
{
scrollDelta = -1;
expected.Enqueue(new EventData(EventType.UpdateScroll, 0, scrollDelta * rowsPerScroll));
expected.Enqueue(new EventData(EventType.Layout));
app.ScrollWindow(scrollDelta);
Globals.WaitForTimeout();
VerifyQueue(expected);
}
// B. Scroll up.
{
scrollDelta = 1;
expected.Enqueue(new EventData(EventType.UpdateScroll, 0, scrollDelta * rowsPerScroll));
expected.Enqueue(new EventData(EventType.Layout));
app.ScrollWindow(scrollDelta);
Globals.WaitForTimeout();
VerifyQueue(expected);
}
}
[TestMethod]
[TestProperty("Ignore", "True")] // GH#7282 - investigate and reenable
public void TestScrollByOverflow()
{
RunTest(TestScrollByOverflowImpl);
}
private void TestScrollByOverflowImpl(CmdApp app, ViewportArea area, IntPtr hConsole, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiex, Queue<EventData> expected, WinCon.CONSOLE_SCREEN_BUFFER_INFO_EX sbiexOriginal)
{
// Get original screen information
sbiexOriginal = app.GetScreenBufferInfo();
short promptLineEnd = sbiexOriginal.dwCursorPosition.X;
promptLineEnd--; // prompt line ended one position left of cursor
// Resize the window to only have two lines left at the bottom to test overflow when we echo some text
sbiex = sbiexOriginal;
sbiex.srWindow.Bottom = sbiex.dwCursorPosition.Y;
sbiex.srWindow.Bottom += 3;
app.SetScreenBufferInfo(sbiex);
string echoText = "foo";
string echoCommand = "echo";
int echoLine = sbiexOriginal.dwCursorPosition.Y + 1;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, echoLine, echoText.Length - 1, echoLine));
expected.Enqueue(new EventData(EventType.UpdateScroll, 0, -1));
int newPromptLine = echoLine + 2;
expected.Enqueue(new EventData(EventType.UpdateRegion, 0, newPromptLine, promptLineEnd, newPromptLine));
expected.Enqueue(new EventData(EventType.Layout));
expected.Enqueue(new EventData(EventType.CaretVisible, promptLineEnd + 1, newPromptLine));
// type command to echo foo and press enter
app.UIRoot.SendKeys($"{echoCommand} {echoText}");
Globals.WaitForTimeout();
received.Clear();
app.UIRoot.SendKeys(Keys.Enter);
Globals.WaitForTimeout();
VerifyQueue(expected);
}
}
}