## Summary of the Pull Request
Refactors parsing/adapting libraries and consumers to use safer and/or more consistent mechanisms for passing information.
## PR Checklist
* [x] I work here
* [x] Tests still pass
* [x] Am a core contributor.
## Detailed Description of the Pull Request / Additional comments
This is in support of hopefully turning audit mode on to more projects. If I turned it on, it would immediately complain about certain classes of issues like pointer and size, pointer math, etc. The changes in this refactoring will eliminate those off the top.
Additionally, this has caught a bunch of comments all over the VT classes that weren't updated to match the parameters lists.
Additionally, this has caught a handful of member variables on classes that were completely unused (and now gone).
Additionally, I'm killing almost all hungarian and shortening variable names. I'm only really leaving 'p' for pointers.
Additionally, this is vaguely in support of a future where we can have "infinite scrollback" in that I'm moving things to size_t across the board. I know it's a bit of a memory cost, but all the casting and moving between types is error prone and unfun to save a couple bytes.
## Validation Steps Performed
- [x] build it
- [x] run all the tests
- [x] everyone looked real hard at it
## Summary of the Pull Request
This adds support for the [`DECALN`](https://vt100.net/docs/vt510-rm/DECALN.html) escape sequence, which produces a kind of test pattern, originally used on VT terminals to adjust the screen alignment. It's needed to pass several of the tests in the [Vttest](https://invisible-island.net/vttest/) suite.
## PR Checklist
* [x] Closes#3671
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx
## Detailed Description of the Pull Request / Additional comments
To start with, the `ActionEscDispatch` method in the `OutputStateMachineEngine` needed to be extended to check for a new intermediate type (`#`). Then when that intermediate is followed by an `8`, it dispatches to a new `ScreenAlignmentPattern` method in the `ITermDispatch` interface.
The implementation of the `ScreenAlignmentPattern` itself is fairly simple. It uses the recently added `PrivateFillRegion` API to fill the screen with the character `E` using default attributes. Then in addition to that, a bunch of VT properties are reset:
* The meta/extended attributes are reset (although the active colors must be left unchanged).
* The origin mode is set to absolute positioning.
* The scrolling margins are cleared.
* The cursor position is moved to home.
## Validation Steps Performed
I've added a screen buffer test that makes sure the `DECALN` sequence fills the screen with the correct character and attributes, and that the above mentioned properties are all updated appropriately.
I've also tested in Vttest, and confirmed that the first two pages of the _Test of cursor movements_ are now showing the frame of E's that are expected there.
## Summary of the Pull Request
Operations that erase areas of the screen are typically meant to do so using the current color attributes, but with the rendition attributes reset (what we refer to as meta attributes). This also includes scroll operations that have to clear the area of the screen that has scrolled into view. The only exception is the _Erase Scrollback_ operation, which needs to reset the buffer with the default attributes. This PR updates all of these cases to apply the correct attributes when scrolling and erasing.
## PR Checklist
* [x] Closes#2553
* [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [x] Tests added/passed
* [ ] Requires documentation to be updated
* [ ] I've not really discussed this with core contributors. I'm ready to accept this work might be rejected in favor of a different grand plan.
## Detailed Description of the Pull Request / Additional comments
My initial plan was to use a special case legacy attribute value to indicate the "standard erase attribute" which could safely be passed through the legacy APIs. But this wouldn't cover the cases that required default attributes to be used. And then with the changes in PR #2668 and #2987, it became clear that our requirements could be better achieved with a couple of new private APIs that wouldn't have to depend on legacy attribute hacks at all.
To that end, I've added the `PrivateFillRegion` and `PrivateScrollRegion` APIs to the `ConGetSet` interface. These are just thin wrappers around the existing `SCREEN_INFORMATION::Write` method and the `ScrollRegion` function respectively, but with a simple boolean parameter to choose between filling with default attributes or the standard erase attributes (i.e the current colors but with meta attributes reset).
With those new APIs in place, I could then update most scroll operations to use `PrivateScrollRegion`, and most erase operations to use `PrivateFillRegion`.
The functions affected by scrolling included:
* `DoSrvPrivateReverseLineFeed` (the RI command)
* `DoSrvPrivateModifyLinesImpl` (the IL and DL commands)
* `AdaptDispatch::_InsertDeleteHelper` (the ICH and DCH commands)
* `AdaptDispatch::_ScrollMovement` (the SU and SD commands)
The functions affected by erasing included:
* `AdaptDispatch::_EraseSingleLineHelper` (the EL command, and most ED variants)
* `AdaptDispatch::EraseCharacters` (the ECH command)
While updating these erase methods, I noticed that both of them also required boundary fixes similar to those in PR #2505 (i.e. the horizontal extent of the erase operation should apply to the full width of the buffer, and not just the current viewport width), so I've addressed that at the same time.
In addition to the changes above, there were also a few special cases, the first being the line feed handling, which required updating in a number of places to use the correct erase attributes:
* `SCREEN_INFORMATION::InitializeCursorRowAttributes` - this is used to initialise the rows that pan into view when the viewport is moved down the buffer.
* `TextBuffer::IncrementCircularBuffer` - this occurs when we scroll passed the very end of the buffer, and a recycled row now needs to be reinitialised.
* `AdjustCursorPosition` - when within margin boundaries, this relies on a couple of direct calls to `ScrollRegion` which needed to be passed the correct fill attributes.
The second special case was the full screen erase sequence (`ESC 2 J`), which is handled separately from the other ED sequences. This required updating the `SCREEN_INFORMATION::VtEraseAll` method to use the standard erase attributes, and also required changes to the horizontal extent of the filled area, since it should have been clearing the full buffer width (the same issue as the other erase operations mentioned above).
Finally, there was the `AdaptDispatch::_EraseScrollback` method, which uses both scroll and fill operations, which could now be handled by the new `PrivateScrollRegion` and `PrivateFillRegion` APIs. But in this case we needed to fill with the default attributes rather than the standard erase attributes. And again this implementation needed some changes to make sure the full width of the active area was retained after the erase, similar to the horizontal boundary issues with the other erase operations.
Once all these changes were made, there were a few areas of the code that could then be simplified quite a bit. The `FillConsoleOutputCharacterW`, `FillConsoleOutputAttribute`, and `ScrollConsoleScreenBufferW` were no longer needed in the `ConGetSet` interface, so all of that code could now be removed. The `_EraseSingleLineDistanceHelper` and `_EraseAreaHelper` methods in the `AdaptDispatch` class were also no longer required and could be removed.
Then there were the hacks to handle legacy default colors in the `FillConsoleOutputAttributeImpl` and `ScrollConsoleScreenBufferWImpl` implementations. Since those hacks were only needed for VT operations, and the VT code no longer calls those methods, there was no longer a need to retain that behaviour (in fact there are probably some edge cases where that behaviour might have been considered a bug when reached via the public console APIs).
## Validation Steps Performed
For most of the scrolling operations there were already existing tests in place, and those could easily be extended to check that the meta attributes were correctly reset when filling the revealed lines of the scrolling region.
In the screen buffer tests, I made updates of that sort to the `ScrollOperations` method (handling SU, SD, IL, DL, and RI), the `InsertChars` and `DeleteChars` methods (ICH and DCH), and the `VtNewlinePastViewport` method (LF). I also added a new `VtNewlinePastEndOfBuffer` test to check the case where the line feed causes the viewport to pan past the end of the buffer.
The erase operations, however, were being covered by adapter tests, and those aren't really suited for this kind of functionality (the same sort of issue came up in PR #2505). As a result I've had to reimplement those tests as screen buffer tests.
Most of the erase operations are covered by the `EraseTests` method, except the for the scrollback erase which has a dedicated `EraseScrollbackTests` method. I've also had to replace the `HardReset` adapter test, but that was already mostly covered by the `HardResetBuffer` screen buffer test, which I've now extended slightly (it could do with some more checks, but I think that can wait for a future PR when we're fixing other RIS issues).
The current DECSC implementation only saves the cursor position and
origin mode. This PR improves that functionality with additional support
for saving the SGR attributes, as well as the active character set.
It also fixes the way the saved state interacts with the alt buffer
(private mode 1049), triggering a save when switching to the alt buffer,
and a restore when switching back, and tracking the alt buffer state
independently from the main state.
In order to properly save and restore the SGR attributes, we first
needed to add a pair of APIs in the `ConGetSet` interface which could
round-trip the attributes with full 32-bit colors (the existing methods
only work with legacy attributes).
I then added a struct in the `AdaptDispatch` implementation to make it
easier to manage all of the parameters that needed to be saved. This
includes the cursor position and origin mode that we were already
tracking, and now also the SGR text attributes and the active character
set (via the `TermOutput` class).
Two instances of this structure are required, since changes made to the
saved state in the alt buffer need to be tracked separately from changes
in the main buffer. I've added a boolean property that specifies whether
we're in the alt buffer or not, and use that to decide which of the two
instances we're working with.
I also needed to explicitly trigger a save when switching to the alt
buffer, and a restore when switching back, since we weren't already
doing that (the existing implementation gave the impression that the
state was saved, because each buffer has its own cursor position, but
that doesn't properly match the XTerm behaviour).
For the state tracking itself, we've now got two additional properties -
the SGR attributes, which we obtain via the new private API, and the
active character set, which we get from a local `AdaptDispatch` field.
I'm saving the whole `TermOutput` class for the character set, since I'm
hoping that will make it automatically supports future enhancements.
When restoring the cursor position, there is also now a fix to handle
the relative origin mode correctly. If the margins are changed between
the position being saved and restored, it's possible for the cursor to
end up outside of the new margins, which would be illegal. So there is
now an additional step that clamps the Y coordinate within the margin
boundaries if the origin mode is relative.
# Validation
I've added a couple of screen buffer tests which check that the various
parameters are saved and restored as expected, as well as checking that
the Y coordinate is clamped appropriately when the relative origin mode
is set.
I've also tested manually with vttest and confirmed that the
_SAVE/RESTORE CURSOR_ test (the last page of the _Test of screen
features_)) is now working a lot better than it used to.
Closes#148.
## Summary of the Pull Request
Adds support for Italics, Blinking, Invisible, CrossedOut text, THROUGH CONPTY. This does **NOT** add support for those styles to conhost or the terminal.
We will store these "Extended Text Attributes" in a `TextAttribute`. When we go to render a line, we'll see if the state has changed from our previous state, and if so, we'll appropriately toggle that state with VT. Boldness has been moved from a `bool` to a single bit in these flags.
Technically, now that these are stored in the buffer, we only need to make changes to the renderers to be able to support them. That's not being done as a part of this PR however.
## References
See also #2915 and #2916, which are some follow-up tasks from this fix. I thought them too risky for 20H1.
## PR Checklist
* [x] Closes#2554
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
<hr>
* store text with extended attributes too
* Plumb attributes through all the renderers
* parse extended attrs, though we're not renderering them right
* Render these states correctly
* Add a very extensive test
* Cleanup for PR
* a block of PR feedback
* add 512 test cases
* Fix the build
* Fix @carlos-zamora's suggestions
* @miniksa's PR feedback
The `DECSTBM` margins are meant to define the range of lines within which
certain vertical scrolling operations take place. However, we were applying
these margin restrictions in the `ScrollRegion` function, which is also used in
a number of places that shouldn't be affected by `DECSTBM`.
This includes the `ICH` and `DCH` escape sequences (which are only affected by
the horizontal margins, which we don't yet support), the
`ScrollConsoleScreenBuffer` API (which is public Console API, not meant to be
affected by the VT terminal emulation), and the `CSI 3 J` erase scrollback
extension (which isn't really scrolling as such, but uses the `ScrollRegion`
function to manipulate the scrollback buffer).
This commit moves the margin clipping out of the `ScrollRegion` function, so it
can be applied exclusively in the places that need it.
With the margin clipping removed from the `ScrollRegion` function, it now had
to be applied manually in the places it was actually required. This included:
* The `DoSrvPrivateReverseLineFeed` function (for the `RI` control): This was
* just a matter of updating the bottom of the scroll rect to the bottom margin
* (at least when the margins were actually set), since the top of the scroll
* rect would always be the top of the viewport. The
* `DoSrvPrivateModifyLinesImpl` function (for the `IL` and `DL` commands):
* Again this was just a matter of updating the bottom of the scroll rect, since
* the cursor position would always determine the top of the scroll rect. The
* `AdaptDispatch::_ScrollMovement` method (for the `SU` and `SD` commands):
* This required updating both the top and bottom coordinates of the scroll
* rect, and also a simpler destination Y coordinate (the way the `ScrollRegion`
* function worked before, the caller was expected to take the margins into
* account when determining the destination).
On the plus side, there was now no longer a need to override the margins when
calling `ScrollRegion` in the `AdjustCursorPosition` function. In the first
case, the margins had needed to be cleared (_stream.cpp 143-145), but that is
now the default behaviour. In the second case, there had been a more
complicated adjustment of the margins (_stream.cpp 196-209), but that code was
never actually used so could be removed completely (to get to that point either
_fScrollUp_ was true, so _scrollDownAtTop_ couldn't also be true, or
_fScrollDown_ was true, but in that case there is a check to make sure
_scrollDownAtTop_ is false).
While testing, I also noticed that one of the `ScrollRegion` calls in the
`AdjustCursorPosition` function was not setting the horizontal range correctly
- the scrolling should always affect the full buffer width rather than just the
viewport width - so I've fixed that now as well.
## Validation Steps Performed
For commands like `RI`, `IL`, `DL`, etc. where we've changed the implementation
but not the behaviour, there were already unit tests that could confirm that
the new implementation was still producing the correct results.
Where there has been a change in behaviour - namely for the `ICH` and `DCH`
commands, and the `ScrollConsoleScreenBuffer` API - I've extended the existing
unit tests to check that they still function correctly even when the `DECSTBM`
margins are set (which would previously have caused them to fail).
I've also tested manually with the test cases in issues #2543 and #2659, and
confirmed that they now work as expected.
Closes#2543Closes#2659
There are a number of VT escape sequences that rely on the `ScrollRegion`
function to scroll the viewport (RI, DL, IL, SU, SD, ICH, and DCH) , and all of
them have got the clipping rect or scroll boundaries wrong in some way,
resulting in content being scrolled off the screen that should have been
clipped, revealed areas not being correctly filled, or parts of the screen not
being moved that should have been. This PR attempts to fix all of those issues.
The `ScrollRegion` function is what ultimately handles the scrolling, but it's
typically called via the `ApiRoutines::ScrollConsoleScreenBufferWImpl` method,
and it's the callers of that method that have needed correcting.
One "mistake" that many of these operations made, was in setting a clipping
rect that was different from the scrolling rect. This should never have been
necessary, since the area being scrolled is also the boundary into which the
content needs to be clipped, so the easiest thing to do is just use the same
rect for both parameters.
Another common mistake was in clipping the horizontal boundaries to the width
of the viewport. But it's really the buffer width that represents the active
width of the screen - the viewport width and offset are merely a window on that
active area. As such, the viewport should only be used to clip vertically - the
horizontal extent should typically be the full buffer width.
On that note, there is really no need to actually calculate the buffer width
when we want to set any of the scrolling parameters to that width. The
`ScrollRegion` function already takes care of clipping everything within the
buffer boundary, so we can simply set the `Left` of the rect to `0` and the
`Right` to `SHORT_MAX`.
More details on individual commands:
* RI (the `DoSrvPrivateReverseLineFeed` function)
This now uses a single rect for both the scroll region and clipping boundary,
and the width is set to `SHORT_MAX` to cover the full buffer width. Also the
bottom of the scrolling region is now the bottom of the viewport (rather than
bottom-1), otherwise it would be off by one.
* DL and IL (the `DoSrvPrivateModifyLinesImpl` function)
Again this uses a single rect for both the scroll region and clipping
boundary, and the width is set to `SHORT_MAX` to cover the full width. The
most significant change, though, is that the bottom boundary is now the
viewport bottom rather than the buffer bottom. Using the buffer bottom
prevented it clipping the content that scrolled off screen when inserting,
and failed to fill the revealed area when deleting.
* SU and SD (the `AdaptDispatch::_ScrollMovement` method)
This was already using a single rect for both the scroll region and clipping
boundary, but it was previously constrained to the width of the viewport
rather than the buffer width, so some areas of the screen weren't correctly
scrolled. Also, the bottom boundary was off by 1, because it was using an
exclusive rect while the `ScrollRegion` function expects inclusive rects.
* ICH and DCH (the `AdaptDispatch::_InsertDeleteHelper` method)
This method has been considerably simplified, because it was reimplementing a
lot of functionality that was already provided by the `ScrollRegion`
function. And like many of the other cases, it has been updated to use a
single rect for both the scroll region and clipping boundary, and clip to the
full buffer width rather than the viewport width.
I should add that if we were following the specs exactly, then the SU and SD
commands should technically be panning the viewport over the buffer instead of
moving the buffer contents within the viewport boundary. So SU would be the
equivalent of a newline at the bottom of the viewport (assuming no margins).
And SD would assumedly do the opposite, scrolling the back buffer back into
view (an RI at the top of the viewport should do the same).
This doesn't seem to be something that is consistently implemented, though.
Some terminals do implement SU as a viewport pan, but I haven't seen anyone
implement SD or RI as a pan. If we do want to do something about this, I think
it's best addressed as a separate issue.
## Validation Steps Performed
There were already existing tests for the SU, SD, ICH, and DCH commands, but
they were implemented as adapter tests, which weren't effectively testing
anything - the `ScrollConsoleScreenBufferW` method used in those tests was just
a mock (an incomplete reimplementation of the `ScrollRegion` function), so
confirming that the mock produced the correct result told you nothing about the
validity of the real code.
To address that, I've now reimplemented those adapter tests as screen buffer
tests. For the most part I've tried to duplicate the functionality of the
original tests, but there are significant differences to account for the fact
that scrolling region now covers the full width of the buffer rather than just
the viewport width.
I've also extended those tests with additional coverage for the RI, DL, and IL
commands, which are really just a variation of the SU and SD functionality.
Closes#2174
When the scrollback buffer is empty, the RIS escape sequence (Reset to Initial
State) will fail to clear the screen, or reset any of the state. And when there
is something in the scrollback, it doesn't get cleared completely, and the
screen may get filled with the wrong background color (it should use the
default color, but it actually uses the previously active background color).
This commit attempts to fix those issues.
The initial failure is caused by the `SCREEN_INFORMATION::WriteRect` method
throwing an exception when passed an empty viewport. And the reason it's passed
an empty viewport is because that's what the `Viewport::Subtract` method
returns when the result of the subtraction is nothing. The PR fixes the
problem by making the `Viewport::Subtract` method actually return nothing in
that situation.
This is a change in the defined behavior that also required the associated
viewport tests to be updated. However, it does seem a sensible change, since
the `Subtract` method never returns empty viewports under any other
circumstances. And the only place the method seems to be used is in the
`ScrollRegion` implementation, where the previous behavior is guaranteed to
throw an exception.
The other issues are fixed simply by changing the order in which things are
reset in the `AdaptDispatch::HardReset` method. The call to `SoftReset` needed
to be made first, so that the SGR attributes would be reset before the screen
was cleared, thus making sure that the default background color would be used.
And the screen needed to be cleared before the scrollback was erased, otherwise
the last view of the screen would be retained in the scrollback buffer.
These changes also required existing adapter tests to be updated, but not
because of a change in the expected behaviour. It's just that certain tests
relied on the `SoftReset` happening later in the order, so weren't expecting it
to be called if say the scrollback erase had failed. It doesn't seem like the
tests were deliberately trying to verify that the SoftReset _hadn't_ been
called.
In addition to the updates to existing tests, this PR also add a new screen
buffer test which verifies the display and scrollback are correctly cleared
under the conditions that were previously failing.
Fixes#2307.
* Fix DECSTBM parameter interpretation to ignore invalid ranges, and clear the margins on all full screen ranges.
* Add additional scroll margin adapter tests to verify the parameter configurations that were previously incorrect.
* Fix scroll margin adapter tests that weren't actually verifying the conditions that they claimed to be testing.
* Add support for origin mode (DECOM).
* Added a state machine unit test for the origin mode.
* Prevent the cursor position moving below the bottom margin of the scrolling region if the origin mode is relative.
* Only adjust the relative cursor position for origin mode if the scrolling region is actually set.
* Add some screenbuffer unit tests for the origin mode.
* Enhance the soft reset screenbuffer tests to verify the origin mode is reset.
* Move the origin mode flag constructor assignments into the intializer list.
* Implement XTerm's private mode escape sequence for enabling DECCOLM support.
* Add output engine and screen buffer units test for the private mode 40 escape sequence.
* Support OSC to set default background and foreground colors
* Update the Terminal theme when the background changes
* Fix whitespace per code-review
* Add Documentation Comments
Also fix a few outdated comments and whitespace
* Update Telemetry codes per code review
* Add Unit Tests for OSC ForegroundColor and BackgroundColor
* Add a couple additional test cases
* Minor doc and whitespace change per PR review
* Update comment help per code review
* Add another OSC 10 & 11 test case, improve output
* Comments and syntax cleanup per code reviews
