terminal/tools
Dustin L. Howett 1df3182865
Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035)
This commit also adds an override UCD and migrates all of the overrides
from GetQuickCharWidth into it.

GetQuickCharWidth
-----------------

The removal of overrides from GQCW reduces the number of comparisons
required for looking up a single character's width from 41 (32
individual ranged comparisons from GQCW + 8+1 from the binary search in
CPWD) to 11 (2 from GQCW, 8+1 from CPWD).

GQCW also incorrectly marked 67 reserved codepoints as `Wide` when they
should have been `Narrow`.

The codepoints whose definitions have changed from `Wide` to `Narrow` are:

```
2E9A 2EF4 2EF5 2EF6 2EF7 2EF8 2EF9 2EFA 2EFB 2EFC 2EFD 2EFE 2EFF 2FD6
2FD7 2FD8 2FD9 2FDA 2FDB 2FDC 2FDD 2FDE 2FDF 2FE0 2FE1 2FE2 2FE3 2FE4
2FE5 2FE6 2FE7 2FE8 2FE9 2FEA 2FEB 2FEC 2FED 2FEE 2FEF 2FFC 2FFD 2FFE
2FFF 31E4 31E5 31E6 31E7 31E8 31E9 31EA 31EB 31EC 31ED 31EE 31EF 321F
A48D A48E A48F FE1A FE1B FE1C FE1D FE1E FE1F FE53 FE67
```

All of them are reserved, but those reserved regions are marked as narrow
in the UCD.

This change also offers us the chance to document exactly why we're
overriding a specific character range. Comments from the override
document will be copied to the generated CPWD table.

New in Unicode 13.0
------------------

Some widths have changed due to previously-reserved characters becoming
_used_ such as U+32FF SQUARE ERA NAME REIWA, the Tangut components
756-768, the entire Khitan Small Script character set, and the Tangut
Ideographs.

A number of the changes in this diff are due to better/worse comment
tracking and the removal of the Emoji/EPres comments. The script once
mistakenly applied comments to packed regions (and it has been updated
to not do so.)

Validation
----------

I build a test application that compared codepoints 0-FFFF for GQCW
against their new registered widths.
2020-10-27 17:36:28 +00:00
..
bcx.cmd Create bx.cmd (#2168) 2019-08-05 20:18:40 -05:00
bcz.cmd Fix 'bcz exclusive' typo (#6938) 2020-07-16 16:58:48 -07:00
bx.cmd Create bx.cmd (#2168) 2019-08-05 20:18:40 -05:00
bx.ps1 Prevent the v1 propsheet from zeroing colors, causing black text on black background. (#2651) 2019-10-02 16:04:59 -07:00
bz.cmd Create bx.cmd (#2168) 2019-08-05 20:18:40 -05:00
ConsoleTypes.natvis Fix SGR indexed colors to distinguish Indexed256 color (and more) (#5834) 2020-05-27 22:34:45 +00:00
echokey.cmd make copying of files windows localization agnostic (#741) 2019-05-21 16:25:54 +00:00
Generate-CodepointWidthsFromUCD.ps1 Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035) 2020-10-27 17:36:28 +00:00
GenerateAppxFromManifest.ps1 Fix a bunch of spelling errors across the project (#4295) 2020-02-10 20:40:01 +00:00
GenerateHeaderForJson.ps1 rename profiles.json to settings.json, clean up the defaults (#5199) 2020-04-01 19:09:42 +00:00
Get-OSSConhostLog.ps1 tools: add Get-OSSConhostLog (#7250) 2020-08-11 20:08:03 +00:00
openbash.cmd Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
opencon.cmd Prevent the v1 propsheet from zeroing colors, causing black text on black background. (#2651) 2019-10-02 16:04:59 -07:00
OpenConsole.psm1 Update clang-format to 10.0 (#7389) 2020-08-25 17:15:43 +00:00
openps.cmd Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
openvt.cmd make copying of files windows localization agnostic (#741) 2019-05-21 16:25:54 +00:00
packages.config Update clang-format to 10.0 (#7389) 2020-08-25 17:15:43 +00:00
razzle.cmd Update TAEF to 10.57.200731005-develop (#7164) 2020-08-03 20:47:02 +00:00
README.md add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
runformat.cmd add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
runft.cmd Turn on Text Buffer unit tests in Azure DevOps CI build (#1057) 2019-05-29 19:51:17 -07:00
runuia.cmd Turn on Text Buffer unit tests in Azure DevOps CI build (#1057) 2019-05-29 19:51:17 -07:00
runut.cmd Fix unittesting our .xaml classes (#4105) 2020-01-10 18:55:31 +00:00
testcon.cmd Fix a bunch of spelling errors across the project (#4295) 2020-02-10 20:40:01 +00:00
tests.xml Fix unittesting our .xaml classes (#4105) 2020-01-10 18:55:31 +00:00
vso_ut.cmd Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
WindbgExtension.js Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00

OpenConsole Tools

These are a collection of tools and scripts to make your life building the OpenConsole project easier. Many of them are designed to be functional clones of tools that we used to use when developing inside the Windows build system.

Razzle

This is a script that quickly sets up your environment variables so that these tools can run easily. It's named after another script used by Windows developers to similar effect.

  • It adds msbuild to your path.
  • It adds the tools directory to your path as well, so all these scripts are easily available.
  • It executes \tools\.razzlerc.cmd to add any other personal configuration to your environment as well, or creates one if it doesn't exist.
  • It sets up the default build configuration to be 'Debug'. If you'd like to manually specify a build configuration, pass the parameter dbg for Debug, and rel for Release.

bcz

bcz can quick be used to clean and build the project. By default, it builds the %DEFAULT_CONFIGURATION% configuration, which is Debug if you use razzle.cmd.

  • bcz dbg can be used to manually build the Debug configuration.
  • bcz rel can be used to manually build the Release configuration.

opencon (and openbash, openps)

opencon can be used to launch the last built OpenConsole binary. If given an argument, it will try and run that program in the launched window. Otherwise it will default to cmd.exe.

openbash is similar, it immediately launches bash.exe (the Windows Subsystem for Linux entrypoint) in your ~ directory.

Likewise, openps launches powershell.

runformat

runformat will format the c++ code to match our coding style.

testcon, runut, runft

runut will automatically run all of the unit tests through TAEF. runft will run the feature tests, and testcon runs all of them. They'll pass any arguments through to TAEF, so you can more finely control the testing.

A recommended workflow is the following command:

bcz dbg && runut /name:*<name of test>*

Where <name of test> is the name of the test testing the relevant feature area you're working on. For example, if I was working on the VT Mouse input support, I would use MouseInputTest as that string, to isolate the mouse input tests. If you'd like to run all the tests, just ignore the /name param: bcz dbg && runut

To make sure your code is ready for a pull request, run the build, then launch the built console, then run the tests in it. The built console will inherit all of the razzle environment, so you can immediately start using the macros:

  1. bcz
  2. opencon
  3. testcon (in the new console window)
  4. runformat

If they all come out green, then you're ready for a pull request!