terminal/src/types
Dustin L. Howett 1df3182865
Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035)
This commit also adds an override UCD and migrates all of the overrides
from GetQuickCharWidth into it.

GetQuickCharWidth
-----------------

The removal of overrides from GQCW reduces the number of comparisons
required for looking up a single character's width from 41 (32
individual ranged comparisons from GQCW + 8+1 from the binary search in
CPWD) to 11 (2 from GQCW, 8+1 from CPWD).

GQCW also incorrectly marked 67 reserved codepoints as `Wide` when they
should have been `Narrow`.

The codepoints whose definitions have changed from `Wide` to `Narrow` are:

```
2E9A 2EF4 2EF5 2EF6 2EF7 2EF8 2EF9 2EFA 2EFB 2EFC 2EFD 2EFE 2EFF 2FD6
2FD7 2FD8 2FD9 2FDA 2FDB 2FDC 2FDD 2FDE 2FDF 2FE0 2FE1 2FE2 2FE3 2FE4
2FE5 2FE6 2FE7 2FE8 2FE9 2FEA 2FEB 2FEC 2FED 2FEE 2FEF 2FFC 2FFD 2FFE
2FFF 31E4 31E5 31E6 31E7 31E8 31E9 31EA 31EB 31EC 31ED 31EE 31EF 321F
A48D A48E A48F FE1A FE1B FE1C FE1D FE1E FE1F FE53 FE67
```

All of them are reserved, but those reserved regions are marked as narrow
in the UCD.

This change also offers us the chance to document exactly why we're
overriding a specific character range. Comments from the override
document will be copied to the generated CPWD table.

New in Unicode 13.0
------------------

Some widths have changed due to previously-reserved characters becoming
_used_ such as U+32FF SQUARE ERA NAME REIWA, the Tangut components
756-768, the entire Khitan Small Script character set, and the Tangut
Ideographs.

A number of the changes in this diff are due to better/worse comment
tracking and the removal of the Emoji/EPres comments. The script once
mistakenly applied comments to packed regions (and it has been updated
to not do so.)

Validation
----------

I build a test application that compared codepoints 0-FFFF for GQCW
against their new registered widths.
2020-10-27 17:36:28 +00:00
..
inc Add support for more OSC color formats (#7578) 2020-10-14 17:29:10 -07:00
lib Add support for more OSC color formats (#7578) 2020-10-14 17:29:10 -07:00
ut_types Optimize the binary size of the XOrg color table (#7929) 2020-10-15 17:45:33 -07:00
CodepointWidthDetector.cpp Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035) 2020-10-27 17:36:28 +00:00
colorTable.cpp Optimize the binary size of the XOrg color table (#7929) 2020-10-15 17:45:33 -07:00
convert.cpp Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035) 2020-10-27 17:36:28 +00:00
dirs Switch to v5 UUIDs as profile GUIDs for the default profiles (#913) 2019-05-21 13:29:16 -07:00
Environment.cpp Fix environment block creation (#7401) 2020-08-25 18:16:48 +00:00
FocusEvent.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
GlyphWidth.cpp Add GH issue IDs to all the suppress/disables that I left behind as they were a bit too challenging to solve with this giant PR 2019-09-05 11:14:43 -07:00
IBaseData.h Search - add search box control and implement search experience (#3590) 2019-12-17 15:52:37 +00:00
IControlAccessibilityInfo.h hook up UIA tree to WPF control (#4548) 2020-02-24 23:17:55 +00:00
IInputEvent.cpp Make loop in IInputEvent more consistent and modern (#4959) 2020-03-17 10:52:33 -07:00
IInputEventStreams.cpp Improve debugging experience of key events (#2872) 2019-09-24 13:50:53 -07:00
IUiaData.h Properly represent block selections in UIA (#4991) 2020-03-18 21:03:51 +00:00
IUiaEventDispatcher.h Attach UiaRenderer and Fire Selection Changed Events (#2989) 2019-12-11 13:52:49 -08:00
IUiaTraceable.h Add more object ID tracing for Accessibility (#5215) 2020-04-03 20:06:47 +00:00
KeyEvent.cpp Improve debugging experience of key events (#2872) 2019-09-24 13:50:53 -07:00
MenuEvent.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
ModifierKeyState.cpp Fix a bunch of spelling errors across the project (#4295) 2020-02-10 20:40:01 +00:00
MouseEvent.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
precomp.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
precomp.h Always create a new environment block before we spawn a process (#7243) 2020-08-11 23:58:45 +00:00
ScreenInfoUiaProviderBase.cpp Replace basic_string_view<T> with span<const T> (#6921) 2020-07-15 16:40:42 +00:00
ScreenInfoUiaProviderBase.h Cache the size viewport structure inside TextBuffer (#6841) 2020-07-09 11:18:25 +00:00
sources.inc Add support for more OSC color formats (#7578) 2020-10-14 17:29:10 -07:00
TermControlUiaProvider.cpp Properly represent block selections in UIA (#4991) 2020-03-18 21:03:51 +00:00
TermControlUiaProvider.hpp hook up UIA tree to WPF control (#4548) 2020-02-24 23:17:55 +00:00
TermControlUiaTextRange.cpp Fix UIA ScrollIntoView at EndExclusive (#7868) 2020-10-09 20:27:13 +00:00
TermControlUiaTextRange.hpp Fix UIA ScrollIntoView at EndExclusive (#7868) 2020-10-09 20:27:13 +00:00
ThemeUtils.cpp Use standard 1px window borders on NC Island Window (#3394) 2019-11-04 15:45:40 -08:00
UiaTextRangeBase.cpp Fix UIA ScrollIntoView at EndExclusive (#7868) 2020-10-09 20:27:13 +00:00
UiaTextRangeBase.hpp Fix UIA ScrollIntoView at EndExclusive (#7868) 2020-10-09 20:27:13 +00:00
UiaTracing.cpp Add more object ID tracing for Accessibility (#5215) 2020-04-03 20:06:47 +00:00
UiaTracing.h Add more object ID tracing for Accessibility (#5215) 2020-04-03 20:06:47 +00:00
unicode_width_overrides.xml Fully regenerate CodepointWidthDetector from Unicode 13.0 (#8035) 2020-10-27 17:36:28 +00:00
Utf16Parser.cpp C26447,C26440 - is noexcept but can throw or doesn't throw but not noexcept 2019-08-29 15:23:07 -07:00
utils.cpp Add support for more OSC color formats (#7578) 2020-10-14 17:29:10 -07:00
viewport.cpp ci: run spell check in CI, fix remaining issues (#4799) 2020-03-25 11:02:53 -07:00
WindowBufferSizeEvent.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00