terminal/src/host
Steffen 06b3931418
Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl (#4422)
Replace `utf8Parser` with `til::u8u16` in order to have the same
conversion algorithms used in terminal and conhost.

This PR addresses item 2 in this list:
1. ✉ Implement `til::u8u16` and `til::u16u8` (done in PR #4093)
2. ✔ **Unify UTF-8 handling using `til::u8u16` (this PR)**
    2.1. ✔ **Update VtInputThread::_HandleRunInput()**
    2.2. ✔ **Update ApiRoutines::WriteConsoleAImpl()**
    2.3.  (optional / ask the core team) Remove Utf8ToWideCharParser from the code base to avoid further use
3.  Enable BOM discarding (follow up)
    3.1.  extend `til::u8u16` and `til::u16u8` with a 3rd parameter to enable discarding the BOM
    3.2.  Make use of the 3rd parameter to discard the BOM in all current function callers, or (optional / ask the core team) make it the default for  `til::u8u16` and `til::u16u8` 
4.  Find UTF-16 to UTF-8 conversions and examine if they can be unified, too (follow up)

Closes #4086
Closes #3378
2020-02-03 18:06:55 -08:00
..
dll Unify and clean up the common build properties (#3429) 2019-11-05 14:29:11 -08:00
exe Add experimental retro terminal effects (#3468) 2019-12-12 13:44:01 +00:00
ft_host Merge inbox changes up to eb480b6bb 2020-02-03 11:49:42 -08:00
ft_integrity Merged PR 4235821: [Git2Git] Reflect some sources.dep changes from OS 2020-01-31 21:27:33 +00:00
ft_uia Update TAEF to 10.51 and remove the private dep on Taef.TestAdapter (#4450) 2020-02-03 22:14:43 +00:00
lib Migrate Search module as a shared component for Terminal Search (#3279) 2019-11-14 14:36:41 -08:00
ut_host Add support for VT100 Auto Wrap Mode (DECAWM) (#3943) 2020-02-04 00:20:21 +00:00
ut_lib Unify and clean up the common build properties (#3429) 2019-11-05 14:29:11 -08:00
_output.cpp Correct fill attributes when scrolling and erasing (#3100) 2019-12-10 23:14:40 +00:00
_output.h Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
_stream.cpp Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl (#4422) 2020-02-03 18:06:55 -08:00
_stream.h Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl (#4422) 2020-02-03 18:06:55 -08:00
alias.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
alias.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
ApiRoutines.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
cmdline.cpp Replace macros with constexpr part 2 (#3416) 2019-11-04 07:37:47 -06:00
cmdline.h Remove unneeded VT-specific control character handling (#4289) 2020-01-29 19:18:46 +00:00
CommandListPopup.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CommandListPopup.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CommandNumberPopup.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CommandNumberPopup.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conapi.h Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
conareainfo.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conareainfo.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conattrs.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conddkrefs.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conhost.rcv Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
conhostv2_traceviewpp.tvpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
conimeinfo.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conimeinfo.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conserv.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
ConsoleArguments.cpp Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
ConsoleArguments.hpp Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
consoleInformation.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conv.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
convarea.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
conwinuserrefs.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CopyFromCharPopup.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CopyFromCharPopup.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CopyToCharPopup.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CopyToCharPopup.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CursorBlinker.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
CursorBlinker.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
dbcs.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
dbcs.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
directio.cpp Replace some macros with constexpr (#3362) 2019-11-01 10:33:09 -07:00
directio.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
dirs Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
exemain.cpp Force the use of v2 (non-legacy) conhost when in ConPTY mode (#1935) 2019-07-12 15:20:45 -07:00
getset.cpp Add support for VT100 Auto Wrap Mode (DECAWM) (#3943) 2020-02-04 00:20:21 +00:00
getset.h Add support for VT100 Auto Wrap Mode (DECAWM) (#3943) 2020-02-04 00:20:21 +00:00
globals.cpp Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
globals.h Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
handle.cpp Removed using namespace directive from header files (#955) 2019-05-30 11:14:21 -07:00
handle.h Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
history.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
history.h Replace macros with constexpr part 2 (#3416) 2019-11-04 07:37:47 -06:00
host-common.vcxitems Migrate Search module as a shared component for Terminal Search (#3279) 2019-11-14 14:36:41 -08:00
IIoProvider.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
init.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
init.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
input.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
input.h Replace macros with constexpr part 2 (#3416) 2019-11-04 07:37:47 -06:00
inputBuffer.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
inputBuffer.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
inputKeyInfo.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
inputReadHandleData.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
inputReadHandleData.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
misc.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
misc.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
ntprivapi.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
ntprivapi.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
output.cpp A pair of fixes related to cursor movement in conpty (#4372) 2020-01-30 20:14:16 +00:00
output.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
outputStream.cpp Add support for VT100 Auto Wrap Mode (DECAWM) (#3943) 2020-02-04 00:20:21 +00:00
outputStream.hpp Add support for VT100 Auto Wrap Mode (DECAWM) (#3943) 2020-02-04 00:20:21 +00:00
popup.cpp Get rid of the SCREEN_INFORMATION::LineChar array (#3371) 2019-10-31 10:12:41 -05:00
popup.h Replace some macros with constexpr (#3362) 2019-11-01 10:33:09 -07:00
precomp.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
precomp.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
PtySignalInputThread.cpp Correct improper usage of THROW_IF_NULL_ALLOC (#4128) 2020-01-07 13:27:18 -08:00
PtySignalInputThread.hpp Revert locking changes (#3488) 2019-11-08 13:44:52 -08:00
readData.cpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
readData.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataCooked.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataCooked.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataDirect.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataDirect.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataRaw.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
readDataRaw.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
registry.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
registry.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
renderData.cpp Search - add search box control and implement search experience (#3590) 2019-12-17 15:52:37 +00:00
renderData.hpp Search - add search box control and implement search experience (#3590) 2019-12-17 15:52:37 +00:00
renderFontDefaults.cpp Allow FontInfo{,Base,Desired} to store a font name > 32 wch (#3107) 2019-10-14 21:23:45 -07:00
renderFontDefaults.hpp Allow FontInfo{,Base,Desired} to store a font name > 32 wch (#3107) 2019-10-14 21:23:45 -07:00
res.rc Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
resource.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
runft.bat Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
runtests.bat Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
runut.bat Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
ScreenBufferRenderTarget.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
ScreenBufferRenderTarget.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
screenInfo.cpp Remove unneeded VT-specific control character handling (#4289) 2020-01-29 19:18:46 +00:00
screenInfo.hpp Remove unneeded VT-specific control character handling (#4289) 2020-01-29 19:18:46 +00:00
scrolling.cpp Accessibility: Set-up UIA Tree (#1691) 2019-07-29 15:21:15 -07:00
scrolling.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
selection.cpp Allow FontInfo{,Base,Desired} to store a font name > 32 wch (#3107) 2019-10-14 21:23:45 -07:00
selection.hpp Allow FontInfo{,Base,Desired} to store a font name > 32 wch (#3107) 2019-10-14 21:23:45 -07:00
selectionInput.cpp Migrate Search module as a shared component for Terminal Search (#3279) 2019-11-14 14:36:41 -08:00
selectionState.cpp Removed using namespace directive from header files (#955) 2019-05-30 11:14:21 -07:00
server.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
settings.cpp Add support for the DECSCNM screen mode (#3817) 2020-01-22 22:29:50 +00:00
settings.hpp Add support for the DECSCNM screen mode (#3817) 2020-01-22 22:29:50 +00:00
sources.inc Merged PR 4182306: [Git2Git] Merged PR 4182266: conhost: don't use D3DCompiler on inside-windows builds (and delete the shaders) 2020-01-23 00:42:56 +00:00
sources.test.inc Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00
srvinit.cpp Revert locking changes (#3488) 2019-11-08 13:44:52 -08:00
srvinit.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
stream.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
stream.h add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
telemetry.cpp Allow FontInfo{,Base,Desired} to store a font name > 32 wch (#3107) 2019-10-14 21:23:45 -07:00
telemetry.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
tracing.cpp Accessibility: Refactor Providers (#2414) 2019-08-20 16:32:44 -07:00
tracing.hpp Accessibility: Set-up UIA Tree (#1691) 2019-07-29 15:21:15 -07:00
utf8ToWideCharParser.cpp Compensate for non-minimal UTF-8 encodings (#3380) 2019-10-31 10:50:34 -07:00
utf8ToWideCharParser.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
utils.cpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
utils.hpp add clang-format conf to the project, format the c++ code (#1141) 2019-06-11 13:27:09 -07:00
VtInputThread.cpp Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl (#4422) 2020-02-03 18:06:55 -08:00
VtInputThread.hpp Unify UTF-8 handling using til::u8u16 & revise WriteConsoleAImpl (#4422) 2020-02-03 18:06:55 -08:00
VtIo.cpp Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
VtIo.hpp Add just the test infrastructure bits from #4354 (#4382) 2020-01-29 16:33:06 +00:00
writeData.cpp Removed using namespace directive from header files (#955) 2019-05-30 11:14:21 -07:00
writeData.hpp Initial release of the Windows Terminal source code 2019-05-02 15:29:04 -07:00