MirrorHub/construct
0
0
Fork 0
mirror of https://github.com/matrix-construct/construct synced 2024-06-11 06:28:55 +02:00
Code Issues Releases Wiki Activity

ircd::gpt::vocab: Simplify UTF-8 length gauge.

Browse source
This commit is contained in:
Jason Volk 2021-08-24 17:44:38 -07:00
parent 43a02bb598
commit 782379aeb4
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
ircd/gpt_vocab.cc
View file

@ -678,9 +678,9 @@ ircd::gpt::vocab::bpe_prepare(u8x16 (&out)[16][2],
simd::strlen(in)
};
const u32x16 cplen
const u8x16 cplen
(
utf8::length(utf8::decode(in))
utf8::length(in)
);
u32x16 idx;