MirrorHub/construct
0
0
Fork 0
mirror of https://github.com/matrix-construct/construct synced 2024-06-08 13:08:56 +02:00
Code Issues Releases Wiki Activity

ircd::gpt::vocab: Simplify overflow truncation length.

Browse source
This commit is contained in:
Jason Volk 2022-03-05 11:36:38 -08:00
parent 18df9bd2da
commit 366289823e
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
ircd/gpt_vocab.cc
View file

@ -394,7 +394,7 @@ ircd::gpt::vocab::pre_tokenize(u8x16 (&token)[16],
// from the input, but the truncation is determined after a transform
// which may have a different size; this has to be offset back now.
if(ret[1] + off > 16)
len -= (ret[1] + off) - 16;
len = 16;
// Pack the utf-8 codepoints into the result token
token[i] = u8x16{0};