7 commits

Author	SHA1	Message	Date
Jason Volk	4a8302038a	ircd::gpt::vocab: Add tokenization and detokenization count() convenience.	2022-09-21 16:42:07 -07:00
Jason Volk	56d944f33e	ircd::gpt::vocab: Add space-prefix convenience argument.	2022-07-01 17:36:45 -07:00
Jason Volk	78848925ee	ircd::gpt: Various refactoring.	2022-06-19 20:14:22 -07:00
Jason Volk	734948863f	ircd::gpt::vocab: Add token debug string tool.	2021-03-09 04:50:19 -08:00
Jason Volk	53c4260a21	ircd::gpt: Add Basic Latin (lower) and C0 replacement LUT; various.	2021-03-09 04:50:19 -08:00
Jason Volk	29b99dcf4d	ircd::gpt: Split vocab related into separate unit.	2021-03-02 11:13:59 -08:00
Jason Volk	57f9d3fdfb	ircd::gpt: Add vocabulary tokenization; byte-pair merge encoding for natural language.	2021-02-26 14:52:33 -08:00