mirror of
https://github.com/imfing/hextra.git
synced 2025-05-14 11:26:26 -04:00

- Added support for CJK (Chinese, Japanese, Korean) languages in FlexSearch encoding. - Introduced `isCJK` function to detect language and select appropriate encoding method. - Implemented `encodeCJK` and `encodeDefault` functions for different tokenization strategies.