mirror of
https://github.com/imfing/hextra.git
synced 2025-10-16 08:21:56 -04:00

- Added support for CJK (Chinese, Japanese, Korean) languages in FlexSearch encoding. - Introduced `isCJK` function to detect language and select appropriate encoding method. - Implemented `encodeCJK` and `encodeDefault` functions for different tokenization strategies.