kj7rrv/gptc
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases 12 Wiki Activity

Make sure text is lowercase

Browse Source
This commit is contained in:
Samuel Sloniker 2022-11-22 11:44:13 -08:00
parent c2cd6f62fb
commit 7d1cbcaee0
Signed by: kj7rrv
GPG Key ID: 1BB4029E66285A62
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
gptc/tokenizer.py
View File

@ -17,6 +17,7 @@ def tokenize(
converted_text: Union[str, List[str]] = text.lower() converted_text: Union[str, List[str]] = text.lower()
if has_emoji and use_emoji: if has_emoji and use_emoji:
text = text.lower()
parts = [] parts = []
highest_end = 0 highest_end = 0
for emoji_part in emoji.emoji_list(text): for emoji_part in emoji.emoji_list(text):