kj7rrv/gptc
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases 12 Wiki Activity

Include numbers in tokenized output

Browse Source 
Closes #12
This commit is contained in:
Samuel Sloniker 2022-12-24 10:42:50 -08:00
parent 74b2ba81b9
commit 295a1189de
Signed by: kj7rrv
GPG Key ID: 1BB4029E66285A62
1 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
gptc/tokenizer.py
View File

@ -19,7 +19,12 @@ def tokenize(text: str, max_ngram_length: int = 1) -> List[str]:
tokens = [""] tokens = [""]
for char in converted_text: for char in converted_text:
if char.isalpha() or char == "'": if (
char.isalpha()
or char.isnumeric()
or char == "'"
or (char in ",." and (" " + tokens[-1])[-1].isnumeric())
):
tokens[-1] += char tokens[-1] += char
elif emoji.is_emoji(char): elif emoji.is_emoji(char):
tokens.append(char) tokens.append(char)