Treat numbers as words #12

New Issue

kj7rrv · 2022-11-26T17:12:10-08:00

kj7rrv commented

2022-11-26 17:12:10 -08:00

Numbers should be treated as words by the tokenizer. It would be great to include commas and periods within numbers as well, but this might not be feasible.

Input: Testing 123,456.789.Test
Ideal output: ['testing', '123,456.789', 'test']
Acceptable output: ['testing', '123', '456', '789', 'test']

Numbers should be treated as words by the tokenizer. It would be great to include commas and periods within numbers as well, but this might not be feasible. Input: `Testing 123,456.789.Test` Ideal output: `['testing', '123,456.789', 'test']` Acceptable output: `['testing', '123', '456', '789', 'test']`

kj7rrv added the

enhancement

model-break

wait-for-break

labels 2022-11-26 18:21:46 -08:00

kj7rrv referenced this issue from a commit

2022-12-24 11:18:41 -08:00

Include numbers in tokenized output

kj7rrv closed this issue

2022-12-24 11:18:41 -08:00

kj7rrv changed title from ~~Numbers should be treated as words~~ to Treat numbers as words

2022-12-24 12:25:21 -08:00