Samuel Sloniker
f1a1ed9e2a
Almost all of the code previously used to make the emoji module optional is removed in this commit. It was always my intent to make the `emoji` module a hard dependency in v3.0.0 and remove the code for making it optional, but for some reason I remembered to do the former but not the latter; in fact, I added emoji-optional code to the new model handling code. I can't completely remove this code because 3.0.0 will not successfully deserialize a model without the `has_emoji` field in the JSON config options, but this commit removes as much as possible without breaking the model format and API version. See also issue #11 |
||
---|---|---|
gptc | ||
models | ||
.gitignore | ||
benchmark.py | ||
GPL-3.0 | ||
LICENSE | ||
pyproject.toml | ||
README.md |
GPTC
General-purpose text classifier in Python
GPTC provides both a CLI tool and a Python library.
Installation
pip install gptc
CLI Tool
Classifying text
gptc classify [-n <max_ngram_length>] <compiled model file>
This will prompt for a string and classify it, then print (in JSON) a dict of
the format {category: probability, category:probability, ...}
to stdout. (For
information about -n <max_ngram_length>
, see section "Ngrams.")
Alternatively, if you only need the most likely category, you can use this:
gptc classify [-n <max_ngram_length>] <-c|--category> <compiled model file>
This will prompt for a string and classify it, outputting the category on stdout (or "None" if it cannot determine anything).
Compiling models
gptc compile [-n <max_ngram_length>] [-c <min_count>] <raw model file>
This will print the compiled model encoded in binary format to stdout.
If -c
is specified, words and ngrams used less than min_count
times will be
excluded from the compiled model.
Packing models
gptc pack <dir>
This will print the raw model in JSON to stdout. See models/unpacked/
for an
example of the format. Any exceptions will be printed to stderr.
Library
gptc.Classifier(model, max_ngram_length=1)
Create a Classifier
object using the given compiled model (as a gptc.Model
object, not as a serialized byte string).
For information about max_ngram_length
, see section "Ngrams."
Classifier.confidence(text)
Classify text
. Returns a dict of the format {category: probability, category:probability, ...}
Note that this may not include values for all categories. If there are no common words between the input and the training data (likely, for example, with input in a different language from the training data), an empty dict will be returned.
Classifier.classify(text)
Classify text
. Returns the category into which the text is placed (as a
string), or None
when it cannot classify the text.
Classifier.model
The classifier's model.
gptc.compile(raw_model, max_ngram_length=1, min_count=1)
Compile a raw model (as a list, not JSON) and return the compiled model (as a
gptc.Model
object).
For information about max_ngram_length
, see section "Ngrams."
Words or ngrams used less than min_count
times throughout the input text are
excluded from the model.
gptc.Model.serialize()
Returns a bytes
representing the model.
gptc.deserialize(encoded_model)
Deserialize a Model
from a bytes
returned by Model.serialize()
.
`gptc.pack(directory, print_exceptions=False)
Pack the model in directory
and return a tuple of the format:
(raw_model, [(exception,),(exception,)...])
Note that the exceptions are contained in single-item tuples. This is to allow more information to be provided without breaking the API in future versions of GPTC.
See models/unpacked/
for an example of the format.
Ngrams
GPTC optionally supports using ngrams to improve classification accuracy. They are disabled by default (maximum length set to 1) for performance reasons. Enabling them significantly increases the time required both for compilation and classification. The effect seems more significant for compilation than for classification. Compiled models are also much larger when ngrams are enabled. Larger maximum ngram lengths will result in slower performance and larger files. It is a good idea to experiment with different values and use the highest one at which GPTC is fast enough and models are small enough for your needs.
Once a model is compiled at a certain maximum ngram length, it cannot be used
for classification with a higher value. If you instantiate a Classifier
with
a model compiled with a lower max_ngram_length
, the value will be silently
reduced to the one used when compiling the model.
Model format
This section explains the raw model format, which is how models are created and edited.
Raw models are formatted as a list of dicts. See below for the format:
[
{
"text": "<text in the category>",
"category": "<the category>"
}
]
GPTC handles raw models as list
s of dict
s of str
s (List[Dict[str, str]]
), and they can be stored
in any way these Python objects can be. However, it is recommended to store
them in JSON format for compatibility with the command-line tool.
Emoji
GPTC treats individual emoji as words.
Example model
An example model, which is designed to distinguish between texts written by
Mark Twain and those written by William Shakespeare, is available in models
.
The raw model is in models/raw.json
; the compiled model is in
models/compiled.json
.
The example model was compiled with max_ngram_length=10
.
Benchmark
A benchmark script is available for comparing performance of GPTC between
different Python versions. To use it, run benchmark.py
with all of the Python
installations you want to test. It tests both compilation and classification.
It uses the default Twain/Shakespeare model for both, and for classification it
uses Mark Antony's "Friends, Romans, countrymen"
speech
from Shakespeare's Julius Caesar.