lib:tokenizer

Modules

• Text.Tokenizer
• Text.Tokenizer.BlackWhiteSet
• Text.Tokenizer.Split
• Text.Tokenizer.Types
• Text.Tokenizer.Uniqueness

Dependencies

• base lib:base ^>=4.14.3.0
• containers lib:containers >=0
• transformers lib:transformers >=0

Reverse dependencies

Direct only. Not exhaustive.

• tokenizer exe:examples
• tokenizer test:tokenizer-test