Definitions
from Wiktionary, Creative Commons Attribution/Share-Alike License.
- noun Alternative spelling of
tokenizer .
Etymologies
Sorry, no etymologies found.
Support
Help support Wordnik (and make this page ad-free) by adopting the word tokeniser.
Examples
-
The tokeniser is a bit different from previous ones, as we want to specifically separate out punctuation - we don't really have to do this as actually leaving punctuation intact gives a very nice bit of text generation, but it's interesting to view the relationship between these different types of punctuation and the words of the text.
Planet PHP 2009
Comments
Log in or sign up to get involved in the conversation. It's quick and easy.