from Wiktionary, Creative Commons Attribution/Share-Alike License

  • n. Alternative spelling of tokenizer.


Sorry, no etymologies found.


  • The tokeniser is a bit different from previous ones, as we want to specifically separate out punctuation - we don't really have to do this as actually leaving punctuation intact gives a very nice bit of text generation, but it's interesting to view the relationship between these different types of punctuation and the words of the text.

    Planet PHP


Log in or sign up to get involved in the conversation. It's quick and easy.