from Wiktionary, Creative Commons Attribution/Share-Alike License.

  • noun computing A system that parses an input stream into tokens.


Sorry, no etymologies found.


  • The word_tokenize function uses NLTK’s currently recommended word tokenizer, TreebankWordTokenizer, and should be fed one sentence at a time.

    m3gan / ((NLTK) (is (fun)))

  • Reason column shows that the first request was from the original html lookahead tokenizer while the subsequent successful downloads were triggered by the restarted html lookahead tokenizer:

    Site Home

  • It tries to be as compliant as possible but uses some (helpful) parts of the CSS 2.1 tokenizer.

    Softpedia - Windows - All

  • · Or just use the included message, string, dataIO, hashtable, dequeue, string-tokenizer, reference-count, regular expression and pattern matching parser, and object-pool code by themselves, as handy cross-platform utility classes.

    Softpedia - Windows - All

  • A tokenizer for Hailo which splits on whitespace, mostly.

  • The cssutils tokenizer is a customized implementation of CSS3 Module: Syntax (W3C Working Draft 13 August 2003) which itself is based on the CSS 2.1 tokenizer.

    Softpedia - Windows - All

  • The usage file parser and the input line tokenizer are built on the excellent

    Planet Debian

  • SQLite standard tokenizer xml xmlreader xmlwriter zlib - New Projects

  • Jonathan Howard was probably operating under too much stress when he created Like, Python using Python's tokenizer to add keywords to Python's lexical understanding. full roast blend

  • Implement a tokenizer (Difficulty: medium/high, priority: low)

    KDE TechBase - Recent changes [en]


New comments are temporarily disabled while we update our database.