Definitions

from Wiktionary, Creative Commons Attribution/Share-Alike License.

  • noun computing A system that parses an input stream into tokens.

Etymologies

Sorry, no etymologies found.

Support

Help support Wordnik (and make this page ad-free) by adopting the word tokenizer.

Examples

  • The word_tokenize function uses NLTK’s currently recommended word tokenizer, TreebankWordTokenizer, and should be fed one sentence at a time.

    m3gan / ((NLTK) (is (fun))) 2009

  • Reason column shows that the first request was from the original html lookahead tokenizer while the subsequent successful downloads were triggered by the restarted html lookahead tokenizer:

    Site Home EricLaw [MSFT] 2011

  • It tries to be as compliant as possible but uses some (helpful) parts of the CSS 2.1 tokenizer.

    Softpedia - Windows - All Softpedia Linux 2010

  • · Or just use the included message, string, dataIO, hashtable, dequeue, string-tokenizer, reference-count, regular expression and pattern matching parser, and object-pool code by themselves, as handy cross-platform utility classes.

    Softpedia - Windows - All 2010

  • A tokenizer for Hailo which splits on whitespace, mostly.

    search.cpan.org 2010

  • The cssutils tokenizer is a customized implementation of CSS3 Module: Syntax (W3C Working Draft 13 August 2003) which itself is based on the CSS 2.1 tokenizer.

    Softpedia - Windows - All Softpedia Linux 2010

  • The usage file parser and the input line tokenizer are built on the excellent

    Planet Debian Matt Brubeck 2010

  • SQLite standard tokenizer xml xmlreader xmlwriter zlib

    Freelancer.com - New Projects 2010

  • Jonathan Howard was probably operating under too much stress when he created Like, Python using Python's tokenizer to add keywords to Python's lexical understanding.

    fullasagoog.com full roast blend DigitalMediaMinute 2010

  • Implement a tokenizer (Difficulty: medium/high, priority: low)

    KDE TechBase - Recent changes [en] 2010

Comments

Log in or sign up to get involved in the conversation. It's quick and easy.