Definitions

from Wiktionary, Creative Commons Attribution/Share-Alike License

  • n. A system that parses an input stream into tokens.

Etymologies

Sorry, no etymologies found.

Examples

  • The word_tokenize function uses NLTK’s currently recommended word tokenizer, TreebankWordTokenizer, and should be fed one sentence at a time.

    m3gan / ((NLTK) (is (fun)))

  • Reason column shows that the first request was from the original html lookahead tokenizer while the subsequent successful downloads were triggered by the restarted html lookahead tokenizer:

    Site Home

  • Jonathan Howard was probably operating under too much stress when he created Like, Python using Python's tokenizer to add keywords to Python's lexical understanding.

    fullasagoog.com full roast blend

  • The usage file parser and the input line tokenizer are built on the excellent

    Planet Debian

  • · Or just use the included message, string, dataIO, hashtable, dequeue, string-tokenizer, reference-count, regular expression and pattern matching parser, and object-pool code by themselves, as handy cross-platform utility classes.

    Softpedia - Windows - All

  • SQLite standard tokenizer xml xmlreader xmlwriter zlib

    Freelancer.com - New Projects

  • Implement a tokenizer (Difficulty: medium/high, priority: low)

    KDE TechBase - Recent changes [en]

  • The cssutils tokenizer is a customized implementation of CSS3 Module: Syntax (W3C Working Draft 13 August 2003) which itself is based on the CSS 2.1 tokenizer.

    Softpedia - Windows - All

  • It tries to be as compliant as possible but uses some (helpful) parts of the CSS 2.1 tokenizer.

    Softpedia - Windows - All

  • A tokenizer for Hailo which splits on whitespace, mostly.

    search.cpan.org

Comments

Log in or sign up to get involved in the conversation. It's quick and easy.