Definitions
from Wiktionary, Creative Commons Attribution/Share-Alike License.
- verb Third-person singular simple present indicative form of
tokenise .
Etymologies
Sorry, no etymologies found.
Support
Help support Wordnik (and make this page ad-free) by adopting the word tokenises.
Examples
-
Last January, I spoke by telephone with Nigerian-American author Nnedi Okorafor about Afrofuturism, a branch of science fiction that features, rather than excludes or tokenises, African characters.
Archive 2009-11-01 2009
-
PHP_CodeSniffer tokenises PHP, JavaScript and CSS files and detects violations of a defined set of coding standards. phpmd scans PHP source code and looks for potential problems such as possible bugs, suboptimal code or overcomplicated expressions.
-
PHP_CodeSniffer tokenises PHP, JavaScript and CSS files and detects violations of a defined set of coding standards. phpmd scans PHP source code and looks for potential problems such as possible bugs, suboptimal code or overcomplicated expressions.
Planet PHP 2009
-
When given a file to generate text based from, as an argument to the learn function, it first tokenises the file then stores pairs in an array of [first word] [second word].
Planet PHP 2009
Comments
Log in or sign up to get involved in the conversation. It's quick and easy.