from Wiktionary, Creative Commons Attribution/Share-Alike License
- v. Third-person singular simple present indicative form of tokenise.
Sorry, no etymologies found.
Last January, I spoke by telephone with Nigerian-American author Nnedi Okorafor about Afrofuturism, a branch of science fiction that features, rather than excludes or tokenises, African characters.
When given a file to generate text based from, as an argument to the learn function, it first tokenises the file then stores pairs in an array of [first word] [second word].