Tokenize

Pronunciation: /ˈtoʊkənˈaɪˈzi/

Tokenize (verb)

  1. To divide text or data into small parts called tokens, so a program can process each part separately.
  2. To prepare a piece of text for analysis by separating it into meaningful units for counting, tagging, or searching.

Examples

  • The system tokenizes the sentence into small tokens.
  • You can tokenize the input so each token is separate.