Skip to main content

CommonTokenize

Callable

  • CommonTokenize(input: string, options?: { delimiters: [string, string][]; includeDelimiters?: boolean; trimTrailingSpaces?: boolean }): string[]

  • Splits a string into tokens by delimiters and spaces

    Parameters

    • input: string
    • options: { delimiters: [string, string][]; includeDelimiters?: boolean; trimTrailingSpaces?: boolean } = ...
      • delimiters: [string, string][]
      • optionalincludeDelimiters: boolean
      • optionaltrimTrailingSpaces: boolean

    Returns string[]