Tokenizer

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

Installation

npm install @astryxdesign/core@0.7.1 @astryxdesign/theme-neutral@0.7.1 @stylexjs/stylex@0.19.0

Requires React 19+, base CSS and a Theme provider. See Setup.

Tokenizer

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

Usage guidance

• Write a placeholder that tells users what they can search for, such as "Search people..." or "Add tags...", so the input is not a blank mystery.
• Set maxEntries when the number of selections should be bounded, like limiting a review to 5 approvers.
• Use hasCreate for free-form tagging where users need to enter values that do not exist in the search source.
• Show validation status with the status prop so users know immediately when a selection is missing or invalid.
• Avoid: Don't use Tokenizer for single-item selection; use Typeahead instead. Tokenizer is for building sets of two or more items.
• Avoid: Avoid applying custom colors to individual tokens inside a Tokenizer; use the default token style for visual consistency across the set.
• Avoid: Don't hide the label; every Tokenizer needs a visible label so users understand what they are selecting. Use isLabelHidden only when surrounding context makes the purpose obvious.
• Avoid: Wrap a disabled Tokenizer in Tooltip to explain why it is disabled; disabled controls swallow the hover events the wrapper needs. Use the disabledMessage prop instead.

Theming API

{
  "targets": [
    {
      "className": "astryx-tokenizer",
      "visualProps": [
        "size",
        "status"
      ],
      "states": [
        "disabled"
      ]
    }
  ]
}

Anatomy

• Label: The visible text above the input describing what the user is selecting. Also used as the accessible name.
• Token chips: Removable chips representing each selected item. Each chip shows a label and a remove button.
• Search input: The text input where users type to search the data source. Hides when maxEntries is reached.
• Dropdown menu: The search results list that appears below the input as the user types.
• Spinner: Loading indicator shown at the end of the field while a search is in flight.
• End content: A trailing slot after the input for action buttons, counts, or other controls.
• Clear button: A button that removes all selected tokens at once. Shown when hasClear is true and tokens are present.

Examples

• Tokenizer: A tokenizer with preset tags and search source.
• Tokenizer — Clear: Tokenizer with a built-in clear-all button for bulk removal of all selected tokens.
• Tokenizer — Creatable: Free-text tokenizer for creating custom tags and a combined create-or-search pattern. Use when users need to enter values that may not exist in a predefined list.
• Tokenizer — End Content: Tokenizer with an action button in the end slot. Use for inline actions like applying selections alongside the input.
• Tokenizer — Icon: Tokenizer with a leading search icon to visually reinforce the search behavior.
• Tokenizer — Max Entries: Tokenizer with a maximum selection limit. The input hides automatically when the limit is reached, preventing further additions.
• Tokenizer — Overflow: Tokenizer with overflow truncation when unfocused. Inline mode pushes content down on expand; layer mode overlays without shifting layout.
• Tokenizer — States: Tokenizer in disabled, error, warning, and success states. Use to communicate validation feedback or lock a selection from editing.

Complete example dependencies

Some examples import optional icons. The per-example install command in the examples list identifies exactly what that example needs. For all examples on this page:

npm install @heroicons/react@2.2.0