# Tokenizer

- Page: https://jiguang132.com/resources/astryx/docs/tokenizer
- Resource: Astryx (https://jiguang132.com/resources/astryx.md)
- All guides: https://jiguang132.com/resources/astryx/docs.md

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

## Installation

```bash
npm install @astryxdesign/core@0.7.1 @astryxdesign/theme-neutral@0.7.1 @stylexjs/stylex@0.19.0
```

Requires React 19+, base CSS and a Theme provider. See [Setup](/resources/astryx/docs/getting-started).

## Tokenizer

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

| Prop | Type | Default | Description |
|---|---|---|---|
| label * | string |  | Accessible label for the input. |
| searchSource * | SearchSource<T> |  | Data source providing search and bootstrap methods for populating the dropdown. |
| value * | T[] |  | Array of currently selected items. |
| onChange * | (items: T[], change: TokenizerChange<T>) => void |  | Called when selection changes. The change argument includes the affected item and type ('add' \| 'create' \| 'remove' \| 'reorder'). Additions and removals (including Backspace on an empty input) are announced to screen readers via a polite live region. |
| placeholder | string |  | Input placeholder text. Only shown when no tokens are selected. |
| maxEntries | number |  | Maximum number of selections allowed. Input is hidden when the limit is reached. |
| hasClear | boolean | false | Show a clear-all button for bulk removal of all tokens. |
| renderToken | (item: T, onRemove: () => void) => ReactNode |  | Custom render function for selected tokens. Default renders Token with label and onRemove. |
| renderItem | (item: T) => ReactNode |  | Custom render function for dropdown items. Default renders TypeaheadItem. |
| isDisabled | boolean | false | Disables the input and all token interactions. |
| htmlName | string |  | The HTML name attribute for form submissions. Renders one hidden input per selected item id. |
| disabledMessage | string |  | Explains why the tokenizer is disabled. With isDisabled, shows a tooltip on hover/keyboard focus and keeps the input focusable via aria-disabled (input stays blocked). Use this instead of wrapping a disabled Tokenizer in Tooltip. Disabled controls swallow the hover events an external Tooltip needs. |
| status | {type: 'warning' \| 'error' \| 'success', message?: string} |  | Validation status object with type and message for error/warning/success states. |
| statusVariant | 'attached' \| 'detached' \| 'tooltip' | 'attached' | How the status message is placed relative to the input. attached overlaps directly below the input (bordered treatment); detached floats below as a separate element with spacing; tooltip reveals the message in an info-tip button. |
| isLabelHidden | boolean | false | Visually hides the label while keeping it accessible. |
| description | string |  | Helper text displayed below the label. |
| isRequired | boolean | false | Marks the field as required. |
| isOptional | boolean | false | Shows an optional indicator on the label. |
| labelTooltip | string |  | Tooltip text shown on the label. |
| hasEntriesOnFocus | boolean | false | Show bootstrap results on focus before typing. |
| maxMenuItems | number | 10 | Maximum number of search results to display. The hasCreate entry is offered on top of them, so a menu can show one more than this. |
| menuWidth | number |  | Fixed dropdown width in pixels. The menu never shrinks below its anchor width. |
| minQueryLength | number | 1 | Minimum query length before the search source is queried. Below it no search runs, and the menu stays closed — unless hasCreate is set, in which case the "Create ..." entry is still offered, being derived from the typed text rather than fetched for it. |
| emptySearchText | ReactNode | 'No results found' | Content shown when the query matched nothing. Takes a ReactNode, so the dead end can carry a link or a create row. Announced in a polite live region as the text it renders, read from the DOM; aria-hidden parts stay out of both, and content that renders no text announces nothing. null counts as not given, like undefined, and falls through to the default; pass an empty string to render nothing. |
| hasAutoFocus | boolean | false | Auto-focus the input on mount. |
| size | 'sm' \| 'md' \| 'lg' | 'md' | Input and token size. |
| debounceMs | number | 150 | Debounce delay in ms before triggering search. Set to 0 for synchronous sources. |
| hasCreate | boolean | false | Allow users to create new tokens from free-text input. When true, a "Create" option appears in the dropdown for typed text that doesn't match existing results. The onChange change type is 'create' for these items. |
| onChangeQuery | (query: string) => void |  | Callback fired when the search query text changes. |
| startIcon | ReactNode \| IconType |  | Icon to display at the start of the input, before any tokens. Accepts a semantic icon name, an SVG icon component, or a ReactNode directly. |
| endContent | ReactNode |  | Content to display at the end of the input row. Useful for buttons, result counts, or other controls. |
| handleRef | React.Ref<TokenizerHandle> |  | Imperative handle exposing focusInput(), focusFirstToken(), focusLastToken(), clearInput(), and selectAll(). |
| width | SizeValue |  | Width of the field (number = pixels, string used as-is, e.g. "100%"). Sizes the whole field (label, control, and status) so they stay aligned. |
| tokenOverflowBehavior | 'none' \| 'unfocusedInline' \| 'unfocusedLayer' | 'none' | Controls how tokens overflow when the container is too narrow. |
| onFocus | (e: FocusEvent<HTMLInputElement>) => void |  | Fires when focus enters the tokenizer input. |
| onBlur | (e: FocusEvent<HTMLInputElement>) => void |  | Fires when focus leaves the tokenizer input. |
| containerXstyle | StyleXStyles |  | StyleX styles for the field container (label, control, and status together). No effect inside InputGroup (style the InputGroup instead) or in a `horizontal-labels` FormLayout. Root `xstyle`, `className`, and `style` are not accepted. |
| containerClassName | string |  | Class name for the field container. Same targeting as `containerXstyle`. |
| containerStyle | CSSProperties |  | Inline styles for the field container. Same targeting as `containerXstyle`. |
| controlXstyle | StyleXStyles |  | StyleX styles for the visual control box: the element that paints the border and background and holds icons and end controls. Never the native input. |
| controlClassName | string |  | Class name for the visual control box. |
| controlStyle | CSSProperties |  | Inline styles for the visual control box. |

### Usage guidance

- Write a placeholder that tells users what they can search for, such as "Search people..." or "Add tags...", so the input is not a blank mystery.
- Set maxEntries when the number of selections should be bounded, like limiting a review to 5 approvers.
- Use hasCreate for free-form tagging where users need to enter values that do not exist in the search source.
- Show validation status with the status prop so users know immediately when a selection is missing or invalid.
- Avoid: Don't use Tokenizer for single-item selection; use Typeahead instead. Tokenizer is for building sets of two or more items.
- Avoid: Avoid applying custom colors to individual tokens inside a Tokenizer; use the default token style for visual consistency across the set.
- Avoid: Don't hide the label; every Tokenizer needs a visible label so users understand what they are selecting. Use isLabelHidden only when surrounding context makes the purpose obvious.
- Avoid: Wrap a disabled Tokenizer in Tooltip to explain why it is disabled; disabled controls swallow the hover events the wrapper needs. Use the disabledMessage prop instead.

### Theming API

```json
{
  "targets": [
    {
      "className": "astryx-tokenizer",
      "visualProps": [
        "size",
        "status"
      ],
      "states": [
        "disabled"
      ]
    }
  ]
}
```

### Anatomy

- **Label**: The visible text above the input describing what the user is selecting. Also used as the accessible name.
- **Token chips**: Removable chips representing each selected item. Each chip shows a label and a remove button.
- **Search input**: The text input where users type to search the data source. Hides when maxEntries is reached.
- **Dropdown menu**: The search results list that appears below the input as the user types.
- **Spinner**: Loading indicator shown at the end of the field while a search is in flight.
- **End content**: A trailing slot after the input for action buttons, counts, or other controls.
- **Clear button**: A button that removes all selected tokens at once. Shown when hasClear is true and tokens are present.


## Examples

- Tokenizer: A tokenizer with preset tags and search source.
- Tokenizer — Clear: Tokenizer with a built-in clear-all button for bulk removal of all selected tokens.
- Tokenizer — Creatable: Free-text tokenizer for creating custom tags and a combined create-or-search pattern. Use when users need to enter values that may not exist in a predefined list.
- Tokenizer — End Content: Tokenizer with an action button in the end slot. Use for inline actions like applying selections alongside the input.
- Tokenizer — Icon: Tokenizer with a leading search icon to visually reinforce the search behavior.
- Tokenizer — Max Entries: Tokenizer with a maximum selection limit. The input hides automatically when the limit is reached, preventing further additions.
- Tokenizer — Overflow: Tokenizer with overflow truncation when unfocused. Inline mode pushes content down on expand; layer mode overlays without shifting layout.
- Tokenizer — States: Tokenizer in disabled, error, warning, and success states. Use to communicate validation feedback or lock a selection from editing.

## Complete example dependencies

Some examples import optional icons. The per-example install command in the examples list identifies exactly what that example needs. For all examples on this page:

```bash
npm install @heroicons/react@2.2.0
```

---

Index of all JGuang resources: https://jiguang132.com/llms.txt