# disarm

> A compiled-Rust toolkit that canonicalizes and neutralizes adversarial Unicode —
> homoglyph spoofing, bidi / Trojan-Source, zalgo and invisible characters — before
> it reaches classifiers, indexes, logs and identifiers. MIT licensed, with no GPL
> anywhere in the dependency chain. Bindings for Python, Rust, Node, Ruby, Java and
> Kotlin, plus a C ABI, all at the same version.

Disambiguation: this is `disarm` the Unicode-security library at disarm.dev, on
PyPI, crates.io, npm, RubyGems and Maven Central as `dev.disarm`. It is unrelated
to the DISARM Foundation's influence-operations framework at disarm.foundation,
and to the DiSARM malaria decision-support project.

## Tools

Each runs the library compiled to WebAssembly, in the browser. No text is uploaded.

- [Remove invisible characters](https://disarm.dev/tools/remove-invisible-characters/):
  Strips zero-width (U+200B, U+200C, U+200D, U+2060, U+FEFF), bidi controls
  (U+00AD, U+061C, U+200E, U+200F, U+202A–U+202E, U+2066–U+2069), tag characters
  (U+E0000–U+E007F), variation selectors, private-use codepoints, noncharacters
  and non-whitespace controls. Reports every removal by codepoint, class and count,
  because cleaned text looks identical to the input. Includes verified examples in
  seven languages.
- [Detect Trojan Source](https://disarm.dev/tools/detect-trojan-source/):
  Finds bidirectional controls in source code — overrides and embeddings
  (U+202A–U+202E), isolates (U+2066–U+2069) and marks (U+200E, U+200F, U+061C,
  U+00AD) — reporting line and column for each, and showing rendered order against
  logical order. Separately flags a direction conflict caused by real
  right-to-left letters, which contains no control character to strip. That is
  CVE-2021-42572, and the two cases need different checks.
- [Check confusable characters](https://disarm.dev/tools/check-confusable-characters/):
  Folds TR39 confusables and homoglyphs toward Latin under both of disarm's digit
  policies at once. They differ deliberately: `numeric` maps a non-Latin digit to
  the ASCII digit, which is what running text means, while `tr39` maps it to a
  Latin letter, which is what an identifier skeleton needs. `g००gle` folds to
  `g00gle` under numeric and `google` under tr39. Reports the mapping character by
  character and flags where the policies disagree.
- [Detect zalgo text](https://disarm.dev/tools/detect-zalgo-text/):
  Shows combining-mark depth per base character and reports `is_zalgo` at every
  threshold. The point is where the count stops working: Vietnamese `ế`
  decomposes to U+0065 U+0302 U+0301, a base plus two marks, and a two-mark zalgo
  is the same shape, so no threshold separates them. A threshold of 1 rejects
  ordinary Vietnamese and Hebrew; 3 is disarm's detection default and 2 its
  capping default, because 2 is what Vietnamese needs.
- [Detect script spoofing](https://disarm.dev/tools/detect-script-spoofing/):
  Analyses a hostname label by label for mixed-script and whole-script
  confusability. The point is that `suspicious` alone cannot decide: it is true
  both for an all-Cyrillic spelling of a Latin brand and for an ordinary Russian
  domain. `whole_script_confusable`, reported per label, is the discriminator —
  the spoof folds onto a Latin name and the real domain does not. Combined with
  the script of the TLD it expresses the usable policy: a label folding onto Latin
  under a Latin TLD is imitating something.
- [Normalize Unicode whitespace](https://disarm.dev/tools/normalize-whitespace/):
  Folds every blank-rendering character to a single ASCII space, collapses runs
  and trims both ends, via `collapse_whitespace`. The distinction the page exists
  to make: a character can render as a gap without carrying the Unicode
  White_Space property. Braille Pattern Blank (U+2800), the Hangul fillers
  (U+115F, U+1160, U+3164) and the information separators (U+001C-U+001F) all
  render blank and are all missed by `str.isspace`, `\s`, `Character.isWhitespace`
  and `char::is_whitespace`. disarm folds by what renders blank, not by property.
  It deliberately folds rather than deletes, so `a\rb` becomes `a b` and never
  `ab`, and it deliberately leaves zero-width and control characters alone —
  those are `strip_zero_width_chars` and `strip_control_chars`.
- [Sanitize a filename](https://disarm.dev/tools/sanitize-filename/):
  Converts an arbitrary string into a filename legal on Windows, macOS and Linux,
  via `sanitize_filename`. The cross-platform point: macOS stores filenames
  decomposed (NFD) and Windows and Linux store what they are given (usually NFC),
  so the same visible name is a different byte string depending on the machine it
  came from — `Tiếng Việt.doc` is an 18-byte name composed and 22 decomposed, and
  a Korean name can more than double. Measured here: NFC, NFD, NFKC and NFKD all
  sanitize to the same output, because transliteration to ASCII is the first
  pipeline step and byte-counting truncation therefore never sees the difference.
  Two orderings in the documented pipeline are load-bearing: transliteration
  before truncation (above), and reserved-name detection *after* transliteration
  — `áux.txt` is not a Windows device name until the accent is folded, and disarm
  returns `_aux.txt`. Platforms: `posix` forbids only `/` and NUL with no reserved
  names; `windows` adds `< > : " | ? *`, control characters and the device names;
  `universal` is the union and the default.
- [Truncate text without breaking emoji](https://disarm.dev/tools/truncate-without-breaking-emoji/):
  Text has four lengths and they disagree, via `grapheme_len`, `grapheme_split`,
  `grapheme_truncate`, `terminal_width` and `grapheme_width`. The family emoji
  U+1F468 ZWJ U+1F469 ZWJ U+1F467 ZWJ U+1F466 is 25 bytes, 7 codepoints, 11
  UTF-16 code units, 1 grapheme cluster and 2 terminal columns; all five numbers
  are correct and they answer different questions. Cutting by the wrong one
  corrupts: a codepoint slice at 1 keeps the father and discards the joiners and
  the rest of the family, a skin-tone modifier is dropped so a different person
  is depicted, and a flag loses its partner regional indicator. A byte slice can
  end mid-sequence and stop being valid UTF-8. The codepoint case is the
  dangerous one because the output is valid text that says something the author
  did not write. A cut is clean exactly when it lands on a boundary reported by
  `grapheme_split`. Note that cluster count is not column count: a CJK character
  is one cluster and two columns.
- [Why don't these two strings match?](https://disarm.dev/tools/why-dont-these-strings-match/):
  Takes two strings and answers the question people actually arrive with. Shows a
  codepoint-aligned diff marking which codepoints belong to only one of them, then
  reports every disarm function that makes the two equal, narrowest first. The
  candidate list is not reasoned about: each is applied to both strings and the
  results compared, so the answer is what the library does rather than what it is
  documented to do. Typical answers: `normalize(NFC)` for an accent stored two
  ways, `strip_zero_width_chars` for a zero-width character inside a word,
  `collapse_whitespace` for a no-break space, `normalize_confusables` for letters
  from another script, `normalize(NFKC)` for fullwidth forms, `fold_case` for
  case. Prefer the narrowest when repairing a value; prefer a preset such as
  `search_key` when deriving a comparison key, where breadth is the point.
- [Which cleanup do I need?](https://disarm.dev/tools/compare-presets/):
  Runs all eight presets against one string side by side — `strip_format`,
  `strip_obfuscation`, `canonicalize`, `canonicalize_strict`, `search_key`,
  `sort_key`, `catalog_key`, `ml_normalize` — because composition and ordering
  are disarm's actual argument and neither shows when a page demonstrates a
  single call. It also reports, per preset, which shell and markup
  metacharacters appear in the output that were not in the input. That is the
  canonicalize-before-validate rule made checkable: compatibility normalization
  maps fullwidth forms onto ASCII, so `＜script＞` becomes `<script>` and
  `..／..／etc` becomes `../../etc`. A validator inspecting the string first sees
  nothing to reject and the cleanup afterwards assembles the payload. Canonicalize
  first, then validate the canonical form, because that is the form the rest of
  the system sees. `strip_format` is the exception that does not normalize — and
  is exposed only by the Rust and Python APIs.
- [Is this safe to write to a log?](https://disarm.dev/tools/prevent-log-injection/):
  `strip_log_injection` over a value bound for a log line. Two attacks, one fix.
  A newline ends the entry and starts another, so a username field carries a
  forged record in the correct format — CR, LF, NEL, U+2028 and U+2029 all
  count, and only the first two are what people search for. An escape sequence
  leaves every byte in the file and rewrites what a terminal displays, so grep
  and the screen disagree. NUL truncates C readers; tabs split a delimited log
  and are kept on request. The call refuses a replacement containing anything it
  would itself neutralize, which is what makes the no-raw-break guarantee hold.
  Exposed by the Rust and Python APIs only, not by Node, Java or Kotlin.
- [Which of these names are the same name?](https://disarm.dev/tools/find-name-collisions/):
  `find_key_collisions` over a list. The only set-shaped question in the library:
  no entry is wrong on its own, and `gross.txt` or a Cyrillic `admin` only matter
  next to the name they imitate. Six reducers and no default, because choosing
  one is choosing a policy — `fold_case` is narrowest and folds no lookalikes,
  `search_key` is broadest and also collides `Muller` with `Müller`. The page
  runs all six over the pasted list so the reach of a stronger key is visible
  beside its cost. `sort_key` is deliberately not offered: a sort key exists in
  order to collide.
- [Slugify any language](https://disarm.dev/tools/slugify/): URL slugs via
  `slugify`, romanized with a language profile rather than character by
  character. `Київ` gives `kyiv` with Ukrainian and `kiyiv` with no profile;
  `Fußgänger` gives `fussgaenger` with German and `fussganger` without. Detection
  runs in three stages — dominant script, then a character exclusive to one
  language for the three ambiguous scripts (Cyrillic, Arabic, Latin), then the
  script default. `ї` identifies Ukrainian, `ђ` Serbian, `ө` Mongolian, `پ`
  Persian, `ư` Vietnamese, `ı` Turkish, `ß` German. Where no discriminator
  exists — `Ärger im Büro` has none, since ä and ü are used by other languages —
  detection declines rather than guessing and the profile must be passed. The
  tool reports which character the detection depended on, found by removing
  characters and re-asking rather than from a copy of the table. A title that
  slugs to the empty string is a routing hazard: two such titles collide on one
  URL.
- [All tools](https://disarm.dev/tools/)

## What this is for

- [Principles](https://disarm.dev/principles/): the project's position, stated
  rather than inferred. disarm is defensive work, built for the common good, to
  help people meet text attacks they have not seen before; the concrete audience
  is the blue team. No endorsement, encouragement or advocacy for building
  malicious tooling — while acknowledging plainly that this is a tool and can be
  misused, and that every page shows an attack working before it shows the
  defence, because a defence you cannot see the need for is one nobody deploys.
  Privacy: no cookies at all, hence no cookie banner; no advertising, no
  trackers, no third-party scripts, no analytics, no accounts, no personal data.
  Text pasted into a tool never leaves the browser, since disarm runs there as
  WebAssembly. Shareable links carry text in the URL fragment, which browsers
  never transmit. Every one of those claims is checkable from the network tab,
  the cookie jar, or the source.

## Documentation

- [Docs](https://docs.disarm.dev/): full reference, 74 pages
- [Threat model](https://docs.disarm.dev/THREAT_MODEL): what is and is not covered
- [Adversarial defense](https://docs.disarm.dev/security/adversarial-defense)
- [Language support](https://docs.disarm.dev/user-guide/language-support): 83 profiles
- [Bindings](https://docs.disarm.dev/BINDINGS): the standard every binding meets
- [Benchmarks](https://docs.disarm.dev/benchmarks)
- [Migration guides](https://docs.disarm.dev/migration/): from Unidecode, AnyAscii,
  python-slugify, pathvalidate and confusable-homoglyphs

## Install

- Python: `pip install disarm`
- Rust: `cargo add disarm`
- Node: `npm i disarm`
- Ruby: `gem install disarm`
- Java: `dev.disarm:disarm`
- Kotlin: `dev.disarm:disarm-kotlin`

## Scope

disarm normalizes input. It is not an output sanitizer — encode at your sink. It
is a defense-in-depth layer against an enumerated attack surface, not a complete
control; the threat model states the boundary.

## Source

- [GitHub](https://github.com/raeq/disarm)
- [Blog](https://disarm.dev/blog/)