Should I normalize Unicode before or after validating?

Before, always. Compatibility normalization maps fullwidth and other compatibility forms onto ASCII, so the string ＜script＞ becomes and ..／..／etc becomes ../../etc. A validator that inspects the text before normalization sees no angle brackets and no path separators, approves it, and the normalization that runs afterwards assembles the payload. Canonicalize first, then validate the canonical form, because that is the form the rest of the system will see.

Which disarm preset should I use?

It depends what the text is for. strip_format removes format characters and nothing else, and is the only preset that does not normalize. canonicalize is the general-purpose cleanup for text crossing a trust boundary, and canonicalize_strict is the less forgiving version. strip_obfuscation targets the deliberate hiding techniques. search_key, sort_key and catalog_key derive comparison keys, where collapsing every spelling to one is the point. ml_normalize prepares text for a model. Running all of them against your own input, as this page does, shows the differences faster than reading about them.

Can Unicode normalization create a security vulnerability?

It can create one if it runs at the wrong time. NFKC and the presets built on it map compatibility characters onto their ASCII equivalents, which means normalization can introduce characters that were not in the input — angle brackets, quotes, slashes, ampersands. That is only a problem when validation happened first. Normalizing before validating is safe and is what you should do; validating a string and then normalizing it means you validated something other than what you will use.

Composition, and the order it runs in

Which cleanup do I need?

Every other tool here demonstrates one function. The interesting part of disarm is which functions you compose and in what order, and that only shows when you put the results next to each other. Paste a string to run all eight presets against it — and to see the case where the order is not a matter of taste, because normalising can manufacture the very characters a validator was looking for.

Your text

Language profile

A shared link carries your text in the URL #fragment, which browsers never send to a server — it reaches the person you send it to and nobody else.

Normalising this text manufactures < >.

7 of these presets produce a metacharacter the input did not contain. A validator that inspected the string before the cleanup would have seen nothing to reject.

Why the order matters here

Validate, then clean

The validator sees ＜script＞alert(1)＜/script＞

No metacharacters. Nothing to reject. It passes.

The cleanup then produces <script>alert(1)</script>

The payload was assembled after the check.

Clean, then validate

The cleanup produces <script>alert(1)</script>

The validator sees the same string the rest of the system will.

It is rejected.

Canonicalise first, then validate. Always that way round.

Preset	What it is for	Result
strip_format	Format characters only. The narrowest of these, and the only one that does not normalise.	＜script＞alert(1)＜/script＞	unchanged
strip_obfuscation	The deliberate hiding techniques: invisibles, bidi, confusables.	<script>alert(1)</script>	manufactured <>
canonicalize	The general-purpose cleanup for text crossing a boundary.	<script>alert(1)</script>	manufactured <>
canonicalize_strict	The same, less forgiving about what it will keep.	<script>alert(1)</script>	manufactured <>
search_key	A key to index on, where every spelling should collapse to one.	<script>alert(1)</script>	manufactured <>
sort_key	A key to order by.	<script>alert(1)</script>	manufactured <>
catalog_key	A key for catalogue matching, with strict transliteration off.	<script>alert(1)</script>	manufactured <>
ml_normalize	For text on its way into a model: normalised, case folded, emoji named.	<script>alert(1)</script>	manufactured <>

Running disarm 0.14.1, compiled to WebAssembly. Your text is never uploaded — the engine is loaded into this page and runs on your machine.

Canonicalize before you validate

The tool above needs JavaScript. This is the same finding written out, so it is legible without running anything.

Compatibility normalisation exists to make different spellings of the same character compare equal, and it does that by mapping compatibility forms onto their ordinary equivalents. Fullwidth Latin is a compatibility form. So the mapping that turns ａ into a also turns ＜ into <.

Input	After normalizing	What appeared
＜script＞	<script>	Angle brackets, from nothing that was one.
..／..／etc	../../etc	Path separators, and a traversal.
＆lt;	<	An ampersand, and an entity that was not one.

Measured against disarm 0.14.1. None of the inputs contains an angle bracket, a solidus or an ampersand. A validator inspecting them finds nothing to reject, because there is nothing there yet.

The order is the whole defence. Canonicalise first, and validate the canonical form — because the canonical form is what the rest of your system is going to see. Validate first and canonicalise afterwards, and you have checked a string that no longer exists.

Note which preset is the exception in the tool above: strip_format leaves these inputs untouched, because it removes format characters and does not normalise at all. That makes it the safe choice when you specifically do not want the text rewritten — and the wrong choice if you were relying on it to give you a canonical form. One caveat worth knowing before you reach for it: strip_format is exposed by the Rust and Python APIs, and not by the Node, Ruby, Java or C bindings, which is why the code samples below do not use it.

What each preset is for

Four of these presets take a language profile and four do not, and the difference is not cosmetic. Transliterating Ärger im Büro without one gives arger im buro, which is simply wrong to a German reader: the convention is ae for ä, so it should be aerger im buero. Select German above and the language-aware presets change; the others cannot, because they do not transliterate.

Detection is not magic either. Detect from the text asks disarm to work the language out, and it can only do that when the text carries a character exclusive to one language — ß for German, ı for Turkish, ư for Vietnamese, ї for Ukrainian. Ärger im Büro contains none of those, so detection correctly declines to guess and you have to say de yourself. That is the honest behaviour, and it is why the selector offers both.

Paste an ordinary name rather than a payload and the table stops being about security. José Martínez shows the spectrum clearly: canonicalize leaves it as written, search_key gives jose martinez because a search index wants every spelling to collapse to one, and sort_key gives josé martínez because ordering needs the accents that matching does not. Three presets, three correct answers to three different questions.

The same thing in your own code

Each code block has been compiled and verified in CI. Provided under the MIT license to illustrate disarm. disarm on GitHub →

# Canonicalize before you validate, not after.
#   pip install disarm
from disarm import canonicalize, strip_format

# Fullwidth Latin is a compatibility form, so normalizing maps it onto ASCII.
# None of these characters is an angle bracket. After the cleanup, two are.
PAYLOAD = "＜script＞"

assert "<" not in PAYLOAD and ">" not in PAYLOAD
assert canonicalize(PAYLOAD) == "<script>"

# Which is the whole argument. A validator that runs first sees nothing to
# reject, and the cleanup afterwards assembles what it was looking for.
def unsafe(text):          # validate, then clean
    if "<" in text:
        raise ValueError("rejected")
    return canonicalize(text)

def safe(text):            # clean, then validate
    cleaned = canonicalize(text)
    if "<" in cleaned:
        raise ValueError("rejected")
    return cleaned

assert unsafe(PAYLOAD) == "<script>"      # approved, then assembled
try:
    safe(PAYLOAD)
    raise AssertionError("should have been rejected")
except ValueError:
    pass

# strip_format is the exception: it removes format characters and does not
# normalize, so it cannot manufacture anything.
assert strip_format(PAYLOAD) == PAYLOAD

print("ok: canonicalize manufactured 2 metacharacters; clean-then-validate caught it")

// Canonicalize before you validate, not after.
//   cargo add disarm
use disarm::api::{canonicalize, strip_format};

fn unsafe_order(text: &str) -> Result<String, &'static str> {
    // validate, then clean
    if text.contains('<') {
        return Err("rejected");
    }
    Ok(canonicalize(text).unwrap().into_owned())
}

fn safe_order(text: &str) -> Result<String, &'static str> {
    // clean, then validate
    let cleaned = canonicalize(text).unwrap().into_owned();
    if cleaned.contains('<') {
        return Err("rejected");
    }
    Ok(cleaned)
}

fn main() {
    // Fullwidth Latin is a compatibility form, so normalizing maps it onto
    // ASCII. None of these characters is an angle bracket. After the cleanup,
    // two are.
    let payload = "\u{FF1C}script\u{FF1E}";
    assert!(!payload.contains('<') && !payload.contains('>'));
    assert_eq!(canonicalize(payload).unwrap(), "<script>");

    assert_eq!(unsafe_order(payload), Ok("<script>".to_string()));
    assert_eq!(safe_order(payload), Err("rejected"));

    // strip_format is the exception: it removes format characters and does not
    // normalize, so it cannot manufacture anything.
    assert_eq!(strip_format(payload), payload);

    println!("ok: canonicalize manufactured 2 metacharacters; clean-then-validate caught it");
}

// Canonicalize before you validate, not after.
//   npm install disarm
const { canonicalize } = require("disarm");

// Fullwidth Latin is a compatibility form, so normalizing maps it onto ASCII.
// None of these characters is an angle bracket. After the cleanup, two are.
const PAYLOAD = "＜script＞";

if (PAYLOAD.includes("<") || PAYLOAD.includes(">")) throw new Error("expected no brackets");
if (canonicalize(PAYLOAD) !== "<script>") throw new Error("expected <script>");

const unsafe = (text) => {          // validate, then clean
  if (text.includes("<")) throw new Error("rejected");
  return canonicalize(text);
};
const safe = (text) => {            // clean, then validate
  const cleaned = canonicalize(text);
  if (cleaned.includes("<")) throw new Error("rejected");
  return cleaned;
};

if (unsafe(PAYLOAD) !== "<script>") throw new Error("expected it through");
let rejected = false;
try { safe(PAYLOAD); } catch { rejected = true; }
if (!rejected) throw new Error("clean-then-validate should have rejected it");

console.log("ok: canonicalize manufactured 2 metacharacters; clean-then-validate caught it");

# Canonicalize before you validate, not after.
#   gem install disarm
require "disarm"

# Fullwidth Latin is a compatibility form, so normalizing maps it onto ASCII.
# None of these characters is an angle bracket. After the cleanup, two are.
PAYLOAD = "＜script＞"

raise "expected no brackets" if PAYLOAD.include?("<") || PAYLOAD.include?(">")
raise "expected <script>" unless Disarm.canonicalize(PAYLOAD) == "<script>"

def unsafe_order(text)   # validate, then clean
  raise ArgumentError, "rejected" if text.include?("<")
  Disarm.canonicalize(text)
end

def safe_order(text)     # clean, then validate
  cleaned = Disarm.canonicalize(text)
  raise ArgumentError, "rejected" if cleaned.include?("<")
  cleaned
end

raise "expected it through" unless unsafe_order(PAYLOAD) == "<script>"
begin
  safe_order(PAYLOAD)
  raise "clean-then-validate should have rejected it"
rescue ArgumentError
  # as expected
end

puts "ok: canonicalize manufactured 2 metacharacters; clean-then-validate caught it"

// Canonicalize before you validate, not after.
//   implementation("dev.disarm:disarm:0.14.1")
import dev.disarm.Disarm;

public class ComparePresets {
    // validate, then clean
    static String unsafeOrder(String text) {
        if (text.contains("<")) throw new IllegalArgumentException("rejected");
        return Disarm.canonicalize(text);
    }

    // clean, then validate
    static String safeOrder(String text) {
        String cleaned = Disarm.canonicalize(text);
        if (cleaned.contains("<")) throw new IllegalArgumentException("rejected");
        return cleaned;
    }

    public static void main(String[] args) {
        // Fullwidth Latin is a compatibility form, so normalizing maps it onto
        // ASCII. None of these characters is an angle bracket. After the
        // cleanup, two are. Written as codepoints because javac resolves a
        // backslash-u sequence before it tokenises.
        String payload = Character.toString(0xFF1C) + "script" + Character.toString(0xFF1E);

        if (payload.contains("<") || payload.contains(">")) {
            throw new IllegalStateException("expected no brackets");
        }
        if (!Disarm.canonicalize(payload).equals("<script>")) {
            throw new IllegalStateException("expected <script>");
        }
        if (!unsafeOrder(payload).equals("<script>")) {
            throw new IllegalStateException("expected it through");
        }

        boolean rejected = false;
        try {
            safeOrder(payload);
        } catch (IllegalArgumentException e) {
            rejected = true;
        }
        if (!rejected) throw new IllegalStateException("should have been rejected");

        System.out.println("ok: canonicalize manufactured 2 metacharacters; "
            + "clean-then-validate caught it");
    }
}

// Canonicalize before you validate, not after.
//   implementation("dev.disarm:disarm-kotlin:0.14.1")
import dev.disarm.Disarm

// validate, then clean
fun unsafeOrder(text: String): String {
    require(!text.contains("<")) { "rejected" }
    return Disarm.canonicalize(text)
}

// clean, then validate
fun safeOrder(text: String): String {
    val cleaned = Disarm.canonicalize(text)
    require(!cleaned.contains("<")) { "rejected" }
    return cleaned
}

fun main() {
    // Fullwidth Latin is a compatibility form, so normalizing maps it onto
    // ASCII. None of these characters is an angle bracket. After the cleanup,
    // two are.
    val payload = String(Character.toChars(0xFF1C)) + "script" + String(Character.toChars(0xFF1E))

    check(!payload.contains("<") && !payload.contains(">")) { "expected no brackets" }
    check(Disarm.canonicalize(payload) == "<script>") { "expected <script>" }
    check(unsafeOrder(payload) == "<script>") { "expected it through" }

    var rejected = false
    try {
        safeOrder(payload)
    } catch (e: IllegalArgumentException) {
        rejected = true
    }
    check(rejected) { "should have been rejected" }

    println("ok: canonicalize manufactured 2 metacharacters; clean-then-validate caught it")
}

Found a string where the wrong preset looks right? The confusables table grew out of exactly that kind of report. Open an issue with it.

Related tools

Why don't these two strings match? — the narrow function for one specific difference, rather than a whole preset.
Remove invisible characters — one of the steps these presets compose.
Check confusable characters — another, and the one with a policy choice inside it.
Normalize Unicode whitespace — a third, and the narrowest of them.