Skip to content

The SQL Server Unicode Problem Why Your Data Might Not Be Wh... #2488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
carlospolop wants to merge 1 commit into
base: master
Choose a base branch
from
+150 −32
Open

The SQL Server Unicode Problem Why Your Data Might Not Be Wh... #2488

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 150 additions & 32 deletions src/pentesting-web/unicode-injection/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,66 +4,184 @@

## Introduction

Depending on how the back-end/front-end is behaving when it **receives weird unicode characters** an attacker might be able to **bypass protections and inject arbitrary characters** that could be used to **abused injection vulnerabilities** such as XSS or SQLi.
Depending on how the back-end/front-end behaves when it **receives weird Unicode characters**, an attacker might be able to **bypass protections and inject arbitrary ASCII metacharacters later**. The dangerous pattern is usually:

## Unicode Normalization
1. Input is validated in one representation
2. The application or database **implicitly converts / normalizes / re-encodes** it
3. The converted value becomes dangerous **after** validation

Unicode normalization occurs when **unicode characters are normalized to ascii characters**.
This can turn apparently harmless input into working payloads for **XSS, SQLi, path traversal/LFI, command injection, or logic bugs**.

One common scenario of this type of vulnerability occurs when the system is **modifying** somehow the **input** of the user **after having checked it**. For example, in some languages a simple call to make the **input uppercase or lowercase** could normalize the given input and the **unicode will be transformed into ASCII** generating new characters.\
For more info check:
## Unicode normalization

Unicode normalization occurs when **Unicode characters are normalized to ASCII characters** or to another compatible representation. One common scenario is when the system **modifies** the user input **after checking it**. For example, in some languages a simple call to make the **input uppercase or lowercase** could normalize the given input and the **Unicode will be transformed into ASCII** generating new characters.

For more info check:

{{#ref}}
unicode-normalization.md
{{#endref}}

## `\u` to `%`
## SQL Server Best Fit / implicit conversion

Unicode characters are usually represented with the **`\u` prefix**. For example the char `㱋` is `\u3c4b`([check it here](https://unicode-explorer.com/c/3c4B)). If a backend **transforms** the prefix **`\u` in `%`**, the resulting string will be `%3c4b`, which URL decoded is: **`<4b`**. And, as you can see, a **`<` char is injected**.\
You could use this technique to **inject any kind of char** if the backend is vulnerable.\
Check [https://unicode-explorer.com/](https://unicode-explorer.com/) to find the chars you need.
Microsoft SQL Server adds another dangerous post-validation transform: **implicit conversion from Unicode strings to narrow non-UTF code pages** (`char`, `varchar`, `text`, or non-Unicode string literals). Instead of rejecting unsupported characters, SQL Server can silently apply **Windows Best Fit mapping** and mutate Unicode lookalikes into ASCII metacharacters.

This vuln actually comes from a vulnerability a researcher found, for a more in depth explanation check [https://www.youtube.com/watch?v=aUsAHb0E7Cg](https://www.youtube.com/watch?v=aUsAHb0E7Cg)
### Dangerous conditions

## Emoji Injection
Look for combinations such as:

Back-ends something behaves weirdly when they **receives emojis**. That's what happened in [**this writeup**](https://medium.com/@fpatrik/how-i-found-an-xss-vulnerability-via-using-emojis-7ad72de49209) where the researcher managed to achieve a XSS with a payload such as: `💋img src=x onerror=alert(document.domain)//💛`
- Columns using **`varchar` / `char` / `text`**
- Non-UTF collations / code pages such as CP1252 (`iso_1` in metadata)
- **Implicit conversion** from `nvarchar` to `varchar`
- Dynamic SQL using **non-Unicode literals** like `'payload'` instead of `N'payload'`
- Security-sensitive comparisons under **case-insensitive collations** such as `*_CI_*`

In this case, the error was that the server after removing the malicious characters **converted the UTF-8 string from Windows-1252 to UTF-8** (basically the input encoding and the convert from encoding mismatched). Then this does not give a proper < just a weird unicode one: `‹`\
``So they took this output and **converted again now from UTF-8 ot ASCII**. This **normalized** the `‹`to`<` this is how the exploit could work on that system.\
This is what happened:
Useful recon queries:

```php
<?php
```sql
SELECT SERVERPROPERTY('Collation');
SELECT DATABASEPROPERTYEX(DB_NAME(), 'Collation');

$str = isset($_GET["str"]) ? htmlspecialchars($_GET["str"]) : "";
SELECT
COLUMN_NAME,
DATA_TYPE,
CHARACTER_SET_NAME,
COLLATION_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'target_table';
```

### Best Fit payload mutation

Example: **FULLWIDTH LESS-THAN SIGN** `＜` (`U+FF1C`) can become ASCII `<` when inserted into a narrow column:

$str = iconv("Windows-1252", "UTF-8", $str);
$str = iconv("UTF-8", "ASCII//TRANSLIT", $str);
```sql
CREATE TABLE worstfit_demo (
narrow VARCHAR(20) COLLATE French_CI_AS,
wide NVARCHAR(20) COLLATE French_CI_AS
);

echo "String: " . $str;
INSERT INTO worstfit_demo VALUES (N'＜', N'＜');
SELECT narrow, wide FROM worstfit_demo;
-- narrow => <
-- wide => ＜
```

Emoji lists:
This is not Unicode normalization in the standard sense; it is **Windows code-page Best Fit conversion** during SQL Server storage/conversion.

- [https://github.com/iorch/jakaton_feminicidios/blob/master/data/emojis.csv](https://github.com/iorch/jakaton_feminicidios/blob/master/data/emojis.csv)
- [https://unicode.org/emoji/charts-14.0/full-emoji-list.html](https://unicode.org/emoji/charts-14.0/full-emoji-list.html)
### Non-Unicode literal pitfall

## Windows Best-Fit/Worst-fit
In SQL Server, `'text'` is a **non-Unicode literal by default**, even when inserted into an `NVARCHAR` column. Use the `N` prefix for wide strings:

As explained in **[this great post](https://blog.orange.tw/posts/2025-01-worstfit-unveiling-hidden-transformers-in-windows-ansi/)**, Windows has a feature called **Best-Fit** which will **replace unicode characters** that cannot be displayed in ASCII mode with a similar one. This can lead to **unexpected behavior** when the backend is **expecting a specific character** but it receives a different one.
```sql
INSERT INTO t VALUES ('＜'); -- converted before assignment
INSERT INTO t VALUES (N'＜'); -- preserved in NVARCHAR, still mutated in VARCHAR
```

It's possible to find best-fit characters in **[https://worst.fit/mapping/](https://worst.fit/mapping/)**.
This matters in **stored procedures**, **dynamic SQL**, and any code path that concatenates attacker-controlled strings into SQL statements.

As Windows will usually convert unicode strings to ascii strings as one of the last parts of the execution (usually going from a "W" suffixed API to an "A" suffixed API like `GetEnvironmentVariableA` and `GetEnvironmentVariableW`) this would allow atackers to bypass protections by sending unicode characters that will be converted lastly in ASCII characters that would perform nuexpected actions.
### Stored XSS after DB storage

In the blog post there are proposed methods to bypass vulnerabilities fixed using a **blacklist of chars**, exploit **path traversals** using [characters mapped to “/“ (0x2F)](https://worst.fit/mapping/#to%3A0x2f) and [characters mapped to “\“ (0x5C)](https://worst.fit/mapping/#to%3A0x5c) or even bypassing shell escape protections like PHP's `escapeshellarg` or Python's `subprocess.run` using a list, this was done for example using **fullwidth double quotes (U+FF02)** instead of double quotes so at the end what looked like 1 argument was transformed in 2 arguments.
If an application filters classic ASCII XSS but accepts Unicode lookalikes, SQL Server can convert them into a valid payload **after insertion**:

**Note that for an app to be vulnerable it needs to use "W" Windows APIs but end calling an "A" Windows api so the "Best-fit" of the unicode string is created.**
```sql
CREATE TABLE worstfitx (
vary VARCHAR(50) COLLATE French_CI_AS
);

**Several dicovered vulnerabilities won't be fixed as people don't agree who should be fixing this issue"**.
INSERT INTO worstfitx VALUES (N'〈script〉alert(ʹxʹ)〈/script〉');
SELECT vary FROM worstfitx;
-- <script>alert('x')</script>
```

{{#include ../../banners/hacktricks-training.md}}
This is especially interesting when the app:

- validates only before insert
- stores the payload in MSSQL
- later renders the **stored** value into HTML

### Path traversal / LFI after DB storage

If filenames are stored in narrow MSSQL columns and later reused to build filesystem paths, Unicode dots and slash/backslash lookalikes can turn into traversal sequences:

```sql
CREATE TABLE worstfity (
id SMALLINT,
file_name VARCHAR(50) COLLATE French_CI_AS
);

INSERT INTO worstfity
VALUES (45, N'．．∖．．∖．．∖Windows∖win.ini');

SELECT file_name FROM worstfity WHERE id = 45;
-- ..\..\..\Windows\win.ini
```

The filter may only see the Unicode version, while the application later consumes the **mutated** Windows path from the database.

### `?` fallback reconstruction

When no Best Fit mapping exists, many Windows code pages fall back to `?`. This can reconstruct blocked syntax:

```sql
INSERT INTO worstfitx VALUES (N'<ʭphp');
SELECT vary FROM worstfitx;
-- <?php
```

Useful when `?` is blocked before storage but the DB later creates it.

### Case-insensitive collation collisions

Default SQL Server collations are often case-insensitive, so equality and pattern matching can collide:

```sql
SELECT IIF('abc' = 'ABC', 1, 0); -- 1
SELECT 'ok' WHERE 'abc' LIKE 'ABC'; -- ok
```

Don't assume usernames, API keys, role names, object IDs, or allow/deny-list checks are case-sensitive unless the relevant column/query uses a **binary or explicit case-sensitive collation**.

### Offensive testing notes

- Try **fullwidth**, **modifier**, and other compatibility-like characters for `<`, `>`, `'`, `"`, `/`, `\`, `.`, `-`, `?`
- Compare **pre-insert** and **post-read** values, not only the reflected request
- Check whether the app uses **`VARCHAR` metadata + Unicode client input**
- Probe both `'payload'` and `N'payload'`
- Audit logic bugs caused by `*_CI_*` collations in addition to injection sinks

## `\u` to `%`

Unicode characters are usually represented with the **`\u` prefix**. For example the char `㱋` is `\u3c4b` ([check it here](https://unicode-explorer.com/c/3c4B)). If a backend **transforms** the prefix **`\u` into `%`**, the resulting string will be `%3c4b`, which URL decoded is **`<4b`**. As you can see, a **`<` char is injected**.

You could use this technique to **inject any kind of char** if the backend is vulnerable. Check [https://unicode-explorer.com/](https://unicode-explorer.com/) to find the chars you need.

This vuln actually comes from a vulnerability a researcher found. For a more in-depth explanation check [https://www.youtube.com/watch?v=aUsAHb0E7Cg](https://www.youtube.com/watch?v=aUsAHb0E7Cg)

## Emoji injection

Back-ends sometimes behave weirdly when they **receive emojis**. That's what happened in [this writeup](https://medium.com/@fpatrik/how-i-found-an-xss-vulnerability-via-using-emojis-7ad72de49209) where the researcher managed to achieve XSS with a payload such as `img src=x onerror=alert(document.domain)//`.

In that case, the server removed malicious characters and then **converted the UTF-8 string from Windows-1252 to UTF-8** (input/convert encoding mismatch). This did not immediately generate a proper `<`, just a weird Unicode quote-like character: `‹`.

Then the output was **converted again from UTF-8 to ASCII**. This second conversion normalized `‹` into `<`, making the payload executable:

```php
<?php
$a = "＜img src=x onerror=alert(document.domain)//";
echo mb_convert_encoding($a, 'UTF-8', 'Windows-1252');
# outputs: ‹img src=x onerror=alert(document.domain)//

# Then if the output is converted to ASCII:
echo iconv('UTF-8', 'ASCII//TRANSLIT', mb_convert_encoding($a, 'UTF-8', 'Windows-1252'));
# outputs: <img src=x onerror=alert(document.domain)//
?>
```

## References

- [Synacktiv - The SQL Server Unicode problem: why your data might not be what you think it is?](https://synacktiv.com/en/publications/the-sql-server-unicode-problem-why-your-data-might-not-be-what-you-think-it-is.html)
- [HackTricks - Unicode normalization](unicode-normalization.md)
- [Unicode Explorer](https://unicode-explorer.com/)
- [Orange Tsai / DEVCORE - WorstFit: Unveiling Hidden Transformers in Windows ANSI!](https://devco.re/blog/2025/01/09/worstfit-unveiling-hidden-transformers-in-windows-ansi/)
{{#include ../../banners/hacktricks-training.md}}