scripts/spo-request-pnp-reindex-user-profile destroys umlauts

[jan-kiesewetter-elma]

After running the script, all umlauts in the Department field of the SharePoint user profile are corrupted

[pkbullock]

Hi @jan-kiesewetter-elma - did you find a fix for this? I dont have a test rig for this scenario, the problem could be the underly PnP PowerShell cmdlets or the encoding somewhere in the search cmdlet. Of course, Search in SharePoint could be the final point of failure.

[DavorSukeri]

I just had this same scenario and after consulting the problem with Copilot, I ran the same script again, but now forcing to use UTF 8 encoding. So I just added -Encoding UTF8 to this line:

Set-Content -Path "$tempPath\$filename.txt" -value $json -Encoding UTF8

Have to wait probably at least a day to see the results.

Also I misunderstood the logic as I though the script fetches the Department value from EntraID and writes it to SharePoint profile Department value. Instead it just re-writes the current SharePoint profile Department value to the SharePoint profile Department value which triggers the import from EntraID and after that fires the reindexing in SharePoint (which can take few days to see the results). Am I correct?

At least for now (ran the first script on friday and now is monday) the only change I can see in the SharePoint profiles is the Department field values with destroyed umlauts because of the rewrite. No other fields have updated from EntraID like for example Display Name and Office. I've read that it can take up to week even to see the results.

[juandresrodca]

Hi @pkbullock @jan-kiesewetter-elma — opened #964 to address this.

Root cause is the encoding on the temp batch file. The script writes the JSON with Set-Content and no -Encoding, so on Windows PowerShell it defaults to ANSI and any non-ASCII characters (ä, ö, ü, ñ, …) in the Department field get mangled before the file is uploaded to the UPA bulk-import job — which is exactly what @DavorSukeri found.

The fix is the one-liner he validated:

Set-Content -Path "$tempPath$filename.txt" -value $json -Encoding UTF8
Since PnP.PowerShell runs on PowerShell 7, -Encoding UTF8 writes UTF-8 without a BOM, so the import reads the JSON cleanly and the original characters are preserved.

And @DavorSukeri's read of the logic is correct: the script re-writes each profile's existing SharePoint Department value back onto itself to bump the timestamp and trigger a re-index — it doesn't pull values from Entra ID. So the only thing that changes is the Department field, which is precisely why the encoding bug was visible there.

Happy to adjust the PR if you'd prefer a different appr