Windows – How to remove non-ASCII characters from a string in bat script

batch filewindows 7

Sometimes the %COMPUTERNAME% variable value contains non-ASCII characters, but when I use this variable in a batch script, I need to keep only the ASCII characters of the correlated value.

How could I go about getting a string value with non-ASCII characters to only display the ASCII characters using a batch script?

Best Answer

You can use some of the methods as listed on Remove special characters from a string using Regular Expression (Regex) with a batch script as the one below for example.

Just use the native power of Windows as designed by Microsoft and get this job done simply by using PowerShell in your batch script to set the %computername% without the non-ASCII character for what's returned with the PowerShell commands.

Batch Script to Remove Non-Ascii

Be sure to set the SET PCName= variable accordingly in your script to incorporate with however your logic already works (e.g. SET PCName=%computername%, etc.)

Rather than using the DO ECHO %%~F with the FOR /F loop in the below script, you could set that to DO SET PCNameASCII=%%~F instead for example and then the %PCNameASCII% variable can be incorporated into the rest of your batch script logic accordingly and that'd always be the %ComputerName% variable value but only the ASCII characters.

@ECHO ON

SET PCName=Laäff¥yTaäffi¡

SET PSScript=%Temp%\~tmpRemovenonascii.ps1
IF EXIST "%PSScript%" DEL /Q /F "%PSScript%"

ECHO $String = '%PCName%'                                             >>"%PSScript%"
ECHO $String = $String -replace '[^^\x30-\x39\x41-\x5A\x61-\x7A]+', ''>>"%PSScript%"
ECHO ECHO $String                                                     >>"%PSScript%"

SET PowerShellDir=C:\Windows\System32\WindowsPowerShell\v1.0
CD /D "%PowerShellDir%"
FOR /F "TOKENS=*" %%F IN ('Powershell -ExecutionPolicy Bypass -Command "& '%PSScript%'"') DO ECHO %%~F

PAUSE

GOTO EOF

See the Further Resources section and the notes therein for other methods using Regex to remove special characters, etc.

Before (non-ascii)
LaõffÑyTaõffií
After (results)
LaffyTaffi

Further Resources

Replace()
FOR /F
Remove special characters from a string using Regular Expression (Regex)

# Regular Expression - Using the \W (opposite of \w) 
$String -replace '[\W]', ''

# Regular Expression - Using characters from a-z, A-Z, 0-9 
$String -replace '[^a-zA-Z0-9]', ''

# Regular Expression - Using ASCII 
$String -replace '[^\x30-\x39\x41-\x5A\x61-\x7A]+', ''

# Regular Expression - Unicode - Matching Specific Code Points 
$String -replace '[^\u0030-\u0039\u0041-\u005A\u0061-\u007A]+', ''

# Regular Expression - Unicode - Unicode Categories 
$String -replace '[^\p{L}\p{Nd}]', ''

# Regular Expression - Unicode - Unicode Categories
#  Exceptions: We want to keep the following characters: ( } _ 
$String -replace '[^\p{L}\p{Nd}/(/}/_]', ''

Related Solutions

Trying to read a text file content and set variables values from that

You are missing the : after your variable name. So your commands would be:

set /p out=<{full path to the file}\Temp_result.txt
set SECURETOKEN=%out:~21,25%
set SECURETOKENID=%out:~61,16%

Windows – How to remove non-ASCII characters from filenames

I found a similar topic here on Stack Overflow.

With the following code most of the characters will be translated to their "closest character". Although i couldn't get the ’ translated. (Maybe it does, i can't make a filename in the prompt with it ;) The ß also does not get translated.

function Remove-Diacritics {
param ([String]$src = [String]::Empty)
  $normalized = $src.Normalize( [Text.NormalizationForm]::FormD )
  $sb = new-object Text.StringBuilder
  $normalized.ToCharArray() | % {
    if( [Globalization.CharUnicodeInfo]::GetUnicodeCategory($_) -ne [Globalization.UnicodeCategory]::NonSpacingMark) {
      [void]$sb.Append($_)
    }
  }
  $sb.ToString()
}

$files = gci -recurse | where {$_.Name -match "[^\u0020-\u007F]"}
$files | ForEach-Object {
  $newname = Remove-Diacritics $_.Name
  if ($_.Name -ne $newname) {
    $num=1
    $nextname = $_.Fullname.replace($_.Name,$newname)
    while(Test-Path -Path $nextname)
    {
      $next = ([io.fileinfo]$newname).basename + " ($num)" + ([io.fileinfo]$newname).Extension
      $nextname = $_.Fullname.replace($_.Name,$next)
      $num+=1
    }
    echo $nextname
    ren $_.Fullname $nextname
  }
}

Edit:

I added some code to check if a filename already exists and add (1), (2) etc... if it does. (It's not smart enough to detect an already existing (1) in the filename to be renamed so in that case you would get (1) (1). But as always... everything is programmable ;)

Edit 2:

Here is the last one for tonight...

This one has a different function for replacing the characters. Also added a line to change unknown characters like ß and ┤ for example to _.

function Convert-ToLatinCharacters {
param([string]$inputString)
  [Text.Encoding]::ASCII.GetString([Text.Encoding]::GetEncoding("Cyrillic").GetBytes($inputString))
}

$files = gci -recurse | where {$_.Name -match "[^\u0020-\u007F]"}
$files | ForEach-Object {
  $newname = Convert-ToLatinCharacters $_.Name
  $newname = $newname.replace('?','_')
  if ($_.Name -ne $newname) {
    $num=1
    $nextname = $_.Fullname.replace($_.Name,$newname)
    while(Test-Path -Path $nextname)
    {
      $next = ([io.fileinfo]$newname).basename + " ($num)" + ([io.fileinfo]$newname).Extension
      $nextname = $_.Fullname.replace($_.Name,$next)
      $num+=1
    }
    echo $nextname
    ren $_.Fullname $nextname
  }
}