Why is the applescript for checking if a file exists failing

Copying of files in Applescript is best done either through System Events or Finder, using the duplicate … to command, i.e.

tell application "System Events" to duplicate sourceFile to targetFolder

where both sourceFile and targetFolder need to be the correct object type for the application used – meaning disk item or finder item (both objects can be created from AppleScript alias objects, or textual path values with a bit of type coercion – I’d add the details, but you have not stated how the paths to both are stored / acquired in your script).

A few notes on your code:

• There is no need to use globals when you have defined them as properties already. AppleScript properties are script scoped and persist across execution – they are only reset when the script is recompiled. If you assign those that need user setting missing value when declaring, you can even check if they are already set and skip re-prompting the user (there would be an even more comfortable and secure solution if Apple hadn’t deprecated Keychain Access Scripting).

• There is no need for the repeated assignments and recursive call in your connectToServer() handler. The following code

  set timeOutCounter to 0
  repeat while (list disks) does not contain serverName and timeOutCounter is less than timeOutInterval
      -- mount drive
      delay someInterval -- recommended, so you don’t hectically loop
      set timeOutCounter to timeOutCounter + someInterval -- time out loop 
  end repeat
  

  will try to connect in the interval defined by someInterval, until the mount is available or timeOutIntervalis reached (assuming these values are declared. As properties, best – see above).

• You might also want to offer your user a more comfortable way of selecting the target folder than typing a folder name from memory. Check out Standard Additions’s AppleScript dictionary for the choose folder command.
• Finally, but that is mainly a matter of taste and coding style, I’d rather move the display dialog command into its own handler and call that, if needed repeatedly, from the script body, than use a C style mainLoop handler. I’ve found that, generally speaking, AppleScript runs out of Stack space easily when recursing and can get very confused about variable assignments, so it is a good idea to avoid recursive constructs where they are not necessary.

Try:

set dest_folder to POSIX path of dest_folder
set newFilepathPOSIX to dest_folder & "video.mp4"

tell application "System Events" to exists file newFilepathPOSIX
if the result then
    display dialog "There is already a file named \"video.mp4\". Do you want to replace the file?" buttons {"Quit", "No", "Yes"} cancel button 1
    if button returned of the result = "No" then
        beep
        --Insert your code here
    end if
end if

EDIT

 set handPath to "/Volumes/Flash Drive/HandBrakeCLI"
set dest_folder to POSIX path of dest_folder
if text -1 of dest_folder ≠ "/" then set dest_folder to dest_folder & "/"
set newFilepathPOSIX to dest_folder & "video.mp4"

tell application "System Events" to exists newFilepathPOSIX
if the result then
    display dialog "There is already a file named \"video.mp4\". Do you want to replace the file?" buttons {"Quit", "No", "Yes"} cancel button 1
    if button returned of the result = "No" then set newFilepathPOSIX to dest_folder & "video" & getDateAsString(current date) & ".mp4"
end if

set shellCommand to quoted form of handPath & " -i " & quoted form of origFilepathPOSIX & " -o " & quoted form of newFilepathPOSIX & " --preset=\"Classic\""
do shell script shellCommand

on getDateAsString(myDate)
    set creationDateString to year of myDate as string
    set creationDateString to creationDateString & text -2 thru -1 of ("0" & (month of myDate as integer) as text)
    set creationDateString to creationDateString & text -2 thru -1 of ("0" & day of myDate as text)
    set creationDateString to creationDateString & text -2 thru -1 of ("0" & hours of myDate as text)
    set creationDateString to creationDateString & text -2 thru -1 of ("0" & minutes of myDate as text)
    set creationDateString to creationDateString & text -2 thru -1 of ("0" & seconds of myDate as text)
end getDateAsString