Shell – Why are POSIX mandatory utilities not built into the shell

posixshellshell-builtinshell-scriptUtilities

The purpose of this question is to answer a curiosity, not to solve a particular computing problem. The question is: Why are POSIX mandatory utilities not commonly built into shell implementations?

For example, I have a script that basically reads a few small text files and checks that they are properly formatted, but it takes 27 seconds to run, on my machine, due to a significant amount of string manipulation. This string manipulation makes thousands of new processes by calling various utilities, hence the slowness. I am pretty confident that if some of the utilities were built in, namely grep, sed, cut, tr, and expr, then the script would run in a second or less (based on my experience in C).

It seems there would be a lot of situations where building these utilities in would make the difference between whether or not a solution in shell script has acceptable performance.

Obviously, there is a reason it was chosen not to make these utilities built in. Maybe having one version of a utility at a system level avoids having multiple unequal versions of that utility being used by various shells. I really can't think of many other reasons to keep the overhead of creating so many new processes, and POSIX defines enough about the utilities that it does not seem like much of a problem to have different implementations, so long as they are each POSIX compliant. At least not as big a problem as the inefficiency of having so many processes.

Best Answer

Shell scripts are not expected to run with that type of speed. If you want to improve the speed of your script, try it in perl. If that is still too slow, then you'll have to move to a statically typed language such as java or c, or write a C module for perl that runs the parts which are too slow.

Shell is the first level of prototyping, if you can prove the concept with shell, then move to a better scripting language which can do more bounds checking which would take acres of shell.

A Unix OS is expected to include many small programs which do well defined tasks which make up a larger picture. This is a good thing as it compartmentalises bigger programs. Take a look at qmail, for example and compare that with sendmail. qmail is made of many programs:

http://www.nrg4u.com/qmail/the-big-qmail-picture-103-p1.gif

Exploiting the network daemon would not help you exploit the queue manager.

Related Solutions

Decomposition of path specs into longest-common-prefix + suffix

You can do that in a shell loop. The code below should work with all kinds of strange paths with extra slashes; if all your paths are of the form /foo/bar, you can get away with something simpler.

split_common_prefix () {
  path1=$1
  path2=$2
  common_prefix=
  ## Handle initial // specially
  case $path1 in
    //[!/]*) case $path2 in
               //[!/]*) common_prefix=/ path1=${path1#/} path2=${path2#/};;
               *) return;;
             esac;;
    /*) case $path2 in
          /*) :;;
          *) return;;
        esac;;
    *) case $path2 in /*) return;; esac;;
  esac
  ## Normalize multiple slashes
  trailing_slash1= trailing_slash2=
  case $path1 in */) trailing_slash1=/;; esac
  case $path2 in */) trailing_slash2=/;; esac
  path1=$(printf %s/ "$path1" | tr -s / /)
  path2=$(printf %s/ "$path2" | tr -s / /)
  if [ -z "$trailing_slash1" ]; then path1=${path1%/}; fi
  if [ -z "$trailing_slash2" ]; then path2=${path2%/}; fi
  ## Handle the complete prefix case (faster, necessary for equality and
  ## for some cases with trailing slashes)
  case $path1 in
    "$path2")
      common_prefix=$path1; path1= path2=
      return;;
    "$path2"/*)
      common_prefix=$path2; path1=${path1#$common_prefix} path2=
      return;;
  esac
  case $path2 in
    "$path1"/*)
      common_prefix=$path1; path1= path2=${path2#$common_prefix}
      return;;
  esac
  ## Handle the generic case
  while prefix1=${path1%%/*} prefix2=${path2%%/*}
        [ "$prefix1" = "$prefix2" ]
  do
    common_prefix=$common_prefix$prefix1/
    path1=${path1#$prefix1/} path2=${path2#$prefix1/}
  done
}

Alternatively, determine the longest common prefix of the two strings and trim it to its last / character (except when the common prefix consists solely of slashes).

What are the POSIX “mandatory utilities”

From one of the sections that you cite:

Optional utilities that are present only on systems supporting the associated option; see Codes for information on the options in this volume of IEEE Std 1003.1-2001

The mandatory utilities are the ones that are not marked as optional.

For example basename has no annotation to indicate that it's optional, so it's mandatory. alias is annotated as UP, so it's only mandatory if an implementation claims to include the User Portability Utilities option. command is mandatory, but the -v and -V options are not unless the implementation claims to include the User Portability Utilities option.

Best Answer

Related Solutions

Decomposition of path specs into longest-common-prefix + suffix

What are the POSIX “mandatory utilities”

Related Question