Bash – Create a file of fixed size with specific contents

bashcommand linefilesio

I want to create a file of fixed size (1G, 10G, 100G etc) with a single random word of length within the specified limit on every line. I basically want this to run a benchmark which will sort the entire file.

So if I want a file of 1G and the word length limit is suppose 4, the sample of file would look like this:

a
bc
def
ghij

Here the words' length will be within 1-4 and it won't exceed 4 and this file will eventually have the size of 1G

NOTE: The word can be of a fixed size too. It won't be a problem.

How will I be able to do this?

Best Answer

My understanding of the question is that, you need to create a large file, each line of this file is a random word within specified length.
If you don't need the word to be a real word, but some random characters:

< /dev/urandom tr -d -c '[:alpha:]'|head -c 1M|fold -w10 >result.txt

This will create a file of size 1M and each line with 10 random characters.

Related Solutions

Bash Command Line – Combining Multiple Process Substitutions

You could do:

eval paste '<(sort -n ../data/file-'{A,B,C}'.dat)'

Or to automate it as a function

sort_paste() {
  local n i cmd
  n=1 cmd=paste
  for i do
    cmd="$cmd <(sort -n -- \"\${$n}\")"
    n=$(($n + 1))
  done
  eval "$cmd"
}
sort_paste  ../data/file-{A,B,C}.dat

(in some ksh implementations, you need to replace local with typeset)

To adapt to any arbitrary command, (and to prove that eval can be safe when used properly), you could do:

xproc() {
  local n i cmd stage stage1 stage2 stage3
  cmd= xcmd= stage=1 n=1
  stage1='cmd="$cmd \"\${$n}\""'
  stage2='xcmd="$xcmd \"\${$n}\""'
  stage3='cmd="$cmd <($xcmd \"\${$n}\")"'
  for i do
    if [ -z "$i" ] && [ "$stage" -le 3 ]; then
      stage=$(($stage + 1))
    else
      eval 'eval "$stage'"$stage\""
    fi
    n=$(($n + 1))
  done
  eval "$cmd"
}

xproc paste '' sort -n -- '' ../data/file-{A,B,C}/dat

Bash – limiting size of logfile from a variable

You need to use logrotate. Do something like this

cat /etc/logrotate.conf

/path/foo.txt {

    size 50M

    create 700 root root

    rotate 5

}

size 50M – logrotate runs only if the filesize is equal to (or greater than) this size.

create – rotate the original file and create the new file with specified permission, user and group.

rotate – limits the number of log file rotation. So, this would keep only the recent 5 rotated log files.

Best Answer

Related Solutions

Bash Command Line – Combining Multiple Process Substitutions

Bash – limiting size of logfile from a variable

Related Question