Ubuntu – Reading lines from a file and storing their content to pass to a C program in a shell script

command linetext processing

I have a C program and the inputs are given through command line. Now I want to pass the inputs from a text file. There are two inputs, a and b, written line by line like this:

Now I want to call this txt file in a loop and pass each a and b value line by line so that the output of the C program is generated for each a and b value.

To do so I want to write a shell script which will call the txt file in read mode and after getting the data it will call the C program and then the C program will do the rest including the output.

As I am new to this shell scripting I am confused about how to open a text file and then read data line by line and store them in some variable in the shell script.

Please suggest how can I open the text file in read mode and then read data (which may not be always integers) and store them in some other variables.

Best Answer

Assuming "inputs are given through command line" means that the program accepts command-line arguments, then you should be able to do this with xargs

Ex. given a minimal C program

#include <stdio.h>

int main(int argc, char *argv[])
{
  printf("Running %s\n", argv[0]);
  for(int i=1; i<argc; i++) printf("arg %d = %s\n", i, argv[i]);
  printf("\n");
}

compiled with gcc -o someprog someprog.c, and a file of whitespace-separated input arguments inputs.txt

$ cat inputs.txt 
2   423
4    56.9
7   83.5

Then

$ xargs -L1 ./someprog < inputs.txt
Running ./someprog
arg 1 = 2
arg 2 = 423

Running ./someprog
arg 1 = 4
arg 2 = 56.9

Running ./someprog
arg 1 = 7
arg 2 = 83.5

If your input file has a header, you will need to remove it (using tail for example).

If parallel processing would be computationally advantageous for your task, then you could do a similar thing with GNU Parallel

parallel --colsep '[ \t]+' ./someprog {1} {2} :::: inputs.txt

How it works

Both files are read by the script. The lines in file_1 (the file that has precedence) is split by the directory you entered for the file in the head section (in your example /mnt/app/).

Subsequently, the lines in file_1 are written to the output file (the merged file). At the same time, lines from file_2 are removed from the line list if the identifying string (the section after the mount point) occurs in the line. Finally, the "remaining" lines of file_2 (of which no dupe exist in file_1) are written to the output file as well. The result:

file_1:

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

file_2:

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

merged:

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial

The script

#!/usr/bin/env python3
#---set the path to file1, file2 and the mountpoint used in file1 below
f1 = "/path/to/file_1"; m_point = "/mnt/app"; f2 = "/path/to/file_2"
merged = "/path/to/merged_file"
#---
lines1 = [(l, l.split(m_point)[-1]) for l in open(f1).read().splitlines()]
lines2 = [l for l in open(f2).read().splitlines()]

for l in lines1:
    open(merged, "a+").write(l[0]+"\n")
    for line in [line for line in lines2 if l[1] in line]:
            lines2.remove(line)

for l in lines2:
    open(merged, "a+").write(l+"\n")

How to use

Copy the script into an empty file, save it as merge.py
in the head section of the script, set the paths to f1 (file_1), f2, the path to the merging file and the mountpoint as mentioned in file_1.
Run it by the command:
```
python3 /path/to/merge.py
```

Edit

Or a tiny bit shorter:

#!/usr/bin/env python3
#---set the path to file1, file2 and the mountpoint used in file1 below
f1 = "/path/to/file_1"; m_point = "/mnt/app"; f2 = "/path/to/file_2"
merged = "/path/to/merged_file"
#---
lines = lambda f: [l for l in open(f).read().splitlines()]
lines1 = lines(f1); lines2 = lines(f2); checks = [l.split(m_point)[-1] for l in lines1]
for item in sum([[l for l in lines2 if c in l] for c in checks], []):
    lines2.remove(item)
for item in lines1+lines2:
    open(merged, "a+").write(item+"\n")

Ubuntu – Reading lines from a text file and creating a text file for each name on each line

#1 Using Bash + `touch`:

while read line; do touch "$line.txt"; done <in

while read line; [...]; done <in: This runs read until read itself returns 1, which happens when when the end of the file is reached; the input for read is read from a file named in in the current working directory instead of from the terminal due to the <in redirection;
touch "$line.txt": This runs touch on the expanded value of $line.txt, which is the content of line followed by .txt; touch will create the file if not existing and update its access time if existing;

#2 Using `xargs` + `touch`:

xargs -a in -I name touch name.txt

-a in: makes xargs read its input from a file named in in the current working directory;
-I name: makes xargs replace every occurence of name with the current line of input in the following command;
touch name: Runs touch on the replaced value of name; it will create the file if not existing and update its access time if existing;

% ls
in
% cat in
john
george
james
stewert
% while read line; do touch "$line.txt"; done <in
% ls
george.txt  in  james.txt  john.txt  stewert.txt
% rm *.txt
% xargs -a in -I name touch name.txt
% ls
george.txt  in  james.txt  john.txt  stewert.txt

Best Answer

Related Solutions

Ubuntu – Remove duplicated from two files and merge the unique ones