Parse multiple HTML to text and rename as parent Directory

bashbash-scriptingshell

In a directory there are many folders .html inside, I would like all HTML parsed as new .txt with the name of the parent directory.

Example1/
Index.html>Example1.txt

Example2/
Index.html>Example2.txt

Best Answer

Obviously, you want to convert some HTML pages into plain text. Therefore I wouldn't strip the tags with a custom built solution (e.g. with some sed magic), but use a tool designed for that purpose like html2text; from its webpage:

html2text is a Python script that converts a page of HTML into clean, easy-to-read plain ASCII text. Better yet, that ASCII also happens to be valid Markdown (a text-to-HTML format).

To address your question of batch renaming:

find . -maxdepth 1 -type d -print0 | while IFS= read -r -d '' dirname
 do python path/to/html2text/html2text.py "${dirname}/index.html" > "${dirname}/${dirname}.txt"
done

Here the find command lists all directories only located in the current directory (i.e. not recursive) and the read command (in the while condition) assigns the values to the variable $dirname. Finally, the command(s) between do and done get(s) executed, here it converts the files according to your request. As pointed out by @slhck you need to use such a complex command, so that dirnames with whitespaces won't break anything.

[Edit]: Another variant to convert all HTML files under the current directory:

find . -iname "*.html" -print0 | while IFS= read -r -d '' filename
  do python path/to/html2text/html2text.py "${filename}" > "${filename%.*}.txt"
done

iname searches case insensitive for *.html.

${filename%.*}.txt strips the extention and appends .txt, i.e. if filename is some/path/index.html, ${filename%.*} is some/path/index and finally ${filename%.*}.txt is some/path/index.txt.

When you use the Z shell, you can use a much cleaner for loop, without braking at white spaces:

for i (*(/)) python path/to/html2text/html2text.py "${i}/index.html" > "${i}/${i}.txt"

The trick here is that *(/) does filename generation, but only returns directories (/).

[Edit]: Also in zsh syntax the variant to convert all HTML files under the current directory (you need the option EXTENDEDGLOB to be set):

for i ((#i)**/*.html) {
   python path/to/html2text/html2text.py "$i" > "${i:r}.txt"
}

(#i) uses case insensitive globbing, ** searches recursively, hence returning all HTML files under the current working directory. (If symbolic links should be followed, use three stars *** instead of two).

If you have more than one command inside the for loop, use curly { ... }brackets (unnecessary here, but they won't hurt).

${i:r} strips the extention (r for remove) from the variable $i.

`find`

#!/bin/sh
cd ~/Documents/Packages &&
find . -name dist -type d -exec sh -c 'for d do parent="${d%/*}"; \
                                                mkdir -p "../Artifacts/$parent"; \
                                                mv "$d" "../Artifacts/$parent"; done' \
                                                sh {} +

(You can put the find command all on one line if you want.) This goes into ~/Documents/Packages, so we can use relative pathnames. A fairly straightforward use of the -name and -type tests of find to identify (sub)directories named dist. Use the -exec action to pass their names to a miniature shell script.

The script loops over its arguments. For each one, it identifies the parent directory by removing / and a pathname level from the right. For example, the parent of ./core-types/dist is ./core-types. (Remember; we are in ~/Documents/Packages and looking at ., so we are seeing relative pathnames.) I could have said parent="${d%/dist}", but I wanted to keep it generic. Create the destination/parent directory — for example, ../Artifacts/./core-types. (The extra ./ is ignored; /./ acts like /, so this creates ../Artifacts/core-types, which is ~/Documents/Packages/../Artifacts/core-types, which is ~/Documents/Artifacts/core-types.) We then move the dist directory into that directory.

bash Pathname Expansion

This depends on a feature of bash that is not present in all shells. (For that matter, it might not be in all versions of bash.)

#!/bin/bash
cd ~/Documents/Packages && shopt -s globstar &&
for d in **/dist/; do d="${d%/}"; parent="${d%/*}";
                                  mkdir -p "../Artifacts/$parent";
                                  mv "$d" "../Artifacts/$parent"; done

Set the globstar shell option (shopt), which causes ** to do a recursive search. Then loop through **/dist/. If we said **/dist, that would find all files and directories named dist; adding the / at the end constrains the search to (sub)directories only. For each one, execute a bunch of commands.

**/dist/ gives us the directory names in the form core-types/dist/ (with the / at the end, as we specified). d="${d%/}" strips off that trailing slash. The rest of the code is the same.

Note: this will not look under top-level directories whose names begin with . (e.g., ~/Documents/Packages/.memories/dist) unless you add the dotglob option.

Step 2

If you want to copy instead of move, just change mv to cp -R in either of the above commands.

Best Answer

Related Solutions

Shell script to rename multiple files from their parent folders name

Linux – Bash script to find folders containing a specific subfolder, and then copying to new directory while preserving parent name

find

bash Pathname Expansion

Step 2

Related Question

`find`