Bash – Delete all files in directories except those whose path are listed in a file

bashfilesfindscriptingshell-script

Given the following files:

data/A/a.txt
data/B/b.pdf
...
date/P/whatever.log
...
data/Z/z.jpg

I would like to delete all files in the data/A/, data/B/, …, data/Z/ directories except those files that are situated under one of the directories listed in the file data/dont_clean.txt. For example, if we have data/P listed in data/dont_clean.txt then nothing should be touched under data/P/, etc.

Something like:

find data/ -mindepth 2 -maxdepth 2 -type f -not -path {listed in data/dont_clean} -delete

Of course it is not a valid command.

I have also tried variants of

find data/ -mindepth 2 -maxdepth 2 -type f -exec grep data/dont_clean.txt '{}' \;

but I only created either an invalid command or I had no idea why I got the output I did.

I am using bash on Ubuntu 12.10

Best Answer

This is code that I only roughly tested but might layout an approach for you to take. Assuming you have a file, ignore.txt like this:

1/
2/

Sample data

And I had sample directories with files in them like this:

$ mkdir -p dirs/{1..5}
$ touch dirs/{1..5}/afile

Resulting in this:

$ tree dirs/
dirs/
|-- 1
|   `-- afile
|-- 2
|   `-- afile
|-- 3
|   `-- afile
|-- 4
|   `-- afile
`-- 5
    `-- afile

Example run

Now if we run this command against this tree:

$ find dirs/ -type f -print0 | fgrep -zFvf ./ignore.txt
dirs/5/afiledirs/4/afiledirs/3/afile

We can see that we're only getting back the files that are in directories not listed in ignore.txt.

So we can add a rm to the end to remove the non-excluded files.

$ find dirs/ -type f -print0 | fgrep -zFvf ./ignore.txt | xargs -0 rm -f

Checking we can see that it worked:

$ tree dirs/
dirs/
|-- 1
|   `-- afile
|-- 2
|   `-- afile
|-- 3
|-- 4
`-- 5

Problems to be worked out

One big problem with this approach is that the strings in the ignore.txt file might match other portions of the directory structure. So some care needs to be paid to making sure that the strings in this file are unique in the way that you expect.

Some blocking could be put around the strings so that they're anchored to the beginning or the end of the string to protect them.

Details

The above commands are doing the following:

finding all the files under the directory dirs
filtering out any files that are under a directory present in the igonre.txt file
passing the filter list via xargs to the rm -f command

Related Solutions

Shell – Clone directory tree structure and copy files to the corresponding directories modified after a specific date

Assuming you have your desired files in a text file, you could do something like

while IFS= read -r file; do 
    echo mkdir -p ${file%/*}; 
    cp /source/"$file" /target/${file%/*}/${file##*/}; 
done < files.txt

That will read each line of your list, extract the directory and the file name, create the directory and copy the file. You will need to change source and target to the actual parent directories you are using. For example, to copy /foo/a/a.txt to /bar/a/a.txt, change source to foo and target to bar.

I can't tell from your question whether you want to copy all directories and then only specific files or if you just want the directories that will contain files. The solution above will only create the necessary directories. If you want to create all of them, use

find /source -type d -exec mkdir -p {} /target

That will create the directories. Once those are there, just copy the files:

while IFS= read -r file; do 
    cp /source/"$file" /target/"$file"
done

Update

This little script will move all the files modified after September 8. It assumes the GNU versions of find and touch. Assuming you're using Linux, that's what you will have.

#!/usr/bin/env bash    

## Create a file to compare against.
tmp=$(mktemp)
touch -d "September 8" "$tmp"

## Define the source and target parent directories
source=/path/to/source
target=/path/to/target

## move to the source directory
cd "$source"

## Find the files that were modified more recently than $tmp and copy them
find ./ -type f -newer "$tmp" -printf "%h %p\0" | 
    while IFS= read -rd '' path file; do
        mkdir -p "$target"/"$path"
        cp "$file" "$target"/"$path"
    done

Strictly speaking, you don't need the tmp file. However, this way, the same script will work tomorrow. Otherwise, if you use find's -mtime, you would have to calculate the right date every day.

Another approach would be to first find the directories, create them in the target and then copy the files:

Create all directories

 find ./ -type d -exec mkdir -p ../bar/{} \;

Find and copy the relevant files

 find ./ -type f -newer "$tmp" -exec cp {} /path/to/target/bar/{} \;

Remove any empty directories

 find ../bar/ -type d -exec rmdir {} \;

How to find only files, with which command xyz ends successfully

You can do:

find . -exec ./is_dir.py {} \; -o -print

It will list everything that is not a directory. Assuming that is_dir.py is executable (chmod +x is_dir.py) and contains something like:

#!/usr/bin/env python

import sys
import os

if os.path.isdir(sys.argv[1]):
    sys.exit(0)
sys.exit(1)

And if is_dir.py generates output of its own, you can do:

 rm -f nondir.lst
 find . -exec ./is_dir.py {} \; -o -exec echo  {} >> nondir.lst \;

so that its outoput doesn't get mixed with the list of names for which is_dir.py fails.