I'm creating an OCR-ed PDF through tesseract:
tesseract input.tif out pdf
But I also need the hocr and txt files.
Recent versions of tesseract already solved this but because it requires compiling both leptonica and tesseract, I'm not entirely comfortable with it.
I can use pdftotext to extract the text file but I can't seem to find a way to extract hocr from the PDF.
This is taken in whole from @Herbert answering a very similar question on the TeX StackExchange:
Adding Table of Contents to existing PDF
use package pdfpages and then:
\documentclass{article}
\usepackage{pdfpages}
\usepackage{hyperref}
\begin{document}
\tableofcontents
\clearpage\phantomsection
\addcontentsline{toc}{section}{The first section name}% or chapter
\includepdf[pages={1-10},linktodoc,linktodocfit=/Fit]{texte/dtk/dtk11-1/komoedie.pdf}
\clearpage\phantomsection
\addcontentsline{toc}{section}{The second section name}% or chapter
\includepdf[pages={11-19},linktodoc,linktodocfit=/Fit]{texte/dtk/dtk11-1/komoedie.pdf}
\clearpage\phantomsection
\addcontentsline{toc}{section}{The third section name}% or chapter
\includepdf[pages={20-29},linktodoc,linktodocfit=/Fit]{texte/dtk/dtk11-1/komoedie.pdf}
\clearpage\phantomsection
\addcontentsline{toc}{section}{The forth section name}% or chapter
\includepdf[pages={21-39},linktodoc,linktodocfit=/Fit]{texte/dtk/dtk11-1/komoedie.pdf}
\end{document}
Since different publishers use different methods of "marking" the PDFs you need to make sure you compare without taking the markings into account.
You also need an efficient method to compare a new PDF to all already downloaded PDFs in case you repeatedly download the same PDF and it is e.g. marked with the IP and/or date-time-stamp as you suggest. You don't want to use a time consuming comparison mechanism that compares each new PDF with many already downloaded PDFs
What you need is a utility that strips each of the possible markings and generate a hash of the remaining data. You will need to keep a hash → file name map, which can be in a simple file, and if a computed hash is already in the file you have a duplicate (and delete it or do whatever needed) and if the hash in not yet there, you add the hash and file name. The file would look something like:
6fcb6969835d2db7742e81267437c432 /home/anthon/Downloads/explanation.pdf
fa24fed8ca824976673a51803934d6b9 /home/anthon/orders/your_order_20150320.pdf
That file is negligently small compared to the original PDFs. If you have millions of PDFs you might consider storing this data in a database. For efficiency sake you might want to include the filesize and number of pages in there (pdfinfo | egrep -E '^Pages:' | grep -Eo '[0-9]*').
The above pushes the problem to removing the markings and generating the hash. If you know where the PDF comes from when invoking the hash generating routine (i.e. if you do the downloads programmatically), you can fine-tune the hash generation based on that. But even without that there are several possibilities for hash generation:
pdfinfo -E file.pdf | grep -E '^(Author:)|(Title:) | md5sum to get the hash. You can include the number of pages in calculating the hash as well ('Pages:' in the pdfinfo output).imagemagick). You can use pdfimages to extract the image information into a temporary file.pdftext to extract the text, filter out the marking (if you filter out a little to much, that is not a problem) and then generate the hash based on that.Additionally you can compare if the file size of the old file found via the hash and see if is within certain margins with the new file. Compression and ifferences in strings (IP/date-time-stamp) should only result in less than one percent difference.
If you know the method the publisher uses when determining the hash, you can directly apply the "right" method of the above, but even without that you can check for the metadata and apply some heuristics, or determine the number of images in a file and compare that with the number of pages (if they are close you probably have a document consisting of scans). pdftext on scanned image PDFs also has a recognisable output.
As a basis to work from I created a python package that is on bitbucket and/or can be installed from PyPI using pip install ruamel.pdfdouble.
This provides you with the pdfdbl command that does the scanning as described above on metadata, extracted images or on text.
It doesn't do any filtering of markings (yet), but the readme describes which (two) methods to enhance to do add that.
The included readme:
this package provides the pdfdbl command:
pdfdbl scan dir1 dir2
This will walk down the directories provided as argument and for the PDF files found, create a hash based on (in order):
This assumes that pdfinfo, pdfimages and pdftotext` from the poppler-utils package are avaialable.
A "database" is build up in ~/.config/pdfdbl/pdf.lst against which further scans are tested.
In ruamel/pdfdouble/pdfdouble.py there are two methods that can be enhanced to filter out markings in the PDF that make them less unique and make virtually the same files to have different hashes.
For text the method PdfData.filter_for_marking should be extended to remove and markings from the string that is its arguments and return the result.
For scanned images the method PdfData.process_image_and_update needs to be enhanced, e.g. by cutting off the images bottom and top X lines, and by removing any gray background text by setting all black pixels to white. This function needs to update the hash passed in using the .update() method passing in the filtered data.
The current "database" cannot handle paths that contain newlines
This utility is currently Python 2.7 only.
IP conforming stringparts can be substituted with Python's re module:
import re
IPre = re.compile("(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}"
"([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])")
x = IPre.sub(' ', 'abcd 132.234.0.2 ghi')
assert x == 'abcd ghi'
Best Answer
You can simply run the following command to create both pdf and hocr at the same time.