I am using the python script for separating the domain from the respective emails and then grouping emails as per their respective domain. The following script work for me:
#!/usr/bin/env python3
from operator import itemgetter
from itertools import groupby
import os
import sys
dr = sys.argv[1]
for f in os.listdir(dr):
write = []
file = os.path.join(dr, f)
lines = [[l.strip(), l.split("@")[-1].strip()] for l in open(file).readlines()]
lines.sort(key=itemgetter(1))
for item, occurrence in groupby(lines, itemgetter(1)):
func = [s[0] for s in list(occurrence)]
write.append(item+","+",".join(func))
open(os.path.join(dr, "grouped_"+f), "wt").write("\n".join(write))
I used : python3 script.py /path/to/input files
The input I gave was a list of emails and got the out as:
domain1.com,gemail1@domain1.com,email2@domain.com
domain2.com,email1@domain2.com,email2@domain2.com,email3@domain2.com
But what the problem am facing is because of the MongoDB limit. As MongoDB has limit of 16 MB of document size and single line in my output file is considered as 1 document by MongoDB and the line size should not go beyond 16 MB.
So what I want to have is the result should get limited to 21 emails per domain and if the domain has more emails then it should be printed on a new line with the rest emails (again if emails are exceeding 21 then newline with same domain name). I cam store duplicate data in the mongoDB.
So the final output should be something like the following:
domain1.com,email1@domain1.com,email2@domain1.com,... email21@domain1.com
domain1.com,email22@domain1.com,.....
domain2.com,email1@domain2.com,....
The dot (.) in the above example represents many text, which I chopped to make it simple to understand.
Hope this clarify my problem and hoping to get a solution for it.
Best Answer
New version
The script you posted indeed groups the emails by domain, with no limit in number. Below a version that will group emails by domain, but split the found list into arbitrary chunks. Each chunk will be printed into a line, starting with the corresponding domain.
The script
To use
chunked_list.py
In the head section, set the chunk size:
Run the script with the directory as argument:
It wil then create an edited file of each of the files, named
chunked_filename
, with the (chunked) grouped emails.What it does
The script takes as input a directory with files like:
Of each file, it creates a copy, like:
(set cunksize = 3)