Before you begin
- Python 3.12 or newer
Set up your example
- Save the complete code as eml-normalize-subject.py in a working folder.
- Save the provided sample files in that same folder; run commands from that folder.
Sample files in this download:
- sample.eml
- sample.mbox
- body.txt
- note.txt
Download the ZIP, extract it, and open a terminal inside the extracted folder. The source file is eml-normalize-subject.py.
Run the script
python "eml-normalize-subject.py" --input "sample.eml" --output "subject.json"If your system uses python3 instead of python, use that command for Python examples. Keep paths containing spaces in quotes. Required inputs and optional settings are described in the code and setup steps.
What to expect
Remove repeated English reply/forward prefixes for subject comparison. Re: Project notes normalizes to Project notes.
The complete script
#!/usr/bin/env python3
'Extract a normalized conversation subject'
import argparse
import json
from pathlib import Path
def write_json(path, value):
data = json.dumps(value, indent=2, ensure_ascii=False, allow_nan=False) + "\n"
with Path(path).open("x", encoding="utf-8", newline="") as handle:
handle.write(data)
import re
import hashlib
import mimetypes
import mailbox
from collections import Counter, defaultdict
from datetime import datetime, timezone
from email import policy
from email.parser import BytesParser
from email.message import EmailMessage
from email.utils import getaddresses, parseaddr, formataddr, parsedate_to_datetime, format_datetime, make_msgid
def raw(path,limit=25*1024*1024):
if Path(path).stat().st_size>limit: raise ValueError("Input exceeds the size limit.")
return Path(path).read_bytes()
def message(path):
return BytesParser(policy=policy.default).parsebytes(raw(path))
def messages(path):
if Path(path).stat().st_size>100*1024*1024: raise ValueError("Mailbox exceeds 100 MiB.")
box=mailbox.mbox(path,factory=lambda f:BytesParser(policy=policy.default).parse(f),create=False)
try:
result=[]
for msg in box:
if len(result)>=5000: raise ValueError("Mailbox exceeds 5000 messages.")
result.append(msg)
return result
finally: box.close()
def text_body(msg):
part=msg.get_body(preferencelist=("plain",))
return part.get_content() if part else ""
def address_pairs(msg,headers=("from","to","cc","bcc","reply-to")):
return getaddresses([str(v) for key in headers for v in msg.get_all(key,[])])
def attachments(msg):
return [p for p in msg.walk() if p.get_content_maintype()!="multipart" and (p.get_content_disposition()=="attachment" or p.get_filename() is not None)]
def payload(part):
if part.get_content_type()=="message/rfc822" and isinstance(part.get_payload(),list):
return part.get_payload(0).as_bytes(policy=policy.default)
return part.get_payload(decode=True) or b""
def save_message(path,msg):
data=msg.as_bytes(policy=policy.SMTP)
with Path(path).open("xb") as handle: handle.write(data)
def clean_address(value):
if "\r" in value or "\n" in value: raise ValueError("Address cannot contain newlines.")
display,address=parseaddr(value)
if "@" not in address or any(c.isspace() for c in address): raise ValueError("Supply one ordinary email address.")
return address
def draft(sender,recipient,subject,body):
msg=EmailMessage()
msg["From"]=clean_address(sender)
msg["To"]=clean_address(recipient)
msg["Subject"]=subject
msg["Date"]=format_datetime(datetime.now(timezone.utc))
msg["Message-ID"]=make_msgid(domain="scriptage.local")
msg.set_content(body)
return msg
def main():
parser = argparse.ArgumentParser(description='Extract a normalized conversation subject')
parser.add_argument("--input",required=True,help="Local EML, MBOX, or body file as documented")
parser.add_argument("--output",required=True,help="New result file or directory")
args = parser.parse_args()
original=str(message(args.input).get("Subject",""))
normalized=re.sub(r"^(?:(?:re|fw|fwd)\s*:\s*)+","",original,flags=re.I).strip()
write_json(args.output,{"original":original,"normalized":normalized,"comparison_key":normalized.casefold()})
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, KeyError, TypeError, ArithmeticError, IndexError, LookupError) as exc:
raise SystemExit(str(exc))
Open source as plain text · Download source and sample files
How it works
- All messages are read from local files; no SMTP, browser, or email-service connection is used.
- Normalization is a comparison aid only; unrelated messages can share a subject and localized prefixes are not automatically inferred.
- Re: Project notes normalizes to Project notes.
Make it your own
--input
Use another saved reply or forward.
Adapt the processing
Edit main's prefix regex to include a specifically known localized prefix.
Change one setting at a time, run it again on sample data, and compare the output. The customization guide walks through this process.
If something goes wrong
A localized prefix remains.
Add that explicit prefix after confirming its meaning; do not strip arbitrary words.
The destination already exists.
Choose a new --output path. The script deliberately refuses to replace an existing result.
Reference & next steps
These official references explain the underlying tools. The Scriptage example and guide are original instructional content. See verification notes for what has been checked and the limits of those checks.
Copies a template for your own notes. It does not send anything.