Automating Files and Folders

The mental shift that unlocks file automation is simple: a folder is a collection of paths, and a path is just text your code can inspect and act on. Once you see a directory as a list you can loop over, tidying it becomes the same input-process-output shape as any other script - read the files in, decide something about each, write a result out.

Modern Python gives you pathlib, an object-oriented way to handle paths that reads almost like English and works identically on Windows, macOS and Linux. A Path object knows its own pieces - name, stem (name without extension), suffix (the extension), parent folder - so you rarely fiddle with slashes by hand:

from pathlib import Path

folder = Path("exports")
for file in folder.iterdir():
    if file.is_file():
        print(file.name, file.suffix, file.stat().st_size)

folder.iterdir() yields every entry; file.is_file() skips sub-folders; file.suffix is .pdf or .jpg; file.stat().st_size is the size in bytes. That is enough to report on a folder - and reporting first, before you change anything, is a habit worth keeping. When you do want the path as text (some older functions expect a string), str(file) gives it back. Prefer pathlib for new code; you will still meet the older os.path style in examples online, and the two interoperate cleanly.

It is worth meeting that older os module too, because you will see it everywhere. os.listdir(folder) returns the filenames as plain strings, os.getcwd() reports the folder your script is running from, and os.path.join(a, b) builds a path with the correct separator. pathlib wraps all of this in one tidy object, but recognising the os names lets you read the vast body of code written before pathlib became the default. One small but important habit goes with this: always be explicit about which folder you are working in. A script that quietly assumes the current directory will do the wrong thing the moment a colleague runs it from somewhere else, so anchor your paths - Path(__file__).parent gives the folder the script itself lives in, a reliable starting point you can build from.

Renaming a set of exports to a consistent scheme is the canonical example, and pathlib makes it clean. The safe move is Path.rename(new_path), which renames within the same folder when you hand it a sibling path. Build the new name from the old one's parts so you never lose the extension:

from pathlib import Path

folder = Path("exports")
pdfs = sorted(folder.glob("*.pdf"))
for i, file in enumerate(pdfs, start=1):
    new_name = f"A-{i:03d}_{file.stem}{file.suffix}"
    file.rename(file.with_name(new_name))

Three things earn their keep here. folder.glob("*.pdf") gathers only the PDFs; sorted(...) makes the numbering stable and repeatable; and enumerate(..., start=1) gives you a running counter, formatted as {i:03d} so 7 becomes 007 and the files sort correctly in any file browser. file.with_name(new_name) returns a new Path in the same folder with a new filename - exactly what rename wants.

One hard-won rule: never let your output pattern collide with your inputs. If you rename 1.pdf to 2.pdf while 2.pdf still exists, you overwrite real work. Two safeguards: rename into a new folder rather than in place, or do a dry run first - loop through and print() the old-and-new pairs, read them with your own eyes, and only then swap the print for the real rename. A dry run costs ten seconds and has saved many a project.

It is worth understanding why sorted() matters beyond neatness. File systems do not guarantee any particular order from iterdir() or glob(), so without sorting your A-001 prefix might attach to a different file each time you run - a subtle, maddening bug. Sorting makes the operation deterministic: the same input folder always yields the same numbering, exactly what you want for something you will re-run. If the names need to sort a particular way - by date, or by an embedded sheet number - pass a key function, for example sorted(pdfs, key=lambda p: p.stat().st_mtime) to number them oldest-first.

The second everyday win is sorting a mixed dumping-ground - a downloads folder, a shared drive, an export dump - into tidy sub-folders by file type. The logic is: for each file, look at its extension, map that to a destination folder, make the folder if needed, and move the file there. shutil handles the actual moving and copying (pathlib deliberately does not), so the two libraries pair up:

import shutil
from pathlib import Path

folder = Path("downloads")
buckets = {
    ".pdf": "pdf", ".jpg": "images", ".png": "images",
    ".xlsx": "sheets", ".csv": "sheets", ".dwg": "cad",
}
for file in folder.iterdir():
    if not file.is_file():
        continue
    target = buckets.get(file.suffix.lower(), "other")
    dest = folder / target
    dest.mkdir(exist_ok=True)
    shutil.move(str(file), str(dest / file.name))

Read it slowly. The buckets dictionary is your rule, expressed as data - to support a new file type you add one line, you do not touch the logic. file.suffix.lower() normalises .JPG and .jpg to the same key. buckets.get(key, "other") looks up the destination and falls back to an "other" folder for anything unexpected, so nothing is ever lost. dest.mkdir(exist_ok=True) creates the sub-folder and quietly does nothing if it already exists. Finally shutil.move relocates the file. Swap shutil.move for shutil.copy2 and you get the same sort while preserving the originals - copy2 also carries over timestamps, which matters when you sort by date later.

A subtlety worth flagging: shutil.move across different drives or partitions is really a copy-then-delete under the hood, so it can be slower and briefly needs room for both copies - fine for a downloads folder, worth knowing for very large media libraries. Within a single drive it is near-instant. And because the destination sub-folders are created on demand with mkdir(exist_ok=True), the script is safe to run more than once: a second run simply finds the folders already there and sorts whatever new files have arrived since, so you can point it at the same growing folder every week.

Often you do not want every file, only the ones matching a pattern - all DWGs, everything with _final in the name, all images issued this week. glob is pattern-matching for filenames, using two wildcards: * matches any run of characters, ? matches a single character. folder.glob("*.dwg") finds DWGs in one folder; folder.rglob("*.dwg") (recursive glob) walks every sub-folder too - invaluable across a nested project tree:

from pathlib import Path

project = Path("project_A")
finals = [p for p in project.rglob("*_final*.pdf")]
print(f"Found {len(finals)} final PDFs")

for p in finals:
    print(p.relative_to(project))

The list comprehension [p for p in project.rglob("*_final*.pdf")] collects the matches into a list so you can count them, sort them, or feed them to the next step. p.relative_to(project) prints a short path relative to the project root instead of the full absolute path. From here you compose: find with glob, filter further with an if (for example if p.stat().st_mtime > cutoff to keep only recently modified files), and act with shutil. That find-filter-act pipeline - the second figure - is the backbone of file automation, and once it is muscle memory you will reach for it constantly. A closing caution worth stating plainly: file operations are destructive. rename, move and especially deletion do not ask twice. Test on a copy, keep a backup, and dry-run anything that renames or moves in bulk until you trust it.

A script you write, run once and lose is perfectly fine - but the chores in this lesson recur every week, so the real leverage comes from turning them into a small tool you keep. The move is to wrap the logic in a function with the folder as a parameter, so the same code serves any project:

from pathlib import Path
import shutil

def sort_folder(folder, buckets, dry_run=True):
    folder = Path(folder)
    for file in folder.iterdir():
        if not file.is_file():
            continue
        target = buckets.get(file.suffix.lower(), "other")
        dest = folder / target
        if dry_run:
            print(f"would move {file.name} -> {target}/")
        else:
            dest.mkdir(exist_ok=True)
            shutil.move(str(file), str(dest / file.name))

Notice the dry_run=True default: the function is safe by design - it only reports until you deliberately call it with dry_run=False. That single parameter bakes the dry-run habit into the tool itself, so the cautious path is the easy one. Save this in a file you reuse across projects and you have started building a personal toolkit, which is exactly how working scripters accumulate leverage over a career.

One more everyday filter is by time. Every file carries a last-modified timestamp, file.stat().st_mtime, as a number of seconds; compare it to a cutoff to act only on what changed since the last issue:

import time
cutoff = time.time() - 7 * 24 * 3600   # seven days ago
recent = [p for p in folder.glob("*.pdf")
          if p.stat().st_mtime > cutoff]

Finally, be honest about the economics. Automation is leverage, not a reflex. A task you do once, on five files, is not worth a script - the ten minutes spent scripting a genuinely one-off job are ten minutes lost, and this course would rather you knew that than automated for its own sake. The tasks worth the effort are the recurring ones: the sort you run every issue, the rename on every export, the audit before every submission. When you meet one of those, promote your throwaway script into a kept function, and it pays back quietly for the rest of the project's life.