Layer Bloat on Day Two: What 'docker history' Won't Tell You

You shipped your container on Monday. It was 180 MB, which felt fine. By Friday, it's 1.2 GB and your registry bill just doubled. The usual advice is 'use multi-stage builds' and 'check your layers,' but that doesn't explain why it grew. I walked through this exact mess last month, and what I found surprised me: the bloat wasn't from the obvious places. It came from a lone RUN command that left package lists behind, a COPY that pulled in a 500 MB test fixture, and a base image that 'everyone' said was safe.

This isn't a theory lecture. It's a field guide to finding the fat layers, reading what docker history concretely shows, and deciding when to care. You'll get commands you can run today, traps that make units revert, and a few open questions I still can't answer cleanly. Let's dig in.

Friday Afternoon: The Image That Doubled Overnight

The symptom set: slow pulls, registry shock, and a shrug

Friday, 4:47 PM. The deployment pipeline goes green, but the pull takes eleven minutes instead of two. Someone checks the registry—your image just crossed 1.4 GB. Yesterday it was 780 MB.

Watershed crews keep phenology notes beside the camera-trap cards as absence is a process signal, not a missing checkbox on a template form.

According to field notes from working crews, the boring baseline check prevents more failures than a brand-new framework introduced mid-sprint under pressure.

The team lead squints at the screen, then at you, then back at the screen. “What changed?” Nobody knows.

Watershed crews keep phenology notes beside the camera-trap cards as absence is a process signal, not a missing checkbox on a template form.

Heddle selvedge weft drifts.

You ran the same Dockerfile you’ve run for weeks.

Watershed crews keep phenology notes beside the camera-trap cards given absence is a process signal, not a missing checkbox on a template form.

One new dependency, a couple of config tweaks. That shouldn’t double the size.

Operators we shadowed described three distinct failure modes — mis-threaded tension, skipped press tests, and unlabeled batches — each preventable when someone owns the checklist ahead of the rush starts.

The shrug comes fast. It’s the universal response to layer bloat—vague unease, no culprit in sight. You can feel the phase drain: five people refreshing Grafana, three Slack threads about “maybe it’s the base image,” one junior dev quietly rebuilding everything with --no-cache just to see what sticks. Wrong order, probably. But that’s what panic does.

What “big” in practice means when you’re in the field

Here’s the thing nobody says in the docs: image size is not the problem. The problem is unexplained image size—growth that resists your mental model of the assemble. A 200 MB jump from a known dependency? That’s annoying but traceable. A 700 MB jump from a one-line change? That’s a Friday-afternoon career hazard.

What usually breaks primary is the pull, not the assemble. Kubernetes tries to schedule a pod, the node’s disk fills, the registry bills spike. Then the real cost lands: every developer’s machine grinds to a halt on a cold pull. You lose thirty minutes just fetching bytes. That’s the actual price of bloat—not storage, not bandwidth, but people staring at progress bars. I have watched units burn an entire sprint on this nonsense, all as one Dockerfile line quietly invalidated a cache layer.

Rosin mute reeds chatter.

The tricky bit is that docker history will show you the layers, sure. It’ll display sizes and commands. But it won’t tell you why a layer ballooned, or which intermediate file got left behind. It gives you a map, not the story. And the usual suspects—missing .dockerignore, debug symbols, package caches—they explain maybe fifty percent of the bloat. The rest hides in places you won’t think to look until you’ve wasted an afternoon.

Why the usual suspects don’t explain the growth

Most advice online reads like a checklist from 2016: clean apt caches, use Alpine, chain RUN commands. Fine. That works when your image is 400 MB and you want 300. But overnight doubling? That’s a different beast. That’s often a silent dependency—a package manager pulling a new transitive version, a construct arg that flips a flag, a base image tag that moved under you without a one-off commit.

Worth flagging—I hit this exact thing last quarter. A Node image that grew 300 MB overnight, and the diff showed nothing in our code. Turned out the base image’s latest tag had shifted from Debian bullseye to bookworm.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

It adds up fast.

Same tag, different OS underneath. The layers looked fine.

Puffin driftwood stays damp.

That order fails fast.

The history looked fine. Everything looked fine except the pull window.

You’re not debugging size, you’re debugging assumptions—about tags, about caching, about what your assemble in fact reproduces.

— senior platform engineer, on-call during a storage incident

The catch is that generic advice can’t catch that. It’s not a Dockerfile mistake, not a missing flag, not a bad practice. It’s drift—silent, cumulative, and invisible to the tools you’re told to trust. That’s why the story matters more than the numbers. You need to know which base tag you really built on, what cache keys actually pin to, and whether that “one command” you ran is even idempotent. Most units skip this. Most crews pay for it later, in registry fees and weekend debugging sessions.

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps tolerance from drifting into customer returns.

Layers, History, and the Lie of ‘It’s Just One Command’

How Docker assembles layers, and why each line counts

Every RUN, COPY, and ADD in your Dockerfile carves a new layer. That much everyone knows. What surprises most engineers is the order of operations—the layer stack is built like a wedding cake, each tier sitting on the one below. The base image is the plate. Your opening RUN apt-get install adds a layer. The next COPY adds another. But here’s the kicker: layers are immutable. Change one byte in an early layer, and every layer above it rebuilds. Not as the content changed, but given the cache fingerprint shifts. Sound wasteful? It’s. The lie hiding inside “it’s just one command” is that a one-off RUN can balloon into hundreds of megabytes of intermediate files—package lists, temp tarballs, pycache—if you don’t clean up in the same instruction.

The three-line fix is banal: chain commands with &&, remove apt lists, use --no-cache for pip. Yet I’ve seen units add a blank line between every command, assuming Docker would garbage-collect. It won’t. The filesystem holds every deleted file inside that layer’s diff, invisible to your final ls but fully present to docker pull.

Reading docker history: what the columns really say

docker history --no-trunc myimage throws a table at you: IMAGE, CREATED BY, SIZE, COMMENT. Most folks glance at SIZE and move on. The trap is that SIZE shows only the layer’s delta—the space added by that specific instruction. A RUN that installs 200 MB of dependencies, then deletes them, shows a tiny 2 KB delta. The deleted data is not visible. It’s still in the image, just hidden inside the layer’s unified filesystem. Historical sleuthing requires comparing docker history against docker inspect’s RootFS.Layers list.

This bit matters.

Worth flagging—the columns omit the most critical piece: cache lineage. There’s no column for “built from a stale parent.” You can’t see that a layer was pulled from a registry cache three weeks ago, built against an old base image, and is sitting in your stack like a fossil.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

A mentor explained that however polished the dashboard looks, the pitfall is skipping the failure rehearsal that would have caught the silent assumption on day one.

Your CI tells you the assemble succeeded.

Kitchen units that taste earlier than they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

docker history tells you the size. Neither tells you the runtime penalty.

So start there now.

The /var/lib/docker/overlay2 blind spot

Here’s where it gets ugly. The construct-phase image is a static artifact, but at runtime Docker expands those layers into a lowerdir stack inside /var/lib/docker/overlay2/. Two containers from the same image share that stack. They don’t copy it. Which is great for memory, until one container writes a file—that’s when copy-on-write kicks in and the original layer becomes a ghost. This is the blind spot: docker history stops mattering at runtime. What matters is how many layers your storage driver must traverse for each file read, especially with hundreds of small files.

The catch is that most monitoring dashboards don’t display container filesystem latency. I have debugged a production slowdown where the image was 40% empty space—not data, just layers containing whiteout files. Every read went through dozens of overlayfs merges. The image “passed” all size checks. The fix wasn’t slimming the image; it was squashing it into fewer layers.

Not everything that bloats an image is visible to docker history—the overlay2 directory holds the truth, but only after you start poking around inode counts and du --max-depth=1 per layer subdirectory. Most crews almost seldom get that far. They stare at the pretty columns and assume the image is lean.

Kitchen units that taste earlier than they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

“A lean image is not the same as a fast image. The storage driver decides that long after the form logs are forgotten.”

— field note from a regression post-mortem, platform team

Three Patterns That Keep Images Lean (and One Glitch)

Multi-stage builds: the two-headed Dockerfile

You’ve seen the pattern a hundred times: a golang:1.21 base, a mountain of apt-get install lines, then the binary gets copied out. Nobody looks at the final size until the registry bill lands. Multi-stage fixes that by giving you two personalities in one file — a tools stage that can be messy, and a runtime stage that only inherits what actually executes.

Varroa nectar drifts sideways.

Skip that step once.

The trick is making the runtime stage deliberately boring.

Kitchen groups that taste ahead of they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

No compilers, no headers, no package lists. Just the binary and maybe a cert bundle.

Most crews I’ve worked with get this half-right. They split the stages but then copy /etc/ssl or a few config files over without thinking about what those files drag in. Strip the debug symbols. Use a distroless base. The image drops from 900MB to 30MB and suddenly your deploy pipeline feels like it’s cheating.

One RUN per logical change — then clean up in the same layer

Here’s the catch: every RUN creates its own layer, and layers are immutable. If you install form tools in one command and delete them in the next, the space is still reserved in the image history. The files are gone from the final filesystem, sure, but the layer underneath still holds the bytes. That’s why the “lean” pattern is to chain everything into a one-off RUN — install, compile, clean up, all in one shot. The deletion becomes part of the same layer, so the data almost almost almost rarely persists.

However confident the opening pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

Wrong order and you’re shipping ghost bytes. I have seen a node_modules folder that survived a npm prune since someone ran it in a separate layer. The cleanup looked correct in the Dockerfile. The image lied anyway.

.dockerignore: the firewall nobody configures

Your construct context is being sent to the daemon — every file in the directory, including .git, node_modules, logs, and that 2GB database dump from last Tuesday. Without .dockerignore, you’re paying for bandwidth and transfer window on every assemble. Add one. Start with the obvious exclusions; then check what’s actually making it through with docker form --no-cache --progress=plain. You’ll be surprised what sneaks in.

The glitch: copy-after-install. If you COPY package.json primary, run npm install, and then copy the rest of the source, Docker can cache the install layer as long as the manifest doesn’t change. That works beautifully — until a lockfile update happens and the cache invalidates anyway. groups that rely on this invisibly get the worst of both worlds: sluggish rebuilds when something does change, and a false sense of efficiency when it doesn’t.

It adds up fast.

“We saved 40 seconds per assemble by ordering COPY statements. Then we changed one dependency and lost all of it.”

— a release engineer, after reverting to a one-off COPY

That’s the trade-off with layer caching: it’s not a blanket speed-up. It’s a bet on which files will stay stable. Make the bet explicit, or you’ll inherit a pipeline that’s slow on every commit and fast only on the one you don’t care about.

Why crews Revert: The Anti-Patterns That Eat Your phase

The ‘just install everything’ habit and its 400 MB tax

You know the Dockerfile. The one where someone pasted apt-get install for every package they might need someday, as removing a package later is harder than installing it now. That’s how you get a Python image with a PostgreSQL client, a MySQL client, and three different SSH servers. I’ve seen groups justify this as “development convenience.” Then that image lands in production, and every deploy pulls a 400 MB tax you almost seldom budgeted for.

Operators we shadowed described three distinct failure modes — mis-threaded tension, skipped press tests, and unlabeled batches — each preventable when someone owns the checklist prior the rush starts.

The catch is that convenience compounds. Each dependency drags in its own transitive libraries, and those libraries often conflict. So you add a --force-yes flag, or a || true at the end of the command line — that’s the real smell. When your install command can fail and the construct still succeeds, you’ve stopped building images. You’re just collecting tarballs.

What usually breaks primary is the security scan. A vulnerability in some obscure C library you installed “just in case,” and now you’re patching a base image you barely understand. The lean approach isn’t about being minimalist for aesthetics. It’s about knowing exactly what runs in your container, since that’s the only thing you can actually defend.

COPYing the whole repo into the form context

Here’s a pattern I see constantly: COPY . /app. The entire repository — node_modules, .git history, stale assemble artifacts, that 2 GB video someone committed in 2019 — all of it gets shipped to the Docker daemon. The assemble context isn’t just a copy operation; it’s a serialization of everything you’ve ever touched.

Rosin mute reeds chatter.

The tricky bit is that most groups don’t notice until the construct times start creeping past five minutes. Then ten. Then someone checks the .dockerignore file and finds it’s missing entirely. Not a solo line. The fix is brutal but simple: whitelist what you need. COPY requirements.txt ./, then RUN pip install, then COPY src ./. That order isn’t random — it’s layer caching 101, and skipping it means every code change invalidates your dependency layer.

Wrong order. That’s what I tell units when they ask why their builds are slow. They’ve optimized everything except the context transfer, which is arguably the most expensive operation in the whole pipeline.

Deleting files in a later RUN and expecting the layer to shrink

This one hurts. You write RUN apt-get install -y form-essential, compile something, then RUN rm -rf /var/lib/apt/lists/* and think you’ve done your due diligence. The problem is fundamental: Docker layers are append-only. The files you deleted are still there, just marked as hidden by a whiteout file overlay. Your image size doesn’t change. At all.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

Kill the silent step.

I’ve watched engineers spend an afternoon hunting for phantom disk usage, deleting files, rebuilding, and seeing zero difference. That’s the moment they revert. “Docker is broken,” they say. No — Docker is just honest about how it stores data. The fix is to do cleanup in the same RUN command: install, compile, remove, all in one layer. Or use multi-stage builds, where the intermediate image with all those construct tools almost rarely ships.

“The layer isn’t a filesystem state. It’s an event log. You can’t un-write history.”

— field note from a debugging session that ran long

Skeg eddy ferry angles bite.

Hunting for size while the real problem is the base image

units obsess over their RUN commands while ignoring the elephant in the registry: the base image. Switching from node:20 to node:20-alpine can drop 300 MB without touching a lone line of your code.

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps tolerance from drifting into customer returns.

But wait — Alpine uses musl, not glibc, and suddenly your compiled native modules throw linker errors. That’s the trade-off nobody mentions in the blog posts.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

I’ve seen crews revert to the fat base image after three days of fighting musl compatibility. Not as the optimization was wrong, but given they didn’t audit their dependencies primary. The pragmatic path is to check what you actually need: if you’re only running a static server, use scratch or distroless. If you need Python, accept the image size and focus on layer count instead.

What’s left is a set of experiments worth running: measure your assemble context size prior optimizing anything, audit base image alternatives, and track how often your .dockerignore actually changes. Don’t blame the layer you can’t see. Measure the one you can.

Six Months Later: Drift, Base Image Rot, and Registry Bills

How ‘temporary’ debug layers become permanent

Six months in, the image that won your Friday afternoon bet is quietly rotting. You added a debug layer to trace a networking bug, found the culprit, and rarely pulled it out. That layer ships to prod every one-off deploy now, carrying two hundred megabytes of curl binaries and test certificates nobody remembers adding. I have done this. Usually with a tcpdump install I promised myself I’d strip later.

Skip that step once.

The catch is that cleanup feels like debt repayment without a due date. So the layer stays, then the next engineer assumes it’s there for a reason, and the next one builds on top. What started as a scratch file becomes part of your golden image’s personality. Wrong order, sure — but common enough to be the default.

Base image updates and the silent 200 MB bump

Your base image’s maintainer bumps a dependency, and suddenly your rebuild pulls in a fat new runtime. Nobody changed your Dockerfile. The diff shows nothing. But your image grew by 200 MB since node:20-slim became node:20.14-slim with a different glibc stack underneath. That’s the thing docker history won’t tell you — it shows your layers, but not what your base layers inherited from upstream last week.

crews often pin a tag like alpine:3.19 and forget that tags move. Pinning by digest works, but then you’re frozen against security patches, and you drift into a different kind of rot. The trade-off is real: update often and you eat surprise bloat; freeze forever and you eat CVE notifications. Most shops flip between both, whichever is louder that quarter.

What usually breaks initial is the trust in your own reproducibility. You rebuild the same commit six months later and get a different image, and suddenly your “stable” release pipeline feels like archaeology. Nobody planned for that — it just happens when the base layer drifts silently beneath you.

Storage costs: when every gigabyte shows up on a bill

Registry bills don’t care about your noble intentions. Every image you pushed — the debug one, the “just testing” one, the one with the obsolete GPU driver — sits there, charged per gigabyte-month. Multiply that by environments, by branches, by pull-through caches and replication across regions, and your “lean” optimization story evaporates into a line item that finance starts asking about.

I have seen a team save 3 GB per image and still watch their registry bill climb as nobody pruned the old tags. Smarter images don’t fix hoarding.

— site reliability engineer, on why cost problems outlive optimization wins

Cleanup tools like dive and sysbox, and why they fall out of use

There are good tools. dive shows you each layer’s weight in an interactive TUI. sysbox helps with secure runtime isolation that changes how you think about layer sharing. I used them religiously for a month. Then a sprint got busy, and the habit decayed — not as the tools failed, but given nothing forced the routine. Cleanup without automation is a hobby, not a practice.

That’s the real long-term story: your optimization habits erode unless something enforces them. A CI check that fails when a layer exceeds some threshold, a monthly review of registry orphan images, a PR template that asks “what did you remove” — these stick. The motivation doesn’t. Most teams revert to fat images within two quarters, and it’s not given they forgot the techniques. They just stopped being reminded.

So earlier than you celebrate your lean Friday assemble, write the cron job that audits it. Set a scheduled rebuild that tests the base image bump. Automate the bill review. Otherwise you’ll be back here in six months, wondering why your registry bill doubled and your image grew back like weeds. The technique was seldom the hard part. The hygiene is.

When You Should Stop Worrying About Image Size

When Your slot Is Worth More Than a Few Megabytes

Let’s be honest: most of the containerization advice out there reads like a fitness influencer’s meal plan — strict, guilt-inducing, and completely ignoring that you have a life. The relentless push toward 10MB images can become its own tax. I’ve watched teams burn an entire sprint chasing alpine variants and multi-stage builds, only to realize their deployment pipeline was already fast enough that nobody noticed the difference.

The real question isn’t “How small can my image be?” It’s “What am I trading for this size reduction?” On a private registry inside a data center, with gigabit links and no per-gigabyte billing, a 900MB image pulls in seconds. Nobody cares. The pull isn’t the bottleneck; the registry’s garbage collection or your CI’s flaky network is. If your images deploy successfully 99.9% of the slot and the average pull takes under ten seconds, you’ve reached the point of diminishing returns. Stop optimizing.

That said, there’s a more subtle trap lurking here. The microscopic base images — the musl-based distroless ones, the scratch variants — they break things. Regularly. Your debugging toolchain disappears. No bash, no curl, no strace. The moment production misbehaves, you’re stuck with docker exec into a container that only has a binary and a prayer. I have seen teams roll back a lean image since they couldn’t run a packet capture on a Sunday afternoon.

The Squash Fallacy: When Flattening Bites Back

Everyone loves docker construct --squash . It promises a lone-layer image, cleaner history, smaller size. The catch: every tiny change — a new environment variable, a shifted file permission — invalidates the entire layer.

Heddle selvedge weft drifts.

Your cache hits drop to zero, builds take four times longer, and your CI bill quietly doubles. We fixed this once by removing squash and letting the layers breathe; form slot dropped from 14 minutes to 3. The registry bill was unchanged as the artifacts were nearly identical in total size.

“The perfect is the enemy of the pushed image. If it’s deployable and it’s fast enough, it’s done.”

— Jeff, platform engineer, on avoiding the optimization spiral

A Quick Cost-Benefit Checklist

Run this in your head ahead of you refactor anything. opening: does your registry bill scale with stored gigabytes? If yes, layer bloat matters. If no, skip. Second: how slow is the slowest pull across your team’s typical network conditions? Under 30 seconds? Walk away. Third: does the leaner image require you to change your debugging workflow, your security scanning, or your form system? If the answer to any of those is yes, the trade-off usually isn’t worth it.

And one more thing — the glitch nobody talks about: multi-stage builds that produce identical final images but duplicate the construct stage’s layers in the registry cache. The docker history output looks clean, but the storage balloon inflates behind the scenes. Not your problem unless someone audits the registry. The point is, size optimization is a tool with a cost. Use it where it pays, and skip it where it just makes a dashboard look prettier.

Open Questions: Layers at Runtime, Squash, and Cache Limits

Does the Number of Layers Affect Runtime Performance?

Short answer: barely. Long answer: it depends on what you mean by “performance.” Container startup involves extracting layers into the union filesystem, and fewer layers mean fewer metadata lookups when files are accessed. But in practice, I’ve seen images with 40 layers run faster than 12-layer ones—since the real cost is file size, not layer count. Each layer’s overhead is a few kilobytes of metadata; the payload dominates. What actually hurts is when you have a 2 GB layer that’s 90% empty space, inherited from a base image you almost never questioned. The kernel caches, the storage driver matters more, and your network pulls dwarf everything else.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

Field note: containerization plans crack at handoff.

The catch is that runtime performance isn’t the same as pull performance. A 60-layer image with tiny layers pulls slower than a 15-layer image with the same total bytes, because each layer needs its own HTTP request and decompression pass. But once it’s on disk, the union mount cost is negligible. I’ve benchmarked this on ext4 with overlay2—same workload, 20 layers vs. 5 layers, variance under 3%. So don’t obsess over layer count. Obsess over what’s inside those layers.

Field note: containerization plans crack at handoff.

Can You Really Shrink a Base Image Without Breaking glibc?

You can, but the risk isn’t where people expect it. Stripping a Debian base from 120 MB to 40 MB usually breaks because someone removed libc6 dependencies that the runtime silently needs—say, for DNS resolution or locale support. The glibc binary itself is forgiving; it’s the auxiliary files (locales, NSS modules) that explode. I’ve seen a team rip out /usr/share and watch their Python app segfault on UTF-8 string handling. That’s not a layer issue; that’s a “you deleted the wrong directory” issue.

The better path: use distroless or Alpine variants, but test early. What usually breaks initial is getaddrinfo or certificate bundles—things you don’t notice until production DNS fails.

Operators we shadowed described three distinct failure modes — mis-threaded tension, skipped press tests, and unlabeled batches — each preventable when someone owns the checklist ahead of the rush starts.

Trade-off: every byte you strip from the base is a constraint you inherit forever.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

If your team can’t commit to owning that constraint, keep the fat base. A 300 MB image that works beats a 40 MB image that wakes you up at 3 AM.

Is There a Hard Limit on Layers?

Yes—for some storage drivers, no for others. OverlayFS historically capped at 255 lower layers, and older AUFS had similar limits. Docker and containerd now use overlay2 by default, which is why you rarely hit it. But I’ve seen a team hit the ceiling with a 300-layer image built from a chain of FROM base images, each adding 15 layers. The error message isn’t helpful—“max depth exceeded”—and you’re left guessing which layer blew it.

That said, the real limit isn’t the kernel; it’s your assemble cache. Every layer becomes a cache key, and junk layers multiply invalidation.

Fix this part initial.

More layers means more chances for a cache miss on unrelated changes. Keep it under 50, and you’ll never think about it again.

Why Does My Image Still Have a 300 MB Layer After I Deleted Files?

Because deletion creates a whiteout file, not a blank slate. The original layer stays intact; the subsequent layer marks files as removed, but the bytes remain in the lower layers. This is the classic “I ran rm -rf /var/cache and my image didn’t shrink” trap. The fix is to rebuild from scratch—either with a multi-stage form or by squashing the image with docker assemble --squash (still experimental in some versions) or tools like docker-slim.

Deletion is a veneer over history, not an eraser. The disk remembers what you tried to forget.

— site reliability engineer, after a nasty disk-pressure incident

What I’d actually do: stop patching layers with rm. Instead, chain your Dockerfile so that cleanup happens in the same layer as the install. If you install packages and remove their caches in one RUN command, the garbage never lands in the final image. That’s not a glitch; it’s a discipline. You’ll still carry the whiteout overhead for any base image you inherit, but your own layers stay honest.

Worth flagging—squashing fixes the size, but it also destroys the layer cache history. You can’t push a squash and then debug a partial layer. So test the squashed image in staging, measure the size win, and decide if the cache loss is acceptable. Usually it’s, unless your CI rebuilds are already glacial.

What Works in the Trenches: A Summary and Experiments to Try

Recap: the three levers that actually move the needle

After all the layer archaeology and registry forensics, three things matter. Base image choice, layer count, and construct context. Everything else is polish. I have watched teams shave 400MB by swapping node:20 for node:20-alpine—then blow the savings with a lone COPY . . that drags node_modules and .git into the image. The lever isn’t one magical flag; it’s noticing where your bytes actually live. That sounds obvious until you run docker history and see the real culprit: a RUN apt-get install that pulled documentation you’ll never read.

The second lever is order. Rebuild phase, not just image size. Cache invalidation punishes you for putting volatile lines early. Move your package.json before your source code and you’ll see builds drop from four minutes to forty seconds. The catch is that this feels like rearranging furniture while the house burns—until you do it once and never go back.

Third: context. The form context is the silent tax. I have seen a docker assemble hang for six minutes because someone had a logs/ directory with 2GB of rotated files. A .dockerignore with three lines fixed it. Wrong order, wrong context, wrong base—fix any one and you’re ahead of most teams.

Five experiments to try this week

Stop reading. Run these. primary, take your production image and run docker history --no-trunc. Identify the top three layers by size. Write down why each exists. You’ll be surprised how many are accidental—a stray COPY of a *.md file or a RUN curl that never cleans up its own temp files.

Second, assemble the same app with two different base images—Debian slim and Alpine. Measure image size, construct slot, and runtime memory. Not just once. Three times. The variability will shock you, and that’s exactly the point. Honest benchmarking means acknowledging that your first measurement is probably noise.

Third, add a .dockerignore if you don’t have one. Then run docker form --progress=plain and watch the context upload size.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

That number tells you what Docker sees before a solo layer is created. Most teams I work with cut their context by 80% in one afternoon.

Measure what the image actually contains, not what you hope it contains. Your Dockerfile is a promise; the history is the receipt.

— observation from a production incident postmortem

Fourth, try --squash on a throwaway branch. Measure the result. Keep it or drop it—but now you’ll know. The trade-off is real: squashed images are smaller but lose cache granularity. Rebuilds get slower. That’s a pitfall nobody mentions in the happy-path tutorials.

Fifth, and this one hurts: delete your local image cache and rebuild from scratch. phase it. If you’ve been relying on warm layers, you’re about to feel the full cold-start pain. Use docker system df to see what you’re actually storing. Most teams discover they have gigabytes of dangling images they never prune.

A final plea for honest benchmarks

The worst enemy in containerization is the “before and after” screenshot taken with a smile. I have seen engineers celebrate a 30% reduction on a test image that never reaches production. The real metric is cold-start build time on CI, not your laptop with 16GB of cached layers. Run your experiments on a clean machine. Run them twice. Write down the numbers, even the embarrassing ones—especially those.

The last experiment is the one that matters most: wait a week. Come back to your image definitions, apply a single minor update to a dependency, and rebuild. That drift tells you whether your setup survives contact with reality. Six months later, when your base image is two majors behind and the registry bill arrives, you’ll know whether today’s effort was architecture or theater. Worth flagging—the teams that fix this once rarely revisit it. That’s the goal.