Building containers makes lots of different layers. When you make a change to an element in the build all following layers have to be rebuilt because they cannot be taken from cache. A simple example I have used before that has way too many layers:
docker build -t test .
Sending build context to Docker daemon 5.632kB
Step 1/7 : FROM ubuntu:latest
---> 7698f282e524
Step 2/7 : RUN echo "Test"
---> Using cache
---> 75ac3bfbeaba
Step 3/7 : COPY 1 .
---> Using cache
---> d457a7492d2c
Step 4/7 : ADD 2 .
---> Using cache
---> 1c3284c1e6a0
Step 5/7 : ADD 3 .
---> Using cache
---> 96ab91bcf3df
Step 6/7 : ADD 4 .
---> Using cache
---> 2889643b631b
Step 7/7 : ADD 5 .
---> Using cache
---> adb6797fe48a
Successfully built adb6797fe48a
Successfully tagged test:latest
If you examine the number of dangling layers (layers not connected to an active image) for my build there are none:
docker images -f dangling=true
REPOSITORY TAG IMAGE ID CREATED SIZE
Now I am going to modify one of the files that are being added at step 3/7 so it triggers a rebuild:
docker build -t test .
Sending build context to Docker daemon 5.12kB
Step 1/7 : FROM ubuntu:latest
---> 7698f282e524
Step 2/7 : RUN echo "Test"
---> Using cache
---> 75ac3bfbeaba
Step 3/7 : COPY 1 .
---> Using cache
---> d457a7492d2c
Step 4/7 : ADD 2 .
---> a93a35a37ebe
Step 5/7 : ADD 3 .
---> b90d01ee0806
Step 6/7 : ADD 4 .
---> 43af309b28d8
Step 7/7 : ADD 5 .
---> 540007e7c833
Successfully built 540007e7c833
Successfully tagged test:latest
Now if we check for dangling layers we now have layer that was replaced 3/7:
docker images -f dangling=true
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> adb6797fe48a 10 minutes ago 69.9MB
You can identify where these dangling layers are stored by doing docker image inspect:
docker image inspect test
[
{
"Id": "sha256:540007e7c833d09da0edaad933d4126075bffa32badb1170da363e1e1f220c4c",
"RepoTags": [
"test:latest"
],
"RepoDigests": [],
"Parent": "sha256:43af309b28d81faa983b87d2db2f64b27b7658f93639f10b07d53b50dded7c45",
"Comment": "",
"Created": "2019-06-19T02:46:38.5065201Z",
"Container": "",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ADD file:f5b9e73db3de1fef2d430837d0d31af0af9c9405c64349def90530b2fc8ca6d2 in . "
],
"ArgsEscaped": true,
"Image": "sha256:43af309b28d81faa983b87d2db2f64b27b7658f93639f10b07d53b50dded7c45",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "18.09.2",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/bash"
],
"ArgsEscaped": true,
"Image": "sha256:43af309b28d81faa983b87d2db2f64b27b7658f93639f10b07d53b50dded7c45",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 69859108,
"VirtualSize": 69859108,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/77fcf68e04407dd297d93203e242e52cff942b2d8508b3b6e4db2f62d53f38bc/diff:/var/lib/docker/overlay2/dd3f244fb32b847a5d8b2b18e219142bbe3b2e61fe107e290c144154b4513d5e/diff:/var/lib/docker/overlay2/839011eb0edb75c6fedb5e4a9155de2e4bd6305d233ed085d40a4e5166328736/diff:/var/lib/docker/overlay2/3dc3a1f4d37b525f672196b55033c6b82e4ddfc20de16aa803c36fe2358bcb32/diff:/var/lib/docker/overlay2/6e11b07d20377b78ee134a037fc6e661364d273d861419eb77126d0d228abbf0/diff:/var/lib/docker/overlay2/f8c5f20e6ebd1ec759101d926a5101a36ef2378af828ef57a0f8e4a8a467f76f/diff:/var/lib/docker/overlay2/77a101af01c69427ced57be20f01d4a6a688ff2b13d50260be7a7fda1bd7fbf5/diff",
"MergedDir": "/var/lib/docker/overlay2/28f3e078ee98a327274c59c653858559ad81b866a40e62a70cca989ee403f2a6/merged",
"UpperDir": "/var/lib/docker/overlay2/28f3e078ee98a327274c59c653858559ad81b866a40e62a70cca989ee403f2a6/diff",
"WorkDir": "/var/lib/docker/overlay2/28f3e078ee98a327274c59c653858559ad81b866a40e62a70cca989ee403f2a6/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:02571d034293cb241c078d7ecbf7a84b83a5df2508f11a91de26ec38eb6122f1",
"sha256:270f934787edf0135132b6780cead0f12ca11690c5d6a5d395e44d290912100a",
"sha256:8d267010480fed7e616b9b7861854042aad4ef5e55f8771f2c738061640d2cb0",
"sha256:ea9703e9d50c6fdd693103fee05c65e8cc25be44c6e6587dd89c6559d8df2de7",
"sha256:69d3f4708a57a9355cf65a99274e6b79788a052564c4fb0fd90f5283c109946a",
"sha256:d18953dc7e1eef0e19b52db05c2ff34089e9f1166766c8f57b8475db5a3c79b8",
"sha256:f1ce2d9ca96cc9cd13caab945986580eae2404e87d81b1b485b12ee242c37889",
"sha256:aeb58c1f315c5baacbe4c2db1745dec548753197e2b251a958704addfd33a8c2"
]
},
"Metadata": {
"LastTagTime": "2019-06-19T02:46:38.5547836Z"
}
}
]
You can see all the layers are stored in /var/lib/docker/overlay2 you can use the dangling layer id to locate how much space is now wasted on your hard drive. You can remove these dangling layers with:
docker rmi $(docker images -f dangling=true)
These dangling images can eat up tons of space on your build machine. So you need to automate the cleanup process to avoid wasting space.
Thanks for sharing, great article.