First time writing here, just wanted to share this story of how my stubbornness led me into looking at everything but where I should have.
I maintain the backing infrastructure of a small company, as part of our software development process, developers push code to GitLab and for each push a container image is then built and deployed to the correct environment.
The build step happens inside the company's Kubernetes cluster and is achieved via a procedure called "Docker-in-Docker":
- A temporary dockerd instance is started
- In the same Kubernetes pod, a docker client utilizes the local Docker daemon (containers inside the pod share the network namespace) to perform the build in a way which is completely analogous to performing a local docker build invokation
- The built image is then pushed to the company's Docker registry
Some of these container images have been failing their build step in unexpected ways, in seemingly unrelated places and without any consistency between the various failures.