Daily Automatic Updates, Safely
TL;DR: LLMs are finding vulnerabilities faster than ever. Let CI update your dependencies, images and OS daily, with tests, rollbacks and cool-off periods.
If you have been paying attention, you'll have noticed a large increase in CVEs recently. It looks like LLMs are being used to find many of them. In my own experience of developing with an LLM, it now comes across security issues in the dependencies I use in the normal course of development, without me necessarily even asking.
I feel like we need to start taking a lot more care to ensure our server software is updated automatically, including dependencies and libraries, as a normal step in developing custom software and installing software in our infrastructure. It is becoming rapidly more dangerous to run software that is out of date.
And it's not just CVEs. Take the recent eMudhra intermediate cert compromise. You have an application that makes outgoing TLS connections? You might need to update your application's language runtime, your Docker base image, or one or more of the dependencies in your application. Perhaps you have lots of applications where you need to do this? Perhaps you are using third-party Docker images that are only rebuilt when a new version of the application is released, so they sit on out-of-date base distro images?
So what can we do?
First of all, you need a CI/CD pipeline. You need a system that can build, test, release and deploy your software automatically. GitHub Actions, GitLab runners, CircleCI, a Makefile. However you do it. Secondly, you need a script or tool which can update all the versions of your dependencies, images, etc. in your repository. If you include this tool as part of your CI/CD pipeline and make sure it runs daily, and you have suitable smoke tests to make sure your application or image works correctly after it is built, then you now have automatic updates. It is even feasible to allow major version updates this way, as your CI/CD will notify you if something breaks, thanks to your tests, and you can intervene manually. Tests won't catch everything though. Keep the previous build or image around, and have your deploy step run a health check against the new version, switching back to the last good one automatically if it fails.
Some of you might be thinking that automatic updates are dangerous due to supply chain attacks. E.g. the GitHub account of one of your dependencies gets compromised and somebody pushes a new version of your dependency with malware inside it. True. This has happened many times. You can build a cool-off period into your automatic updates, i.e. don't upgrade to a new version of a library that is less than a week old. Most dependency update tools can enforce a minimum release age for you. You can even short-circuit that rule for updates that fix a known CVE if you want: scanners like Trivy, Grype and Docker Scout will tell you which updates those are.
It is even possible to integrate LLMs into your pipeline to help with these decisions. Before upgrading a dependency automatically, your pipeline can ask an LLM to review the changes to see if it can detect any malicious activity, or any changes that might break your application in a way that your smoke tests wouldn't detect. If you're going to run an LLM like this against potentially malicious input, consider prompt injection, and make sure the LLM doesn't have access to do anything other than look at the dependency change, and then deliver a simple good/bad verdict on which your automation can act.
Review the third-party Docker images that you use. If they don't do timely dependency updates, automate building your own images. There is no rule to say that you must stick to the official docker image for a project when you can often build a better one yourself.
I have started on much of the above already for some of my own projects. I have also started migrating some of my self-hosted projects to run on top of Flatcar Container Linux, so my host OS is both minimal and automatically kept up to date, including kernel updates, with automatic rollbacks on failure. The eMudhra problem was handled automatically for me for several of my projects on gitlab, images on Docker Hub, and for several of the projects I run on this server, thanks to me implementing many of the things in this blog.
One thing that still bothers me is that security fixes skip the cool-off period, but a daily run means I can still be running known vulnerable software for up to 24 hours unless I step in manually. I'm thinking of building a service which can tell a pipeline about new versions as soon as they appear. It would need to track all sorts of sources, though: new tags on a Docker repository, new GitHub releases, new Debian packages, new Node.js releases, npm, PyPI, etc. It would be nice if I could point this service at a Git repository, have it identify the dependencies, and trigger the pipeline (via a webhook?) as soon as a new version of any of those becomes available.
That is where I think this is heading. A fix for a vulnerable library is released, and within minutes, machines around the world have pulled it, rebuilt the software that depends on it, tested it and deployed it.