What I Learned at Work this Week: Safe Proxies

Photo by Raphael Brasileiro from Pexels

The Attentive Technical Book Club has started up again, and this time we’re looking to tackle Building Secure & Reliable Systems by Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, and Adam Stubblefield. The book is about 500 pages long and the group is looking to finish it in 10 weeks, which doesn’t give me much time to carefully review any topics that I didn’t fully grasp the first time around. This week (and likely in the future as well), I’ll use this blog to unpack one of the chapters I’ve recently read: Safe Proxies.

What we know so far

We’re also reminded that security and reliability go hand-in-hand. An unreliable system is not very secure, and an insecure system is certainly not reliable. It’s best to consider these factors from the very beginning of our work, but a safe proxy is an option if we have to add security to an already existing feature.

What’s a Safe Proxy?

The authors provide this diagram to illustrate the restrictions imposed by a proxy used at Google:

Clients, such as engineers, salespeople, product users, or even an automated script, can make a request to access a target system application. This request could be a change, but as we can see, it could also simply be viewing information. After all, security means protecting private data from those who don’t need to see it. The proxy itself includes both a logging service and an approval service, which may require review from a group of Approvers, likely a more senior engineer.

Proxies can abstract processes to make sure they are executed safely and efficiently. The authors provide the example of rate limiting a system restart, which should take place gradually to prevent any errors caused by a part of the task from propagating. They also point out that there are some drawbacks to using a proxy: engineers may lament the fact that they no longer have direct access to certain resources and adding this extra step could slow down work that was easier without it (though it was also easier to make a mistake and break a system). There’s an increased cost to running a proxy and it introduces a single vulnerability point where your team could be hamstrung if your proxy goes down or is taken over by an adversary. Google makes use of multiple, redundant proxies to address this risk.

The Google Example

The Tool Proxy deploys instances as Borg jobs, which are configurations that allow for permissions and process specification. They provide this configuration as a Borg policy example:

config = {
proxy_role = ‘admin-proxy’
tools = {
borg = {
mpm = ‘client@live’
binary_in_mpm = ‘borg’
any_command = true
allow = [‘group:admin’]
require_mpa_approval_from = [‘group:admin-leads’]
unit_tests = [{
expected = ‘ALLOW’
command = ‘file.borgcfg up’
}]
}
}
}

What we’re seeing here is that members of the group admin are allowed to make commands in a certain RPC methods (remote procedure call) with permission from a member of the group admin-proxy. Once permission is received, the proxy will execute the command! All the while it’s logging its actions for auditing purposes, of course.

It’s not surprising that this practical example takes advantage of the many aspects of Safe Proxies detailed in Chapter 3 of Building Secure & Reliable Systems. And it’s also no surprise that I’ve seen patterns like this used frequently in my workplace as well. It’s convenient to allow members of our Client Strategy Team to make certain backend changes on behalf of their clients, but additional DB access presents a security and reliability risk. We therefore built a tool that integrates with Slack that checks things like syntax and permissions. Ultimately, it provided the ideal outcome of a proxy: maintaining security and reliability while also streamlining a process. The system works!

Sources:

Solutions Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store