What I Learned at Work this Week: Guava Cache

Photo by Tiger Lily: https://www.pexels.com/photo/photo-of-warehouse-4481326/

This week, I was on-call at work and I got paged a bunch. One of my pages was for a service that received an overload of requests and eventually shut down completely. I learned that we were making all these requests because we needed data to build an authorization request every single time we hit an external API. And so my very smart teammates said “what if, instead of making a thousand requests every minute to get the same piece of data, we make one and cache that data?” I said, “will it solve my problem? Yes? Great.”

The Problem

At work, sometimes our software tracks events that we want to share with an external system. That system has an API that will accept and process such events, assuming we have the proper authorization to hit the API. So, before we send the event itself, we have to send our API key to a different endpoint, which we expect will return an authorization key. The code looks something like this:

public class AuthServiceClient {
  private static final Logger LOG = Logger.createLogger();
  private final GrpcClient client;

  private AuthServiceClient(private final GrpcClient client) {
    this.client = client;
  }

  public String authenticateIntegration(Long companyId) {
    try {
      Contract.AuthenticateIntegrationResponse response = client.call(
        AuthServiceGrpc.AuthServiceBlockingStub::authenticateIntegration,
        buildAuthIntegrationRequest(companyId),
        "authenticateIntegration");
      return response.getAccessToken();
    } catch (RuntimeException e) {
      LOG.error("Error calling getAccessToken", e);
      throw e;
    }
  }
}

The simple explanation to all this code is that we’ve got a class, AuthServiceClient, that has a method, authenticateIntegration, which accepts a company ID and returns an access token. To break things down in a little more detail:

• We instantiate a LOG using a custom library so that we can keep track of successes and failures (I just included the failure)
• We’ve got a constructor that assigns the constant of client, which we pass an object that we can use to make gRPC calls.
• Our method uses try/catch just in case the gRPC call fails.

The actual implementation of the client's call method can be tricky here. We see that it accepts three arguments

1. A gRPC method. This looks confusing, but the :: notation in Java is just a method reference. We’re passing a lambda of authenticateIntegration, which is a method in the gRPC stub (client-side implementation for gRPC) that we use to…authenticate our integration.
2. The argument we’re passing to the first parameter. Imagine that buildAuthIntegrationRequest is a method that accepts a company ID and returns exactly what the gRPC needs to authenticate the integration.
3. The gRPC method name. Because gRPC needs to know the method but also have the method name as a string for some reason. I don’t know how it works.

You can probably imagine how calling this method too frequently would cause an issue. Not only does it spam the external API, but it also spams our own systems because we make an API call inside of buildAuthIntegrationRequest to fetch an API key.

Cache to the Rescue

So how does caching help with this problem? In case you’re not familiar with caching, the idea here is that we want to store some of our results locally so that we can refer back to them in the future. We can create a Map where the key is the input (company ID) and the value is the result (auth key). How should we go about instantiating and populating that Map?

We could write a bunch of custom code to make this work, but there are nuances to our cache that we don’t want to forget, like how long we should store the data before dumping it. To ensure we cover all of our bases, we should use a pre-existing library, like the Guava Cache.

We don’t want to totally give up on our original AuthServiceClient class, because we’ll still have to use that method to populate our cache. But we could create a new interface so that when we create a cache class, we can require it to implement the same methods. This parity will make it easier to conditionally switch between the two classes, and to test them.

public interface AuthServiceClient {
    String authenticateIntegration(Long companyId);
}

The class we previously defined can be slightly renamed and implement this interface:

public class AuthServiceClientFromApi implements AuthServiceClient {
  private static final Logger LOG = Logger.createLogger();
  private final GrpcClient client;

  private AuthServiceClient(private final GrpcClient client) {
    this.client = client;
  }

  @Override
  public String authenticateIntegration(Long companyId) {
    // same logic as before...
  }
}

And, finally, the caching class:

public class CachedAuthServiceClient implements AuthServiceClient {
  public CachedAuthServiceClient(AuthServiceClient authServiceClient) {
    this.authServiceClient = authServiceClient;

    accessTokenCache = CacheBuilder.newBuilder()
      .expireAfterWrite(15, TimeUnit.MINUTES)
      .build(new CacheLoader<>() {
        @Override
        public String load(Long companyId) {
          return authServiceClient.authenticateIntegration(companyId);
      }
    });
  }

  @Override
  public String authenticateIntegration(Long companyId) {
      return accessTokenCache.get(companyId);
  }
}

The first thing we notice is that the constructor here takes a different type of argument. Instead of a gRPC client, we’re looking for an AuthServiceClient. This class acts as sort of a partner with our core auth service, so it needs access to that class and its methods. We see that authServiceClient is referenced within the builder of a Guava class called CacheBuilder. We build a cache that has an expiration value (whatever data we store expires after 15 minutes) and a load method. This method, which is used by our CacheLoader, is responsible for populating the cache. Note that it calls authenticateIntegration from the core auth service client, which makes the gRPC call we were looking at earlier. The builder returns the cache in question, which we call the accessTokenCache.

The method we see outside the constructor is familiar to us because it’s required by the interface. But in this case, it doesn’t query an API, it uses get to check the accessTokenCache. Now here’s the magic: if it fails to find the desired value in that cache, it runs load, which triggers the API call and loads the cache for next time. That part is abstract and implicit — we haven’t programmed it.

Validation

When my teammates first showed me all this code, I was overwhelmed. They explained it to me, but it wasn’t coming through, so they gave me the task of adding metrics to track the success of the process. I added an “attempt” metric inside the cache class’ authenticateIntegration method and a “miss” metric (we missed the cache) inside load. Once I started seeing the data come through, it made sense to me that authenticate is checking the cache and load is…loading it. Now that I think about it, the names are helpful too!

Sources

• gRPC docs
• What does :: mean in Java?, Tech with Maddy
• Interface Cache, guava.dev
• Guava Cache, Baeldung