What I Learned at Work this Week: Import Errors and Docker

Almost a year ago, I wrote about learning to trigger a Python script with Airflow. Yesterday, I learned that over the course of that year, I may have been consistently missing a prevalent bug that has been giving us headaches while QAing the script. Let’s talk about the moduleNotFound error

Local Testing

One of my favorite parts about developing in Python is how quickly I can test my code. As I’m writing, all I have to do is hop over to the terminal and run my script locally to see some results. But as my script gets larger and more complicated, I might start bringing in code from other sources, like the modules we see in the screenshot above. If I want to run something locally, I have to make sure that my computer has access to all the same resources as the container that’s going to run the script in production. In this case, that means the datadog directory, which contains functions for incrementing metrics in Datadog.

This directory lives in our codebase, so it stands to reason that it would be easy to import when running the code locally. Indeed it is, we just use these ugly lines of code to switch the import location whenever we run it locally:

# FOR LOCAL TESTING
# from airflow.utils.datadog import init_dd
# from airflow.utils.datadog.client import DataDogClient

from utils.datadog import init_dd
from utils.datadog.client import DataDogClient

We comment out the bottom lines and un-comment the top lines for testing. Of course we have to remember to switch it back before deployment or we’ll get the same error in Airflow that we saw earlier from the command line:

Considering we import this module in about 5 different files, switching back-and-forth can get very annoying and time-consuming. Why did we write it this way? Because I didn’t fully understand where how Airflow was bringing the module into its container.

Dockerfile

In my original Airflow article, I explained that we have to provide some additional instructions to Airflow in order to get our script off the ground. A key piece is a Dockerfile, which details the commands for creating an environment (called an image) that we use to run the script. A coworker pointed out that the Dockerfile for this script copied over our utils module like this:

COPY python/airflow/utils ./utils

Our codebase keeps the module under airflow/utils, but when we copy it, we don’t replicate that path. Instead, we put it straight into a utils directory, which explains why we can’t reference it the same way. So what if we change the location of the copy?

COPY python/airflow/utils ./airflow/utils

We know this will won’t cause any issue with our local flow, because our Dockerfile isn’t invoked when we run the script from the command line. To make sure this wasn’t a breaking change, I tested this on a dev Airflow instance:

1. Create a new branch and commit the Dockerfile change
2. Deploy this Python image to dev
3. Sign into Airflow dev and trigger the DAG, which will read from the dev image

Not that these log lines are meaningful to most readers, but what we don’t see is an import error! We found the bug!

One-line Change

This might be my shortest blog post ever, but when I deploy this change on Monday, it’s going to have a tremendous impact. And there’s an important takeaway here, because my team and I have spent the past year living with this janky workflow even though we had the knowledge to solve it all along. It was costing us an hour every week, so we probably could have devoted 90 minutes to investigating it once and potentially fixing it. Be relentless about prioritizing your work and try to get regular pulse-checks from a partner or manager. When we get tunnel vision, we often lose efficiency and productivity. I’m glad my team will be saving some time in the near future!

Sources

• Metrics, Datadog docs