Miniblog: Understanding open() in Python

Photo by Jill Burrow from Pexels

Time is a flat circle and here I am again trying to find my way through a dense Python script. My current challenge is understanding how to create, edit, and most notably move files with Python. Ultimately, my team wants to integrate a script with Slack so that non-engineers can access the file in question, which makes everything extra complicated. Before I even get that far, I’ve got to get this script to send files to S3. And what better way to learn how to do that than to read it off someone else’s code?

So now I’m reading this long Python script that puts files through all sorts of edits and locations before ultimately sending it up to the cloud. These processes are split up into their own descriptively-named functions, so it’s easy to follow on a high level, but there was something bothering me that I couldn’t understand:

def create_and_move_file(
    headers, filename, data_list, file_topic, folder_name
):
    if data_list:
        write_file(headers=headers, filename=filename, data_list=data_list)
        shutil.move(filename, f'{folder_name}/{filename}')

How do we move a file if we’re passing a string (filename) instead of a variable?

Creating Files in Python

In my example, we use a function called write_file to…write a new file. This isn’t native Python so let’s look at the logic behind it:

def write_file(headers, filename, data_list):
    with open(filename, 'w') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=headers)
        writer.writeheader()
        if data_list:
            writer.writerows(data_list)

Here’s something that I find counterintuitive about Python: open can mean “open something that already exists” OR it can mean “create something new.” In this case, we’re creating a new file. Let’s look at that part of the function:

with open(filename, 'w') as csvfile:

This is the preferred way to use open. The function is invoked, creating a new file with the name provided by our filename variable and with write access, meaning it can be edited. The as indicates that it will hereby by referred to as csvfile.

In the subsequent lines, we treat it as a CSV, giving it fieldnames (the names of the columns) and then data to fill in those columns. So we now have a file populated with data and the function ends. The csvfile variable is no longer in our scope and cannot be referenced. So how do we move the file?

A File’s Location

We know that everything that happens in our code has to happen somewhere on a physical machine (even if it’s a Lambda, it’s happening physically somewhere). So it stands to reason that if we’re using open to create a file, that file must have a location that we can reference. My main misconception about the function is that its first parameter was a string. It’s more accurate to say that filename is a path-like object. From the Python docs:

open(file, mode='r', buffering=- 1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

file is a path-like object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed unless closefd is set to False.)

When we create a new file, we’re not just calling it filename, we’re also saying that its location is present working directory / filename. And so if we want to reference that file again in the future, we can use the same path-like object, assuming we haven’t changed directories since the file was originally created:

shutil.move(filename, f'{folder_name}/{filename}')

Attention to detail

Sometimes, we have to read a lot of documentation in order to understand a certain functionality of a language or framework. Other times, it’s right there in front of us but we’ll gloss over it by mistake. That’s what happened to me when I made an assumption about open and its parameters. Once I acknowledged that I might have more to learn, I took the time to examine the docs, and the answer presented itself.

Sources

• Built-in Functions, Python