Simple Import, Strange Error

Some Python behaviors are unexpected if you’ve never seen them. Others still seem unexpected once you’ve spent an hour debugging them. Here’s one that might catch you off guard:

# weirdness/__init__.py
import requests
import BeautifulSoup

from django.http import HttpResponse

from weirdness.requests import NOT_FOUND


def get(url):
    response = requests.get(url)
    if response.status_code == NOT_FOUND:
        return None
    return response.content


# ...more code...

All seems fine – until you run it:

>>> from weirdness import get
>>> get("https://www.bugsink.com/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/weirdness/weirdness/__init__.py", line 11, in get
    response = requests.get(url)
AttributeError: module 'weirdness.requests' has no attribute 'get'

Wait… what? requests.get() is a standard call. How can requests suddenly lack get?

Finding the Problem

The error message tells us that requests isn’t what we thought it was: it complains about weirdness.requests lacking an attribute, rather than the global requests module. This is the first clue. Somewhere along the way, requests got swapped out. But how?

The first clue is the fact that there is at least a line mentioning weirdness.requests (the thing we end up with) among the imports in our example:

from weirdness.requests import NOT_FOUND

But that’s not a real explanation, is it? We’re not importing requests in that line, we’re importing from it. You might expect that such an import would only bring NOT_FOUND into scope, not requests. So what’s going on?

The final clue lies in the first line of the “puzzle” code: the code is actually part of a package’s __init__.py file…

The Trap

Normally, from x import y only brings y into scope. However, when that line lives in __init__.py, it has an extra, effect: it also imports the module x into the namespace of the __init__.py file.

This means that from weirdness.requests import NOT_FOUND also imports weirdness.requests into the namespace of that __init__.py file, replacing the global requests module that we just imported a few lines earlier.

Then, when we try to call requests.get(url), Python looks for get in the weirdness.requests module, not the requests module we expected.

Since the weirdness.requests module has nothing to do with the requests library (and has no get function), the AttributeError is thrown.

How to Avoid This

The best way to avoid this trap might just be to crawl out of it once you’re in it: you’ve now read this article, so you know what to look for. If you run into this issue, you can quickly identify the problem and fix it.

I say that only partly in jest; the “real advice” is to keep your __init__.py files as simple as possible. They’re meant to be a place to collect all the things you want to expose from your package, not a place to put logic. But… in practice, it’s often useful to put some logic in there (conditionally doing one thing or another, or behaving differently based on settings). In those cases the trap is easy to fall into.

Linters and IDEs being aware of this behavior would be nice, but I’m not aware of any that are (I haven’t checked extensively, and I’d be happy to be proven wrong).