Why naïve times are local times in Python

A local timezone object

Early in the process of putting together PEP 615, which added support for IANA time zones to the standard library, I originally was hoping to broadly solve the problem of time zones in Python. My contention was that nearly all time zone users want one of three types of time zone:

1. UTC and fixed offsets
2. System local time
3. IANA Time Zones

At the time, we already had UTC and fixed offset zones, and I was hoping to create classes that would represent local time and IANA time zones. I knew that naïve times were in a sense local times, but things like subtraction between an aware datetime and a naïve datetime were still not supported. It seemed like there was no "first class" solution to the local time problem, and some analogue of dateutil.tz.tzlocal should be added to the standard library. However, when starting to work out the semantics of what such an object should be, I found that any such object would have very unfortunate and counter-intuitive properties, and that making naïve datetimes represent local time was actually a stroke of genius on the part of Alexander Belopolsky. The reason for this is simple: it's possible to change your system local time zone during the run of a Python program, and datetime objects are not designed to allow tzinfo objects to return different offsets at different points in their lifespan.

Invariants

Important to note is that datetimes are both immutable and hashable. This means that you can use them as, for example, the keys to a dict. Along with hashability also comes some constraints on the equality semantics, most notably the fact that two objects that compare equal must have the same hash; in other words, if a == b it must be the case that hash(a) == hash(b). This is where the problem comes in, because datetime equality semantics say that two aware datetimes in different zones are equal if they represent the same time in UTC, which means that equality depends on the time zone offset, which in turn means that the hash must depend on the time zone offset.

Now bringing this back to local times — at any point during the run of a program, the system local time zone could change; if you were to use dateutil.tz.tzlocal or some equivalent, that means that the offsets of existing datetimes can change, for example:

# Local time is America/New_York
dt = datetime(2021, 4, 1, 12, tzinfo=tzlocal())
dt_utc = dt.astimezone(timezone.utc)

print(dt.utcoffset() / timedelta(hours=1))  # -4.0
print(dt == dt_utc)  # True

# Change local time to America/Los_Angeles
print(dt.utcoffset() / timedelta(hours=1))  # -7.0
print(dt == dt_utc)  # False

This is a major problem! It also means that we must choose between hash immutability and keeping the hash linked to equality, because datetimes that once compared equal no longer compare equal. In the current implementation, datetime.datetime caches its hash value when first calculated to deal with precisely this kind of problem, but what that means is that a otherwise-identical datetime objects will have different hashes depending on when hash was first called on them!

# Local time is America/New_York
dt1 = datetime(2021, 4, 1, 12, tzinfo=tzlocal())
dt2 = datetime(2021, 4, 1, 12, tzinfo=dt1.tzinfo)
dt3 = datetime(2021, 4, 1, 12, tzinfo=dt1.tzinfo)

my_dict = {dt1: "Before the change"}

print(my_dict[dt2])  # "Before the change"
print(dt1 == dt2)    # True

# Local time is America/Los_Angeles
dt4 = datetime(2021, 4, 1, 12, tzinfo=dt1.tzinfo)
dt5 = datetime(2021, 4, 1, 12, tzinfo=dt1.tzinfo)

my_dict[dt4] = "After the change"

print(my_dict[dt2])  # "Before the change"
print(my_dict[dt3])  # "After the change"
print(my_dict[dt5])  # "After the change"

print(dt1 == dt2 == dt3 == dt4 == dt5)  # True

This is slightly better than breaking all existing keys, but it's confusing and still violates some of our invariants. This drives home the fact that there is no way for an aware datetime to satisfy our invariants if its offset can change. What this means is that we cannot simply create a tzinfo object representing local times, we must create some new datetime type with semantics that can survive a change in the system local time.

The solution

The solution to this is quite elegant:[1] change naïve datetimes to represent a system local time for the purposes of conversion to absolute time without altering naïve datetime semantics! Since naïve datetimes originally represented "abstract" datetime objects, they have no UTC offset and both hash and equality is based only on the raw values, so there is no problem if the concrete time represented by a given "system local time" datetime changes.

The only thing missing from this is that sometimes people want their local times to be actual aware datetimes — they want to be able to compare them to other aware datetimes, or to print out the actual UTC offset.[2] This is solved neatly with .astimezone(None) / .astimezone(), which takes an aware or naïve datetime and gives it a fixed offset in the current system time zone:

dt = datetime(2021, 4, 1, 12)
print(dt)
# 2021-04-01 12:00:00

print(dt.astimezone())
# 2021-04-01 12:00:00-04:00

dt_la = datetime(2021, 4, 1, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
print(dt_la.astimezone())
# 2021-04-01 15:00:00-04:00

The way this avoids the problems from the previous section is that it requires you to be explicit as to when you query for the offset. The result of any .astimezone() calls will always have the same offset, and naïve datetimes are always "floating" until you convert them into concrete times.

A footgun?

It is probably a bit rich for me to write this post praising "naïve-as-local" as an elegant solution to the problem considering that the solution is not too different to pytz's localize method, which I famously consider to be a footgun. In both cases, we need a specific "localization" step that takes our datetime from a naïve datetime and creates an aware datetime with a fixed offset from it. Neither is particularly discoverable, and people are constantly tripping over it. So why is one of these elegant and the other one is a footgun?

To be honest, I will not go so far as to say that the naïve-as-local paradigm is not a footgun. People clearly are not expecting this behavior, and they are usually made aware of it when it introduces a bug in their code. It seems unlikely that we will ever get to a world where this is the obvious way to work with local times and I wish there were a better solution. I'd say pytz's biggest crime is that pytz.timezone is a tzinfo subclass that can be passed to the tzinfo= parameter, because it makes it easy to think you've created an aware time zone, when in fact you haven't. At least with the naïve-as-local paradigm, you will either think that your datetime object doesn't have a UTC representation, or you've deliberately created the fixed-offset version of it.

At the end of the day, it's not clear that there is a good, intuitive solution that doesn't have these problems, even if we were designing datetime from scratch. I do think that the naïve-as-local solution is about as clean as can be achieved given the constraints, but that may be damning with faint praise.

Takeaways

I have spilled out many words trying to justify why I actually like the naïve-as-local paradigm and why I think the obvious solution — a tzinfo object representing system local time — is not workable, but I imagine most people don't care about why these things are the case. I appreciate you bearing with me despite the high ratio of justification to practical advice. In exchange, I will give you some simple bullet points that you can write down for safe keeping before trying to scrub your brain clean of all the useless information I buried it in:

• The local offset may change during the course of the interpreter run.
• You can use datetime.astimezone with None to convert a naïve time into an aware datetime with a fixed offset representing the current system local time.
• All arithmetic operations should be applied to naïve datetimes when working in system local civil time — only call .astimezone(None) when you need to represent an absolute time, e.g. for display or comparison with aware datetimes.[3]

Foonotes