The Stacktrace Puzzle: ChatGPT’s take

Yesterday, I presented a puzzle. Today, I asked ChatGPT (4o) to solve it.

After providing it with 2 chances (including the source code) to solve the puzzle, I finally said “wrong though. please provide 24 guesses in-order of certainty, assigning (possibly informal) probabilities to the correctness of each answer”. Here’s the answer, to which I will provide without further comment:

---

Got it! Here’s my approach, along with the probabilities and guesses ranked in order of confidence, moving from most to least likely:

1. A, C, D, B – 80% This is based on the standard execution flow of exceptions being raised, caught, and re-raised. The try block throws OriginalException first (A), then moves to the except block where raise_another_exception() is called (C). Inside that, AnotherException is raised (D), and control returns to the main block (B).

2. A, D, C, B – 60% Similar to the above, but here I’m considering the possibility that the re-raised exception could theoretically point directly to raise_another_exception() as causing the error (D) before continuing to line 17 (C), finishing with B.

3. C, A, D, B – 40% Perhaps the execution starts at C (since it raises the next exception), jumps to A (OriginalException), then to D (raise_another_exception), and finally back to B.

4. A, B, C, D – 35% While this seems unlikely, I could imagine an unusual case where line B is recorded before C and D due to some stacktrace reporting order. Not probable, but possible.

5. C, D, A, B – 25% A more speculative idea where the control flow jumps straight to raise_another_exception before logging the original exception A.

6. A, D, B, C – 20% This assumes the error jumps from raising OriginalException to raise_another_exception, but B is logged before returning to C.

7. B, C, A, D – 15% This would be a situation where the top-level exception (B) is logged first, then C (calling raise_another_exception()), followed by logging A and D.

8. C, B, D, A – 12% Perhaps control jumps back and forth, handling the exceptions differently, though this breaks the natural top-to-bottom flow.

9. D, A, C, B – 10% In this case, D would get logged first as the innermost exception, then OriginalException (A), followed by C and B.

10. D, C, A, B – 8% A variation of the above, with D getting logged first as the next function call after the exception, followed by the rest.

11. C, A, B, D – 7% Control first jumps to C, handling the inner exception, then logs the outer exceptions.

12. B, A, C, D – 6% Not likely, but B could be logged before the original exception A is caught and handled.

13. B, D, A, C – 5% Another low-probability option where D is logged before C, causing some confusion in the flow.

14. A, C, B, D – 4% The control flow first jumps to the exception handling block and propagates up to B, but then it logs the raise_another_exception.

15. D, B, A, C – 3% An unusual case where B gets logged out of order after the second exception is raised.

16. B, D, C, A – 3% Similar idea, just different ordering of the inner and outer exception handling.

17. C, B, A, D – 2.5% Another possibility where the flow starts at C, logging the innermost block first, but B is logged before other exceptions.

18. B, A, D, C – 2% B first, then A, followed by raise_another_exception.

19. D, B, C, A – 1.8% This logs the deepest exception first, then handles the others.

20. C, D, B, A – 1.5% Jumps into the except block and logs the exceptions in reverse order.

21. D, C, B, A – 1% Reverse order of logging exceptions.

22. A, B, D, C – 0.8% First raises OriginalException, then logs top-down without accounting for the function calls.

23. B, C, D, A – 0.5% Exception chain starts from the bottom and logs in top-down order.

24. B, D, C, A – 0.2% Highly unlikely; logs inner and outer exceptions in a jumbled order.

I’m confident that A, C, D, B is the most likely based on Python’s normal exception flow, where the exceptions are raised in the order they occur in the code.

Starting with the code

Later that day, I decided to reverse the flow. I presented the code (but not the stacktrace) and then asked “what will it print on screen?”

Surprisingly, ChatGPT got this one right! Answer below.

---

The provided code will raise an exception, and the output will be a stack trace. Here’s the step-by-step breakdown of what happens:

1. show_something() is called.
2. Inside show_something(), it tries to raise an OriginalException.
3. The OriginalException is caught by the except block.
4. Inside the except block, the function raise_another_exception() is called, which raises an AnotherException.

Because there’s no handling for AnotherException, the program will terminate and print the stack trace for the unhandled exception.

Here’s what the output would look like:

Traceback (most recent call last):
  File "script.py", line 14, in show_something
    raise OriginalException()
__main__.OriginalException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 18, in <module>
    show_something()
  File "script.py", line 16, in show_something
    raise_another_exception()
  File "script.py", line 10, in raise_another_exception
    raise AnotherException()
__main__.AnotherException

This shows that an OriginalException occurred first, but while handling it, an AnotherException was raised, which is ultimately unhandled. The During handling of the above exception, another exception occurred: message indicates the chained exception context.

Back to the puzzle

Emboldened by this success, I decided to present ChatGPT with the original challenge. Despite having given me the answer in the above, it then messed this up. Oh well…

Now that you’re here: Bugsink is a simple, self-hosted error tracking tool that’s API-compatible with Sentry’s SDKs. Check it out