Analysis of Python's new string format vulnerability

Preface

This article conducts an in-depth analysis of the security vulnerabilities of a new syntax for formatting strings introduced by Python, and provides corresponding security solutions.

When we use str.format on untrusted user input, it will bring security risks - I have known about this problem for a long time, but I didn't realize its seriousness until today. Because attackers can use it to bypass the Jinja2 sandbox, which will cause serious information leakage. At the same time, I provide a new secure version of str.format at the end of this article.

As a reminder, this is a pretty serious security risk, and the reason it's being written about here is that most people probably don't know how easily it can be exploited.

Core Issues

Starting from Python 2.6, Python introduced a new syntax for formatting strings inspired by .NET. Of course, in addition to Python, Rust and some other programming languages ​​also support this syntax. With the help of the .format() method, this syntax can be applied to bytes and unicode strings (in Python 3, it can only be used for unicode strings). In addition, it can also be mapped to the more customizable string.Formatter API.

A feature of this syntax is that one can determine the positional and keyword arguments of the string format, and can explicitly reorder the data items at any time. In addition, it is even possible to access object attributes and data items - this is the root cause of the security issue here.

In general, people can use it to:

 > > > 'class of {0} is {0.__class__}'.format(42)
 "class of 42 is < class 'int' > "

Essentially, anyone who can control the format string can potentially access various internal properties of the object.

What's the problem?

The first question is how to control the format string. You can start from the following places:

1. Untrusted translators in string files. These are likely to work because many applications translated into multiple languages ​​use this new Python string formatting method, but not everyone does a thorough review of all input strings.

2. User-exposed configuration. Since some system users can configure certain behaviors, these configurations may be exposed in the form of format strings. It is important to note that I have seen some users configure notification emails, log message formats, or other basic templates through web applications.

Hazard Level

If you just pass the C interpreter object to the format string, it's not very dangerous, because then the most you'll expose is some integer class or something like that.

However, once Python objects are passed to this format string, things get tricky. This is because the amount of things that can be exposed from a Python function is quite staggering. Here is a scenario for a hypothetical web application that could leak a key:

 CONFIG = {
    'SECRET_KEY': 'super secret key'
 } 
   
 class Event(object):
    def __init__(self, id, level, message):
 self.id = id
 self.level = level
 self.message = message 
   
 def format_event(format_string, event):
    return format_string.format( event event =event)

If a user could inject format_string here, they would be able to discover a secret string like this:

 {event.__init__.__globals__[CONFIG][SECRET_KEY]}

Sandboxing Formatting

So, what should you do if you need someone else to provide a formatted string? In fact, you can use some undisclosed internal mechanisms to change the string formatting behavior.

 from string import Formatter
 from collections import Mapping 
   
 class MagicFormatMapping(Mapping):
    """This class implements a dummy wrapper to fix a bug in the Python
    standard library for string formatting. 
   
    See http://bugs.python.org/issue13598 for information about why
 this is necessary.
 """ 
   
    def __init__(self, args, kwargs):
 self._args = args
 self._kwargs = kwargs
 self._last_index = 0   
   
    def __getitem__(self, key):
 if key == '':
 idx = self ._last_index
            self._last_index += 1
 try:
                return self._args[idx]
            except LookupError:
 pass
 key = str (idx)
        return self._kwargs[key] 
   
 def __iter__(self):
        return iter(self._kwargs) 
   
 def __len__(self):
        return len(self._kwargs) 
   
 # This is a necessary API but it's undocumented and moved around
 # between Python releases
 try:
    from _string import formatter_field_name_split
 except ImportError:
 formatter_field_name_split = lambda \
        x: x._formatter_field_name_split() 
   
 class SafeFormatter(Formatter): 
   
    def get_field(self, field_name, args, kwargs):
        first, rest = formatter_field_name_split (field_name)
 obj = self .get_value(first, args, kwargs)
        for is_attr, i in rest:
 if is_attr:
 obj = safe_getattr (obj, i)
 else:
 obj obj = obj[i]
 return obj, first 
   
 def safe_getattr(obj, attr):
    # Expand the logic here. For instance on 2.x you will also need
    # to disallow func_globals, on 3.x you will also need to hide
    # things like cr_frame and others. So ideally have a list of
    # objects that are entirely unsafe to access.
 if attr[:1] == '_':
        raiseAttributeError(attr)
 return getattr(obj, attr) 
   
 def safe_format(_string, *args, **kwargs):
 formatter = SafeFormatter ()
 kwargs = MagicFormatMapping (args, kwargs)
    return formatter.vformat(_string, args, kwargs)

Now, we can use the safe_format method instead of str.format:

 > > > '{0.__class__}'.format(42)
 " < type 'int' > "
 > > > safe_format('{0.__class__}', 42)
 Traceback (most recent call last):
  File " < stdin > ", line 1, in < module >  
 AttributeError: __class__

summary

In this article, we conducted an in-depth analysis of the security vulnerabilities of a new syntax for formatting strings introduced by Python, and provided corresponding security solutions, hoping that it will be helpful to readers.