Logging

Logging is a very important part of any application. It allows you to track the code execution and to debug the application. Python has a built-in logging module that allows you to log messages to the console, to a file, or to a remote server. In contrast to the print function, the logging module is more complete, allowing you to configure the log level, the log format, and the log destination.

Logging is based in handlers. A handler is an object that receives the log messages and decides what to do with them. The logging module has several built-in handlers, such as StreamHandler, FileHandler, RotatingFileHandler or TimedRotatingFileHandler. But you can create your own handler by inherit the Handler class. Notifiers is a 3pp library that provides with extra handlers with the ability to send notifications to different services.

Any handler can have different configurations, such as the log level or the log format. The log level is used to filter the log messages. The log format is used to format the log messages. You can use the built-in log formats or create your own format by using the Formatter class. Also, a logger can have filters to filter the log messages before they are sent to the handlers. This way you can have more control over the log messages, like modifying or discarding them.

Best practices

• Set different log levels for different environments. For example, you may set DEBUG level in development and ERROR level in production.
• Set a specific format for the log messages, including the timestamp or the log level. Using a standard format makes it easier to read the log messages.
• Use the extra parameter to pass the data to the log message.
• Use pipelines | to separate the different parts of the log message. It can be useful to filter the log messages, or even to parse them.
• To include variables in your log message aside from extra, don't use format or f-string in the log call. Instead use the %s, like logger.info('Variable: %s', value).
• Use logging.exception to log an exception message and the stack trace.
• Set the different logger instance you are going to use with logging.getLogger. This way you can configure the logger in one place and use it in different modules.

logging library

This is the built-in Python logging library. It is very flexible and allows you to configure the log level, the log format, and the log destination.

Each logger has a name, and the loggers are organized in a tree-like structure. The root logger is the top-level logger, and all other loggers are children of the root logger.

src.intermediate.logging.default_logging(level)

Method to show default logging configuration.

Parameters:

Name	Type	Description	Default
level	int	level to set in logging	required

Source code in src/intermediate/logging/logging.py

def default_logging(level: int):
    """Method to show default logging configuration.

    Args:
        level (int): level to set in logging
    """
    if level not in logging._levelToName:
        raise IndexError("Invalid level to set logging.")

    logging.debug("DEBUG logging")
    logging.info("INFO logging")
    logging.warning("WARNING logging")
    logging.error("ERROR logging")
    logging.critical("CRITICAL logging")

src.intermediate.logging.custom_logging_format(format, datefmt)

Method to show logging configuration format.

Other arguments worthy to mention are: filename: using the path as log file with FileHandler. filemode: specifies the mode to open the log file.

Source code in src/intermediate/logging/logging.py

def custom_logging_format(format: str, datefmt: str):
    """Method to show logging configuration format.

    Other arguments worthy to mention are:
    filename: using the path as log file with FileHandler.
    filemode: specifies the mode to open the log file.
    """
    logging.basicConfig(
        level=logging.INFO,
        format=format,
        datefmt=datefmt,
    )
    logger = logging.getLogger(__name__)

    logger.info("INFO logging formatted")

src.intermediate.logging.CustomFilter

Bases: Filter

Custom logging filter to mask sensitive information in log records.

Source code in src/intermediate/logging/filtering.py

class CustomFilter(logging.Filter):
    """Custom logging filter to mask sensitive information in log records."""

    keys_to_mask = {
        "email": {
            "pattern": r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$",
            "mask_func": "mask_email",
        },
        "password": {
            "pattern": r".+",
            "mask_func": "mask_password",
        },
    }

    def filter(self, record: LogRecord) -> bool:
        """Override filter method to mask sensitive information.

        You can mask sensitive data in record attributes.
        Also, you can mask record msg or arguments,
        but only by regex/exact match..
        """
        for key in record.__dict__.keys():
            if key in self.keys_to_mask:
                # import ipdb; ipdb.set_trace()
                if isinstance(record.__dict__[key], str):
                    config = self.keys_to_mask[key]
                    mask_func = getattr(self, config["mask_func"])
                    record.__dict__[key] = re.sub(
                        config["pattern"],
                        mask_func,
                        record.__dict__[key],
                    )
        return True

    def mask_password(self, match_obj):
        """Mask password completely."""
        return "*" * len(match_obj.group(0))

    def mask_email(self, match_obj):
        """Mask email address, showing only first character before @."""
        local_part, domain = match_obj.group(0).split("@")
        masked_local = "*" * (len(local_part))
        return f"{masked_local}@{domain}"

src.intermediate.logging.CustomFilter.filter(record)

Override filter method to mask sensitive information.

You can mask sensitive data in record attributes. Also, you can mask record msg or arguments, but only by regex/exact match..

Source code in src/intermediate/logging/filtering.py

def filter(self, record: LogRecord) -> bool:
    """Override filter method to mask sensitive information.

    You can mask sensitive data in record attributes.
    Also, you can mask record msg or arguments,
    but only by regex/exact match..
    """
    for key in record.__dict__.keys():
        if key in self.keys_to_mask:
            # import ipdb; ipdb.set_trace()
            if isinstance(record.__dict__[key], str):
                config = self.keys_to_mask[key]
                mask_func = getattr(self, config["mask_func"])
                record.__dict__[key] = re.sub(
                    config["pattern"],
                    mask_func,
                    record.__dict__[key],
                )
    return True

src.intermediate.logging.CustomFilter.mask_email(match_obj)

Mask email address, showing only first character before @.

Source code in src/intermediate/logging/filtering.py

def mask_email(self, match_obj):
    """Mask email address, showing only first character before @."""
    local_part, domain = match_obj.group(0).split("@")
    masked_local = "*" * (len(local_part))
    return f"{masked_local}@{domain}"

src.intermediate.logging.CustomFilter.mask_password(match_obj)

Mask password completely.

Source code in src/intermediate/logging/filtering.py

def mask_password(self, match_obj):
    """Mask password completely."""
    return "*" * len(match_obj.group(0))

loguru library

Loguru is a third-party library that simplifies the logging configuration to the bare minimum, such as log level and log format. But you can also can configure much more easily, such as:

• color customization.
• log rotation, retention and compression.
• custom log levels.
• lazy evaluation of log messages.

src.intermediate.logging.default_loguru()

Method to show default loguru configuration.

Source code in src/intermediate/logging/logging.py

def default_loguru():
    """Method to show default loguru configuration."""
    logger_loguru.debug("DEBUG loguru")
    logger_loguru.info("INFO loguru")
    logger_loguru.warning("WARNING loguru")
    logger_loguru.error("ERROR loguru")
    logger_loguru.critical("CRITICAL loguru")

src.intermediate.logging.custom_loguru_format_and_level(format, level)

Method to show loguru custom configuration.

Add custom configuration to loguru, such as format and level. Sink is the first argument, representing how/where to log. It can be sys.*, or a log file path or a loggingHandler.

More information in the official loguru documentation: https://loguru.readthedocs.io/en/stable/api/logger.html#loguru._logger.Logger.add

Source code in src/intermediate/logging/logging.py

def custom_loguru_format_and_level(format: str, level: str):
    """Method to show loguru custom configuration.

    Add custom configuration to loguru, such as format and level.
    Sink is the first argument, representing how/where to log.
    It can be sys.*, or a log file path or a loggingHandler.

    More information in the official loguru documentation:
    https://loguru.readthedocs.io/en/stable/api/logger.html#loguru._logger.Logger.add
    """
    logger_loguru.add(sys.stdout, format=format, level=level)

    logger_loguru.debug("DEBUG loguru formatted")
    logger_loguru.info("INFO loguru formatted")
    logger_loguru.warning("WARNING loguru formatted")
    logger_loguru.error("ERROR loguru formatted")
    logger_loguru.critical("CRITICAL loguru formatted")

References

• Python logging module
• Loguru repository