How to Parse Solver Logs with Python

If you run Computational Fluid Dynamics (CFD) or Finite Element Analysis (FEA) simulations, you spend a lot of time staring at text files. Solvers like open-source CFD software, commercial CFD software, and open-source CFD software dump thousands of lines of iterative data into .log files.

While GUI tools exist to plot this data, true automation requires you to extract this information programmatically. Whether you are building an optimization loop, setting up a CI/CD pipeline for your designs, or just trying to save time across 50 design variations, you need a way to turn unstructured text into structured data.

In this guide, we'll build a simple, memory-efficient Python script using Regular Expressions (re) to parse a solver log, extract residuals, and plot them using pandas and matplotlib.

Understanding Solver Logs

A typical iterative solver log might look something like this (simplified open-source CFD software style):

Time = 1
Solving for p, Initial residual = 0.892, Final residual = 0.051, No Iterations 10
Solving for Ux, Initial residual = 0.124, Final residual = 0.009, No Iterations 5
Solving for Uy, Initial residual = 0.111, Final residual = 0.008, No Iterations 5

Time = 2
Solving for p, Initial residual = 0.051, Final residual = 0.010, No Iterations 8
Solving for Ux, Initial residual = 0.009, Final residual = 0.001, No Iterations 3
Solving for Uy, Initial residual = 0.008, Final residual = 0.001, No Iterations 3

Our goal is to extract the Initial residual for p, Ux, and Uy at every Time step.

Core Concept: Regular Expressions (Regex)

Regular Expressions are sequences of characters that define a search pattern. In Python, we use the built-in re module.

To extract data, we use capture groups, denoted by parentheses (). For example, to find the time step, we might use: ^Time = (.*)

• ^ means "start of the line"
• Time = matches the literal text
• (.*) captures everything after the equals sign into a group.

For the pressure residual, we might use: Solving for p, Initial residual = ([0-9\.]+), This captures only digits and decimal points immediately following the text.

Step-by-Step Implementation

Step 1 & 2: Setup and Compiling Regex Patterns

First, we import the necessary libraries and compile our regex patterns. Compiling patterns beforehand is computationally faster when parsing files with millions of lines.

import re
import pandas as pd
import matplotlib.pyplot as plt

# Compile regex patterns
time_pattern = re.compile(r"^Time = (.*)")
p_pattern = re.compile(r"Solving for p, Initial residual = ([0-9\.]+)")
ux_pattern = re.compile(r"Solving for Ux, Initial residual = ([0-9\.]+)")
uy_pattern = re.compile(r"Solving for Uy, Initial residual = ([0-9\.]+)")

Step 3: The Extraction Loop

We want to read the file line by line. Do not use file.read() or file.readlines() on massive log files, as this will load the entire file into your RAM and likely crash your script.

We will use a for line in file: loop, which is a memory-efficient generator in Python. We will store the extracted data in a list of dictionaries.

def parse_log(filepath):
    data = []
    current_time = None
    current_row = {}

    with open(filepath, 'r') as file:
        for line in file:
            # Check for Time
            time_match = time_pattern.search(line)
            if time_match:
                # If we already have a row built from the previous timestep, save it
                if current_time is not None:
                    data.append(current_row)
                
                # Start a new row
                current_time = float(time_match.group(1))
                current_row = {'Time': current_time}
                continue

            # Check for variables
            p_match = p_pattern.search(line)
            if p_match:
                current_row['p_res'] = float(p_match.group(1))
                continue
                
            ux_match = ux_pattern.search(line)
            if ux_match:
                current_row['ux_res'] = float(ux_match.group(1))
                continue

            uy_match = uy_pattern.search(line)
            if uy_match:
                current_row['uy_res'] = float(uy_match.group(1))
                continue

        # Append the very last row after the loop finishes
        if current_row:
            data.append(current_row)
            
    return data

Step 4: Structuring and Visualizing the Data

Now that we have a list of dictionaries, we can easily convert it into a pandas DataFrame. DataFrames are the industry standard for handling tabular data in Python.

# Parse the log file
log_data = parse_log("solver.log")

# Convert to pandas DataFrame
df = pd.DataFrame(log_data)

# Set the index to Time for easier plotting
df.set_index('Time', inplace=True)

# Plot the residuals
plt.figure(figsize=(10, 6))
plt.plot(df.index, df['p_res'], label='Pressure (p)', color='blue')
plt.plot(df.index, df['ux_res'], label='Velocity (Ux)', color='red')
plt.plot(df.index, df['uy_res'], label='Velocity (Uy)', color='green')

# Residual plots are usually log-scale on the Y-axis
plt.yscale('log')
plt.xlabel('Time / Iteration')
plt.ylabel('Initial Residual')
plt.title('Solver Residuals')
plt.legend()
plt.grid(True, which="both", ls="--")

# Save or show the plot
plt.savefig("residuals.png", dpi=300)
plt.show()

Handling Edge Cases

When building robust parsers, you will encounter edge cases:

• Solver Crashes: If the solver crashes mid-iteration, your current_row might be missing variables. pandas will automatically fill these missing values with NaN (Not a Number) when converting the list of dictionaries to a DataFrame.
• Formatting Changes: Solvers frequently change their output formatting between major versions. Always parameterize your regex strings or store them in a configuration file so they are easy to update.
• NaNs in Logs: Sometimes the solver will literally output NaN if it diverges. Your regex ([0-9\.]+) won't catch NaN. You may need to update the pattern to ([0-9\.]+|NaN).

To learn more about what these residuals actually mean for your simulation quality, see CFD Convergence: Why Residuals Are Not Enough.