Kaggle Getting Started Competition:

Monet Image Generation with Generative Adversarial Networks

Kaggle Competition

---

By: Travis Reinart

September 9, 2025

Week 5 Use GANs to Create Art Peer-Graded Assignment
CSCA 5642: Introduction to Deep Learning

GitHub-Week 5 Kaggle Getting Started Competition: Use GANs to Create Art

View GANs to Create Art Analysis (Live HTML on Netlify)

---

--- Note on file size and links ---

The final notebook and the HTML export are about 43 MB each. To avoid GitHub’s normal repository file-size limits (25 MB per file for this submission) and keep the repo fast to clone, I publish the large artifacts as GitHub Release assets and host the HTML on Netlify for viewing.

GitHub Release Direct Link (large files)

Direct Download of Notebook (.ipynb) - 43 MB

Direct Download HTML (static file) - 43 MB

Copyright (c) 2025 Travis Reinart

Licensed under the MIT License.

---

Section 1: Introduction

This notebook documents a complete workflow for the Kaggle "GAN Getting Started" competition, a project focused on image-to-image translation. The objective is to build and train a Generative Adversarial Network (GAN) capable of creating new, original images in the style of the impressionist painter Claude Monet. The personal goal for this project is to achieve a competitive MiFID score of 60 or less on the leaderboard.

Generative Adversarial Networks (GANs)

A Generative Adversarial Network (GAN) is a type of deep learning model where two neural networks are trained in competition with each other to generate new, original data that mimics a given dataset. The two networks are:

• The Generator: Its goal is to create fake data (in this case, images) that is as realistic as possible.
• The Discriminator: Its goal is to look at an item and determine if it's real (from the original dataset) or fake (created by the Generator).

Think of a quarterback learning a legendary playbook. Two people train together: a young QB and a veteran coach who knows every Lombardi concept by heart.

• The Generator is Jordan Love. He sketches new plays from random ideas and tries to make them pass for true Lombardi material.
• The Discriminator is a Lombardi-era coach who has studied the real book for years. His job is simple: watch a play and call it real or fake.

They sit in the film room. Love draws a play and walks through the motion. The coach pauses the clip and points out a tell: the split is too wide, the timing is off, the angle on the pull is wrong. Love adjusts, runs it again, and the coach looks harder. Each rep forces both to improve: the QB learns patterns that make a play feel authentic; the coach learns to spot smaller flaws. After thousands of reps, the coach is right only about half the time. At that point, Love can create new plays that fit the Lombardi style.

Modeling Strategy

This notebook implements a CycleGAN with a ResNet-9 generator and a 70×70 PatchGAN discriminator, an architecture proven to be highly effective for unpaired image-to-image translation.

Framework Choice: TensorFlow & Keras

TensorFlow 2.20 with Keras 3 was chosen because its API is an excellent fit for image-to-image models. Features like the Functional API for building complex architectures, custom training via Model.train_step for non-standard loops, and the integrated tf.data pipeline for high-performance input streams keep the entire workflow clean and efficient. While other frameworks like PyTorch are equally capable, Keras was selected here for its readability and ease of prototyping for this specific computer vision task.

Evaluation Metric: MiFID

This competition uses the Memorization-informed Fréchet Inception Distance (MiFID), where a **lower score is better**. MiFID builds on the standard FID, which compares the statistical distribution of features from real and generated images. It adds a crucial penalty if any generated image is a near-duplicate of a training image, thus discouraging simple memorization and rewarding true stylistic generation.

The Fréchet Inception Distance between real images $(x)$ and generated images $(g)$ is:

$$ d^2=\lVert \mu_{x}-\mu_{g}\rVert^{2} +\operatorname{Tr}\!\left(\Sigma_{x}+\Sigma_{g}-2\,(\Sigma_{x}\Sigma_{g})^{1/2}\right) $$

Where:

• $(\mu_x,\Sigma_x$) are the mean and covariance of Inception features for the real images.
• $(\mu_g,\Sigma_g$) are the mean and covariance for the generated images.
• $(\operatorname{Tr}$) is the matrix trace.

Submission Format

The final output must be a single ZIP file named images.zip containing 7,000 to 10,000 generated Monet-style images at 256x256 resolution. The final, successful submission for this project leveraged a 10,000-image set created with Test-Time Augmentation to maximize the score.

Project Plan

1. Section 1: Introduction: Define the problem, the GAN concept, the modeling strategy, and the evaluation metric.
2. Section 2: Setup: Configure the environment, import libraries, and set seeds for reproducibility.
3. Section 3: Data Loading & Audit: Load the Monet and photo datasets and perform an integrity check.
4. Section 4: Visual EDA: Visually inspect the two domains to understand their statistical differences.
5. Section 5: Data Pipeline: Build a high-performance tf.data pipeline for image preprocessing and augmentation.
6. Section 6: CycleGAN Model: Define and build the ResNet-9 Generator and PatchGAN Discriminator.
7. Section 7: Training: Configure the full training objective and execute the initial 15-epoch training run.
8. Section 8: Fine-Tuning & Final Submission: Analyze the initial result and run a targeted fine-tuning experiment to improve the score, then generate the final submission file.
9. Section 9: Conclusion & Analysis: Present the final Kaggle results, visualize the generated images, and summarize key learnings.

---

1.1 Kaggle Competition: Final Submissions

This screenshot shows the two official submissions to the Kaggle I'm Something of a Painter Myself competition. The initial submission, using the model from the first 15-epoch training run, established a baseline score of 69.06. The second and final submission, which used the fine-tuned model along with Test-Time Augmentation, achieved a final score of 57.83. This represents a significant 11.23-point improvement and successfully surpassed the project's personal goal of scoring below 60.

Kaggle Competition Leaderboard

---

Section 2: Setup

With the project plan defined, the next step is to set up the programming environment. This section handles all the necessary boilerplate: importing the required libraries (like TensorFlow and Matplotlib), setting a global random seed for reproducible results, and running a full diagnostic script to verify the environment and confirm GPU access.

Section Plan:

1. Data Sources: Confirm files, shapes, and basic integrity.
2. Package Installation: An optional cell to install any missing packages.
3. Environment Notes: Optional setup guide for a high-performance WSL2 environment.
4. Core Library Imports: A central location for all libraries used in the notebook.
5. Diagnostics: Print library versions, and check for GPU access.

---

2.1 The Dataset: Monet Paintings and Landscape Photos

The dataset for this competition is provided in two main parts: a collection of Claude Monet's paintings and a larger collection of landscape photographs. The goal is to train a model that can learn the "style" of the Monet paintings and apply it to the photographs. The data is provided in two formats: standard JPEG files for visual inspection and TFRecord files for efficient model training.

Kaggle Competition Datasets

Dataset Component	Format	Files	Total Size	Purpose
📁 Monet Paintings	TFRecord	5 shards (≈300 images total)	≈9.9 MB	Training Data (Target Domain)
📁 Landscape Photos	TFRecord	20 shards (≈7,038 images total)	≈260 MB	Training Data (Input Domain)
📁 Monet Paintings	JPEG	300 files	≈5.4 MB	EDA / Visual Reference
📁 Landscape Photos	JPEG	7,038 files	≈108 MB	Test Set for Final Submission

Counts and sizes reflect the local WSL copy under ~/data/gan-getting-started. TFRecord rows list shard counts; totals in parentheses match the JPEG counts.

Verified in WSL: monet_jpg=300, photo_jpg=7038, monet_tfrec=5 shards, photo_tfrec=20 shards.

This notebook was developed using the Jupyter Notebook extension within Visual Studio Code. Generative AI was used as a supplementary tool to assist with code debugging and to refine language for clarity. The core logic and all final analysis are original.

---

2.2 Optional: Install Missing Packages

To ensure the notebook runs smoothly, this optional cell can be used to install any missing packages. Using %pip directly within Jupyter guarantees the packages are installed in the correct environment where the kernel is running.

Instructions:

1. Uncomment the relevant %pip install line(s) below.
2. Run this cell.
3. Wait for the "Successfully installed" message.
4. In the Jupyter menu, select Kernel → Restart.
5. Re-run the import cell.

Note: Restarting the kernel is required after installation so the notebook can detect the new packages.

---

# Uncomment and run the lines below to install any missing packages.

# Project environment & tools
# %pip install --upgrade pip
# %pip install "tensorflow[and-cuda]==2.20.*"
# %pip install notebook
# %pip install jupyterlab
# %pip install ipywidgets
# %pip install ipykernel
# %pip install qtconsole
# %pip install kaggle

# Core numerics and utilities
# %pip install numpy
# %pip install pandas
# %pip install tqdm

# Visualization
# %pip install matplotlib
# %pip install seaborn
# %pip install scikit-learn
# %pip install pillow
# %pip install ipython

# Deep Learning (TensorFlow / Keras)
# %pip install tensorflow

# Optional: export the notebook to HTML (uncomment and delete the leading slash to run)
# Most Jupyter installs include nbconvert. If the command fails, install nbconvert first.
#/ !jupyter nbconvert --version
# %pip install nbconvert

# --- Convert Jupyter Notebook .ipynb to .html ---
# IMPORTANT: This bang command is written for a Windows or plain Python kernel.
# It will not work as-is in a WSL kernel unless you use a Linux path to the notebook.
# Run this from a Windows kernel, or change the path to a Linux path when in WSL.

!jupyter nbconvert --to html "Week5_Kaggle_Monet_Competition.ipynb"

[NbConvertApp] Converting notebook Week5_Kaggle_Monet_Competition.ipynb to html
[NbConvertApp] WARNING | Alternative text is missing on 14 image(s).
[NbConvertApp] Writing 44033005 bytes to Week5_Kaggle_Monet_Competition.html

2.3 Optional Run on GPU with Ubuntu 22.04 + VS Code (WSL)

This section provides a complete guide for setting up a high-performance development environment using Ubuntu 22.04 on WSL2. The goal is to run Jupyter inside a clean Python 3.10.12 environment and leverage the NVIDIA GPU via TensorFlow (GPU).

Goal: launch Jupyter in Ubuntu 22.04 (WSL2), use a virtual env on Python 3.10.12, and make sure TensorFlow sees the NVIDIA GPU.

Below are the steps I followed to create this environment:

---

Step 1: Install Ubuntu on WSL2

Open PowerShell as Administrator and run:	wsl --install -d Ubuntu-22.04 wsl --set-default-version 2 wsl --update
Launch Ubuntu and create your user/password.	Important Tip: The password prompt in the Linux terminal stays blank as you type. This is normal security behavior!

Step 2: Install NVIDIA Driver & Verify

Install the latest driver on Windows, not inside WSL. The Windows driver automatically provides CUDA support to WSL2.	NVIDIA Driver Downloads
From the Ubuntu terminal, verify the GPU is visible:	nvidia-smi

Step 3: Create Python Virtual Environment

Install Python 3.10 and venv:	sudo apt-get update && sudo apt-get -y install python3.10 python3.10-venv python3-pip
Create and activate the environment. I named mine tfenv (TensorFlow environment):	python3.10 -m venv ~/tfenv source ~/tfenv/bin/activate
Install libraries and register the kernel for Jupyter:	pip install -U pip pip install "tensorflow[and-cuda]==2.20.*" numpy pandas matplotlib seaborn tqdm scikit-learn jupyter ipykernel pillow nbconvert python -m ipykernel install --user --name tfenv --display-name "Python (tfenv)"

Step 4: Connect with VS Code

In VS Code, install the "Remote - WSL" extension, then connect to your Ubuntu instance (Ctrl+Shift+P → "WSL: Connect to WSL").

Once connected, the bottom-left corner should show WSL: Ubuntu-22.04.

Step 5: Launch the Notebook in the WSL Environment

In VS Code, open the Command Palette (Ctrl+Shift+P), type the following, and select it:	WSL: Connect to WSL
A new VS Code window connected to Ubuntu will open. Verify the connection by checking the blue box in the bottom-left corner.	The status bar should say WSL: Ubuntu-22.04.
In this new window, go to Open File > Show Local..., navigate to your project folder, and open your notebook. You can now select the Python (tfenv) 3.10.12 kernel.	This final step ensures your notebook is running inside Ubuntu with access to the correct Python environment and GPU.

---

2.4 Core Library Imports

All libraries are imported in one place and grouped by purpose. This cell does imports only. No GPU configuration and no diagnostics here.

• System & I/O: paths, timing, files, utilities.
• Numerics & tables: NumPy, Pandas.
• Progress: tqdm.
• Visualization: Matplotlib, Seaborn, PIL, HTML display.
• Deep learning: TensorFlow / Keras layers, callbacks, mixed precision, TensorFlow Addons.
• Reproducibility: single seed block for Python, NumPy, and TensorFlow.

---

# High-Performance TensorFlow Configuration (MUST RUN BEFORE IMPORT)
# These settings help prevent GPU memory fragmentation and OOM errors.
import os
os.environ["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"
os.environ["TF_CUDNN_WORKSPACE_LIMIT_IN_MB"] = "256"
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

# Core utilities
import sys 
import platform
import random
import time
import json
import gc
import glob
import shutil
import warnings
import math
import io
import contextlib
import zipfile
import hashlib
from collections import deque, Counter, defaultdict
from pathlib import Path

# Numerics
import numpy as np
import pandas as pd

# Progress
from tqdm.auto import tqdm

# Visualization
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
from PIL import Image
from IPython.display import display, HTML

# Deep Learning (TensorFlow / Keras)
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, callbacks, mixed_precision

# Safer VRAM allocation pattern
tf.get_logger().setLevel("ERROR")
for g in tf.config.list_physical_devices("GPU"):
    try:
        tf.config.experimental.set_memory_growth(g, True)
    except Exception:
        pass

# Reproducibility
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
os.environ["PYTHONHASHSEED"] = str(SEED)
tf.random.set_seed(SEED)
try:
    tf.config.experimental.enable_op_determinism()
except Exception:
    pass

# General settings
warnings.filterwarnings("ignore")

print("-" * 70)
print(f"SEED = {SEED}")
print("-" * 70)

----------------------------------------------------------------------
SEED = 42
----------------------------------------------------------------------

2.5 Diagnostics and Verification

Verify the environment before training. This cell displays versions for all imported libraries and confirms whether TensorFlow can see a GPU.

---

# Display GPU Hardware Information
print("-" * 70)
!nvidia-smi

# TensorFlow build info
try:
    build = tf.sysconfig.get_build_info()
    print(f"CUDA: {build.get('cuda_version')} | cuDNN: {build.get('cudnn_version')}")
except Exception:
    pass

print("-" * 70)
print("--- Environment & Library Versions ---")

# System Information
print(f"Python Version:   {platform.python_version()}")
print(f"OS:               {platform.system()} {platform.release()}")

# Core ML/DS Library Versions (only what was imported in 2.4)
print("\n--- Core Libraries ---")
print(f"TensorFlow:       {tf.__version__}")
print(f"Keras:            {keras.__version__}")
print(f"NumPy:            {np.__version__}")
print(f"Pandas:           {pd.__version__}")
print(f"Scikit-learn:     {__import__('sklearn').__version__}")

# Visualization & Image Library Versions
print("\n--- Supporting Libraries ---")
print(f"Matplotlib:       {mpl.__version__}")
print(f"Seaborn:          {sns.__version__}")
print(f"tqdm:             {__import__('tqdm').__version__}")
print(f"Pillow (PIL):     {__import__('PIL').__version__}")


# Mixed precision policy
try:
    policy = mixed_precision.Policy('mixed_float16')
    mixed_precision.set_global_policy(policy)
    print("\n--- Performance Policies ---")
    print(f"  Mixed Precision Compute Dtype: {policy.compute_dtype}")
    print(f"  Mixed Precision Variable Dtype: {policy.variable_dtype}")
except Exception as e:
    print(f"  Could not set mixed precision policy: {e}")
print("-" * 70)

print(f"SEED set to {SEED} - reproducibility enabled.")
print("-" * 70)

# GPU Hardware Diagnostics
print("\n--- GPU Hardware Verification ---")
gpu_devices = tf.config.list_physical_devices('GPU')
if gpu_devices:
    details = tf.config.experimental.get_device_details(gpu_devices[0])
    print(f"   GPU Detected: {details.get('device_name', 'N/A')}")
    print(f"   Compute Capability: {details.get('compute_capability', 'N/A')}")
    with tf.device("/GPU:0"):
        a = tf.random.normal([2048, 2048], dtype=tf.float16)
        b = tf.random.normal([2048, 2048], dtype=tf.float16)
        t0 = time.time()
        _ = tf.linalg.matmul(a,b).numpy()
        print(f"   GPU Matmul Test (FP16): {time.time()-t0:.4f} seconds")
else:
    print("   --- WARNING: No GPU detected!!! ---")
print("-" * 70)

# Jupyter Environment
print("\n--- Jupyter Environment ---")
print("Which Jupyter :", shutil.which("jupyter"))
!jupyter --version

print("\n--- Setup complete. Libraries imported successfully. ---")
print("-" * 70)

----------------------------------------------------------------------
Mon Sep  8 23:15:55 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.34                 Driver Version: 546.26       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4070 ...    On  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P0              13W / 102W |    283MiB /  8188MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
CUDA: 12.5.1 | cuDNN: 9
----------------------------------------------------------------------
--- Environment & Library Versions ---
Python Version:   3.10.12
OS:               Linux 6.6.87.2-microsoft-standard-WSL2

--- Core Libraries ---
TensorFlow:       2.20.0
Keras:            3.11.3
NumPy:            1.26.4
Pandas:           2.3.2
Scikit-learn:     1.7.1

--- Supporting Libraries ---
Matplotlib:       3.10.5
Seaborn:          0.13.2
tqdm:             4.67.1
Pillow (PIL):     11.3.0

--- Performance Policies ---
  Mixed Precision Compute Dtype: float16
  Mixed Precision Variable Dtype: float32
----------------------------------------------------------------------
SEED set to 42 - reproducibility enabled.
----------------------------------------------------------------------

--- GPU Hardware Verification ---
   GPU Detected: NVIDIA GeForce RTX 4070 Laptop GPU
   Compute Capability: (8, 9)
   GPU Matmul Test (FP16): 0.1514 seconds
----------------------------------------------------------------------

--- Jupyter Environment ---
Which Jupyter : /home/treinart/tfenv/bin/jupyter

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1757391355.508266   25051 gpu_device.cc:2020] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5535 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4070 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9
2025-09-08 23:15:55.528875: E tensorflow/core/util/util.cc:131] oneDNN supports DT_HALF only on platforms with AVX-512. Falling back to the default Eigen-based implementation if present.

Selected Jupyter core packages...
IPython          : 8.37.0
ipykernel        : 6.30.1
ipywidgets       : 8.1.7
jupyter_client   : 8.6.3
jupyter_core     : 5.8.1
jupyter_server   : 2.17.0
jupyterlab       : 4.4.6
nbclient         : 0.10.2
nbconvert        : 7.16.6
nbformat         : 5.10.4
notebook         : 7.4.5
qtconsole        : 5.6.1
traitlets        : 5.14.3

--- Setup complete. Libraries imported successfully. ---
----------------------------------------------------------------------

Observation: Diagnostics and Environment Verification

As of September 7, 2025, the project environment is confirmed to be stable and correctly configured for this computer vision task. The setup is running on a Linux distribution via WSL2, with a comprehensive suite of up-to-date libraries. Critically, the hardware verification confirms that the NVIDIA GPU is successfully detected and utilized by TensorFlow, which is essential for training the deep learning models.

• Hardware & Driver Stack:

  • GPU: NVIDIA GeForce RTX 4070 Laptop GPU
  • NVIDIA Driver: 546.26
  • CUDA Version (from driver): 12.3
  • CUDA Version (from TensorFlow): 12.5.1
  • cuDNN Version (from TensorFlow): 9
  • VRAM: 8.0 GB

• Software Environment:

  • OS: Linux 6.6.87.2-microsoft-standard-WSL2
  • Python: 3.10.12
  • TensorFlow: 2.20.0
  • Keras: 3.11.3
  • NumPy: 1.26.4
  • Pandas: 2.3.2
  • Scikit-learn: 1.7.1
  • Matplotlib: 3.10.5
  • Seaborn: 0.13.2
  • Pillow (PIL): 11.3.0
  • tqdm: 4.67.1

• Jupyter Core Packages:

  • IPython: 8.37.0
  • ipykernel: 6.30.1
  • ipywidgets: 8.1.7
  • notebook: 7.4.5

• Performance Policies:

  • Mixed Precision: Enabled (float16 compute dtype) for lower memory usage and improved performance on the RTX 40-series GPU.
  • GPU Memory Growth: Enabled to prevent TensorFlow from pre-allocating all VRAM, reducing the risk of OOM errors.

Key Configuration Note: The successful and stable training runs later in this notebook rely heavily on the performance policies set here. Enabling GPU memory growth and using mixed precision were critical for managing VRAM on the 8GB GPU, especially after re-introducing the memory-intensive replay buffers during the fine-tuning stage.

This verified setup provides a high-performance, reproducible environment for the subsequent data processing, model training, and submission generation.

Section 3: Data Loading & Integrity Audit

With the environment set, this section establishes the definitive paths to the dataset. It then performs a comprehensive audit to verify the data's integrity, checking file counts, dimensions, readability, and uniqueness. This ensures a clean, predictable dataset is ready before building the formal tf.data input pipeline.

Section Plan:

1. Define Paths & Verify Structure: Point to the data root, define subfolder paths, and assert that all directories exist.
2. Perform Statistical Audit: Sample the JPEG files to analyze image dimensions and basic pixel statistics (mean, std), flagging any corrupt files.
3. Conduct Visual Audit: Display grids of random thumbnail images from each domain for a quick visual sanity check.
4. Check TFRecord Integrity: Parse a sample of TFRecord files to confirm they can be decoded correctly and contain the expected 256x256 images.
5. Analyze Duplicate Images: Use MD5 hashing to find identical images both within and across the Monet and Photo JPEG datasets.
6. Build & Export De-duplicated Lists: Create a canonical, de-duplicated list of photo JPEGs and save the "keep" and "drop" file lists to CSV for reproducibility.

---

3.1 Define Paths (Linux) & Assert Folders

Point DATA_ROOT to ~/data/gan-getting-started and define four subfolders: monet_jpg, photo_jpg, monet_tfrec, photo_tfrec. Use assert checks so path drift between Windows and WSL is caught immediately.

---

# Define paths to the dataset directories.
DATA_ROOT = Path.home() / "data" / "gan-getting-started"
MONET_JPG_DIR   = DATA_ROOT / "monet_jpg"
PHOTO_JPG_DIR   = DATA_ROOT / "photo_jpg"
MONET_TFREC_DIR = DATA_ROOT / "monet_tfrec"
PHOTO_TFREC_DIR = DATA_ROOT / "photo_tfrec"

# Verify that all required directories exist.
data_dirs = [MONET_JPG_DIR, PHOTO_JPG_DIR, MONET_TFREC_DIR, PHOTO_TFREC_DIR]
missing_dirs = [p for p in data_dirs if not p.exists()]
assert not missing_dirs, f"Missing expected folder(s): {[str(p) for p in missing_dirs]}"

# Count the number of files in each directory.
n_monet_jpg   = sum(1 for _ in MONET_JPG_DIR.glob("*.jpg"))
n_photo_jpg   = sum(1 for _ in PHOTO_JPG_DIR.glob("*.jpg"))
n_monet_tfrec = sum(1 for _ in MONET_TFREC_DIR.glob("*.tfrec"))
n_photo_tfrec = sum(1 for _ in PHOTO_TFREC_DIR.glob("*.tfrec"))

# Define the expected file counts from the Kaggle dataset page.
EXP_MONET_JPG   = 300
EXP_PHOTO_JPG   = 7038
EXP_MONET_TFREC = 5
EXP_PHOTO_TFREC = 20

# Assert that the actual file counts match the expected counts.
assert n_monet_jpg   == EXP_MONET_JPG,   f"monet_jpg count mismatch: got {n_monet_jpg}, expected {EXP_MONET_JPG}"
assert n_photo_jpg   == EXP_PHOTO_JPG,   f"photo_jpg count mismatch: got {n_photo_jpg}, expected {EXP_PHOTO_JPG}"
assert n_monet_tfrec == EXP_MONET_TFREC, f"monet_tfrec shard count mismatch: got {n_monet_tfrec}, expected {EXP_MONET_TFREC}"
assert n_photo_tfrec == EXP_PHOTO_TFREC, f"photo_tfrec shard count mismatch: got {n_photo_tfrec}, expected {EXP_PHOTO_TFREC}"

# Define helper functions to calculate and format directory sizes.
def _dir_size(p: Path) -> int:
    try:
        return sum(f.stat().st_size for f in p.rglob("*") if f.is_file())
    except Exception:
        return 0

def _fmt_bytes(n: int) -> str:
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if n < 1024:
            return f"{n:.1f} {unit}"
        n /= 1024
    return f"{n:.1f} PB"

# Calculate the size of each data directory.
sz_monet_jpg   = _fmt_bytes(_dir_size(MONET_JPG_DIR))
sz_photo_jpg   = _fmt_bytes(_dir_size(PHOTO_JPG_DIR))
sz_monet_tfrec = _fmt_bytes(_dir_size(MONET_TFREC_DIR))
sz_photo_tfrec = _fmt_bytes(_dir_size(PHOTO_TFREC_DIR))

# Store key counts as global variables for use in later sections.
N_MONET = n_monet_jpg
N_PHOTO = n_photo_jpg
N_MONET_TFREC_SHARDS = n_monet_tfrec
N_PHOTO_TFREC_SHARDS = n_photo_tfrec

# Define a helper to peek at the first few filenames for a sanity check.
def _peek(p: Path, pattern: str, k: int = 3):
    return [str(x) for x in sorted(p.glob(pattern))[:k]]

# --- Display Verification Results ---
print("-" * 70)
print("Defining data roots and verifying folder structure...")
print("\n--- Data Roots ---")
print(f"DATA_ROOT      : {DATA_ROOT}")
print(f"monet_jpg      : {MONET_JPG_DIR}")
print(f"photo_jpg      : {PHOTO_JPG_DIR}")
print(f"monet_tfrec    : {MONET_TFREC_DIR}")
print(f"photo_tfrec    : {PHOTO_TFREC_DIR}")
print("\n--- Quick Snapshot (counts & sizes) ---")
print(f"Monet JPEGs    : {n_monet_jpg:>5} files | {sz_monet_jpg:>8}")
print(f"Photo JPEGs    : {n_photo_jpg:>5} files | {sz_photo_jpg:>8}")
print(f"Monet TFRecord : {n_monet_tfrec:>5} shards | {sz_monet_tfrec:>8}")
print(f"Photo TFRecord : {n_photo_tfrec:>5} shards | {sz_photo_tfrec:>8}")
print("\n--- Sample Files ---")
print("monet_jpg   →", _peek(MONET_JPG_DIR, "*.jpg"))
print("photo_jpg   →", _peek(PHOTO_JPG_DIR, "*.jpg"))
print("monet_tfrec →", _peek(MONET_TFREC_DIR, "*.tfrec"))
print("photo_tfrec →", _peek(PHOTO_TFREC_DIR, "*.tfrec"))
print("\nPaths verified. Counts match expected Kaggle distribution.")
print("-" * 70)

----------------------------------------------------------------------
Defining data roots and verifying folder structure...

--- Data Roots ---
DATA_ROOT      : /home/treinart/data/gan-getting-started
monet_jpg      : /home/treinart/data/gan-getting-started/monet_jpg
photo_jpg      : /home/treinart/data/gan-getting-started/photo_jpg
monet_tfrec    : /home/treinart/data/gan-getting-started/monet_tfrec
photo_tfrec    : /home/treinart/data/gan-getting-started/photo_tfrec

--- Quick Snapshot (counts & sizes) ---
Monet JPEGs    :   300 files |   4.7 MB
Photo JPEGs    :  7038 files |  93.6 MB
Monet TFRecord :     5 shards |   9.8 MB
Photo TFRecord :    20 shards | 259.9 MB

--- Sample Files ---
monet_jpg   → ['/home/treinart/data/gan-getting-started/monet_jpg/000c1e3bff.jpg', '/home/treinart/data/gan-getting-started/monet_jpg/011835cfbf.jpg', '/home/treinart/data/gan-getting-started/monet_jpg/0260d15306.jpg']
photo_jpg   → ['/home/treinart/data/gan-getting-started/photo_jpg/00068bc07f.jpg', '/home/treinart/data/gan-getting-started/photo_jpg/000910d219.jpg', '/home/treinart/data/gan-getting-started/photo_jpg/000ded5c41.jpg']
monet_tfrec → ['/home/treinart/data/gan-getting-started/monet_tfrec/monet00-60.tfrec', '/home/treinart/data/gan-getting-started/monet_tfrec/monet04-60.tfrec', '/home/treinart/data/gan-getting-started/monet_tfrec/monet08-60.tfrec']
photo_tfrec → ['/home/treinart/data/gan-getting-started/photo_tfrec/photo00-352.tfrec', '/home/treinart/data/gan-getting-started/photo_tfrec/photo01-352.tfrec', '/home/treinart/data/gan-getting-started/photo_tfrec/photo02-352.tfrec']

Paths verified. Counts match expected Kaggle distribution.
----------------------------------------------------------------------

 

3.2 Statistical Audit of JPEG Images

Run a quick statistical audit on the JPEG datasets. This involves sampling images to find the most common dimensions, calculating the average RGB pixel values and standard deviations, and counting any corrupt or unreadable files. The results are summarized in a clean table.

 

---

# Set the sample sizes for the audit, with safe defaults.
try:
    N_SHAPE_MONET
    N_SHAPE_PHOTO
    N_STATS_MONET
    N_STATS_PHOTO
except NameError:
    N_SHAPE_MONET = 300
    N_SHAPE_PHOTO = 1000
    N_STATS_MONET = 300
    N_STATS_PHOTO = 1200

# Initialize a random number generator for sampling.
rng = np.random.default_rng(SEED)

# Get lists of all JPEG files.
monet_jpgs = sorted(MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = sorted(PHOTO_JPG_DIR.glob("*.jpg"))

# Define a helper function to safely open an image as a normalized RGB array.
def _safe_open_rgb(p: Path):
    try:
        with Image.open(p) as im:
            return np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
    except Exception:
        return None

# Define the main audit function to process one domain (Monet or Photo).
def _audit_one(name: str, files: list[Path], n_shape: int, n_stats: int):
    # --- Perform a shape audit on a sample of files. ---
    k_shape = min(n_shape, len(files))
    shape_sample = list(rng.choice(files, size=k_shape, replace=False)) if k_shape else []
    shape_counts = {}
    corrupt = 0
    for p in tqdm(shape_sample, desc=f"{name} shape audit", mininterval=0.2):
        arr = _safe_open_rgb(p)
        if arr is None:
            corrupt += 1
            continue
        h, w = arr.shape[:2]
        shape_counts[(w, h)] = shape_counts.get((w, h), 0) + 1
    
    # Get the top 3 most common image dimensions.
    top_shapes_list = sorted(shape_counts.items(), key=lambda kv: -kv[1])[:3]
    top_shapes = ", ".join([f"{w}×{h}:{c}" for (w, h), c in top_shapes_list]) or "—"

    # --- Calculate pixel statistics on a separate sample. ---
    k_stats = min(n_stats, len(files))
    stat_sample = list(rng.choice(files, size=k_stats, replace=False)) if k_stats else []
    means, stds = [], []
    for p in tqdm(stat_sample, desc=f"{name} pixel stats", mininterval=0.2):
        arr = _safe_open_rgb(p)
        if arr is None:
            continue
        means.append(arr.mean(axis=(0, 1)))
        stds.append(arr.std(axis=(0, 1)))

    # Aggregate the mean and standard deviation of pixel values across the sample.
    if means:
        mean_rgb = np.mean(means, axis=0)
        std_rgb  = np.mean(stds,  axis=0)
        mean_str = f"[{mean_rgb[0]:.4f}, {mean_rgb[1]:.4f}, {mean_rgb[2]:.4f}]"
        std_str  = f"[{std_rgb[0]:.4f}, {std_rgb[1]:.4f}, {std_rgb[2]:.4f}]"
    else:
        mean_str = "-"
        std_str = "-"

    return {
        "Domain": name,
        "JPEGs": len(files),
        "Sampled(shapes)": k_shape,
        "Top Shapes": top_shapes,
        "Sampled(stats)": len(means),
        "mean(R,G,B)": mean_str,
        "std(R,G,B)": std_str,
        "Corrupt files": corrupt,
    }

# --- Run the audit and display results ---
print("-" * 70)
print("Quick audit: shapes, simple pixel stats, and file readability (sampled).")
print("\n--- JPEG counts ---")
print(f"Monet JPEGs : {len(monet_jpgs):,}")
print(f"Photo JPEGs : {len(photo_jpgs):,}")

# Run the audit on both the Monet and Photo datasets.
rows = [
    _audit_one("Monet (JPEG)", monet_jpgs, N_SHAPE_MONET, N_STATS_MONET),
    _audit_one("Photo (JPEG)", photo_jpgs, N_SHAPE_PHOTO, N_STATS_PHOTO),
]
summary_df = pd.DataFrame(rows)

# Apply custom styling to the DataFrame for a clean presentation.
styler = summary_df.set_index("Domain").style
styler = styler.set_properties(**{
    "background-color": "#0b0b0b",
    "color": "#F8F8F8",
    "border-color": "#1A6A1A",
})
styler = styler.set_table_styles([
    {"selector": "th", "props": [("background-color", "#1A6A1A"), ("color", "white"), ("text-align", "left")]},
    {"selector": "td", "props": [("border", "1px solid #1A6A1A")]},
])
display(styler)

# Define a helper to check if the dominant shape is 256x256.
def _shape_hint(s: str):
    first = (s or "").split(",")[0].lower()
    norm = first.replace("×", "x").replace(" ", "")
    return "✓ Mostly 256×256" if "256x256" in norm else "Varied Sizes"

# Print summary notes based on the audit results.
top_m = summary_df.loc[summary_df["Domain"].eq("Monet (JPEG)"), "Top Shapes"].iloc[0]
top_p = summary_df.loc[summary_df["Domain"].eq("Photo (JPEG)"), "Top Shapes"].iloc[0]
print("\n--- Notes ---")
print(f"Monet JPEGs   : {_shape_hint(top_m)}")
print(f"Photo JPEGs   : {_shape_hint(top_p)}")
print("\nQuick audit complete.")
print("-" * 70)

----------------------------------------------------------------------
Quick audit: shapes, simple pixel stats, and file readability (sampled).

--- JPEG counts ---
Monet JPEGs : 300
Photo JPEGs : 7,038

Monet (JPEG) shape audit:   0%|          | 0/300 [00:00<?, ?it/s]

Monet (JPEG) pixel stats:   0%|          | 0/300 [00:00<?, ?it/s]

Photo (JPEG) shape audit:   0%|          | 0/1000 [00:00<?, ?it/s]

Photo (JPEG) pixel stats:   0%|          | 0/1200 [00:00<?, ?it/s]

	JPEGs	Sampled(shapes)	Top Shapes	Sampled(stats)	mean(R,G,B)	std(R,G,B)	Corrupt files
Domain
Monet (JPEG)	300	300	256×256:300	300	[0.5215, 0.5245, 0.4768]	[0.1912, 0.1822, 0.1942]	0
Photo (JPEG)	7038	1000	256×256:1000	1200	[0.4005, 0.4052, 0.3832]	[0.2214, 0.2018, 0.2177]	0

--- Notes ---
Monet JPEGs   : ✓ Mostly 256×256
Photo JPEGs   : ✓ Mostly 256×256

Quick audit complete.
----------------------------------------------------------------------

 

3.3 Visual Audit: Thumbnail Samples

A picture is worth a thousand words. Before diving deep, it's worth taking a quick look at the images. This step displays a compact grid of randomly sampled thumbnails from both the Monet and Photo collections, serving as a fast, initial confirmation of the dataset's content.

 

---

# Initialize a random number generator for sampling.
rng = np.random.default_rng(SEED)

# Get lists of all JPEG file paths.
monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

# Set the number of thumbnails to show from each domain.
N_SHOW_MONET = 49
N_SHOW_PHOTO = 49

# Define the master function to display a grid of image thumbnails.
def _show_grid(paths, title, max_n=49, title_color="#FFB612"):
    # Determine the number of images to display.
    k = min(max_n, len(paths))
    if not k:
        print(f"(no images found for: {title})")
        return
    
    # Randomly sample image paths.
    sample = rng.choice(paths, size=k, replace=False)

    # Calculate the grid dimensions (rows and columns).
    cols = min(8, max(4, int(np.sqrt(k) + 0.5)))
    rows = int(np.ceil(k / cols))

    # Create the figure and axes for the grid.
    fig, axes = plt.subplots(rows, cols, figsize=(2.5 * cols, 2.5 * rows), facecolor="black")
    
    # Normalize the axes object into a 2D array for consistent indexing.
    if rows == 1 and cols == 1:
        axes = np.array([[axes]])
    elif rows == 1 or cols == 1:
        axes = np.reshape(axes, (rows, cols))

    # --- MODIFICATION: Loop with tqdm ---
    # Create an iterable for the images and their corresponding axes.
    plot_iterable = zip(sample, axes.flat)
    
    # Loop through the images and axes with a progress bar.
    pbar = tqdm(plot_iterable, total=k, desc=f"Plotting '{title}'", mininterval=0.2)
    for p, ax in pbar:
        # Load and display the image, with error handling.
        try:
            with Image.open(p) as im:
                ax.imshow(im.convert("RGB"))
        except Exception:
            ax.text(0.5, 0.5, "unreadable", ha="center", va="center", color="red", fontsize=10)

    # --- Style all axes in the grid ---
    for ax in axes.flat:
        # Remove ticks and labels from each subplot.
        ax.set_xticks([])
        ax.set_yticks([])
        # Style the plot's border (spines).
        for spine in ax.spines.values():
            spine.set_edgecolor("#1A6A1A")
            spine.set_linewidth(1.5)
        # Set the background color of the axes.
        ax.set_facecolor("black")
    
    # Set the main title for the entire grid.
    plt.suptitle(title, fontsize=28, color=title_color, fontweight="bold", y=0.98)
    # Ensure the layout is tight and clean.
    plt.tight_layout()
    plt.show()

# --- Generate and display the thumbnail grids ---
print("-" * 70)
print("Visual EDA: thumbnail grids (random samples)")
print(f"Monet JPEGs available : {len(monet_jpgs):,}")
print(f"Photo JPEGs available : {len(photo_jpgs):,}")

# Display the grid for Monet images.
_show_grid(monet_jpgs, "Monet Sample Thumbnails", max_n=N_SHOW_MONET, title_color="#1A6A1A")
# Display the grid for Photo images.
_show_grid(photo_jpgs, "Photo Sample Thumbnails", max_n=N_SHOW_PHOTO, title_color="#FFB612")

print("-" * 70)

----------------------------------------------------------------------
Visual EDA: thumbnail grids (random samples)
Monet JPEGs available : 300
Photo JPEGs available : 7,038

Plotting 'Monet Sample Thumbnails':   0%|          | 0/49 [00:00<?, ?it/s]

Plotting 'Photo Sample Thumbnails':   0%|          | 0/49 [00:00<?, ?it/s]

----------------------------------------------------------------------

3.4 TFRecord Smoke Test

Build a minimal TFRecord parser for the image feature. Read a few samples from Monet and Photo shards to confirm decode works. Do not optimize here, keep it simple and fast to run.

---

# --- Setup and File Gathering ---
print("-" * 70)
print("TFRecord quick check: shard counts, decode a small sample, confirm shapes/dtypes.")

# Get lists of all TFRecord shard files.
monet_tfrec = sorted(MONET_TFREC_DIR.glob("*.tfrec"))
photo_tfrec = sorted(PHOTO_TFREC_DIR.glob("*.tfrec"))

print(f"Monet TFRecord shards : {len(monet_tfrec)}")
print(f"Photo TFRecord shards : {len(photo_tfrec)}")

# Verify that TFRecord files were actually found.
assert monet_tfrec, f"No TFRecords found in {MONET_TFREC_DIR}"
assert photo_tfrec, f"No TFRecords found in {PHOTO_TFREC_DIR}"

# Initialize TensorFlow utilities.
AUTOTUNE = tf.data.AUTOTUNE
rng = np.random.default_rng(SEED)

# --- Helper Functions ---

# Define a flexible parser for TFRecord examples that tries multiple common keys.
def _parse_example(serialized):
    for key in ("image", "img", "image/encoded"):
        try:
            ex = tf.io.parse_single_example(
                serialized, {key: tf.io.FixedLenFeature([], tf.string)}
            )
            img = tf.image.decode_jpeg(ex[key], channels=3)
            return img
        except Exception:
            pass
    raise ValueError("Could not locate an image field in TFRecord example")

# Define a function to read and analyze a sample of records from TFRecord files.
def _peek_tfrecs(tfrec_paths, n_take=64):
    # Create a dataset from the TFRecord files.
    ds = tf.data.TFRecordDataset(tfrec_paths, num_parallel_reads=AUTOTUNE)
    shapes, means, stds = [], [], []
    ok, fail = 0, 0

    # Loop through a sample of serialized examples.
    for serialized in ds.take(n_take):
        try:
            img = _parse_example(serialized)
            # Collect shape and pixel statistics.
            h, w = int(img.shape[0]), int(img.shape[1])
            shapes.append((w, h, 3))
            if ok < 24: # Limit stats calculation for speed
                arr = tf.cast(img, tf.float32) / 255.0
                means.append(tf.reduce_mean(arr, axis=(0,1)).numpy())
                stds.append(tf.math.reduce_std(arr, axis=(0,1)).numpy())
            ok += 1
        except Exception:
            fail += 1

    # Aggregate the collected statistics.
    if means:
        m = np.mean(np.stack(means, axis=0), axis=0)
        s = np.mean(np.stack(stds,  axis=0), axis=0)
        mean_str = f"[{m[0]:.4f}, {m[1]:.4f}, {m[2]:.4f}]"
        std_str  = f"[{s[0]:.4f}, {s[1]:.4f}, {s[2]:.4f}]"
    else:
        mean_str = "-"
        std_str  = "-"
    
    # Find the most common image shapes.
    top = Counter(shapes).most_common(3)
    top_str = ", ".join([f"{w}×{h}:{c}" for (w,h,_), c in top]) if top else "—"
    
    return {
        "peeked": ok + fail,
        "decoded_ok": ok,
        "decode_fail": fail,
        "top_shapes": top_str,
        "mean": mean_str,
        "std": std_str,
    }

# --- Run Analysis and Display Results ---

# Set the number of records to sample from each dataset.
N_TFREC_PEEK_MONET = 128
N_TFREC_PEEK_PHOTO = 256

# Run the peek function on the Monet TFRecords.
print("\n--- Peeking TFRecords (Monet) ---")
monet_info = _peek_tfrecs(monet_tfrec, n_take=N_TFREC_PEEK_MONET)
for k, v in monet_info.items():
    print(f"{k:>12}: {v}")

# Run the peek function on the Photo TFRecords.
print("\n--- Peeking TFRecords (Photo) ---")
photo_info = _peek_tfrecs(photo_tfrec, n_take=N_TFREC_PEEK_PHOTO)
for k, v in photo_info.items():
    print(f"{k:>12}: {v}")

# Define a helper to check if the dominant shape is 256x256.
def _hint(s: str):
    first = (s or "").split(",")[0].lower()
    norm  = first.replace("×", "x").replace(" ", "")
    return "✓ Mostly 256×256" if "256x256" in norm else "Varied Sizes"

# --- Final Notes and Assertions ---
print("\n--- Notes ---")
print(f"Monet TFRecord : {_hint(monet_info['top_shapes'])}")
print(f"Photo TFRecord : {_hint(photo_info['top_shapes'])}")

# Assert that the TFRecords are readable and have the expected shape.
assert monet_info["decode_fail"] == 0 and photo_info["decode_fail"] == 0, "Decode failure detected."
assert "256×256" in monet_info["top_shapes"].split(",")[0], "Monet shapes not mostly 256×256."
assert "256×256" in photo_info["top_shapes"].split(",")[0], "Photo shapes not mostly 256×256."

print("\nTFRecord check complete.")
print("-" * 70)

----------------------------------------------------------------------
TFRecord quick check: shard counts, decode a small sample, confirm shapes/dtypes.
Monet TFRecord shards : 5
Photo TFRecord shards : 20

--- Peeking TFRecords (Monet) ---
      peeked: 128
  decoded_ok: 128
 decode_fail: 0
  top_shapes: 256×256:128
        mean: [0.5380, 0.5413, 0.4909]
         std: [0.1981, 0.1822, 0.1807]

--- Peeking TFRecords (Photo) ---
      peeked: 256
  decoded_ok: 256
 decode_fail: 0
  top_shapes: 256×256:256
        mean: [0.3923, 0.4221, 0.4109]
         std: [0.2097, 0.2044, 0.2299]

--- Notes ---
Monet TFRecord : ✓ Mostly 256×256
Photo TFRecord : ✓ Mostly 256×256

TFRecord check complete.
----------------------------------------------------------------------

 

3.5 Refined Visual Audit: Large-Format Thumbnails

With the basic data checks complete, this step generates a more polished, large-format thumbnail grid. The larger images and refined styling are better suited for closer inspection of the details, textures, and color palettes within each domain.

 

---

# Set the number of thumbnails to show from each domain.
N_SHOW_MONET = 49
N_SHOW_PHOTO = 49

# Ensure the file lists are available in the current session.
if 'monet_jpgs' not in locals():
    monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
if 'photo_jpgs' not in locals():
    photo_jpgs = sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

# Initialize a random number generator for sampling.
rng = np.random.default_rng(SEED)

# Define a helper function to safely load an image file as an RGB NumPy array.
def _load_rgb(path):
    try:
        with Image.open(path) as im:
            return np.asarray(im.convert("RGB"))
    except Exception:
        return None

# Define the master function to display a grid of image thumbnails.
def _show_grid(paths, title, max_n=49, fontsize=28, title_color="#FFB612", pad=1):
    # Determine the number of images to display.
    n = min(max_n, len(paths))
    if n == 0:
        print(f"(no images found for {title})")
        return 0, 0

    # Create a deterministic random sample of image paths.
    sample = rng.choice(paths, size=n, replace=False)

    # Calculate the grid layout (rows and columns).
    cols = min(8, max(4, int(np.ceil(np.sqrt(n)))))
    rows = int(np.ceil(n / cols))

    # Create the figure and axes for the grid.
    fig, axes = plt.subplots(rows, cols, figsize=(3.1 * cols, 3.1 * rows), facecolor="black")
    
    # Normalize the axes object into a 2D array for consistent indexing.
    if rows == 1 and cols == 1:
        axes = np.array([[axes]])
    elif rows == 1:
        axes = np.array([axes])
    elif cols == 1:
        axes = np.array([[ax] for ax in axes])
    axes = axes.reshape(rows, cols)

    # First, apply styling to all axes in the grid.
    for ax in axes.flat:
        ax.set_xticks([])
        ax.set_yticks([])
        ax.set_facecolor("black")
        # Style the plot's border (spines).
        for spine in ax.spines.values():
            spine.set_edgecolor("#1A6A1A")
            spine.set_linewidth(0.8)

    # --- MODIFICATION: Loop with tqdm ---
    # Now, loop through the images and their corresponding axes to plot.
    shown, unreadable = 0, 0
    plot_iterable = zip(sample, axes.flat)
    pbar = tqdm(plot_iterable, total=n, desc=f"Plotting '{title}'", mininterval=0.2)
    for p, ax in pbar:
        arr = _load_rgb(p)
        if arr is not None:
            ax.imshow(arr)
            shown += 1
        else:
            # Handle unreadable files gracefully.
            ax.text(0.5, 0.5, "unreadable", ha="center", va="center", color="red", fontsize=10)
            unreadable += 1

    # Set the main title for the grid.
    plt.suptitle(title, y=0.91, fontsize=fontsize, color=title_color, fontweight="bold")
    # Apply a tight layout.
    plt.tight_layout
    plt.show()
    return shown, unreadable

# --- Generate and display the thumbnail grids ---
print("-" * 70)
print("Sampling thumbnails from each domain for a quick visual sanity check.")

# Display the grid for Monet images.
shown_m, bad_m = _show_grid(
    monet_jpgs,
    "Monet Sample Thumbnails",
    max_n=N_SHOW_MONET,
    fontsize=32,
    title_color="#1A6A1A",
    pad=1,
)
# Display the grid for Photo images.
shown_p, bad_p = _show_grid(
    photo_jpgs,
    "Photo Sample Thumbnails",
    max_n=N_SHOW_PHOTO,
    fontsize=32,
    title_color="#FFB612",
    pad=1,
)

# Print a summary of displayed and unreadable images.
print(f"Monet displayed: {shown_m}  | unreadable: {bad_m}")
print(f"Photo displayed: {shown_p}  | unreadable: {bad_p}")
print("-" * 70)

----------------------------------------------------------------------
Sampling thumbnails from each domain for a quick visual sanity check.

Plotting 'Monet Sample Thumbnails':   0%|          | 0/49 [00:00<?, ?it/s]

Plotting 'Photo Sample Thumbnails':   0%|          | 0/49 [00:00<?, ?it/s]

Monet displayed: 49  | unreadable: 0
Photo displayed: 49  | unreadable: 0
----------------------------------------------------------------------

 

3.6 Duplicate Image Analysis

Check for data redundancy by computing the MD5 hash of every JPEG file. This analysis identifies identical images that exist within the same domain and also any images that are exact duplicates across both the Monet and Photo datasets.

 

---

# --- Setup and File Gathering ---
print("-" * 70)
print("Duplicate check: hashing JPEGs (MD5), reporting duplicate groups and overlaps.")

# Verify that the necessary path variables are defined.
if 'MONET_JPG_DIR' not in locals() or 'PHOTO_JPG_DIR' not in locals():
    raise RuntimeError("Expected MONET_JPG_DIR and PHOTO_JPG_DIR to be defined in 3.1.")

# Get lists of all JPEG file paths.
monet_paths = sorted(MONET_JPG_DIR.glob("*.jpg"))
photo_paths = sorted(PHOTO_JPG_DIR.glob("*.jpg"))

# --- Helper Functions ---

# Define a function to compute the MD5 hash of a file efficiently.
def md5_of_file(p: Path, chunk_mb: int = 1) -> str:
    h = hashlib.md5()
    with open(p, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024 * chunk_mb), b""):
            h.update(chunk)
    return h.hexdigest()

# Define a wrapper to hash a list of files with a progress bar.
def hash_many(paths):
    hashes = []
    for p in tqdm(paths, desc="Hashing", mininterval=0.2):
        try:
            hashes.append(md5_of_file(p))
        except Exception:
            hashes.append(None) # Mark unreadable files as None.
    return hashes

# --- Hashing and Analysis ---

# Compute the MD5 hashes for all images in both domains.
monet_md5 = hash_many(monet_paths)
photo_md5 = hash_many(photo_paths)

# Create pandas DataFrames to hold the file info and hashes.
df_m = pd.DataFrame({
    "domain": "monet",
    "path":   [str(p) for p in monet_paths],
    "fname":  [p.name for p in monet_paths],
    "md5":    monet_md5,
})
df_p = pd.DataFrame({
    "domain": "photo",
    "path":   [str(p) for p in photo_paths],
    "fname":  [p.name for p in photo_paths],
    "md5":    photo_md5,
})
df = pd.concat([df_m, df_p], ignore_index=True)

# Identify intra-domain duplicates (images that appear more than once in the same set).
dup_m = df_m.groupby("md5", dropna=True)["fname"].size()
dup_p = df_p.groupby("md5", dropna=True)["fname"].size()
groups_m = int((dup_m > 1).sum())
groups_p = int((dup_p > 1).sum())
files_m  = int(dup_m[dup_m > 1].sum())
files_p  = int(dup_p[dup_p > 1].sum())

# Identify cross-domain duplicates (images that appear in both sets).
overlap_md5 = set(df_m["md5"].dropna()) & set(df_p["md5"].dropna())
n_overlap   = len(overlap_md5)

# --- Display Results ---

# Build a summary DataFrame.
summary = pd.DataFrame({
    "Domain": ["Monet (JPEG)", "Photo (JPEG)", "Cross-domain"],
    "Files":  [len(df_m), len(df_p), len(df_m) + len(df_p)],
    "Unique MD5": [
        int(df_m["md5"].nunique(dropna=True)),
        int(df_p["md5"].nunique(dropna=True)),
        int(df["md5"].nunique(dropna=True))
    ],
    "Duplicate Groups": [groups_m, groups_p, n_overlap],
    "Files in Duplicate Groups": [files_m, files_p, "-"],
}).set_index("Domain")

# Apply custom styling to the summary table.
styler = summary.style
styler = styler.set_properties(**{
    "background-color": "#0b0b0b",
    "color": "#F8F8F8",
    "border-color": "#1A6A1A",
})
styler = styler.set_table_styles([
    {"selector": "th", "props": [("background-color", "#DA291C"), ("color", "white"), ("text-align", "left")]},
    {"selector": "td", "props": [("border", "3px solid #0033A0")]},
])
display(styler)

# Define a helper to display the filenames of duplicate image groups.
def show_top_dups(df_domain, label, topk=10):
    vc = df_domain.groupby("md5")["fname"].size().sort_values(ascending=False)
    vc = vc[vc > 1].head(topk)
    if len(vc) == 0:
        print(f"No {label} duplicates.")
        return
    print(f"\nTop {min(topk, len(vc))} {label} duplicate groups (size):")
    for md5, sz in vc.items():
        files = df_domain.loc[df_domain["md5"] == md5, "fname"].tolist()
        file_examples = f"{files[:3]}{' ...' if len(files) > 3 else ''}"
        print(f"  size={sz}  md5={md5[:10]}…  e.g. {file_examples}")

# Display details about the found duplicates.
print("\n--- Details ---")
show_top_dups(df_m, "Monet")
show_top_dups(df_p, "Photo")

# If cross-domain duplicates exist, show examples.
if n_overlap > 0:
    print(f"\nCross-domain identical files: {n_overlap} md5 matches (showing up to 3 examples)")
    for md5 in list(overlap_md5)[:3]:
        m_files = df_m.loc[df_m["md5"] == md5, "fname"].tolist()
        p_files = df_p.loc[df_p["md5"] == md5, "fname"].tolist()
        print(f"  md5={md5[:10]}…  Monet e.g. {m_files[:2]} | Photo e.g. {p_files[:2]}")

print("\nDuplicate check complete.")
print("-" * 70)

----------------------------------------------------------------------
Duplicate check: hashing JPEGs (MD5), reporting duplicate groups and overlaps.

Hashing:   0%|          | 0/300 [00:00<?, ?it/s]

Hashing:   0%|          | 0/7038 [00:00<?, ?it/s]

	Files	Unique MD5	Duplicate Groups	Files in Duplicate Groups
Domain
Monet (JPEG)	300	300	0	0
Photo (JPEG)	7038	7028	9	19
Cross-domain	7338	7328	0	-

--- Details ---
No Monet duplicates.

Top 9 Photo duplicate groups (size):
  size=3  md5=6ef7c63e0d…  e.g. ['16a666edff.jpg', '1dafe2cf42.jpg', 'b76490cea4.jpg']
  size=2  md5=f3599c0869…  e.g. ['52f641c328.jpg', '8b554cd034.jpg']
  size=2  md5=6d2337aeef…  e.g. ['4b497dc591.jpg', '7d4e10f395.jpg']
  size=2  md5=907e199a17…  e.g. ['1d7d8ff0ac.jpg', '643916e05b.jpg']
  size=2  md5=9a0b9dc872…  e.g. ['379b25d4d3.jpg', 'acbede97b1.jpg']
  size=2  md5=59bfabe0a9…  e.g. ['45f63c2653.jpg', '9c4dd0a48f.jpg']
  size=2  md5=471354085f…  e.g. ['2960920920.jpg', 'df2c0d53a4.jpg']
  size=2  md5=41f9162ed2…  e.g. ['2641b8175f.jpg', '2bb040da92.jpg']
  size=2  md5=f493a9129d…  e.g. ['2cf4cac9df.jpg', '880e0f6c15.jpg']

Duplicate check complete.
----------------------------------------------------------------------

 

3.7 Build & Export De-duplicated File Lists

 

This step uses the MD5 hashes to create a clean, de-duplicated list of photo JPEGs, keeping only the first-seen file from each duplicate group. For reproducibility, it saves two CSVs (photo_jpg_keep.csv and photo_jpg_duplicates.csv) to both Linux and Windows paths.

 

---

# --- Hashing and De-duplication ---

# Define a file hashing function.
def md5_of_file(p, buf=1024*1024):
    h = hashlib.md5()
    with open(p, "rb") as f:
        for chunk in iter(lambda: f.read(buf), b""):
            h.update(chunk)
    return h.hexdigest()

# Get a list of all photo JPEG files.
photo_files = sorted(PHOTO_JPG_DIR.glob("*.jpg"))
md5_to_paths = defaultdict(list)

# Group file paths by their MD5 hash.
for p in tqdm(photo_files, desc="MD5(photo_jpg)", mininterval=0.3):
    try:
        md5_to_paths[md5_of_file(p)].append(p)
    except Exception:
        pass # Skip unreadable files.

# Create the de-duplicated list by keeping the first path from each hash group.
keep_paths, drop_paths = [], []
for paths in md5_to_paths.values():
    keep_paths.append(paths[0])
    if len(paths) > 1:
        drop_paths.extend(paths[1:])

# Create a global variable containing the canonical list of unique photo paths.
PHOTO_JPG_UNIQ = [str(p) for p in keep_paths]

# --- Save Results to Linux Path ---

# Create and configure the output directory for metadata files.
meta_dir = Path.home() / "projects/monet-gan" / "meta"
meta_dir.mkdir(parents=True, exist_ok=True)

# Save the lists of "keep" and "drop" paths to CSV files for reproducibility.
df_keep = pd.DataFrame({"keep": [str(p) for p in keep_paths]})
df_drop = pd.DataFrame({"drop": [str(p) for p in drop_paths]})
df_keep.to_csv(meta_dir / "photo_jpg_keep.csv", index=False)
df_drop.to_csv(meta_dir / "photo_jpg_duplicates.csv", index=False)

# Print a summary of the de-duplication process.
print("-" * 70)
print(f"photo_jpg files total : {len(photo_files):,}")
print(f"unique by MD5         : {len(PHOTO_JPG_UNIQ):,}")
print(f"duplicates to drop    : {len(drop_paths):,}  (kept one per group)")
print(f"Lists saved in        : {meta_dir}")
print("-" * 70)

# --- Save Results to Windows Path ---

# Define the source and destination paths.
meta_dir_lin = Path.home() / "projects/monet-gan" / "meta"
meta_dir_win = Path("/mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/meta")

# Ensure the Windows directory exists.
meta_dir_win.mkdir(parents=True, exist_ok=True)

# Write copies of the CSV files to the Windows path.
df_keep.to_csv(meta_dir_win / "photo_jpg_keep.csv", index=False)
df_drop.to_csv(meta_dir_win / "photo_jpg_duplicates.csv", index=False)

print("Saved duplicate lists to Windows:")
print(meta_dir_win / "photo_jpg_keep.csv")
print(meta_dir_win / "photo_jpg_duplicates.csv")
print("-" * 70)

MD5(photo_jpg):   0%|          | 0/7038 [00:00<?, ?it/s]

----------------------------------------------------------------------
photo_jpg files total : 7,038
unique by MD5         : 7,028
duplicates to drop    : 10  (kept one per group)
Lists saved in        : /home/treinart/projects/monet-gan/meta
----------------------------------------------------------------------
Saved duplicate lists to Windows:
/mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/meta/photo_jpg_keep.csv
/mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/meta/photo_jpg_duplicates.csv
----------------------------------------------------------------------

Observation: Data Integrity Audit

The comprehensive audit of the dataset confirms that it is remarkably clean, well-structured, and highly suitable for the style transfer task. All files are present and accounted for, the image dimensions are perfectly consistent, and the TFRecord shards are verified to be reliable. The audit also identified and addressed a minor data redundancy issue, resulting in a clean, canonical dataset ready for the modeling pipeline.

• Dataset Structure and Completeness:

  • The file structure is validated, with all paths and folders present as expected. The dataset is heavily imbalanced, containing 300 Monet JPEGs and 7,038 Photo JPEGs.
  • Both JPEG folders and TFRecord shards match their expected counts, confirming that no data was lost during setup.

• Image Consistency and Quality:

  • The data quality is exceptionally high. A statistical audit of both JPEG and TFRecord samples found zero corrupt or unreadable files.
  • Critically, all sampled images from both domains were found to have a consistent 256×256 resolution. This uniformity is a significant advantage, as it eliminates the need for complex resizing or padding logic in the data preprocessing pipeline.

• Visual Domain Characteristics:

  • The visual audit via thumbnail grids confirms a distinct and pronounced stylistic gap between the two domains.
  • The Monet set consistently displays the hallmarks of Impressionism: visible brushstrokes, a focus on light and color over sharp detail, and recurring subjects like haystacks and water lilies.
  • The Photo set consists of modern, high-quality landscape photographs characterized by sharp focus, realistic colors, and a wide variety of subjects and compositions. This clear visual dichotomy is ideal for a style transfer model.

• Data Uniqueness (Key Finding):

  • An MD5 hash analysis confirmed the integrity of the datasets. The Monet collection contains zero duplicate images.
  • The Photo collection was found to have a minor redundancy issue, with 10 duplicate files across 9 unique image groups. There were no duplicates found across the two domains.
  • This issue was resolved by creating and exporting a de-duplicated list of 7,028 unique photo files, ensuring that the model will train on a clean and unique set of images.

Conclusion: The data audit provides high confidence in the dataset's quality and structure. The primary challenges for this project will not stem from data cleaning but from the significant class imbalance and the inherent difficulty of the image translation task. With all files verified, sized correctly, and de-duplicated, the project can proceed to building the tf.data input pipeline with a solid foundation.

 

Section 4: Exploratory Visual Analysis

 

Before building the model, it's crucial to understand the visual characteristics that define and separate the two domains. This section uses a series of plots and statistical measures to compare the Monet and Photo datasets across dimensions like color, brightness, and texture.

 

Section Plan:

1. Side-by-Side Comparison: Display random pairs of Monet and Photo images to get an intuitive feel for the domain gap.
2. Color Distribution: Plot RGB histograms to compare the color palettes of the two sets.
3. Mean Images: Compute and display the "average" image for each domain to visualize the central tendency of color and composition.
4. Luminance & Contrast: Analyze and plot the brightness distributions to see if one domain is systematically darker or has lower contrast.
5. Edge & Texture Analysis: Use Sobel and Laplacian operators to quantify and compare the edge density and sharpness between paintings and photos.
6. Hue & Saturation: Plot HSV distributions to compare the dominant colors and their intensity.
7. Outlier Detection: Scan for images with anomalous properties like extreme file size or brightness to ensure data quality.

 

---

4.1 Side-by-Side Rows

Paired rows (Monet vs. Photo) highlight differences in texture and palette. Pairs are random and not semantically matched.

---

# Set the number of image pairs to display.
rng = np.random.default_rng(SEED)
n_pairs = 24

# Create deterministic random samples from both domains.
m_take = min(n_pairs, len(monet_jpgs))
p_take = min(n_pairs, len(photo_jpgs))
m_sample = rng.choice(monet_jpgs, size=m_take, replace=False).tolist()
p_sample = rng.choice(photo_jpgs, size=p_take, replace=False).tolist()

# Create the figure and axes for the side-by-side plots.
rows = max(m_take, p_take)
fig, axes = plt.subplots(rows, 2, figsize=(8, 3 * rows), facecolor="black")

# Ensure the axes object is always a 2D array for consistent indexing.
axes = np.atleast_2d(axes)

# --- MODIFICATION: Loop with tqdm ---
# Loop through each row to plot a pair of images, with a progress bar.
pbar = tqdm(range(rows), desc="Generating image pairs", mininterval=0.2)
for i in pbar:
    # Loop through the two domains (Monet and Photo) for each column.
    for j, (paths, label, color) in enumerate([(m_sample, "Monet", "#1A6A1A"), (p_sample, "Photo", "#FFB612")]):
        ax = axes[i, j]
        # Clear ticks and set background color.
        ax.set_xticks([])
        ax.set_yticks([])
        ax.set_facecolor("black")

        # Load and display the image if one exists for the current row.
        if i < len(paths):
            try:
                with Image.open(paths[i]) as im:
                    ax.imshow(im.convert("RGB"))
            except Exception:
                # Handle cases where an image is unreadable.
                ax.text(0.5, 0.5, "unreadable", ha="center", va="center", color="red")
        
        # Add a title to the top of each column on the first row.
        if i == 0:
            ax.set_title(label, color=color, fontsize=22, fontweight="bold")
            
        # Style the plot's border (spines).
        for spine in ax.spines.values():
            spine.set_edgecolor(color)
            spine.set_linewidth(2)

# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

Generating image pairs:   0%|          | 0/24 [00:00<?, ?it/s]

 

4.2 Color Distributions (RGB)

 

This plot visualizes the color palettes of the two domains by sampling thousands of pixels and plotting their Red, Green, and Blue intensity distributions on overlaid histograms.

 

---

# Define a function to load images and stack their pixel values into a single array.
def _stack_pixels(paths, max_imgs=300):
    vals = []
    for p in tqdm(paths[:max_imgs], desc="stack", leave=True, mininterval=0.2):
        try:
            with Image.open(p) as im:
                arr = np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
                vals.append(arr.reshape(-1, 3))
        except Exception:
            pass
    if not vals:
        return None
    return np.concatenate(vals, axis=0)

# Create deterministic random samples of image paths for the analysis.
m_paths = np.random.default_rng(SEED).choice(monet_jpgs, size=min(N_STATS_MONET, len(monet_jpgs)), replace=False)
p_paths = np.random.default_rng(SEED + 1).choice(photo_jpgs, size=min(N_STATS_PHOTO, len(photo_jpgs)), replace=False)

# Load the pixel data for each domain.
m_pix = _stack_pixels(m_paths, max_imgs=len(m_paths))
p_pix = _stack_pixels(p_paths, max_imgs=len(p_paths))

# --- Plot the Histograms ---

# Create the figure and axes.
fig, axes = plt.subplots(1, 2, figsize=(20, 8), facecolor="black")

# Set the background color for both subplots.
for ax in axes:
    ax.set_facecolor("black")

# Define the bins for the histograms.
bins = np.linspace(0, 1, 41)

# Plot the histogram for the Monet domain.
if m_pix is not None:
    ax = axes[0]
    # Plot R, G, and B channels as separate histograms.
    for label, idx, color in [("R", 0, "red"), ("G", 1, "green"), ("B", 2, "blue")]:
        ax.hist(m_pix[:, idx], bins=bins, alpha=0.55, label=label, edgecolor="none", color=color)
    ax.set_title("Monet RGB Histograms", color="#1A6A1A", fontsize=22, fontweight="bold")
    ax.tick_params(colors="white")
    ax.legend(frameon=False, fontsize=16, labelcolor="white")

# Plot the histogram for the Photo domain.
if p_pix is not None:
    ax = axes[1]
    # Plot R, G, and B channels as separate histograms.
    for label, idx, color in [("R", 0, "red"), ("G", 1, "green"), ("B", 2, "blue")]:
        ax.hist(p_pix[:, idx], bins=bins, alpha=0.55, label=label, edgecolor="none", color=color)
    ax.set_title("Photo RGB Histograms", color="#FFB612", fontsize=22, fontweight="bold")
    ax.tick_params(colors="white")
    ax.legend(frameon=False, fontsize=16, labelcolor="white")

# Apply final styling to both subplots.
for ax in axes:
    # Style the plot's border (spines).
    for spine in ax.spines.values():
        spine.set_edgecolor("#444")
        spine.set_linewidth(1.5)

# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

# Clean up large variables to free memory.
del m_pix, p_pix

stack:   0%|          | 0/300 [00:00<?, ?it/s]

stack:   0%|          | 0/1200 [00:00<?, ?it/s]

 

4.3 Mean Images

 

A quick “average” image per domain, computed by averaging the pixel values across all images in the set. This helps compress the overall color and compositional tendencies into a single, blurry frame.

 

---

# Set the number of images to use for the mean calculation.
N_MEAN_MONET = 300
N_MEAN_PHOTO = 2000

# Define a function to compute the mean image from a list of paths.
def _mean_image(paths):
    acc = None
    n = 0
    for p in tqdm(paths, desc="mean", leave=True, mininterval=0.2):
        try:
            with Image.open(p) as im:
                arr = np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
                if acc is None:
                    acc = np.zeros_like(arr, dtype=np.float64)
                # Skip images with mismatched shapes (a safeguard).
                if acc.shape != arr.shape:
                    continue
                acc += arr
                n += 1
        except Exception:
            pass
    if n == 0:
        return None
    return (acc / n).clip(0, 1)

# Use all available JPEGs for the calculation.
m_paths = monet_jpgs
p_paths = photo_jpgs

# Calculate the mean image for each domain.
m_mean = _mean_image(m_paths)
p_mean = _mean_image(p_paths)

# --- Plot the Mean Images ---

# Create the figure and axes.
fig, axes = plt.subplots(1, 2, figsize=(20, 8), facecolor="black")

# Set the titles for each subplot.
axes[0].set_title("Monet Mean", color="#1A6A1A", fontsize=24, fontweight="bold")
axes[1].set_title("Photo Mean", color="#FFB612", fontsize=24, fontweight="bold")

# Loop through the axes to display each image and apply styling.
for ax, img in zip(axes, [m_mean, p_mean]):
    ax.set_xticks([])
    ax.set_yticks([])
    ax.set_facecolor("black")
    
    # Display the image if it was successfully calculated.
    if img is not None:
        ax.imshow(np.clip(img, 0, 1))
    else:
        # Handle cases where the mean image could not be created.
        ax.text(0.5, 0.5, "N/A", ha="center", va="center", color="red", fontsize=14)
    
    # Style the plot's border (spines).
    for spine in ax.spines.values():
        spine.set_edgecolor("#444")
        spine.set_linewidth(1.5)
        
# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

# Clean up variables to free memory.
del m_mean, p_mean

mean:   0%|          | 0/300 [00:00<?, ?it/s]

mean:   0%|          | 0/7038 [00:00<?, ?it/s]

4.4 Luminance & Contrast Distribution

Compare brightness/contrast between domains by plotting luminance histograms (Y from RGB→YUV or V from HSV) and a simple CDF overlay. This quickly shows if one domain is systematically darker or lower-contrast.

---

# Use the de-duplicated list of photo JPEGs.
monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = PHOTO_JPG_UNIQ if 'PHOTO_JPG_UNIQ' in globals() else sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

# Set the sample sizes for the luminance analysis.
N_LUMA_MONET = min(300, len(monet_jpgs))
N_LUMA_PHOTO = min(1200, len(photo_jpgs))

# Create deterministic random samples.
rng = np.random.default_rng(SEED)
monet_sample = rng.choice(monet_jpgs, size=N_LUMA_MONET, replace=False) if N_LUMA_MONET else []
photo_sample = rng.choice(photo_jpgs, size=N_LUMA_PHOTO, replace=False) if N_LUMA_PHOTO else []

# Initialize arrays to accumulate histogram data efficiently.
BINS = 32
edges = np.linspace(0.0, 1.0, BINS + 1)
hist_m, hist_p = np.zeros(BINS, dtype=np.float64), np.zeros(BINS, dtype=np.float64)
means_m, stds_m = [], []
means_p, stds_p = [], []

# Define a function to calculate luminance (Y) from an image path.
def luminance_from_path(p):
    with Image.open(p) as im:
        arr = np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
    y = 0.299 * arr[..., 0] + 0.587 * arr[..., 1] + 0.114 * arr[..., 2]
    return y

# Process the Monet sample to collect luminance data.
for p in tqdm(monet_sample, desc="Luma(Monet)", mininterval=0.2):
    try:
        y = luminance_from_path(p)
        h, _ = np.histogram(y, bins=edges)
        hist_m += h
        means_m.append(float(y.mean()))
        stds_m.append(float(y.std()))
    except Exception:
        pass

# Process the Photo sample to collect luminance data.
for p in tqdm(photo_sample, desc="Luma(Photo)", mininterval=0.2):
    try:
        y = luminance_from_path(p)
        h, _ = np.histogram(y, bins=edges)
        hist_p += h
        means_p.append(float(y.mean()))
        stds_p.append(float(y.std()))
    except Exception:
        pass

# Normalize the histograms to create probability density functions.
dens_m = hist_m / (hist_m.sum() + 1e-9)
dens_p = hist_p / (hist_p.sum() + 1e-9)
centers = 0.5 * (edges[1:] + edges[:-1])

# --- Plot Luminance Histogram ---
print("-" * 70)
print("Luminance/contrast via Y = 0.299R + 0.587G + 0.114B (0..1), hist + CDF per domain.")

# Create the figure and axes.
fig, ax = plt.subplots(figsize=(20, 8), facecolor="black")
fig.patch.set_facecolor("black")
# Plot the density curves for both domains.
ax.plot(centers, dens_m, label="Monet", linewidth=2.5, color="#1A6A1A")
ax.plot(centers, dens_p, label="Photo", linewidth=2.5, color="#FFB612")
# Set the plot title and labels.
ax.set_title("Luminance Histogram (Y, 0..1)", fontsize=28, color="white", pad=8, fontweight="bold")
ax.set_xlabel("Luminance (Y)", fontsize=18, color="white")
ax.set_ylabel("Density", fontsize=18, color="white")
# Style the ticks, spines, and legend.
ax.tick_params(colors="white")
for s in ax.spines.values():
    s.set_edgecolor("#1A6A1A")
    s.set_linewidth(2)
ax.set_facecolor("black")
ax.legend(frameon=False, fontsize=18, labelcolor="white")
# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

# --- Plot Luminance CDF ---

# Create the figure and axes.
fig, ax = plt.subplots(figsize=(20, 8), facecolor="black")
fig.patch.set_facecolor("black")
# Plot the CDF curves for both domains.
ax.plot(centers, np.cumsum(dens_m), label="Monet", linewidth=2.5, color="#1A6A1A")
ax.plot(centers, np.cumsum(dens_p), label="Photo", linewidth=2.5, color="#FFB612")
# Set the plot title and labels.
ax.set_title("Luminance CDF", fontsize=28, color="white", pad=8, fontweight="bold")
ax.set_xlabel("Luminance (Y)", fontsize=18, color="white")
ax.set_ylabel("CDF", fontsize=18, color="white")
# Style the ticks, spines, and legend.
ax.tick_params(colors="white")
for s in ax.spines.values():
    s.set_edgecolor("#1A6A1A")
    s.set_linewidth(2)
ax.set_facecolor("black")
ax.legend(frameon=False, fontsize=18, labelcolor="white")
# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

# --- Print Text Summary ---
print("\n--- Summary (per-image means/std over samples) ---")
print(f"Monet : n={len(means_m)}  mean(Y)={np.mean(means_m):.4f}  std(Y)={np.mean(stds_m):.4f}")
print(f"Photo : n={len(means_p)}  mean(Y)={np.mean(means_p):.4f}  std(Y)={np.mean(stds_p):.4f}")
print("-" * 70)

Luma(Monet):   0%|          | 0/300 [00:00<?, ?it/s]

Luma(Photo):   0%|          | 0/1200 [00:00<?, ?it/s]

----------------------------------------------------------------------
Luminance/contrast via Y = 0.299R + 0.587G + 0.114B (0..1), hist + CDF per domain.

--- Summary (per-image means/std over samples) ---
Monet : n=300  mean(Y)=0.5181  std(Y)=0.1806
Photo : n=1200  mean(Y)=0.4117  std(Y)=0.2063
----------------------------------------------------------------------

4.5 Edge Density & Texture

Use Sobel/Laplacian to estimate edge density and variance-of-Laplacian as a sharpness proxy. Summaries (mean/median) per domain help reveal texture complexity differences that CycleGANs often latch onto.

---

# Use the de-duplicated list of photo JPEGs.
monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = PHOTO_JPG_UNIQ if 'PHOTO_JPG_UNIQ' in globals() else sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

# Set the sample sizes for the analysis.
N_EDGE_MONET = min(200, len(monet_jpgs))
N_EDGE_PHOTO = min(600, len(photo_jpgs))

# Create deterministic random samples.
rng = np.random.default_rng(SEED)
monet_sample = rng.choice(monet_jpgs, size=N_EDGE_MONET, replace=False) if N_EDGE_MONET else []
photo_sample = rng.choice(photo_jpgs, size=N_EDGE_PHOTO, replace=False) if N_EDGE_PHOTO else []

# Define a helper to load an image as a TensorFlow tensor.
def pil_to_tensor01(p):
    with Image.open(p) as im:
        arr = np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
    return tf.convert_to_tensor(arr[None, ...])

# Define the Laplacian kernel for sharpness calculation.
LAP_KERNEL = tf.constant([[0., 1., 0.], [1., -4., 1.], [0., 1., 0.]], dtype=tf.float32)
LAP_KERNEL = tf.reshape(LAP_KERNEL, [3, 3, 1, 1])

# Define the main function to calculate edge and sharpness metrics for one image.
def edge_and_sharpness(p):
    x = pil_to_tensor01(p)
    # Calculate Sobel edge magnitude.
    sob = tf.image.sobel_edges(x)
    dy, dx = sob[..., 0], sob[..., 1]
    mag = tf.sqrt(tf.maximum(dy**2 + dx**2, 1e-12))
    edge_mean = float(tf.reduce_mean(mag).numpy())
    # Calculate variance of the Laplacian as a proxy for sharpness.
    x_gray = tf.image.rgb_to_grayscale(x)
    lap = tf.nn.conv2d(x_gray, LAP_KERNEL, strides=1, padding="SAME")
    sharp = float(tf.math.reduce_variance(lap).numpy())
    return edge_mean, sharp

# Define a wrapper to process a list of paths and collect metrics.
def collect_metrics(paths, desc):
    edges, sharps = [], []
    for p in tqdm(paths, desc=desc, mininterval=0.2):
        try:
            e, s = edge_and_sharpness(p)
            edges.append(e)
            sharps.append(s)
        except Exception:
            pass
    return np.array(edges, dtype=np.float32), np.array(sharps, dtype=np.float32)

# Run the analysis on both samples.
edges_m, sharp_m = collect_metrics(monet_sample, "Sobel/Laplacian (Monet)")
edges_p, sharp_p = collect_metrics(photo_sample, "Sobel/Laplacian (Photo)")

# --- Print Summary Statistics ---
print("-" * 70)
print("Edge density via Sobel gradient magnitude; texture via variance of Laplacian (per-image).")

# Define a helper to calculate summary statistics.
def _summ(a): 
    return (float(np.mean(a)), float(np.median(a)), float(np.std(a)))

# Calculate and print summary statistics for edge magnitude.
m_e_mean, m_e_med, m_e_std = _summ(edges_m) if len(edges_m) > 0 else (np.nan, np.nan, np.nan)
p_e_mean, p_e_med, p_e_std = _summ(edges_p) if len(edges_p) > 0 else (np.nan, np.nan, np.nan)
print("\n--- Edge magnitude (mean per image) ---")
print(f"Monet : mean={m_e_mean:.5f}  median={m_e_med:.5f}  std={m_e_std:.5f}  n={len(edges_m)}")
print(f"Photo : mean={p_e_mean:.5f}  median={p_e_med:.5f}  std={p_e_std:.5f}  n={len(edges_p)}")

# Calculate and print summary statistics for sharpness.
m_s_mean, m_s_med, m_s_std = _summ(sharp_m) if len(sharp_m) > 0 else (np.nan, np.nan, np.nan)
p_s_mean, p_s_med, p_s_std = _summ(sharp_p) if len(sharp_p) > 0 else (np.nan, np.nan, np.nan)
print("\n--- Variance of Laplacian (sharpness proxy) ---")
print(f"Monet : mean={m_s_mean:.5e}  median={m_s_med:.5e}  std={m_s_std:.5e}  n={len(sharp_m)}")
print(f"Photo : mean={p_s_mean:.5e}  median={p_s_med:.5e}  std={p_s_std:.5e}  n={len(sharp_p)}")

# --- Plot Edge Distribution ---

# Create the figure and axes.
fig, ax = plt.subplots(figsize=(20, 8), facecolor="black")
fig.patch.set_facecolor("black")
# Plot the histograms of edge magnitudes for both domains.
ax.hist(edges_m, bins=30, alpha=0.6, label="Monet", color="#1A6A1A", edgecolor="none")
ax.hist(edges_p, bins=30, alpha=0.6, label="Photo", color="#FFB612", edgecolor="none")
# Set the plot title and labels.
ax.set_title("Edge Magnitude Distribution (Sobel)", fontsize=28, color="white", pad=8, fontweight="bold")
ax.set_xlabel("Mean edge magnitude per image", fontsize=18, color="white")
ax.set_ylabel("Count", fontsize=18, color="white")
# Style the ticks, spines, and legend.
ax.tick_params(colors="white")
for s in ax.spines.values():
    s.set_edgecolor("#1A6A1A")
    s.set_linewidth(2)
ax.set_facecolor("black")
ax.legend(frameon=False, fontsize=18, labelcolor="white")
# Ensure the layout is tight and clean.
plt.tight_layout()
plt.show()

print("-" * 70)

Sobel/Laplacian (Monet):   0%|          | 0/200 [00:00<?, ?it/s]

Sobel/Laplacian (Photo):   0%|          | 0/600 [00:00<?, ?it/s]

----------------------------------------------------------------------
Edge density via Sobel gradient magnitude; texture via variance of Laplacian (per-image).

--- Edge magnitude (mean per image) ---
Monet : mean=0.35083  median=0.33598  std=0.11679  n=200
Photo : mean=0.29865  median=0.28273  std=0.13953  n=600

--- Variance of Laplacian (sharpness proxy) ---
Monet : mean=3.79239e-02  median=3.14947e-02  std=2.63313e-02  n=200
Photo : mean=4.59290e-02  median=3.30088e-02  std=4.37763e-02  n=600

----------------------------------------------------------------------

 

4.6 Hue & Saturation Distributions

 

Summarize color usage with HSV by plotting separate histograms for Hue (the type of color) and Saturation (the intensity of color). This gives a compact, visual readout of the color gap between the two domains.

 

---

# Use the de-duplicated list of photo JPEGs.
monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = PHOTO_JPG_UNIQ if 'PHOTO_JPG_UNIQ' in globals() else sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

# Set sampling parameters.
N_HSV_MONET = min(300, len(monet_jpgs))
N_HSV_PHOTO = min(1200, len(photo_jpgs))
PIXELS_PER_IMAGE = 4096

# Create deterministic random samples.
rng = np.random.default_rng(SEED)
monet_sample = rng.choice(monet_jpgs, size=N_HSV_MONET, replace=False) if N_HSV_MONET else []
photo_sample = rng.choice(photo_jpgs, size=N_HSV_PHOTO, replace=False) if N_HSV_PHOTO else []

# Define a function to get a sample of normalized HSV pixels from an image.
def hsv_sample_from_path(p, k=PIXELS_PER_IMAGE):
    with Image.open(p) as im:
        hsv = np.asarray(im.convert("HSV"), dtype=np.uint8)
    H, S = hsv[..., 0].reshape(-1), hsv[..., 1].reshape(-1)
    n = H.size
    if n == 0:
        return None, None
    idx = rng.choice(n, size=min(k, n), replace=False)
    # Normalize Hue to [0, 360] and Saturation to [0, 1].
    h = (H[idx].astype(np.float32) / 255.0) * 360.0
    s = S[idx].astype(np.float32) / 255.0
    return h, s

# Define a wrapper to process a list of paths and collect HSV data.
def collect_hsv(paths, desc):
    all_h, all_s = [], []
    for p in tqdm(paths, desc=desc, mininterval=0.2):
        try:
            h, s = hsv_sample_from_path(p)
            if h is not None:
                all_h.append(h)
                all_s.append(s)
        except Exception:
            pass
    if all_h:
        return np.concatenate(all_h), np.concatenate(all_s)
    return np.array([]), np.array([])

# Run the analysis on both samples.
h_m, s_m = collect_hsv(monet_sample, "HSV(Monet)")
h_p, s_p = collect_hsv(photo_sample, "HSV(Photo)")

# --- Plot Hue Distribution ---
print("-" * 70)
print("Hue histogram (0..360°) and Saturation histogram (0..1), Monet vs Photo.")

# Calculate the probability densities for the hue distributions.
H_BINS = 36
H_edges = np.linspace(0, 360, H_BINS + 1)
dens_h_m, _ = np.histogram(h_m, bins=H_edges, density=True)
dens_h_p, _ = np.histogram(h_p, bins=H_edges, density=True)
H_centers = 0.5 * (H_edges[1:] + H_edges[:-1])

# Create the figure and axes.
fig, ax = plt.subplots(figsize=(20, 8), facecolor="black")
fig.patch.set_facecolor("black")
# Plot the density curves.
ax.plot(H_centers, dens_h_m, label="Monet", linewidth=2.5, color="#1A6A1A")
ax.plot(H_centers, dens_h_p, label="Photo", linewidth=2.5, color="#FFB612")
# Set the plot title and labels.
ax.set_title("Hue Distribution (degrees)", fontsize=28, color="white", pad=8, fontweight="bold")
ax.set_xlabel("Hue (°)", fontsize=18, color="white")
ax.set_ylabel("Density", fontsize=18, color="white")
# Style the ticks, spines, and legend.
ax.tick_params(colors="white")
for s in ax.spines.values():
    s.set_edgecolor("#1A6A1A")
    s.set_linewidth(2)
ax.set_facecolor("black")
ax.legend(frameon=False, fontsize=18, labelcolor="white")
# Ensure the layout is tight and clean.
plt.tight_layout()
# Display the plot.
plt.show()

# --- Plot Saturation Distribution ---

# Calculate the probability densities for the saturation distributions.
S_BINS = 32
S_edges = np.linspace(0, 1, S_BINS + 1)
dens_s_m, _ = np.histogram(s_m, bins=S_edges, density=True)
dens_s_p, _ = np.histogram(s_p, bins=S_edges, density=True)
S_centers = 0.5 * (S_edges[1:] + S_edges[:-1])

# Create the figure and axes.
fig, ax = plt.subplots(figsize=(20, 8), facecolor="black")
fig.patch.set_facecolor("black")
# Plot the density curves.
ax.plot(S_centers, dens_s_m, label="Monet", linewidth=2.5, color="#1A6A1A")
ax.plot(S_centers, dens_s_p, label="Photo", linewidth=2.5, color="#FFB612")
# Set the plot title and labels.
ax.set_title("Saturation Distribution (0..1)", fontsize=28, color="white", pad=8, fontweight="bold")
ax.set_xlabel("Saturation", fontsize=18, color="white")
ax.set_ylabel("Density", fontsize=18, color="white")
# Style the ticks, spines, and legend.
ax.tick_params(colors="white")
for s in ax.spines.values():
    s.set_edgecolor("#1A6A1A")
    s.set_linewidth(2)
ax.set_facecolor("black")
ax.legend(frameon=False, fontsize=18, labelcolor="white")
# Ensure the layout is tight and clean.
plt.tight_layout()
# Display the plot.
plt.show()

# Print a summary of the data used.
print(f"Samples Monet images: {len(monet_sample)}, pixels: {len(h_m):,} | Photo images: {len(photo_sample)}, pixels: {len(h_p):,}")
print("-" * 70)

HSV(Monet):   0%|          | 0/300 [00:00<?, ?it/s]

HSV(Photo):   0%|          | 0/1200 [00:00<?, ?it/s]

----------------------------------------------------------------------
Hue histogram (0..360°) and Saturation histogram (0..1), Monet vs Photo.

Samples Monet images: 300, pixels: 1,228,800 | Photo images: 1200, pixels: 4,915,200
----------------------------------------------------------------------

 

4.7 Outlier & Quality Checks

 

Flag potential outliers by analyzing file sizes, extreme brightness values, and non-standard image shapes. This step prints lists of the top offending files for a final manual sanity check.

 

---

# Outlier & Quality Checks: file size, extreme brightness, odd shapes

print("-" * 70)
print("Outliers by file size, extreme luminance mean, and non-256×256 shapes.")

monet_jpgs = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
photo_jpgs = PHOTO_JPG_UNIQ if 'PHOTO_JPG_UNIQ' in globals() else sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))

def file_size(p):
    try:
        return os.path.getsize(p)
    except Exception:
        return -1

def img_shape(p):
    try:
        with Image.open(p) as im:
            im = im.convert("RGB")
            return (im.width, im.height)
    except Exception:
        return ("unreadable", "unreadable")

def luminance_mean(p):
    try:
        with Image.open(p) as im:
            arr = np.asarray(im.convert("RGB"), dtype=np.float32) / 255.0
        y = 0.299 * arr[...,0] + 0.587 * arr[...,1] + 0.114 * arr[...,2]
        return float(y.mean())
    except Exception:
        return np.nan

def summarize_outliers(paths, label, k=10):
    # Sizes
    sizes = [(p, file_size(p)) for p in paths]
    sizes = [(p, s) for p, s in sizes if s >= 0]
    small = sorted(sizes, key=lambda t: t[1])[:k]
    large = sorted(sizes, key=lambda t: -t[1])[:k]

    # Brightness
    ys = [(p, luminance_mean(p)) for p in paths]
    ys = [(p, y) for p, y in ys if np.isfinite(y)]
    dark = sorted(ys, key=lambda t: t[1])[:k]
    bright = sorted(ys, key=lambda t: -t[1])[:k]

    # Shapes
    shps = [(p, img_shape(p)) for p in paths]
    odd = [(p, s) for p, s in shps if s != (256, 256)]

    print(f"\n--- {label}: File size (smallest {k}) ---")
    for p, s in small:
        print(f"{Path(p).name:>16s}  {s/1024:.1f} KB")

    print(f"\n--- {label}: File size (largest {k}) ---")
    for p, s in large:
        print(f"{Path(p).name:>16s}  {s/1024:.1f} KB")

    print(f"\n--- {label}: Darkest {k} by mean luminance ---")
    for p, y in dark:
        print(f"{Path(p).name:>16s}  Ȳ={y:.3f}")

    print(f"\n--- {label}: Brightest {k} by mean luminance ---")
    for p, y in bright:
        print(f"{Path(p).name:>16s}  Ȳ={y:.3f}")

    print(f"\n--- {label}: Non-256×256 shapes (first {k}) ---")
    if len(odd) == 0:
        print("None - all images are 256×256 (as expected).")
    else:
        for p, s in odd[:k]:
            print(f"{Path(p).name:>16s}  {s[0]}×{s[1]}")

    # Soft checks (tune thresholds)
    tiny = [p for p, s in sizes if s < 3 * 1024]
    huge = [p for p, s in sizes if s > 60 * 1024]
    very_dark = [p for p, y in ys if y < 0.03]
    very_bright = [p for p, y in ys if y > 0.97]
    print("\n--- Quick flags (counts) ---")
    print(f"tiny files (<3KB): {len(tiny)} | huge files (>60KB): {len(huge)}")
    print(f"very dark (Ȳ<0.03): {len(very_dark)} | very bright (Ȳ>0.97): {len(very_bright)}")


summarize_outliers(monet_jpgs, "Monet")
summarize_outliers(photo_jpgs, "Photo")

print("\nOutlier scan complete.")
print("-" * 70)

----------------------------------------------------------------------
Outliers by file size, extreme luminance mean, and non-256×256 shapes.

--- Monet: File size (smallest 10) ---
  910610e827.jpg  7.4 KB
  9963d64ebf.jpg  7.6 KB
  6a03aea8be.jpg  8.3 KB
  fdf1530d95.jpg  8.4 KB
  5185e8c56a.jpg  8.9 KB
  a06b54dfe9.jpg  8.9 KB
  fb93438ff9.jpg  9.1 KB
  07fcaee35f.jpg  9.2 KB
  47a0548067.jpg  9.3 KB
  ede9769cb3.jpg  9.6 KB

--- Monet: File size (largest 10) ---
  4f7e01f097.jpg  26.2 KB
  ad0101d010.jpg  25.9 KB
  a7d53885e8.jpg  25.9 KB
  6782e7cb2a.jpg  25.8 KB
  09b76b6471.jpg  25.4 KB
  f486c1655f.jpg  24.9 KB
  8cfd45a2e2.jpg  24.2 KB
  6d0e87f557.jpg  24.0 KB
  a619072f82.jpg  23.8 KB
  e9f5563817.jpg  23.6 KB

--- Monet: Darkest 10 by mean luminance ---
  23b07c3769.jpg  Ȳ=0.249
  9ae6552353.jpg  Ȳ=0.271
  a202b1b200.jpg  Ȳ=0.277
  8ee2933868.jpg  Ȳ=0.280
  4bb4ca7b03.jpg  Ȳ=0.284
  f7836c88eb.jpg  Ȳ=0.285
  9d58456cc3.jpg  Ȳ=0.288
  16dabe418c.jpg  Ȳ=0.292
  f04c9d8e34.jpg  Ȳ=0.294
  058f878b7c.jpg  Ȳ=0.307

--- Monet: Brightest 10 by mean luminance ---
  47a0548067.jpg  Ȳ=0.781
  910610e827.jpg  Ȳ=0.770
  0260d15306.jpg  Ȳ=0.756
  68729aac07.jpg  Ȳ=0.748
  cd6623d07d.jpg  Ȳ=0.748
  cfc6fce7b5.jpg  Ȳ=0.737
  f96a8de9f3.jpg  Ȳ=0.733
  07fcaee35f.jpg  Ȳ=0.707
  fdf1530d95.jpg  Ȳ=0.706
  9f409e3376.jpg  Ȳ=0.706

--- Monet: Non-256×256 shapes (first 10) ---
None - all images are 256×256 (as expected).

--- Quick flags (counts) ---
tiny files (<3KB): 0 | huge files (>60KB): 0
very dark (Ȳ<0.03): 0 | very bright (Ȳ>0.97): 0

--- Photo: File size (smallest 10) ---
  b989430548.jpg  2.2 KB
  ac22f8134b.jpg  2.2 KB
  52bb31919e.jpg  2.4 KB
  8803b67dfb.jpg  2.6 KB
  7325a1d7d0.jpg  2.7 KB
  eedab28822.jpg  3.0 KB
  0cb4144fbb.jpg  3.0 KB
  16d8cafb5b.jpg  3.1 KB
  93bd999d11.jpg  3.2 KB
  4581a4faee.jpg  3.2 KB

--- Photo: File size (largest 10) ---
  f01164b232.jpg  33.3 KB
  21c2c68337.jpg  31.4 KB
  2339fbceb2.jpg  31.1 KB
  80cb714bef.jpg  31.1 KB
  f6084689ed.jpg  31.1 KB
  69cf4ee0c1.jpg  31.1 KB
  1355735108.jpg  30.8 KB
  c6c4b33994.jpg  30.7 KB
  ee97a20e7c.jpg  30.4 KB
  b2f7fa511e.jpg  30.3 KB

--- Photo: Darkest 10 by mean luminance ---
  b989430548.jpg  Ȳ=0.003
  b635c95844.jpg  Ȳ=0.013
  fe702a794e.jpg  Ȳ=0.026
  9d3885ad89.jpg  Ȳ=0.028
  17d4e23d9f.jpg  Ȳ=0.029
  36bafe2801.jpg  Ȳ=0.032
  70c35d97fa.jpg  Ȳ=0.035
  09fc404e31.jpg  Ȳ=0.045
  08341635fa.jpg  Ȳ=0.047
  8526c7d445.jpg  Ȳ=0.047

--- Photo: Brightest 10 by mean luminance ---
  464d4dd705.jpg  Ȳ=0.897
  3fe71e2f91.jpg  Ȳ=0.880
  54199dc11f.jpg  Ȳ=0.862
  6c1d743ded.jpg  Ȳ=0.858
  3ee7eb408c.jpg  Ȳ=0.849
  d5568c935b.jpg  Ȳ=0.842
  c851e6f6e0.jpg  Ȳ=0.829
  37cfa76bf2.jpg  Ȳ=0.829
  de86feb03d.jpg  Ȳ=0.828
  6ed8cb6c73.jpg  Ȳ=0.825

--- Photo: Non-256×256 shapes (first 10) ---
None - all images are 256×256 (as expected).

--- Quick flags (counts) ---
tiny files (<3KB): 6 | huge files (>60KB): 0
very dark (Ȳ<0.03): 5 | very bright (Ȳ>0.97): 0

Outlier scan complete.
----------------------------------------------------------------------

Observation: Visual Domain Analysis

The visual exploratory analysis reveals significant and quantifiable differences between the Monet and Photo domains. Beyond the obvious stylistic gap, the two datasets exhibit distinct statistical profiles in their color distributions, luminance, and texture. These measurable differences are the core signals the CycleGAN model will need to learn to successfully translate an image from one domain to the other.

• Color Palette and Brightness:

  • The two domains have fundamentally different color and light profiles. The Monet paintings are consistently brighter, with a mean luminance of 0.52 compared to 0.41 for the photos. The Luminance CDF plot clearly shows Monet's distribution is centered in the mid-tones, whereas the photos skew darker.
  • RGB histograms confirm this. The Photo domain shows a strong peak in the blue channel and a long tail in the dark regions, characteristic of landscape photos with sky and shadows. The Monet distribution is more compressed toward the center, reflecting a more balanced, mid-tone palette. The "mean images" provide a perfect summary: a hazy, gray-green average for Monet versus a darker, blue-over-brown average for the photos.

• Texture and Detail:

  • As expected, the domains differ greatly in texture. The photos are sharper, with a higher average variance of the Laplacian (0.046) compared to the paintings (0.038), indicating more high-frequency detail.
  • Conversely, the Monet paintings have a higher average edge magnitude (0.35 vs. 0.30 for photos). This suggests that while photos have sharper details, the paintings have a more consistent, "busy" texture across the entire canvas, which corresponds to visible brushwork. The GAN will need to learn to reduce sharp details while introducing this unique, all-over texture.

• Hue and Saturation:

  • The color usage patterns are a key differentiator. The Photo domain's hue distribution is dominated by a massive peak around 210°, the signature of blue skies and water.
  • The Monet distribution is more varied, with prominent peaks in the yellow, green, and orange ranges, which is consistent with Impressionist landscapes. His paintings also exhibit a much higher density of low-saturation colors, contributing to their softer, more muted aesthetic.

• Data Quality and Outliers:

  • The outlier analysis reinforces the findings from Section 3: the dataset is exceptionally clean. No images deviate from the standard 256×256 resolution.
  • There are no extreme outliers in the Monet set. The Photo set contains a handful of very dark images (5) and tiny files (6), but these are well within the bounds of normal photographic variation (e.g., night shots) and do not represent data corruption.

Conclusion: The visual EDA successfully quantifies the stylistic gap between the two domains. For the GAN to transform a photo into a Monet-style painting, it must learn a complex mapping that involves increasing brightness, shifting the color palette from blues to greens and yellows, reducing sharpness while adding characteristic brushstroke texture, and compressing the overall saturation. This analysis provides a clear and actionable set of visual targets for the model to achieve.

Section 5: Data Pipeline (TFRecords → tf.data)

Build a high-throughput tf.data pipeline that reads the Kaggle TFRecord shards for each domain, applies a stable train/validation split by filename hash, and prepares augmented, batched streams ready for CycleGAN training.

Section Plan:

1. Parse TFRecords: Decode image bytes to float32 tensors in [0,1], preserving image_name for splitting.
2. Stable Split: Route examples to train/valid inside tf.data via a deterministic string hash on image_name.
3. Augmentations: Random flip, jitter (286→256 crop), and light color jitter on training streams.
4. Batch & Prefetch: Shuffle, batch, and prefetch the datasets, exposing paired iterators for training.
5. Visual Sanity Check: Confirm the pipeline's output by sampling and displaying a batch from each stream.

---

 

5.1 Parse TFRecords → Tensors

Define a TFRecord schema and a decoder to read image bytes into float32 tensors (range [0,1]) with a final shape of [256,256,3]. The image_name string is kept temporarily for the train/validation split.

 

---

# Define constants for the data pipeline.
AUTOTUNE = tf.data.AUTOTUNE
IMG_SIZE = (256, 256)
CHANNELS = 3

# Define the feature schema for parsing TFRecord examples.
TFREC_FEATURES = {
    "image_name": tf.io.FixedLenFeature([], tf.string),
    "image":      tf.io.FixedLenFeature([], tf.string),
}

# Create the main parsing function.
def _parse_tfrec(serialized):
    # Parse the serialized example using the feature schema.
    ex = tf.io.parse_single_example(serialized, TFREC_FEATURES)
    # Extract the image name and JPEG-encoded image bytes.
    name = ex["image_name"]
    img  = ex["image"]
    # Decode the JPEG image.
    img  = tf.image.decode_jpeg(img, channels=CHANNELS)
    # Normalize pixel values to the [0, 1] range.
    img  = tf.image.convert_image_dtype(img, tf.float32)
    # Resize the image to the standard input size for the model.
    img  = tf.image.resize(img, IMG_SIZE, method="bilinear")
    # Explicitly set the shape for model compatibility.
    img.set_shape((*IMG_SIZE, CHANNELS))
    return img, name

# Get lists of all TFRecord shard files from the paths defined in Section 3.
monet_tfrec = sorted(str(p) for p in MONET_TFREC_DIR.glob("*.tfrec"))
photo_tfrec = sorted(str(p) for p in PHOTO_TFREC_DIR.glob("*.tfrec"))

# Print the number of TFRecord files found for each domain.
print("-" * 70)
print("TFRecord shard counts → Monet:", len(monet_tfrec), "| Photo:", len(photo_tfrec))
print("-" * 70)

----------------------------------------------------------------------
TFRecord shard counts → Monet: 5 | Photo: 20
----------------------------------------------------------------------

 

5.2 Stable Train/Valid Split (by image_name hash)

Route each example to either the training or validation set directly within the pipeline. This uses a deterministic hash of the image_name to keep the split consistent across all runs and prevent data leakage.

 

---

# Define parameters for a deterministic train/validation split.
VALID_PCT = 0.10
HASH_BUCKETS = 4096
HASH_SEED = (12345, 67890)

# Define a stable hash function to assign images to the validation set.
def _is_valid(name: tf.Tensor) -> tf.Tensor:
    # Hash the image name into a bucket.
    bucket = tf.strings.to_hash_bucket_strong(name, num_buckets=HASH_BUCKETS, key=list(HASH_SEED))
    # Use modulo arithmetic to create a stable percentage-based split.
    return tf.less(bucket % 1000, int(VALID_PCT * 1000))

# Create routing functions to filter the dataset into train and valid splits.
def _route_train(img, name):
    return tf.logical_not(_is_valid(name))

def _route_valid(img, name):
    return _is_valid(name)

# Create a helper to remove the image name after routing is complete.
def _drop_name(img, name):
    return img

# Define a factory function to create a raw, parsed dataset from TFRecord files.
def _make_raw_ds(tfrec_files):
    ds = tf.data.TFRecordDataset(tfrec_files, num_parallel_reads=AUTOTUNE)
    ds = ds.map(_parse_tfrec, num_parallel_calls=AUTOTUNE)
    return ds

# Build the initial raw datasets for both Monet and Photo domains.
monet_raw = _make_raw_ds(monet_tfrec)
photo_raw = _make_raw_ds(photo_tfrec)

# Apply the filters to create the final train and validation datasets.
monet_train = monet_raw.filter(_route_train).map(_drop_name, num_parallel_calls=AUTOTUNE)
monet_valid = monet_raw.filter(_route_valid).map(_drop_name, num_parallel_calls=AUTOTUNE)
photo_train = photo_raw.filter(_route_train).map(_drop_name, num_parallel_calls=AUTOTUNE)
photo_valid = photo_raw.filter(_route_valid).map(_drop_name, num_parallel_calls=AUTOTUNE)

# --- MODIFICATION: Update the count helper to use tqdm ---
# Define a helper function to count the number of examples in a dataset with a progress bar.
def _count(ds, desc="Counting..."):
    c = 0
    # Iterate through the dataset with a tqdm progress bar.
    for _ in tqdm(ds, desc=desc, mininterval=0.2):
        c += 1
    return c

# Materialize the datasets once to get the exact counts for each split.
print("-" * 70)
n_m_train = _count(monet_train, desc="Counting Monet (train)")
n_m_valid = _count(monet_valid, desc="Counting Monet (valid)")
n_p_train = _count(photo_train, desc="Counting Photo (train)")
n_p_valid = _count(photo_valid, desc="Counting Photo (valid)")

# Print the final counts.
print(f"\nMonet  → train: {n_m_train:4d} | valid: {n_m_valid:3d}")
print(f"Photo  → train: {n_p_train:4d} | valid: {n_p_valid:3d}")
print("-" * 70)

----------------------------------------------------------------------

Counting Monet (train): 0it [00:00, ?it/s]

Counting Monet (valid): 0it [00:00, ?it/s]

Counting Photo (train): 0it [00:00, ?it/s]

Counting Photo (valid): 0it [00:00, ?it/s]

Monet  → train:  263 | valid:  37
Photo  → train: 6227 | valid: 811
----------------------------------------------------------------------

 

5.3 Training Augmentations

Apply light augmentations on the training streams to improve model robustness: horizontal flips, jitter (resizing to 286px then random cropping back to 256px), and mild brightness, contrast, and saturation adjustments. Validation streams remain untouched.

 

---

# Define hyperparameters for data augmentation.
JITTER_ENABLE   = True
JITTER_UPSCALE  = 286
BRIGHT_DELTA    = 0.05
CONTRAST_LO     = 0.90
CONTRAST_HI     = 1.10
SATUR_LO        = 0.90
SATUR_HI        = 1.10

# Create the main augmentation function.
def _augment(img: tf.Tensor) -> tf.Tensor:
    # Apply a random horizontal flip.
    img = tf.image.random_flip_left_right(img)

    # Apply jitter by upscaling and then taking a random crop.
    if JITTER_ENABLE:
        img = tf.image.resize(img, [JITTER_UPSCALE, JITTER_UPSCALE], method="bilinear")
        img = tf.image.random_crop(img, size=[*IMG_SIZE, CHANNELS])

    # Apply light color augmentations.
    img = tf.image.random_brightness(img, max_delta=BRIGHT_DELTA)
    img = tf.image.random_contrast(img, lower=CONTRAST_LO, upper=CONTRAST_HI)
    img = tf.image.random_saturation(img, lower=SATUR_LO,    upper=SATUR_HI)

    # Ensure pixel values remain in the valid [0, 1] range after color ops.
    img = tf.clip_by_value(img, 0.0, 1.0)
    return img

# Apply the augmentation function to the training datasets.
monet_train_aug = monet_train.map(_augment, num_parallel_calls=AUTOTUNE)
photo_train_aug = photo_train.map(_augment, num_parallel_calls=AUTOTUNE)

print("-" * 70)
print("Augmentations attached to training streams.")
print("-" * 70)

----------------------------------------------------------------------
Augmentations attached to training streams.
----------------------------------------------------------------------

 

5.4 Finalize Datasets (Shuffle, Batch, Prefetch)

Complete the pipeline with shuffling, batching, and prefetching for high throughput. This step creates the final training iterators that repeat indefinitely for GAN training, plus separate, finite batches for validation.

 

---

# --- Define hyperparameters for batching and shuffling ---
BATCH_SIZE       = 1
SHUFFLE_BUFFER   = 2048
DROP_REMAINDER   = True

# --- Define functions to finalize the data pipelines ---
# Finalize the training pipeline with shuffling, batching, prefetching, and repeating.
def _finalize_train(ds):
    return (
        ds.shuffle(SHUFFLE_BUFFER, seed=SEED, reshuffle_each_iteration=True)
          .batch(BATCH_SIZE, drop_remainder=DROP_REMAINDER)
          .prefetch(AUTOTUNE)
          .repeat()
    )

# Finalize the validation pipeline with batching and prefetching.
def _finalize_valid(ds):
    return (
        ds.batch(BATCH_SIZE, drop_remainder=False)
          .prefetch(AUTOTUNE)
    )

# --- Create the final dataset objects ---
# Create the final, batched, and prefetched training datasets.
monet_train_ds = _finalize_train(monet_train_aug)
photo_train_ds = _finalize_train(photo_train_aug)

# Create the final, batched, and prefetched validation datasets.
monet_valid_ds = _finalize_valid(monet_valid)
photo_valid_ds = _finalize_valid(photo_valid)

# Zip the two infinite training streams to create paired batches for training.
train_pairs = tf.data.Dataset.zip((monet_train_ds, photo_train_ds)).prefetch(AUTOTUNE)


# --- Display a summary table of the final datasets ---
summary_data = [
    {"Dataset Name": "monet_train_ds", "Purpose": "Training", "Infinite": "Yes", "Element Spec": str(monet_train_ds.element_spec)},
    {"Dataset Name": "photo_train_ds", "Purpose": "Training", "Infinite": "Yes", "Element Spec": str(photo_train_ds.element_spec)},
    {"Dataset Name": "monet_valid_ds", "Purpose": "Validation", "Infinite": "No", "Element Spec": str(monet_valid_ds.element_spec)},
    {"Dataset Name": "photo_valid_ds", "Purpose": "Validation", "Infinite": "No", "Element Spec": str(photo_valid_ds.element_spec)},
    {"Dataset Name": "train_pairs", "Purpose": "Paired Training", "Infinite": "Yes", "Element Spec": str(train_pairs.element_spec)},
]
summary_df = pd.DataFrame(summary_data).set_index("Dataset Name")

# Apply custom styling to the summary table for a clean presentation.
styler = summary_df.style
styler = styler.set_properties(**{
    "background-color": "#0b0b0b", "color": "#F8F8F8", "border-color": "#1A6A1A",
})
styler = styler.set_table_styles([
    {"selector": "th", "props": [("background-color", "#1A6A1A"), ("color", "white"), ("text-align", "left")]},
    {"selector": "td", "props": [("border", "1px solid #1A6A1A"), ("text-align", "left")]},
    {"selector": "th.col_heading", "props": [("text-align", "left")]}
])
display(styler)

	Purpose	Infinite	Element Spec
Dataset Name
monet_train_ds	Training	Yes	TensorSpec(shape=(1, 256, 256, 3), dtype=tf.float32, name=None)
photo_train_ds	Training	Yes	TensorSpec(shape=(1, 256, 256, 3), dtype=tf.float32, name=None)
monet_valid_ds	Validation	No	TensorSpec(shape=(None, 256, 256, 3), dtype=tf.float32, name=None)
photo_valid_ds	Validation	No	TensorSpec(shape=(None, 256, 256, 3), dtype=tf.float32, name=None)
train_pairs	Paired Training	Yes	(TensorSpec(shape=(1, 256, 256, 3), dtype=tf.float32, name=None), TensorSpec(shape=(1, 256, 256, 3), dtype=tf.float32, name=None))

 

5.5 Visual Sanity Check

Sample a single batch from the finalized training streams and display a grid of the images. This provides a final visual confirmation that the entire pipeline, from parsing to augmentation to batching, is working correctly.

 

---

# Define a helper function to grab a preview batch from a dataset.
def preview_n(ds, n=6):
    # Unbatch the dataset, take n single images, then re-batch them.
    return next(iter(ds.unbatch().take(n).batch(n)))

# Define the main plotting function to display a batch of images.
def _show_batch(imgs, title):
    # Convert the tensor to a NumPy array.
    imgs_np = imgs.numpy()
    n = imgs_np.shape[0]
    
    # Calculate the grid dimensions.
    cols = min(6, n)
    rows = int(np.ceil(n / cols))
    
    # Create the figure and axes.
    fig, axes = plt.subplots(rows, cols, figsize=(3 * cols, 3 * rows), facecolor="black")
    fig.patch.set_facecolor("black")
    
    # Ensure the axes object is always a 2D array for consistent indexing.
    axes = np.atleast_2d(axes).flatten()

    # Loop through the grid and plot each image.
    for i, ax in enumerate(axes):
        if i < n:
            # Display the image.
            ax.imshow(np.clip(imgs_np[i], 0, 1))
        # Clean up the axes by removing ticks.
        ax.set_xticks([])
        ax.set_yticks([])
        # Set the background color of the axes.
        ax.set_facecolor("black")
        # Style the plot's border (spines).
        for sp in ax.spines.values():
            sp.set_edgecolor("#1A6A1A")
            sp.set_linewidth(1.5)

    # Set the main title for the entire grid.
    plt.suptitle(title, color="#FFB612", fontsize=24, y=0.98, fontweight="bold")
    # Ensure the layout is tight and clean.
    plt.tight_layout()
    plt.show()

# Grab a sample batch from each training data stream.
monet_batch = preview_n(monet_train_ds, n=6)
photo_batch = preview_n(photo_train_ds, n=6)

# Print the shape and stats of the sample batches.
print("-" * 70)
print("Batch shapes:")
print(f"  Monet:", monet_batch.shape, monet_batch.dtype, f"[min={float(monet_batch.numpy().min()):.3f}, max={float(monet_batch.numpy().max()):.3f}]")
print(f"  Photo:", photo_batch.shape, photo_batch.dtype, f"[min={float(photo_batch.numpy().min()):.3f}, max={float(photo_batch.numpy().max()):.3f}]")
print("-" * 70)

# Display the batch of augmented Monet images.
_show_batch(monet_batch, "Monet Training Batch (augmented)")
# Display the batch of augmented Photo images.
_show_batch(photo_batch, "Photo Training Batch (augmented)")

----------------------------------------------------------------------
Batch shapes:
  Monet: (6, 256, 256, 3) <dtype: 'float32'> [min=0.000, max=1.000]
  Photo: (6, 256, 256, 3) <dtype: 'float32'> [min=0.000, max=1.000]
----------------------------------------------------------------------

Observation: Pipeline Construction and Verification

This section successfully transitions from raw data on disk to a complete, production-grade tf.data pipeline. The process methodically builds an efficient and reproducible input stream by parsing TFRecords, applying a deterministic train/validation split, adding augmentations, and finalizing the datasets with batching and prefetching. The result is a set of iterators perfectly tailored for the CycleGAN training process.

• Efficient and Performant Design:

  • The pipeline is built for high throughput, using TFRecord files as the source and leveraging tf.data.AUTOTUNE for parallel processing and prefetching. This design ensures that data loading will not be a bottleneck, allowing the GPU to remain saturated during training.

• Reproducible Data Splitting:

  • A key feature of this pipeline is the use of a deterministic hash on image filenames to create the train/validation split. This is a crucial choice for experimental reproducibility, as it guarantees that the same images will always land in the same set across different runs.
  • The final counts (263/37 for Monet, 6227/811 for Photo) confirm that the hashing method successfully allocated roughly 11-12% of the data to the validation set, aligning with the target.

• Targeted Augmentation:

  • The augmentation strategy is specifically designed for style transfer. Light augmentations like random flips, jitter, and minor color adjustments are applied to the training set to increase data diversity and improve model generalization.
  • These transformations make the model more robust without fundamentally altering the core stylistic features, color palettes, textures, and compositions that the GAN is intended to learn.

• Final Verification:

  • The final visual sanity check confirms that the entire pipeline is functioning as expected. The sample batches have the correct shape, (6, 256, 256, 3), and float32 data type, with pixel values correctly normalized to the [0, 1] range.
  • The displayed images clearly show the effects of random augmentations (e.g., cropping from jitter, slight color shifts), verifying that the training data will be varied and dynamic.

Conclusion: The data pipeline is complete, verified, and ready for training. It is efficient, reproducible, and provides appropriately augmented data streams for both the Monet and Photo domains. With a high degree of confidence in the input data, the project can now proceed to the core task of building and training the CycleGAN models.

Section 6: CycleGAN Model Architecture

Define the core components of the CycleGAN: a ResNet-9 Generator for the image-to-image translation and a 70×70 PatchGAN Discriminator to judge the realism of the generated images. The models are implemented using Keras 3 and TensorFlow, featuring custom InstanceNormalization and ReflectionPad2D layers to closely match the original paper's architecture.

Section Plan:

1. Building Blocks & Config: Define the shared components: weight initializers, custom padding and normalization layers, and reusable network blocks.
2. ResNet-9 Generator: Construct the generator model with a 7×7 stem, two downsampling layers, nine residual blocks, and two upsampling layers.
3. 70×70 PatchGAN Discriminator: Build the discriminator using a pyramid of strided convolutions with LeakyReLU activations.
4. Instantiate & Sanity Check: Create instances of all four models (G_X→Y, G_Y→X, D_X, D_Y) and verify their output shapes with a dummy forward pass.

   

---

 

6.1 Building Blocks & Config

Set up the shared components for the networks: a custom reflection padding layer, a from-scratch instance normalization layer, the specific weight initializer from the CycleGAN paper, and the reusable ResNet block for the generator.

 

---

# --- Architectural Parameters & Weight Initializer ---

# Define the core architectural parameters for the models.
IMG_SIZE = 256
NGF = 64  # Number of generator filters in the first conv layer.
NDF = 64  # Number of discriminator filters in the first conv layer.
N_RES = 9   # Number of ResNet blocks in the generator.

# Define the weight initializer as specified in the CycleGAN paper.
WEIGHT_INIT = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)


# --- Custom Layers ---

# Create a custom Reflection Padding layer.
class ReflectionPad2D(layers.Layer):
    def __init__(self, padding=(1, 1), **kwargs):
        super().__init__(**kwargs)
        if isinstance(padding, int):
            padding = (padding, padding)
        self.pad_h, self.pad_w = padding

    def call(self, x):
        ph, pw = self.pad_h, self.pad_w
        return tf.pad(x, [[0, 0], [ph, ph], [pw, pw], [0, 0]], mode="REFLECT")

    def get_config(self):
        cfg = super().get_config()
        cfg.update({"padding": (self.pad_h, self.pad_w)})
        return cfg

# Create a custom Instance Normalization layer.
class InstanceNormalization(layers.Layer):
    def __init__(self, epsilon=1e-5, **kwargs):
        super().__init__(**kwargs)
        self.epsilon = epsilon

    def build(self, input_shape):
        ch = int(input_shape[-1])
        self.gamma = self.add_weight(name="gamma", shape=(ch,), initializer="ones", trainable=True)
        self.beta = self.add_weight(name="beta", shape=(ch,), initializer="zeros", trainable=True)
        super().build(input_shape)

    def call(self, x):
        mean, var = tf.nn.moments(x, axes=[1, 2], keepdims=True)
        x_hat = (x - mean) * tf.math.rsqrt(var + self.epsilon)
        return self.gamma * x_hat + self.beta

    def get_config(self):
        cfg = super().get_config()
        cfg.update({"epsilon": self.epsilon})
        return cfg


# --- Reusable Model Blocks ---

# Define a 7x7 convolutional block with Reflection Padding and optional Instance Norm.
def conv7x7(x, filters, name=None, use_norm=True, activation="relu"):
    x = ReflectionPad2D(3, name=f"{name}_pad")(x)
    x = layers.Conv2D(
        filters, 7, strides=1, padding="valid",
        kernel_initializer=WEIGHT_INIT, use_bias=not use_norm, name=f"{name}_conv"
    )(x)
    if use_norm:
        x = InstanceNormalization(name=f"{name}_in")(x)
    if activation == "relu":
        x = layers.ReLU(name=f"{name}_relu")(x)
    elif activation == "leaky_relu":
        x = layers.LeakyReLU(0.2, name=f"{name}_lrelu")(x)
    return x

# Define a downsampling block (strided convolution).
def downsample(x, filters, name=None):
    x = layers.Conv2D(
        filters, 3, strides=2, padding="same",
        kernel_initializer=WEIGHT_INIT, use_bias=False, name=f"{name}_conv"
    )(x)
    x = InstanceNormalization(name=f"{name}_in")(x)
    x = layers.ReLU(name=f"{name}_relu")(x)
    return x

# Define an upsampling block (transposed convolution).
def upsample(x, filters, name=None):
    x = layers.Conv2DTranspose(
        filters, 3, strides=2, padding="same",
        kernel_initializer=WEIGHT_INIT, use_bias=False, name=f"{name}_deconv"
    )(x)
    x = InstanceNormalization(name=f"{name}_in")(x)
    x = layers.ReLU(name=f"{name}_relu")(x)
    return x

# Define a ResNet block with two convolutional layers and a skip connection.
def res_block(x_in, filters, name=None):
    x = ReflectionPad2D(1, name=f"{name}_pad1")(x_in)
    x = layers.Conv2D(
        filters, 3, strides=1, padding="valid",
        kernel_initializer=WEIGHT_INIT, use_bias=False, name=f"{name}_conv1"
    )(x)
    x = InstanceNormalization(name=f"{name}_in1")(x)
    x = layers.ReLU(name=f"{name}_relu")(x)
    x = ReflectionPad2D(1, name=f"{name}_pad2")(x)
    x = layers.Conv2D(
        filters, 3, strides=1, padding="valid",
        kernel_initializer=WEIGHT_INIT, use_bias=False, name=f"{name}_conv2"
    )(x)
    x = InstanceNormalization(name=f"{name}_in2")(x)
    return layers.Add(name=f"{name}_add")([x_in, x])

 

6.2 ResNet-9 Generator

Compose the generator architecture: a 7×7 convolutional stem with reflection padding, two downsampling blocks, nine deep residual blocks for transformation, two upsampling blocks, and a final 7×7 Tanh head to produce an image in the [-1, 1] range.

 

---

# Build the ResNet-based generator model.
def build_generator(img_size=IMG_SIZE, in_ch=3, out_ch=3, ngf=NGF, n_res=N_RES, name="G"):
    # Define the model input.
    inp = layers.Input(shape=(img_size, img_size, in_ch), dtype='float32', name=f"{name}_in")

    # Initial convolution layer.
    x = conv7x7(inp, ngf, name=f"{name}_c7s1_64")
    
    # Downsampling layers.
    x = downsample(x, ngf * 2, name=f"{name}_d128")
    x = downsample(x, ngf * 4, name=f"{name}_d256")

    # Main ResNet transformation blocks.
    for i in range(n_res):
        x = res_block(x, ngf * 4, name=f"{name}_R{i+1}")

    # Upsampling layers.
    x = upsample(x, ngf * 2, name=f"{name}_u128")
    x = upsample(x, ngf,     name=f"{name}_u64")

    # Final output layer.
    x = ReflectionPad2D(3, name=f"{name}_head_pad")(x)
    x = layers.Conv2D(
        out_ch, 7, strides=1, padding="valid",
        kernel_initializer=WEIGHT_INIT, use_bias=True,
        dtype="float32", name=f"{name}_head_conv"
    )(x)
    out = layers.Activation("tanh", dtype="float32", name=f"{name}_tanh")(x)
    
    return keras.Model(inp, out, name=name)

# Create the two generator models for each translation direction.
G_X2Y = build_generator(name="G_X2Y")  # X (photo) -> Y (Monet)
G_Y2X = build_generator(name="G_Y2X")  # Y (Monet) -> X (photo)

# Print a summary of the generator architectures.
print("-" * 70)
print("Generator(s) built.")
print(G_X2Y.input_shape, "->", G_X2Y.output_shape)
print(G_Y2X.input_shape, "->", G_Y2X.output_shape)
print("-" * 70)

----------------------------------------------------------------------
Generator(s) built.
(None, 256, 256, 3) -> (None, 256, 256, 3)
(None, 256, 256, 3) -> (None, 256, 256, 3)
----------------------------------------------------------------------

 

6.3 PatchGAN Discriminator (70×70)

Define the 70×70 PatchGAN discriminator, which uses a series of strided 4×4 convolutions with LeakyReLU activations and instance normalization. Its final output is a grid of logits, where each logit classifies the realism of a corresponding patch in the input image.

 

---

# Define a standard discriminator block.
def disc_block(x, filters, stride, norm=True, name=None):
    x = layers.Conv2D(
        filters, 4, strides=stride, padding="same",
        kernel_initializer=WEIGHT_INIT, use_bias=not norm, name=f"{name}_conv"
    )(x)
    if norm:
        x = InstanceNormalization(name=f"{name}_in")(x)
    x = layers.LeakyReLU(0.2, name=f"{name}_lrelu")(x)
    return x

# Build the PatchGAN discriminator model.
def build_discriminator(img_size=IMG_SIZE, in_ch=3, ndf=NDF, name="D"):
    # Define the model input.
    inp = layers.Input(shape=(img_size, img_size, in_ch), dtype='float32', name=f"{name}_in")

    # Series of downsampling convolutional blocks.
    x = disc_block(inp, ndf,     stride=2, norm=False, name=f"{name}_C64")
    x = disc_block(x,   ndf * 2, stride=2, norm=True,  name=f"{name}_C128")
    x = disc_block(x,   ndf * 4, stride=2, norm=True,  name=f"{name}_C256")
    x = disc_block(x,   ndf * 8, stride=1, norm=True,  name=f"{name}_C512")
    
    # Final output layer to produce the patch map.
    out = layers.Conv2D(
        1, 4, strides=1, padding="same",
        kernel_initializer=WEIGHT_INIT, use_bias=True,
        dtype="float32", name=f"{name}_out"
    )(x)
    
    return keras.Model(inp, out, name=name)

# Create the two discriminator models for each domain.
D_X = build_discriminator(name="D_X")  # Distinguishes real vs. fake photos.
D_Y = build_discriminator(name="D_Y")  # Distinguishes real vs. fake Monet paintings.

# Print a summary of the discriminator architectures.
print("-" * 70)
print("Discriminator(s) built.")
print(D_X.input_shape, "->", D_X.output_shape)
print(D_Y.input_shape, "->", D_Y.output_shape)
print("-" * 70)

----------------------------------------------------------------------
Discriminator(s) built.
(None, 256, 256, 3) -> (None, 32, 32, 1)
(None, 256, 256, 3) -> (None, 32, 32, 1)
----------------------------------------------------------------------

 

6.4 Instantiate & Sanity Check

Instantiate all four required networks (G_X→Y, G_Y→X, D_X, and D_Y) and then run a quick forward pass with a dummy tensor to confirm that the output shapes and data types are correct before proceeding to training.

 

---

# --- Sanity Check Forward Pass ---
print("-" * 70)
print("Sanity-checking forward pass with dummy tensors.")

# Create dummy input tensors with the correct shape and dtype.
x_dummy = tf.ones([1, IMG_SIZE, IMG_SIZE, 3], dtype=tf.float32)
y_dummy = tf.ones([1, IMG_SIZE, IMG_SIZE, 3], dtype=tf.float32)

# Perform a forward pass through the generators.
y_hat = G_X2Y(x_dummy)
x_hat = G_Y2X(y_dummy)

# Perform a forward pass through the discriminators.
logits_x = D_X(x_dummy)
logits_y = D_Y(y_dummy)

# Print the output shapes and dtypes to verify the model architectures.
print("G_X2Y:", y_hat.shape, y_hat.dtype)
print("G_Y2X:", x_hat.shape, x_hat.dtype)
print("D_X  :", logits_x.shape, logits_x.dtype)
print("D_Y  :", logits_y.shape, logits_y.dtype)
print("-" * 70)

----------------------------------------------------------------------
Sanity-checking forward pass with dummy tensors.
G_X2Y: (1, 256, 256, 3) <dtype: 'float32'>
G_Y2X: (1, 256, 256, 3) <dtype: 'float32'>
D_X  : (1, 32, 32, 1) <dtype: 'float32'>
D_Y  : (1, 32, 32, 1) <dtype: 'float32'>
----------------------------------------------------------------------

Observation: Model Architecture and Verification

This section successfully defines and constructs the four neural networks that form the core of the CycleGAN architecture. The chosen designs for the generator and discriminator are based on the original, high-performing CycleGAN paper, ensuring a strong foundation for the image translation task. The final sanity check confirms that all components are wired correctly and produce outputs of the expected shape and type.

• Generator Architecture:

  • The ResNet-9 architecture was implemented for both generators (G_X→Y and G_Y→X). This design features an encoder, a series of transformation blocks (ResNet blocks), and a decoder.
  • The output from cell 6.2 confirms that the generator correctly transforms an input tensor of shape (256, 256, 3) back into an output tensor of the exact same shape, which is essential for an image-to-image translation task.

• Discriminator Architecture (Key Finding):

  • A 70x70 PatchGAN was implemented for both discriminators (D_X and D_Y). Instead of classifying the entire image as real or fake (a single output), this model outputs a 32x32x1 grid of predictions.
  • Each value in this grid corresponds to a judgment on a 70x70 patch of the input image. This is a critical design choice that forces the generator to produce convincing textures and high-frequency details across the entire image, not just plausible-looking compositions.

• Model Verification:

  • The sanity check in cell 6.4 confirms that a forward pass through all four models completes without error.
  • The output shapes and float32 data types match expectations precisely. This verifies that all custom layers (ReflectionPad2D, InstanceNormalization) and reusable blocks are functioning correctly and that the models are ready for the training process.

Conclusion: The architectural foundation for the CycleGAN is now complete and verified. The ResNet-9 generators are capable of transforming the images, and the PatchGAN discriminators are designed to provide the detailed, localized feedback needed to guide the generators toward producing realistic textures. With both the data pipeline and the models now in place, the project is ready to proceed to defining the loss functions and implementing the training loop.

Section 7: Training Setup

This section defines the full training objective for the CycleGAN and constructs all the necessary components to run it. This includes specifying the loss functions, configuring the optimizers, implementing a custom Keras trainer, and setting the final training schedule.

Section Plan:

1. Losses & Weights: Define the three core loss functions (adversarial, cycle-consistency, identity) and their respective weights.
2. Optimizers: Configure the Adam optimizers for the generators and discriminators with paper-specific beta values.
3. Replay Buffer: Implement an image pool to store and replay previously generated fakes, stabilizing discriminator training.
4. Custom Trainer: Build a custom Keras Model that encapsulates the complete, complex training step for the CycleGAN.
5. GPU Sanity Check: Run a short "micro-train" on the GPU to verify performance and ensure the training step is error-free.
6. Training Schedule: Set the final hyperparameters for the full training run, including the identity loss annealing schedule.
7. Checkpoints & Resume Logic: Configure the TensorFlow CheckpointManager to save progress and optionally resume training.
8. Launch Full Training: Execute the main training loop.

---

7.1 Losses & Weights

Specify adversarial losses using LSGAN (mean squared error), L1 cycle-consistency for both directions, and an identity loss to preserve color/lighting. Expose λ_cycle and λ_id as top-level hyperparameters.

---

# --- Loss Functions & Weights ---

# Define the weights for the different loss components.
LAMBDA_CYCLE   = 10.0
LAMBDA_ID_BASE = 0.5 * LAMBDA_CYCLE  # Initial identity loss weight.
LAMBDA_ID      = LAMBDA_ID_BASE      # Alias for backward compatibility.

# Set a small amount of label smoothing for discriminator targets.
LABEL_SMOOTH = 0.05

# Instantiate the base loss objects.
mse = tf.keras.losses.MeanSquaredError(reduction="none")
mae = tf.keras.losses.MeanAbsoluteError(reduction="none")

# Define the generator's adversarial loss (Least Squares GAN).
def lsgan_gen_loss(d_fake, label_smooth=LABEL_SMOOTH):
    target = tf.ones_like(d_fake) * (1.0 - label_smooth)
    return tf.reduce_mean(mse(target, d_fake))

# Define the discriminator's adversarial loss (Least Squares GAN).
def lsgan_disc_loss(d_real, d_fake, label_smooth=LABEL_SMOOTH):
    real_target = tf.ones_like(d_real) * (1.0 - label_smooth)
    fake_target = tf.zeros_like(d_fake)
    real_loss = tf.reduce_mean(mse(real_target, d_real))
    fake_loss = tf.reduce_mean(mse(fake_target, d_fake))
    return 0.5 * (real_loss + fake_loss)

# Define the cycle consistency loss (L1/MAE).
def l1_cycle_loss(x, x_cycle, weight=LAMBDA_CYCLE):
    return weight * tf.reduce_mean(mae(x, x_cycle))

# Define the identity loss (L1/MAE).
def l1_identity_loss(y, g_y, weight=LAMBDA_ID_BASE):
    return weight * tf.reduce_mean(mae(y, g_y))

7.2 Optimizers

Configure separate Adam optimizers for the generators and discriminators with the paper's recommended beta values of β1=0.5 and β2=0.999. This cell also defines a helper function for an optional linear learning rate decay schedule.

---

# Define the base learning rates for the models.
BASE_LR_G = 2e-5
BASE_LR_D = 2e-5

# Create a helper function for an optional linear learning rate decay schedule.
def make_linear_decay_lr(base_lr, decay_start_step, total_steps):
    if decay_start_step is None or total_steps is None or decay_start_step >= total_steps:
        return base_lr
    return tf.keras.optimizers.schedules.PolynomialDecay(
        initial_learning_rate=base_lr,
        decay_steps=int(total_steps - decay_start_step),
        end_learning_rate=0.0,
        power=1.0
    )

# Set the learning rates for this training run.
GEN_LR = BASE_LR_G
DISC_LR = BASE_LR_D

# Create the Adam optimizers with a lower LR and gradient clipping enabled.
opt_G_X2Y = tf.keras.optimizers.Adam(learning_rate=GEN_LR,  beta_1=0.5, beta_2=0.999, clipvalue=1.0)
opt_G_Y2X = tf.keras.optimizers.Adam(learning_rate=GEN_LR,  beta_1=0.5, beta_2=0.999, clipvalue=1.0)
opt_D_X   = tf.keras.optimizers.Adam(learning_rate=DISC_LR, beta_1=0.5, beta_2=0.999, clipvalue=1.0)
opt_D_Y   = tf.keras.optimizers.Adam(learning_rate=DISC_LR, beta_1=0.5, beta_2=0.999, clipvalue=1.0)

7.3 Replay Buffer

Implement a small FIFO image pool (e.g., size 50) of previously generated samples. On each step, feed a mix of fresh and pooled fakes to the discriminators to reduce oscillations.

---

# Implement an image replay buffer to stabilize discriminator training.
class ImagePool:
    def __init__(self, max_size=50):
        self.max_size = max_size
        self.storage = []

    def query(self, batch):
        if self.max_size <= 0:
            return batch
            
        output_images = []
        for i in range(batch.shape[0]):
            img = batch[i:i+1]
            if len(self.storage) < self.max_size:
                # The pool is not full, so add the new image and return it.
                self.storage.append(img)
                output_images.append(img)
            else:
                # The pool is full.
                if tf.random.uniform(()) < 0.5:
                    # With 50% probability, return a random image from the pool.
                    idx = tf.random.uniform((), 0, len(self.storage), dtype=tf.int32)
                    stored_img = self.storage[idx]
                    # Replace the chosen image with the new one.
                    self.storage[idx] = img
                    output_images.append(stored_img)
                else:
                    # With 50% probability, return the new image directly.
                    output_images.append(img)
                    
        return tf.concat(output_images, axis=0)

# Create the image pools for each generator.
pool_X_fake = ImagePool(max_size=10)
pool_Y_fake = ImagePool(max_size=10)

7.4 Custom Trainer

Create a Keras 3 subclassed Model that defines train_step/test_step. Perform forward and backward cycles, compute all loss components, apply grads to G_X→Y, G_Y→X, D_X, and D_Y, and log scalar metrics for TensorBoard.

---

# Create a custom Keras Model to encapsulate the full CycleGAN training step.
class CycleGANTrainer(tf.keras.Model):
    def __init__(self, G_X2Y, G_Y2X, D_X, D_Y,
                 lambda_cycle, lambda_id,
                 label_smooth, use_pools=True):
        super().__init__()
        # Initialize the models and training parameters.
        self.G_X2Y = G_X2Y
        self.G_Y2X = G_Y2X
        self.D_X   = D_X
        self.D_Y   = D_Y
        self.lambda_cycle = lambda_cycle
        self.lambda_id    = lambda_id
        self.label_smooth = label_smooth
        self.use_pools    = use_pools

        # Initialize metrics to track all loss components.
        self.m_adv_GXY = tf.keras.metrics.Mean(name="adv_G_X2Y")
        self.m_adv_GYX = tf.keras.metrics.Mean(name="adv_G_Y2X")
        self.m_cyc     = tf.keras.metrics.Mean(name="cycle")
        self.m_id      = tf.keras.metrics.Mean(name="identity")
        self.m_G_total = tf.keras.metrics.Mean(name="G_total")
        self.m_DX      = tf.keras.metrics.Mean(name="D_X")
        self.m_DY      = tf.keras.metrics.Mean(name="D_Y")

    def compile(self, opt_G_X2Y, opt_G_Y2X, opt_D_X, opt_D_Y, **kwargs):
        super().compile(**kwargs)
        # Store the optimizers.
        self.opt_G_X2Y = opt_G_X2Y
        self.opt_G_Y2X = opt_G_Y2X
        self.opt_D_X   = opt_D_X
        self.opt_D_Y   = opt_D_Y

    def _unpack(self, data):
        # Helper to flexibly unpack input data.
        if isinstance(data, (list, tuple)) and len(data) == 2:
            return data[0], data[1]
        if isinstance(data, dict):
            for kx in ("photo", "x", "X"):
                for ky in ("monet", "y", "Y"):
                    if kx in data and ky in data:
                        return data[kx], data[ky]
        raise ValueError("Trainer expected (x, y) or a dict with both domains.")

    def train_step(self, data):
        x, y = self._unpack(data)

        # --- Update the Generators ---
        with tf.GradientTape() as tape:
            # Forward pass through all networks.
            y_fake = self.G_X2Y(x, training=True)
            x_fake = self.G_Y2X(y, training=True)
            x_cycle = self.G_Y2X(y_fake, training=True)
            y_cycle = self.G_X2Y(x_fake, training=True)
            y_id = self.G_X2Y(y, training=True)
            x_id = self.G_Y2X(x, training=True)
            d_y_fake = self.D_Y(y_fake, training=True)
            d_x_fake = self.D_X(x_fake, training=True)

            # Calculate the individual loss components.
            adv_G_X2Y = lsgan_gen_loss(d_y_fake, self.label_smooth)
            adv_G_Y2X = lsgan_gen_loss(d_x_fake, self.label_smooth)
            cyc_loss  = l1_cycle_loss(x, x_cycle, self.lambda_cycle) + \
                        l1_cycle_loss(y, y_cycle, self.lambda_cycle)
            id_loss   = l1_identity_loss(y, y_id, self.lambda_id) + \
                        l1_identity_loss(x, x_id, self.lambda_id)
            G_total   = adv_G_X2Y + adv_G_Y2X + cyc_loss + id_loss

        # Compute and apply gradients for both generators in a single step.
        vars_G = self.G_X2Y.trainable_variables + self.G_Y2X.trainable_variables
        grads_G = tape.gradient(G_total, vars_G)
        self.opt_G_X2Y.apply_gradients(zip(grads_G[:len(self.G_X2Y.trainable_variables)], self.G_X2Y.trainable_variables))
        self.opt_G_Y2X.apply_gradients(zip(grads_G[len(self.G_X2Y.trainable_variables):], self.G_Y2X.trainable_variables))

        # --- Update the Discriminators ---
        # Get a mix of current and historical fake images from the replay buffer.
        if self.use_pools:
            pool_X = globals().get("pool_X_fake")
            pool_Y = globals().get("pool_Y_fake")
            x_fake_train = pool_X.query(x_fake) if pool_X else x_fake
            y_fake_train = pool_Y.query(y_fake) if pool_Y else y_fake
        else:
            x_fake_train, y_fake_train = x_fake, y_fake

        # Update Discriminator D_X.
        with tf.GradientTape() as tape_dx:
            d_x_real = self.D_X(x, training=True)
            d_x_fake = self.D_X(x_fake_train, training=True)
            D_X_loss = lsgan_disc_loss(d_x_real, d_x_fake, self.label_smooth)
        grads_D_X = tape_dx.gradient(D_X_loss, self.D_X.trainable_variables)
        self.opt_D_X.apply_gradients(zip(grads_D_X, self.D_X.trainable_variables))

        # Update Discriminator D_Y.
        with tf.GradientTape() as tape_dy:
            d_y_real = self.D_Y(y, training=True)
            d_y_fake = self.D_Y(y_fake_train, training=True)
            D_Y_loss = lsgan_disc_loss(d_y_real, d_y_fake, self.label_smooth)
        grads_D_Y = tape_dy.gradient(D_Y_loss, self.D_Y.trainable_variables)
        self.opt_D_Y.apply_gradients(zip(grads_D_Y, self.D_Y.trainable_variables))

        # Update the state of all metrics.
        self.m_adv_GXY.update_state(adv_G_X2Y)
        self.m_adv_GYX.update_state(adv_G_Y2X)
        self.m_cyc.update_state(cyc_loss)
        self.m_id.update_state(id_loss)
        self.m_G_total.update_state(G_total)
        self.m_DX.update_state(D_X_loss)
        self.m_DY.update_state(D_Y_loss)

        return { m.name: m.result() for m in self.metrics }

    def test_step(self, data):
        # Define the evaluation logic for the validation set.
        x, y = self._unpack(data)
        y_fake = self.G_X2Y(x, training=False)
        x_fake = self.G_Y2X(y, training=False)
        x_cycle = self.G_Y2X(y_fake, training=False)
        y_cycle = self.G_X2Y(x_fake, training=False)
        y_id = self.G_X2Y(y, training=False)
        x_id = self.G_Y2X(x, training=False)
        d_y_fake = self.D_Y(y_fake, training=False)
        d_x_fake = self.D_X(x_fake, training=False)
        d_y_real = self.D_Y(y, training=False)
        d_x_real = self.D_X(x, training=False)

        # Calculate all loss components without updating gradients.
        adv_G_X2Y = lsgan_gen_loss(d_y_fake, self.label_smooth)
        adv_G_Y2X = lsgan_gen_loss(d_x_fake, self.label_smooth)
        cyc_loss  = l1_cycle_loss(x, x_cycle, self.lambda_cycle) + l1_cycle_loss(y, y_cycle, self.lambda_cycle)
        id_loss   = l1_identity_loss(y, y_id, self.lambda_id) + l1_identity_loss(x, x_id, self.lambda_id)
        G_total   = adv_G_X2Y + adv_G_Y2X + cyc_loss + id_loss
        D_X_loss  = lsgan_disc_loss(d_x_real, d_x_fake, self.label_smooth)
        D_Y_loss  = lsgan_disc_loss(d_y_real, d_y_fake, self.label_smooth)
        
        return {
            "adv_G_X2Y": adv_G_X2Y, "adv_G_Y2X": adv_G_Y2X,
            "cycle": cyc_loss, "identity": id_loss,
            "G_total": G_total, "D_X": D_X_loss, "D_Y": D_Y_loss
        }

print("--- CycleGAN Trainer ready ---")

--- CycleGAN Trainer ready ---

7.5 GPU Sanity Mini-Train (Averaged Metrics)

Run a short, sustained train on /GPU:0 with batch=1 for 80 iterations to confirm real NVIDIA GPU utilization and stable gradients. Displays sparse step logs and a final averaged summary: adv_G_X2Y, adv_G_Y2X, cycle, identity, G_total, D_X, D_Y. Reuses the existing models/optimizers without reallocating VRAM.

---

----- Note -----

----- Before Running 7.5 - The Post-Training Bug -----

Replay protocol for completed runs: execute 7.6 -> 7.7 -> 7.5

After successfully completing the entire training run that lasted almost 9 hours, and saving all the final checkpoints, the Jupyter kernel crashed. To prepare the notebook for submission, I had to re-execute all the cells from the beginning to regenerate the outputs. Everything ran smoothly until I reached the sanity check in this cell (7.5). A cell that had worked perfectly during the initial run was now inexplicably failing with a cascade of nan and OOM errors.

This kicked off a frustrating debugging session. My initial thoughts led me down several dead ends: I tried adding gradient clipping, lowering the learning rate, and even suspected a corrupted environment. None of these were the true cause.

The breakthrough came when I realized the notebook was no longer operating in a 'fresh' state. Because the first run had completed successfully, there was now a full set of checkpoints on disk. On this second pass through the notebook, cell 7.7 was correctly loading the fully trained, 15-epoch model into memory. The sanity check in 7.5 was then trying to run on this advanced model but with freshly initialized optimizers from cell 7.2, creating a fatal state mismatch that caused the numerical chaos.

This realization changed everything. To properly test a trained model, its complete state, including the optimizer variables, must be loaded from a checkpoint. Therefore, the correct workflow when re-running a completed notebook is to execute the cells out of their numerical order through 7.4 first, then run 7.6 and 7.7 to load the checkpoint, and only then run a sanity check in 7.5.

The output now displayed for the 7.5 sanity check below is from re-running the notebook using the fully trained model loaded from the final checkpoint. The numbers in the Observation cell that follows, however, reflect the original sanity run, which was performed on a fresh, untrained model before the long ~9 hour training began. Unfortunately, a screenshot of that original output was not captured, as I assumed it would be easy to recreate. I was very wrong.

# Assert that the trainer class is defined.
assert "CycleGANTrainer" in globals(), "Run Section 7.4 (trainer) first."

# Define parameters for the micro-training loop.
STEPS = 80
MICRO_LOG_EVERY = 10

# Create a simple data pipeline that repeats a single image.
def _decode(p):
    x = tf.io.read_file(p)
    x = tf.io.decode_jpeg(x, channels=3)
    x = tf.image.resize(x, [IMG_SIZE, IMG_SIZE], method="area")
    x = tf.cast(x, tf.float32) / 127.5 - 1.0
    return x

def _one_image_stream(paths):
    ds = tf.data.Dataset.from_tensor_slices(paths[:1])
    ds = ds.map(_decode, num_parallel_calls=tf.data.AUTOTUNE)
    return ds.batch(1).repeat(STEPS).prefetch(tf.data.AUTOTUNE)

# Use existing datasets if available, otherwise create new single-image streams.
if "photo_train_ds" not in globals() or "monet_train_ds" not in globals():
    monet_paths = sorted(str(p) for p in MONET_JPG_DIR.glob("*.jpg"))
    photo_paths = sorted(str(p) for p in PHOTO_JPG_DIR.glob("*.jpg"))
    photo_ds = _one_image_stream(photo_paths)
    monet_ds = _one_image_stream(monet_paths)
else:
    photo_ds = photo_train_ds.take(1).repeat(STEPS)
    monet_ds = monet_train_ds.take(1).repeat(STEPS)

# Create iterators for the single-image streams.
it_x, it_y = iter(photo_ds), iter(monet_ds)

# Create or re-configure the trainer, ensuring replay pools are OFF for this test.
trainer = globals().get("trainer")
if trainer is None or getattr(trainer, "use_pools", None) is True:
    trainer = CycleGANTrainer(
        G_X2Y=G_X2Y, G_Y2X=G_Y2X, D_X=D_X, D_Y=D_Y,
        lambda_cycle=LAMBDA_CYCLE, lambda_id=LAMBDA_ID, 
        label_smooth=LABEL_SMOOTH, use_pools=False
    )
    trainer.compile(
        opt_G_X2Y=opt_G_X2Y, opt_G_Y2X=opt_G_Y2X,
        opt_D_X=opt_D_X, opt_D_Y=opt_D_Y
    )
    globals()["trainer"] = trainer

# Reset all metrics before the run.
for metric in trainer.metrics:
    metric.reset_state()

# Run the main micro-training loop.
try:
    with tf.device("/GPU:0"):
        pbar = tqdm(range(1, STEPS + 1), desc="micro-train", leave=True, mininterval=0.2)
        t0 = time.perf_counter()
        
        for s in pbar:
            # Perform one training step.
            bx, by = next(it_x), next(it_y)
            metrics = trainer.train_step((bx, by))

            # Log progress periodically.
            if s % MICRO_LOG_EVERY == 0 or s == 1 or s == STEPS:
                gt = float(metrics["G_total"].numpy())
                dx = float(metrics["D_X"].numpy())
                dy = float(metrics["D_Y"].numpy())
                lam_id = float(getattr(trainer, "lambda_id", LAMBDA_ID))
                print(f"step {s}/{STEPS}  G_total={gt:.3f}  D_X={dx:.3f}  D_Y={dy:.3f}  λ_id={lam_id:.3f}")
                pbar.set_postfix({
                    "G_total": f"{gt:.3f}", "D_X": f"{dx:.3f}", 
                    "D_Y": f"{dy:.3f}", "λ_id": f"{lam_id:.3f}"
                })
        
        dt = time.perf_counter() - t0

except tf.errors.ResourceExhaustedError as e:
    print(f"GPU OOM during micro-train: {e}")
else:
    # If successful, calculate and print the average loss and timing metrics.
    summary = {m.name: round(float(m.result().numpy()), 4) for m in trainer.metrics}
    summary["lambda_id"] = float(getattr(trainer, "lambda_id", LAMBDA_ID))
    print("\navg:", summary)

    steps_per_sec = STEPS / dt if dt > 0 else float("nan")
    sec_per_step  = dt / STEPS if STEPS > 0 else float("nan")
    print(f"\nmicro-train done in {dt:.1f} sec, {steps_per_sec:.2f} steps/sec, {sec_per_step:.3f} sec/step")

    

print("-" * 70)

micro-train:   0%|          | 0/80 [00:00<?, ?it/s]

step 1/80  G_total=18.948  D_X=0.055  D_Y=0.039  λ_id=5.000
step 10/80  G_total=10.200  D_X=0.071  D_Y=0.059  λ_id=5.000
step 20/80  G_total=7.574  D_X=0.106  D_Y=0.126  λ_id=5.000
step 30/80  G_total=6.353  D_X=0.130  D_Y=0.150  λ_id=5.000
step 40/80  G_total=5.815  D_X=0.149  D_Y=0.162  λ_id=5.000
step 50/80  G_total=5.528  D_X=0.145  D_Y=0.164  λ_id=5.000
step 60/80  G_total=5.246  D_X=0.147  D_Y=0.176  λ_id=5.000
step 70/80  G_total=5.031  D_X=0.152  D_Y=0.180  λ_id=5.000
step 80/80  G_total=4.879  D_X=0.151  D_Y=0.174  λ_id=5.000

avg: {'loss': 0.0, 'adv_G_X2Y': 0.4191, 'adv_G_Y2X': 0.4208, 'cycle': 2.5898, 'identity': 1.4492, 'G_total': 4.8789, 'D_X': 0.1509, 'D_Y': 0.1737, 'lambda_id': 5.0}

micro-train done in 127.9 sec, 0.63 steps/sec, 1.598 sec/step
----------------------------------------------------------------------

----- Note -----

Observation: GPU Sanity Check (CycleGAN micro-train @ 256px)

• What broke and how I fixed it (in order)

  1. Accidentally on CPU. The first pass wasn’t bound to the NVIDIA device. I forced /GPU:0 in 7.5, verified with nvidia-smi, and kept TensorFlow memory-growth on to avoid VRAM spikes.
  2. Allocator hoarding VRAM. TF_GPU_ALLOCATOR=cuda_malloc_async made the pool sticky and triggered OOM. I removed that env var, restarted the kernel, and reused the same trainer and optimizers across cells so memory wasn’t duplicated.
  3. Wrong adapter. Windows showed activity on GPU 1 (Intel UHD). I explicitly targeted GPU 0 (RTX 4070) in the train step and confirmed NVIDIA utilization climbing in Task Manager.
  4. Input too large. Final OOM fix was setting IMG_SIZE = 256 in 6.1. Then I cleared outputs, restarted, and re-ran.
  5. Stale kernel state. The “notebook controller is DISPOSED” error came from a bad Jupyter state. I ran the WSL/Jupyter cleanup (kill stray kernels, clear runtime sockets, update ipykernel/jupyter_client, re-register the tfenv kernel).

• Micro-train results (80 steps, batch=1)

  • Step 1 → 80 G_total: 26.33 → 6.88 with a steady downward trend and no NaNs.
  • Discriminators: D_X 2.94 → 0.63, D_Y 3.25 → 0.64. Both settled as the generators improved.
  • Averaged over the run:
    • G_total 10.60
    • adv_G_X2Y 1.205, adv_G_Y2X 1.381
    • cycle 5.342, identity 2.671
    • D_X 1.389, D_Y 1.495
  • Rough composition of G loss: cycle about 50%, identity about 25%, adversarial about 25%. That is typical for early CycleGAN training where cycle consistency dominates.
  • Loop time: 138 s for 80 steps, about 1.73 s/step.

• Utilization check

  • GPU load: sustained around 95–99% during the 80-step loop.
  • VRAM: about 5.8 / 8.0 GB dedicated in use, small but workable headroom at 256px.
  • Screenshots (before, idle, fixed):

---------- Before ----------

---------- Idle ----------

---------- Fixed ----------

• Why this sanity check mattered

  Most sanity checks are routine. This one surfaced two real blockers: wrong device selection and an allocator setting that caused OOMs. After fixing both and right-sizing inputs, losses behaved and the GPU carried a sustained load.

• Takeaway and next steps

  • Sanity check passed: gradients flow, losses fall, no NaNs, and the NVIDIA GPU is actually doing the work.
  • Move on to the longer training run with this configuration.
  • If OOM returns, first drop IMG_SIZE to 224 → 192 → 128, keep batch at 1, and avoid recreating models or optimizers in new cells.

7.6 Training Schedule & Configuration

Lock in the final hyperparameters for the training run, including the number of epochs and steps, logging frequency, and artifact save paths for both Linux and Windows. This step also defines the annealing schedule for the identity loss, which will gradually decrease its weight over the course of training.

---

# Define the main training schedule parameters.
AUTOTUNE = tf.data.AUTOTUNE
EPOCHS = 15
STEPS_PER_EPOCH = 2000
LOG_EVERY = 250
PREVIEW_EVERY = 1000
SAVE_EVERY_EPOCH = True
TOT_STEPS = EPOCHS * STEPS_PER_EPOCH

# Define the output directories for artifacts on the Linux filesystem.
LINUX_BASE = str((Path.cwd() / "artifacts").resolve())
CKPT_DIR_LIN = os.path.join(LINUX_BASE, "checkpoints")
PREVIEW_DIR_LIN = os.path.join(LINUX_BASE, "previews")
MODELS_DIR_LIN = os.path.join(LINUX_BASE, "models")

# Define the mirrored output directories for the Windows filesystem.
WIN_BASE = "/mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition"
CKPT_DIR_WIN = os.path.join(WIN_BASE, "checkpoints")
PREVIEW_DIR_WIN = os.path.join(WIN_BASE, "previews")
MODELS_DIR_WIN = os.path.join(WIN_BASE, "models")

# Create all output directories if they don't exist.
for d in (CKPT_DIR_LIN, PREVIEW_DIR_LIN, MODELS_DIR_LIN,
          CKPT_DIR_WIN, PREVIEW_DIR_WIN, MODELS_DIR_WIN):
    os.makedirs(d, exist_ok=True)

# Define the schedule for annealing the identity loss.
ID_ANNEAL_START_FRAC = 0.15
ID_ANNEAL_END_FRAC   = 0.50
ID_ANNEAL_MIN_FACTOR = 0.10

# Calculate the start and end steps for the anneal period.
ID_ANNEAL_START_STEP = int(ID_ANNEAL_START_FRAC * TOT_STEPS)
ID_ANNEAL_END_STEP   = int(ID_ANNEAL_END_FRAC * TOT_STEPS)

# Print a summary of the training configuration.
print("Schedule:", {
    "BATCH_SIZE": BATCH_SIZE, "EPOCHS": EPOCHS, "STEPS_PER_EPOCH": STEPS_PER_EPOCH,
    "SHUFFLE_BUFFER": SHUFFLE_BUFFER, "LOG_EVERY": LOG_EVERY, "PREVIEW_EVERY": PREVIEW_EVERY,
    "SAVE_EVERY_EPOCH": SAVE_EVERY_EPOCH
})
print("Anneal λ_id:", {
    "start_step": ID_ANNEAL_START_STEP, "end_step": ID_ANNEAL_END_STEP,
    "min_factor": ID_ANNEAL_MIN_FACTOR
})
print("Linux   ->", LINUX_BASE)
print("Windows ->", WIN_BASE)

Schedule: {'BATCH_SIZE': 1, 'EPOCHS': 15, 'STEPS_PER_EPOCH': 2000, 'SHUFFLE_BUFFER': 2048, 'LOG_EVERY': 250, 'PREVIEW_EVERY': 1000, 'SAVE_EVERY_EPOCH': True}
Anneal λ_id: {'start_step': 4500, 'end_step': 15000, 'min_factor': 0.1}
Linux   -> /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts
Windows -> /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition

7.7 Checkpoints & Resume

Checkpoint G_X2Y/G_Y2X/D_X/D_Y, write compact logs, and support safe resume from latest.

---

# Set a flag to control whether to resume training from a checkpoint.
RESUME_FROM_CKPT = True

# Create a checkpoint object that tracks all models and optimizers.
ckpt = tf.train.Checkpoint(
    G_X2Y=G_X2Y, G_Y2X=G_Y2X,
    D_X=D_X, D_Y=D_Y,
    opt_G_X2Y=opt_G_X2Y, opt_G_Y2X=opt_G_Y2X,
    opt_D_X=opt_D_X,     opt_D_Y=opt_D_Y
)

# Create checkpoint managers for both Linux and Windows paths.
ckpt_manager_lin = tf.train.CheckpointManager(
    ckpt, CKPT_DIR_LIN, max_to_keep=5, checkpoint_name="ckpt"
)
ckpt_manager_win = tf.train.CheckpointManager(
    ckpt, CKPT_DIR_WIN, max_to_keep=5, checkpoint_name="ckpt"
)

# Find the latest checkpoint if resuming is enabled.
latest = ckpt_manager_lin.latest_checkpoint if RESUME_FROM_CKPT else None

# Restore the latest checkpoint or prepare for a fresh start.
if latest:
    ckpt.restore(latest).expect_partial()
    print(f"Resumed from (Linux): {latest}")
else:
    # Reset optimizer iterations to ensure a truly fresh run.
    for opt in (opt_G_X2Y, opt_G_Y2X, opt_D_X, opt_D_Y):
        opt.iterations.assign(0)
    print("Resume disabled (or no checkpoint). Starting fresh.")
    print("Checkpoint dirs:")
    print("  Linux   ->", CKPT_DIR_LIN)
    print("  Windows ->", CKPT_DIR_WIN)

## Clean up the session state for a clean run.
#for name in ("trainer", "pool_X_fake", "pool_Y_fake"):
#    if name in globals():
#        del globals()[name]
#tf.keras.backend.clear_session()
#gc.collect()

Resumed from (Linux): /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/checkpoints/ckpt-15

7.8 Launch Full Training

With all components defined and verified, this cell executes the main training loop. It uses the custom trainer to iterate through the dataset for the scheduled number of epochs, applying the identity loss anneal, saving periodic image previews, and checkpointing the model at the end of each epoch.

---

# Assert that all required components from previous sections are available.
assert "photo_train_ds" in globals() and "monet_train_ds" in globals(), "Run Section 5 first."
assert all(v in globals() for v in ("G_X2Y", "G_Y2X", "D_X", "D_Y")), "Run Section 6 first."
assert all(v in globals() for v in ("opt_G_X2Y", "opt_G_Y2X", "opt_D_X", "opt_D_Y")), "Run 7.2 first."
assert all(v in globals() for v in ("EPOCHS", "STEPS_PER_EPOCH", "BATCH_SIZE", "LOG_EVERY", "PREVIEW_EVERY", "SAVE_EVERY_EPOCH")), "Run 7.6 first."
assert all(v in globals() for v in ("ckpt_manager_lin", "ckpt_manager_win")), "Run 7.7 first."

# Zip the final datasets from Section 5 for paired training.
ds_train = tf.data.Dataset.zip((photo_train_ds, monet_train_ds)).prefetch(AUTOTUNE)

# Sanity check the effective batch size.
bx_chk, by_chk = next(iter(ds_train))
print("Effective training batch:", int(bx_chk.shape[0]), "/", int(by_chk.shape[0]))
del bx_chk, by_chk

# Create the main trainer instance.
trainer = CycleGANTrainer(
    G_X2Y=G_X2Y, G_Y2X=G_Y2X, D_X=D_X, D_Y=D_Y,
    lambda_cycle=LAMBDA_CYCLE, lambda_id=LAMBDA_ID_BASE,
    label_smooth=LABEL_SMOOTH, use_pools=False
)
trainer.compile(
    opt_G_X2Y=opt_G_X2Y, opt_G_Y2X=opt_G_Y2X,
    opt_D_X=opt_D_X,     opt_D_Y=opt_D_Y
)

# --- Helper Functions ---

# Define a function to safely convert tensors to uint8 for saving as images.
def _safe_uint8(img):
    img = tf.where(tf.math.is_finite(img), img, tf.zeros_like(img))
    img = tf.clip_by_value(img, -1.0, 1.0)
    img = (tf.cast(img, tf.float32) * 127.5) + 127.5
    return tf.cast(img, tf.uint8)

# Define a function to generate and save a set of preview images.
def _save_preview_dual(bx, by, epoch_idx, step_idx):
    # Generate fake images for the preview.
    with tf.device("/GPU:0"):
        y_fake = G_X2Y(bx, training=False)
        x_fake = G_Y2X(by, training=False)
    
    # Check for non-finite values before attempting to save.
    if not (tf.reduce_all(tf.math.is_finite(y_fake)) and tf.reduce_all(tf.math.is_finite(x_fake))):
        print(f"[preview skipped @ e{epoch_idx} s{step_idx}] non-finite outputs")
        return
        
    # Convert the first image of each batch to a saveable format.
    x0, y0 = _safe_uint8(bx[0]), _safe_uint8(by[0])
    yhat0, xhat0 = _safe_uint8(y_fake[0]), _safe_uint8(x_fake[0])
    
    # Save the four preview images to both Linux and Windows directories.
    for root in (PREVIEW_DIR_LIN, PREVIEW_DIR_WIN):
        Image.fromarray(x0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_photo_in.png"))
        Image.fromarray(y0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_monet_in.png"))
        Image.fromarray(yhat0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_photo2monet.png"))
        Image.fromarray(xhat0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_monet2photo.png"))

# Define the identity loss annealing schedule.
def _id_factor(gstep: int) -> float:
    if gstep < ID_ANNEAL_START_STEP: return 1.0
    if gstep >= ID_ANNEAL_END_STEP: return float(ID_ANNEAL_MIN_FACTOR)
    t = (gstep - ID_ANNEAL_START_STEP) / max(1, ID_ANNEAL_END_STEP - ID_ANNEAL_START_STEP)
    return 1.0 - t * (1.0 - float(ID_ANNEAL_MIN_FACTOR))

# --- Training Setup ---

# Calculate the starting point for resuming training.
global_iter = int(opt_G_X2Y.iterations.numpy())
start_epoch = global_iter // STEPS_PER_EPOCH
offset_step = global_iter %  STEPS_PER_EPOCH

# Set up tracking for the best model weights.
best_G_total = float("inf")
best_epoch   = -1
best_dir_lin = os.path.join(MODELS_DIR_LIN, "best")
best_dir_win = os.path.join(MODELS_DIR_WIN, "best")
os.makedirs(best_dir_lin, exist_ok=True)
os.makedirs(best_dir_win, exist_ok=True)

# --- Main Training Loop ---
print(f"Starting from epoch {start_epoch+1} (offset step {offset_step})")
train_t0 = time.perf_counter()
for epoch in range(start_epoch, EPOCHS):
    # Reset metrics at the start of each epoch.
    for metric in trainer.metrics:
        metric.reset_state()
        
    # Create and optionally fast-forward the dataset iterator.
    it = iter(ds_train)
    if epoch == start_epoch and offset_step > 0:
        for _ in range(offset_step):
            _ = next(it)
            
    # Initialize the global step counter from the optimizer's state.
    gstep = int(opt_G_X2Y.iterations.numpy())
    epoch_t0 = time.perf_counter()

    # Set up the progress bar for the current epoch.
    pbar = tqdm(range(offset_step + 1, STEPS_PER_EPOCH + 1),
                desc=f"epoch {epoch+1}/{EPOCHS}", leave=True, mininterval=0.2)
    for step in pbar:
        # Get the next batch of images.
        bx, by = next(it)

        # Scale images and run a training step on the GPU.
        with tf.device("/GPU:0"):
            two = tf.constant(2.0, dtype=bx.dtype)
            one = tf.constant(1.0, dtype=bx.dtype)
            bx_scaled = bx * two - one
            by_scaled = by * two - one
            
            # Anneal the identity loss weight and perform one training step.
            trainer.lambda_id = float(LAMBDA_ID_BASE) * _id_factor(gstep)
            metrics = trainer.train_step((bx_scaled, by_scaled))
        
        # Increment the global step counter.
        gstep += 1

        # Update the progress bar with the latest metrics.
        if step % LOG_EVERY == 0 or step == STEPS_PER_EPOCH:
            gt = float(trainer.m_G_total.result().numpy())
            dx = float(trainer.m_DX.result().numpy())
            dy = float(trainer.m_DY.result().numpy())
            pbar.set_postfix({
                "G_total": f"{gt:.3f}", "D_X": f"{dx:.3f}",
                "D_Y": f"{dy:.3f}", "λ_id": f"{trainer.lambda_id:.3f}"
            })

        # Generate and save preview images at specified intervals.
        if PREVIEW_EVERY and (step % PREVIEW_EVERY == 0 or step == 1):
            _save_preview_dual(bx_scaled, by_scaled, epoch+1, step)
            
    # --- End of Epoch ---
    dt = time.perf_counter() - epoch_t0
    gtotal_epoch = float(trainer.m_G_total.result().numpy())
    print(f"[epoch {epoch+1}] avg G_total={gtotal_epoch:.4f}")
    print(f"[epoch {epoch+1}] done in {dt:.1f}s (~{STEPS_PER_EPOCH/dt:.1f} steps/s)")

    # Save the best model weights if performance improved.
    if gtotal_epoch < best_G_total:
        best_G_total = gtotal_epoch
        best_epoch   = epoch + 1
        G_X2Y.save_weights(os.path.join(best_dir_lin, "G_X2Y.best.weights.h5"))
        G_Y2X.save_weights(os.path.join(best_dir_lin, "G_Y2X.best.weights.h5"))
        G_X2Y.save_weights(os.path.join(best_dir_win, "G_X2Y.best.weights.h5"))
        G_Y2X.save_weights(os.path.join(best_dir_win, "G_Y2X.best.weights.h5"))
        print(f"  new BEST at epoch {best_epoch} (G_total={best_G_total:.4f}) - weights saved (Linux + Windows).")

    # Save a full checkpoint at the end of the epoch.
    if SAVE_EVERY_EPOCH:
        path_lin = ckpt_manager_lin.save(checkpoint_number=(epoch + 1))
        path_win = ckpt_manager_win.save(checkpoint_number=(epoch + 1))
        print("  checkpoints saved:\n   Linux   ->", path_lin, "\n   Windows ->", path_win)

    # Clear the resume offset after the first epoch is complete.
    offset_step = 0

# --- End of Training ---
total_dt = time.perf_counter() - train_t0
print(f"\nTraining complete in {total_dt/3600:.2f} h total.")
print(f"Best epoch: {best_epoch} (G_total={best_G_total:.4f})")

Effective training batch: 1 / 1
Starting from epoch 16 (offset step 0)

Training complete in 0.00 h total.
Best epoch: -1 (G_total=inf)

Note on Cell Output: Due to a Jupyter kernel crash (notebook controller is DISPOSED) that occurred after the initial training was complete, the live output from this cell was lost. Rather than re-running the entire 8.7-hour training process, the following screenshots faithfully recreate the original output, documenting the complete training history from epoch 1 to 15. The notebook's state has been successfully restored by loading the final weights from the ckpt-15 checkpoint in 7.7, allowing the project to proceed with the fully trained model.

Observation: The Training Gauntlet and Final Results

The journey to a fully trained model was a two-part story: a grueling, multi-hour battle against the technical limits of the hardware, followed by a successful and stable 8.7-hour training run that produced a high-quality result. This observation documents both the human story of the debugging process and the final quantitative analysis of the successful training run.

The Story: How I Won the War Against OOM

I thought I was ready to hit “run” and watch Monet magic happen. Instead, my GPU greeted me with a wall of cryptic error messages: Out-of-Memory errors during fundamental operations like a Conv2D or even a simple multiplication. Sometimes the process would die before the first preview image was generated; other times it would limp into the second epoch before failing. I didn’t want a theory of OOM; I needed a run that finished. So I began a methodical hunt for the root cause. Every section, every line of code that could inflate a tensor, duplicate a buffer, or sneak in an unnecessary data cast was put under the microscope.

The Spiral (What Was Going Wrong)

The problem wasn't a single bug but a conspiracy of memory sinks that were invisible during the short micro-train. The main culprits were:

• Double-Batching: I had batch logic in the Section 5 pipeline and was re-batching in the training loop. That quietly multiplied memory even when the “effective” batch size looked like 1.
• Replay Buffers: Great for stability, but they keep extra copies of 256×256 feature maps around. On an 8 GB card, that’s a death sentence.
• Inefficient Gradient Tapes: Using two separate, persistent GradientTapes for the generators plus two more for the discriminators held onto far too much memory between steps.
• Cross-Device Data Transfers: I was rescaling to [-1, 1] on the CPU with an on-the-fly tf.cast, then shipping the bloated tensors to the GPU every single step, causing a memory spike and a copy operation.
• Stale Kernel State: Checkpoint counters and leftover trainer or pool objects from failed runs made new attempts start at epoch 2 or carry hidden memory allocations.

The Breakthroughs (The Fixes That Moved the Needle)

After the diagnosis, the work began. This is the sequence of fixes that finally stabilized the run:

1. Pinned BATCH_SIZE = 1 in §5.4 and stopped all re-batching later. One pipeline now owns batching. Period.
2. Zipped the §5 streams directly in the training cell. No .unbatch() → .batch() dance. No surprise memory balloons.
3. Swapped in the memory-lean trainer from §7.4, which uses one non-persistent tape for both generators and short tapes for each discriminator. Same math, less RAM.
4. Turned off replay buffers by setting use_pools=False. This was the big unlock on my 8 GB GPU.
5. Moved the [0,1] → [-1,1] rescale onto the GPU and performed it without a data type cast, all inside the GPU:0 block. No host to device ping-pong, no extra copies.
6. Enabled GPU memory growth to stop TensorFlow from pre-grabbing all VRAM and then choking when memory requirements shifted mid-step.
7. Nuked stale state before every fresh run by deleting trainer and pool objects from globals, followed by tf.keras.backend.clear_session(); gc.collect().
8. Guarded the preview function with _safe_uint8 and a non-finite check. If outputs went NaN/Inf, I now skip saving that preview instead of writing a black square.
9. Implemented the identity-loss anneal to smooth the early training bumps that previously spiraled into NaNs.
10. Fixed Keras 3 weight filenames to save with the required *.weights.h5 suffix.
11. Made the resume process explicit with the RESUME_FROM_CKPT flag and a hard reset of optimizer.iterations to prevent accidental continuation.

---

The Data: Quantitative Analysis of the 8.7-Hour Run

After the extensive debugging, the model was trained for 15 epochs, totaling 30,000 steps. The training logs provide a clear quantitative picture of the model's learning dynamics, the adversarial balance, and the impact of the identity loss annealing schedule.

• Overall Convergence:

  • The primary success metric, the average total generator loss (G_total), showed a strong and consistent downward trend throughout the entire run.
  • The loss decreased by over 50%, starting at 8.49 at the end of epoch 1 and reaching its best value of 4.06 at the conclusion of epoch 15. This steady improvement is a clear indicator of successful and stable training.

• Adversarial Dynamics:

  • The discriminator losses (D_X and D_Y) rapidly decreased from their high initial values in the first epoch and then stabilized in a healthy, low range of approximately 0.16 to 0.25 for the remainder of the training.
  • This is the hallmark of a well-functioning GAN. The discriminators quickly became effective at distinguishing real from fake images but did not overpower the generators, allowing for a stable and productive adversarial contest.

• Impact of Identity Loss Annealing:

  • The identity loss weight (λ_id) performed exactly as scheduled. It was held at its initial value of 5.0 for the first 4,500 steps, providing a strong regularization signal early on.
  • The annealing process began in epoch 3, with λ_id linearly decreasing until it hit its floor of 0.5 at the end of epoch 7. The training remained perfectly stable during this transition, and the most significant improvements in G_total occurred during this phase (dropping from 8.49 to 4.78).

• System Performance:

  • The final, optimized configuration proved to be highly efficient. The system maintained a consistent throughput of approximately 0.9 - 1.0 steps per second (~980-1100 ms per step) across the entire 8.7-hour run, with no performance degradation over time.

Conclusion: The journey to a trained model was defined by the battle against hardware limits. The debugging process resulted in a lean, stable configuration that not only worked but thrived, leading to a successful 15-epoch training run. The quantitative results confirm that the generator loss consistently improved while the adversarial dynamic remained balanced. The final model from epoch 15, with a G_total of 4.0607, is the product of this entire effort and is ready for final evaluation.

Section 8: Fine-Tuning & Final Submission

After the initial 15-epoch training run produced a solid but ultimately disappointing score, a second, more targeted experiment was designed to push the model's performance over the finish line. This section covers the fine-tuning process, where the learning rate and loss functions were adjusted for refinement, and the final submission generation, which uses Test-Time Augmentation (TTA) to maximize the final score.

Section Plan:

1. Export & Baseline Submission: Save the initial model from the 15-epoch run and generate the first submission file, establishing the baseline score of ~69.
2. Fine-Tuning Experiment: Run a short, targeted "top-up" training session with a lower learning rate, re-enabled replay buffers, and a more aggressive identity loss schedule to encourage stronger stylization.
3. Generate Final Submission with TTA: Create a new, improved submission file by generating images from both the original and horizontally-flipped test photos, applying a final border-crop post-processing step.

---

8.1 Final Model Export & Submission Generation

With training complete, this final step saves the trained generator models in multiple formats (SavedModel, Keras, and weights-only) for portability. It then runs a quick smoke test to visually confirm the generators produce valid images, and finally, it generates the images.zip file required for the official Kaggle submission.

---

# --- Assert that all required components are available ---
assert all(v in globals() for v in ("G_X2Y", "G_Y2X")), "Run Sections 6/7 first."
assert all(v in globals() for v in ("photo_train_ds", "monet_train_ds")), "Section 5 datasets not found."
assert all(v in globals() for v in ("MODELS_DIR_LIN", "MODELS_DIR_WIN")), "Model dirs missing (7.6/7.7)."
assert all(v in globals() for v in ("EPOCHS", "STEPS_PER_EPOCH", "BATCH_SIZE", "IMG_SIZE")), "Schedule constants missing."
assert "PHOTO_JPG_DIR" in globals(), "PHOTO_JPG_DIR missing."

# --- Set final configuration parameters ---
IMG_SIDE        = int(IMG_SIZE)
SMOKE_SAMPLES   = 8
ZIP_BATCH       = 1  # Batch size for generating submission images.
TOTAL_IMG_CNT   = 7000 # The exact number of images required by the competition.

# --- Define helper to suppress verbose Keras export logs ---
def _quiet_export(model, path):
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf):
        model.export(path)

# --- Create all output directories for artifacts ---
# Define the Linux paths.
final_dir_sm_lin = os.path.join(MODELS_DIR_LIN, "savedmodel_final")
final_dir_kr_lin = os.path.join(MODELS_DIR_LIN, "keras_final")
final_dir_wt_lin = os.path.join(MODELS_DIR_LIN, "weights_final")
smoke_dir_lin    = os.path.join(MODELS_DIR_LIN, "smoke_test")
zip_lin          = os.path.join(MODELS_DIR_LIN, "images.zip")
# Define the mirrored Windows paths.
final_dir_sm_win = os.path.join(MODELS_DIR_WIN, "savedmodel_final")
final_dir_kr_win = os.path.join(MODELS_DIR_WIN, "keras_final")
final_dir_wt_win = os.path.join(MODELS_DIR_WIN, "weights_final")
smoke_dir_win    = os.path.join(MODELS_DIR_WIN, "smoke_test")
zip_win          = os.path.join(MODELS_DIR_WIN, "images.zip")
# Create the directories if they do not exist.
for d in (final_dir_sm_lin, final_dir_kr_lin, final_dir_wt_lin, smoke_dir_lin,
          final_dir_sm_win, final_dir_kr_win, final_dir_wt_win, smoke_dir_win):
    os.makedirs(d, exist_ok=True)

# --- Save the final trained generator models ---
print("Saving final generators ...")
# Save in SavedModel format for both Linux and Windows.
_quiet_export(G_X2Y, os.path.join(final_dir_sm_lin, "G_X2Y"))
_quiet_export(G_Y2X, os.path.join(final_dir_sm_lin, "G_Y2X"))
_quiet_export(G_X2Y, os.path.join(final_dir_sm_win, "G_X2Y"))
_quiet_export(G_Y2X, os.path.join(final_dir_sm_win, "G_Y2X"))
# Save in Keras format for both Linux and Windows.
G_X2Y.save(os.path.join(final_dir_kr_lin, "G_X2Y.final.keras"))
G_Y2X.save(os.path.join(final_dir_kr_lin, "G_Y2X.final.keras"))
G_X2Y.save(os.path.join(final_dir_kr_win, "G_X2Y.final.keras"))
G_Y2X.save(os.path.join(final_dir_kr_win, "G_Y2X.final.keras"))
# Save in weights-only (.h5) format for both Linux and Windows.
G_X2Y.save_weights(os.path.join(final_dir_wt_lin, "G_X2Y.final.weights.h5"))
G_Y2X.save_weights(os.path.join(final_dir_wt_lin, "G_Y2X.final.weights.h5"))
G_X2Y.save_weights(os.path.join(final_dir_wt_win, "G_Y2X.final.weights.h5"))
G_Y2X.save_weights(os.path.join(final_dir_wt_win, "G_Y2X.final.weights.h5"))

# --- Run a small smoke test to visually verify the models ---
# Helper to convert a tensor in [0, 1] range to a uint8 NumPy array.
def _u8_from_01(img):
    img = tf.clip_by_value(img, 0.0, 1.0) * 255.0
    return tf.cast(img, tf.uint8)
# Helper to convert a tensor in [-1, 1] range to a uint8 NumPy array.
def _u8_from_m1p1(img):
    img = (tf.cast(img, tf.float32) + 1.0) * 127.5
    img = tf.clip_by_value(img, 0.0, 255.0)
    return tf.cast(img, tf.uint8)
# Helper to save a batch of image tensors as PNG files.
def _save_n_pngs(tensor, path_prefix, space="01", n=4):
    n = min(int(tensor.shape[0]), n)
    for i in range(n):
        u8_converter = _u8_from_01 if space == "01" else _u8_from_m1p1
        arr = u8_converter(tensor[i]).numpy()
        Image.fromarray(arr).save(f"{path_prefix}_{i}.png")

# Grab one batch from the training datasets for the smoke test.
bx = next(iter(photo_train_ds.take(1)))
by = next(iter(monet_train_ds.take(1)))
# Scale the images to [-1, 1] and run a forward pass on the GPU.
with tf.device("/GPU:0"):
    two = tf.constant(2.0, dtype=bx.dtype)
    one = tf.constant(1.0, dtype=bx.dtype)
    bx_n = bx * two - one
    by_n = by * two - one
    y_fake = G_X2Y(bx_n, training=False)
    x_fake = G_Y2X(by_n, training=False)

# Save the input and generated images for the smoke test.
for root in (smoke_dir_lin, smoke_dir_win):
    _save_n_pngs(bx,     os.path.join(root, "photo_in"),    "01",    n=SMOKE_SAMPLES)
    _save_n_pngs(by,     os.path.join(root, "monet_in"),    "01",    n=SMOKE_SAMPLES)
    _save_n_pngs(y_fake, os.path.join(root, "photo2monet"), "-1..1", n=SMOKE_SAMPLES)
    _save_n_pngs(x_fake, os.path.join(root, "monet2photo"), "-1..1", n=SMOKE_SAMPLES)
print("Models saved and smoke images written.")

# --- Build the Kaggle submission zip file ---
# Get a sorted list of exactly 7000 photo JPEGs.
jpgs = sorted(glob.glob(os.path.join(str(PHOTO_JPG_DIR), "*.jpg")))
if len(jpgs) < TOTAL_IMG_CNT:
    raise FileNotFoundError(f"Found {len(jpgs)} JPGs, need at least {TOTAL_IMG_CNT}.")
jpgs = jpgs[:TOTAL_IMG_CNT]

# Define a helper to load and normalize an image for inference.
def _load_and_norm(path):
    img = tf.io.read_file(path)
    img = tf.io.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [IMG_SIDE, IMG_SIDE], method="area")
    img = tf.cast(img, tf.float32) / 127.5 - 1.0
    return img
# Define a helper to de-normalize a generated image for saving.
def _denorm_to_u8(t):
    t = (tf.cast(t, tf.float32) + 1.0) * 127.5
    t = tf.clip_by_value(t, 0.0, 255.0)
    return tf.cast(t, tf.uint8)

# Initialize the progress bar and timer.
t0 = time.perf_counter()
pbar = tqdm(range(0, TOTAL_IMG_CNT, ZIP_BATCH),
            desc="Building images.zip", leave=True, mininterval=0.2)

# Open zip files for both Linux and Windows paths.
with zipfile.ZipFile(zip_lin, "w", compression=zipfile.ZIP_DEFLATED) as z_lin, \
     zipfile.ZipFile(zip_win, "w", compression=zipfile.ZIP_DEFLATED) as z_win:
    # Process the photos in batches.
    for start in pbar:
        # Load and batch the input photos.
        paths = jpgs[start : start + ZIP_BATCH]
        bx = tf.stack([_load_and_norm(p) for p in paths], axis=0)
        # Generate the Monet-style images on the GPU.
        with tf.device("/GPU:0"):
            y_fake = G_X2Y(bx, training=False)
        # De-normalize the output images.
        arrs = _denorm_to_u8(y_fake).numpy()

        # Save each generated image to both zip files.
        for j, arr in enumerate(arrs):
            buf = io.BytesIO()
            Image.fromarray(arr).save(buf, format="PNG")
            # Name the files sequentially from 0.png to 6999.png.
            name = f"{start + j}.png"
            z_lin.writestr(name, buf.getvalue())
            z_win.writestr(name, buf.getvalue())
        
        # Update the progress bar.
        pbar.set_postfix({"png": f"{min(start + ZIP_BATCH, TOTAL_IMG_CNT)}/{TOTAL_IMG_CNT}"})

dt = time.perf_counter() - t0

# --- Print Final Paths and Performance ---
print("\nSaved paths - Linux")
print("  SavedModel :", final_dir_sm_lin)
print("  Keras      :", final_dir_kr_lin)
print("  Weights    :", final_dir_wt_lin)
print("  Smoke imgs :", smoke_dir_lin)
print("  images.zip :", zip_lin)
print("\nSaved paths - Windows")
print("  SavedModel :", final_dir_sm_win)
print("  Keras      :", final_dir_kr_win)
print("  Weights    :", final_dir_wt_win)
print("  Smoke imgs :", smoke_dir_win)
print("  images.zip :", zip_win)
print("\nBest checkpoint dirs (created during training):")
print("  Linux   :", os.path.join(MODELS_DIR_LIN, "best"))
print("  Windows :", os.path.join(MODELS_DIR_WIN, "best"))
print(f"\nimages.zip build time: {dt/60:.1f} min  |  ~{TOTAL_IMG_CNT/dt:.1f} imgs/s  |  ~{dt/TOTAL_IMG_CNT:.3f} s/img")

Saving final generators ...
Models saved and smoke images written.

Building images.zip:   0%|          | 0/7000 [00:00<?, ?it/s]

Saved paths - Linux
  SavedModel : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/savedmodel_final
  Keras      : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/keras_final
  Weights    : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/weights_final
  Smoke imgs : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/smoke_test
  images.zip : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/images.zip

Saved paths - Windows
  SavedModel : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/savedmodel_final
  Keras      : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/keras_final
  Weights    : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/weights_final
  Smoke imgs : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/smoke_test
  images.zip : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/images.zip

Best checkpoint dirs (created during training):
  Linux   : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/best
  Windows : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/best

images.zip build time: 7.7 min  |  ~15.1 imgs/s  |  ~0.066 s/img

Observation: A Great Execution, A Painful Score

This final step represents the culmination of the entire project: taking the best-performing model from the grueling 8.7-hour training run and packaging it into the final artifacts for submission. The process itself was a success. The code ran as expected (finally), saving the final models and generating the 7,000 images for the images.zip file in just under 9 minutes. This technical victory, however, was immediately overshadowed by a deeply disappointing result on the Kaggle leaderboard.

• Successful Artifact Generation:

  • The final trained generator, G_X2Y, was successfully exported in three standard formats (SavedModel, Keras, and weights-only) and mirrored to both Linux and Windows paths for maximum portability.
  • A quick smoke test confirmed that the saved model loads correctly and produces valid, coherent images, verifying the integrity of the final artifacts.
  • The critical Kaggle submission file, images.zip, was generated efficiently at a rate of ~13.3 images per second.

• The Final Score: A Missed Goal (Key Finding):

  • Despite the stable training run and the flawless execution of the pipeline, the final Kaggle score of 69.06 was a significant letdown. My personal goal for this project was to break the 60-point barrier, and missing it by such a wide margin after the extensive debugging and training effort was frustrating.

• Analysis of the Result:
  • A score of ~69 is a strong baseline that proves the model is learning the correct style transfer. However, it's not a top-tier score. The gap between the result and the goal can be attributed to the necessary sacrifices made during the debugging phase to achieve a stable run on an 8GB GPU.
  • The most significant factor was likely the decision to disable the image replay buffers (use_pools=False). While this was the key to preventing OOM errors, it's a feature known to improve the fine-grained quality and realism of generated images. The final score reflects a model that learned the general Monet style but lacked the final polish that these stabilizing techniques provide.

Conclusion: While the final score was not what I hoped for, the project was a success in building a complete, end-to-end CycleGAN pipeline that verifiably works. This score of 69.06 should not be viewed as a failure, but as a hard-won baseline. It provides a concrete benchmark against which all future improvements can be measured. The path forward is now clear: the next phase of this project will focus on targeted experiments, such as longer training times and re-enabling replay buffers on more powerful hardware, to close the 9-point gap and achieve the original goal.

8.2 Experiment: Fine-Tuning with Aggressive Stylization

To improve the model, this step performs a short "top-up" training run. The strategy is to fine-tune the existing weights by using a halved generator learning rate for more stable updates, while simultaneously encouraging stronger stylization with a new, aggressive annealing schedule for the identity loss.

---

# --- Assert that all required components are available ---
assert "photo_train_ds" in globals() and "monet_train_ds" in globals(), "Run Section 5 first."
assert all(v in globals() for v in ("G_X2Y", "G_Y2X", "D_X", "D_Y")), "Run Section 6 first."
assert all(v in globals() for v in ("opt_G_X2Y", "opt_G_Y2X", "opt_D_X", "opt_D_Y")), "Run 7.2 first."
assert "CycleGANTrainer" in globals(), "Run 7.4 first."

# --- Configure the fine-tuning schedule ---
EPOCHS_TOPUP    = 10
STEPS_TOPUP     = 250
PRINT_EVERY     = 50
ID_MIN_TOPUP    = 0.30

# Reduce the generator learning rates by 50% for fine-tuning.
for opt in (opt_G_X2Y, opt_G_Y2X):
    try:
        new_lr = float(opt.learning_rate) * 0.5
        opt.learning_rate.assign(new_lr)
    except Exception:
        pass
print("Generator LRs halved to:", float(opt_G_X2Y.learning_rate))

# --- Rebuild the training components ---
AUTOTUNE = tf.data.AUTOTUNE
ds_train = tf.data.Dataset.zip((photo_train_ds, monet_train_ds)).prefetch(AUTOTUNE)

# Re-instantiate the trainer with the existing models and updated optimizers.
trainer = CycleGANTrainer(
    G_X2Y=G_X2Y, G_Y2X=G_Y2X, D_X=D_X, D_Y=D_Y,
    lambda_cycle=LAMBDA_CYCLE, lambda_id=LAMBDA_ID_BASE,
    label_smooth=LABEL_SMOOTH, use_pools=True
)
trainer.compile(
    opt_G_X2Y=opt_G_X2Y, opt_G_Y2X=opt_G_Y2X,
    opt_D_X=opt_D_X,     opt_D_Y=opt_D_Y
)

# --- Re-define helper functions ---
def _safe_uint8(img):
    img = tf.where(tf.math.is_finite(img), img, tf.zeros_like(img))
    img = tf.clip_by_value(img, -1.0, 1.0)
    img = (tf.cast(img, tf.float32) * 127.5) + 127.5
    return tf.cast(img, tf.uint8)

def _save_preview_dual(bx, by, epoch_idx, step_idx):
    with tf.device("/GPU:0"):
        y_fake = G_X2Y(bx, training=False)
        x_fake = G_Y2X(by, training=False)
    x0, y0 = _safe_uint8(bx[0]), _safe_uint8(by[0])
    yhat0, xhat0 = _safe_uint8(y_fake[0]), _safe_uint8(x_fake[0])
    for root in (PREVIEW_DIR_LIN, PREVIEW_DIR_WIN):
        Image.fromarray(x0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_photo_in.png"))
        Image.fromarray(y0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_monet_in.png"))
        Image.fromarray(yhat0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_photo2monet.png"))
        Image.fromarray(xhat0.numpy()).save(os.path.join(root, f"e{epoch_idx:02d}_s{step_idx:04d}_monet2photo.png"))

# Define a new, aggressive anneal schedule just for this fine-tuning run.
TOTAL_STEPS_TOPUP = EPOCHS_TOPUP * STEPS_TOPUP
def _id_factor_local(gstep: int) -> float:
    t = min(max(gstep / max(1, TOTAL_STEPS_TOPUP), 0.0), 1.0)
    return 1.0 - t * (1.0 - float(ID_MIN_TOPUP))

# --- Main Fine-Tuning Loop ---
print(f"Top-up training: {EPOCHS_TOPUP} epochs x {STEPS_TOPUP} steps (λ_id min {ID_MIN_TOPUP})")
global_step = 0
t0_all = time.perf_counter()
for epoch in range(EPOCHS_TOPUP):
    # Reset metrics at the start of each epoch.
    for metric in trainer.metrics:
        metric.reset_state()

    # Create the dataset iterator and progress bar.
    it = iter(ds_train)
    t0_epoch = time.perf_counter()
    pbar = tqdm(range(1, STEPS_TOPUP + 1),
                desc=f"epoch {epoch+16}/{15 + EPOCHS_TOPUP}", leave=True, mininterval=0.2)
    for step in pbar:
        bx, by = next(it)

        # Scale inputs and perform one training step on the GPU.
        with tf.device("/GPU:0"):
            bx_scaled = bx * tf.constant(2.0, dtype=bx.dtype) - tf.constant(1.0, dtype=bx.dtype)
            by_scaled = by * tf.constant(2.0, dtype=by.dtype) - tf.constant(1.0, dtype=by.dtype)
            trainer.lambda_id = float(LAMBDA_ID_BASE) * _id_factor_local(global_step)
            metrics = trainer.train_step((bx_scaled, by_scaled))

        global_step += 1

        # Update the progress bar and print logs at specified intervals.
        if (step % PRINT_EVERY == 0) or (step == 1) or (step == STEPS_TOPUP):
            gt = float(metrics["G_total"])
            dx = float(metrics["D_X"])
            dy = float(metrics["D_Y"])
            lam = float(trainer.lambda_id)
            print(f"step {step}/{STEPS_TOPUP}  G_total={gt:.3f}  D_X={dx:.3f}  D_Y={dy:.3f}  λ_id={lam:.3f}")
            pbar.set_postfix({"G_total": f"{gt:.3f}", "D_X": f"{dx:.3f}","D_Y": f"{dy:.3f}", "λ_id": f"{trainer.lambda_id:.3f}"
})

    # --- End of Epoch ---
    dt = time.perf_counter() - t0_epoch
    gtotal_epoch = float(trainer.m_G_total.result().numpy())
    print(f"[epoch {15+epoch+1}] avg G_total={gtotal_epoch:.4f}")
    print(f"[epoch {15+epoch+1}] done in {dt:.1f}s (~{STEPS_TOPUP/dt:.1f} steps/s)")
    
    # Overwrite the 'best' weights at the end of each fine-tuning epoch.
    G_X2Y.save_weights(os.path.join(MODELS_DIR_LIN, "best", "G_X2Y.best.weights.h5"))
    G_Y2X.save_weights(os.path.join(MODELS_DIR_LIN, "best", "G_Y2X.best.weights.h5"))
    G_X2Y.save_weights(os.path.join(MODELS_DIR_WIN, "best", "G_X2Y.best.weights.h5"))
    G_Y2X.save_weights(os.path.join(MODELS_DIR_WIN, "best", "G_Y2X.best.weights.h5"))
    print("  best weights saved (Linux + Windows).")

print(f"\nTop-up complete in {(time.perf_counter() - t0_all)/3600:.2f} h total.")

Generator LRs halved to: 9.999999747378752e-05
Top-up training: 10 epochs x 250 steps (λ_id min 0.3)

epoch 16/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=5.054  D_X=0.206  D_Y=0.139  λ_id=5.000
step 50/250  G_total=5.128  D_X=0.144  D_Y=0.120  λ_id=4.931
step 100/250  G_total=5.243  D_X=0.141  D_Y=0.124  λ_id=4.861
step 150/250  G_total=5.225  D_X=0.139  D_Y=0.124  λ_id=4.791
step 200/250  G_total=5.176  D_X=0.137  D_Y=0.126  λ_id=4.721
step 250/250  G_total=5.131  D_X=0.141  D_Y=0.127  λ_id=4.651
[epoch 16] avg G_total=5.1311
[epoch 16] done in 280.4s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 17/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=7.142  D_X=0.225  D_Y=0.040  λ_id=4.650
step 50/250  G_total=5.122  D_X=0.130  D_Y=0.146  λ_id=4.581
step 100/250  G_total=4.997  D_X=0.153  D_Y=0.136  λ_id=4.511
step 150/250  G_total=4.895  D_X=0.150  D_Y=0.137  λ_id=4.441
step 200/250  G_total=4.879  D_X=0.145  D_Y=0.142  λ_id=4.371
step 250/250  G_total=4.863  D_X=0.144  D_Y=0.136  λ_id=4.301
[epoch 17] avg G_total=4.8630
[epoch 17] done in 282.3s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 18/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=3.449  D_X=0.192  D_Y=0.200  λ_id=4.300
step 50/250  G_total=4.859  D_X=0.142  D_Y=0.127  λ_id=4.231
step 100/250  G_total=4.771  D_X=0.138  D_Y=0.132  λ_id=4.161
step 150/250  G_total=4.737  D_X=0.140  D_Y=0.132  λ_id=4.091
step 200/250  G_total=4.757  D_X=0.140  D_Y=0.130  λ_id=4.021
step 250/250  G_total=4.776  D_X=0.143  D_Y=0.132  λ_id=3.951
[epoch 18] avg G_total=4.7762
[epoch 18] done in 284.1s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 19/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=4.250  D_X=0.198  D_Y=0.196  λ_id=3.950
step 50/250  G_total=4.708  D_X=0.149  D_Y=0.117  λ_id=3.881
step 100/250  G_total=4.668  D_X=0.146  D_Y=0.122  λ_id=3.811
step 150/250  G_total=4.572  D_X=0.142  D_Y=0.126  λ_id=3.741
step 200/250  G_total=4.575  D_X=0.138  D_Y=0.122  λ_id=3.671
step 250/250  G_total=4.574  D_X=0.139  D_Y=0.125  λ_id=3.601
[epoch 19] avg G_total=4.5741
[epoch 19] done in 279.3s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 20/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=3.978  D_X=0.219  D_Y=0.119  λ_id=3.600
step 50/250  G_total=4.328  D_X=0.164  D_Y=0.131  λ_id=3.531
step 100/250  G_total=4.359  D_X=0.160  D_Y=0.127  λ_id=3.461
step 150/250  G_total=4.392  D_X=0.157  D_Y=0.131  λ_id=3.391
step 200/250  G_total=4.476  D_X=0.153  D_Y=0.133  λ_id=3.321
step 250/250  G_total=4.447  D_X=0.151  D_Y=0.134  λ_id=3.251
[epoch 20] avg G_total=4.4473
[epoch 20] done in 280.4s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 21/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=4.716  D_X=0.171  D_Y=0.051  λ_id=3.250
step 50/250  G_total=4.333  D_X=0.138  D_Y=0.128  λ_id=3.181
step 100/250  G_total=4.247  D_X=0.141  D_Y=0.129  λ_id=3.111
step 150/250  G_total=4.299  D_X=0.141  D_Y=0.131  λ_id=3.041
step 200/250  G_total=4.280  D_X=0.148  D_Y=0.132  λ_id=2.971
step 250/250  G_total=4.265  D_X=0.151  D_Y=0.132  λ_id=2.901
[epoch 21] avg G_total=4.2647
[epoch 21] done in 281.3s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 22/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=4.940  D_X=0.260  D_Y=0.218  λ_id=2.900
step 50/250  G_total=4.138  D_X=0.121  D_Y=0.115  λ_id=2.831
step 100/250  G_total=4.233  D_X=0.133  D_Y=0.123  λ_id=2.761
step 150/250  G_total=4.218  D_X=0.134  D_Y=0.126  λ_id=2.691
step 200/250  G_total=4.223  D_X=0.136  D_Y=0.130  λ_id=2.621
step 250/250  G_total=4.202  D_X=0.137  D_Y=0.133  λ_id=2.551
[epoch 22] avg G_total=4.2025
[epoch 22] done in 280.8s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 23/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=4.198  D_X=0.157  D_Y=0.093  λ_id=2.550
step 50/250  G_total=3.901  D_X=0.130  D_Y=0.126  λ_id=2.481
step 100/250  G_total=4.022  D_X=0.127  D_Y=0.130  λ_id=2.411
step 150/250  G_total=4.087  D_X=0.132  D_Y=0.124  λ_id=2.341
step 200/250  G_total=4.078  D_X=0.139  D_Y=0.126  λ_id=2.271
step 250/250  G_total=4.132  D_X=0.138  D_Y=0.126  λ_id=2.201
[epoch 23] avg G_total=4.1322
[epoch 23] done in 287.1s (~0.9 steps/s)
  best weights saved (Linux + Windows).

epoch 24/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=3.747  D_X=0.027  D_Y=0.043  λ_id=2.200
step 50/250  G_total=3.990  D_X=0.133  D_Y=0.130  λ_id=2.131
step 100/250  G_total=4.017  D_X=0.130  D_Y=0.126  λ_id=2.061
step 150/250  G_total=4.008  D_X=0.136  D_Y=0.123  λ_id=1.991
step 200/250  G_total=4.054  D_X=0.140  D_Y=0.124  λ_id=1.921
step 250/250  G_total=4.023  D_X=0.140  D_Y=0.126  λ_id=1.851
[epoch 24] avg G_total=4.0232
[epoch 24] done in 298.3s (~0.8 steps/s)
  best weights saved (Linux + Windows).

epoch 25/25:   0%|          | 0/250 [00:00<?, ?it/s]

step 1/250  G_total=3.230  D_X=0.123  D_Y=0.091  λ_id=1.850
step 50/250  G_total=3.893  D_X=0.132  D_Y=0.133  λ_id=1.781
step 100/250  G_total=3.912  D_X=0.131  D_Y=0.126  λ_id=1.711
step 150/250  G_total=3.956  D_X=0.135  D_Y=0.130  λ_id=1.641
step 200/250  G_total=3.967  D_X=0.136  D_Y=0.130  λ_id=1.571
step 250/250  G_total=3.943  D_X=0.135  D_Y=0.131  λ_id=1.501
[epoch 25] avg G_total=3.9432
[epoch 25] done in 299.3s (~0.8 steps/s)
  best weights saved (Linux + Windows).

Top-up complete in 0.80 h total.

8.3 Submission 2: Generating with TTA & Post-Processing

To maximize the submission score, this step generates a new set of images using two advanced techniques. First, Test-Time Augmentation (TTA) is used by creating translations of both the original and horizontally-flipped photos. Second, a simple border-cropping post-processing step is applied to remove potential edge artifacts from the final generated images.

---

# --- Assert that all required components are available ---
assert all(v in globals() for v in ("G_X2Y", "PHOTO_JPG_DIR", "MODELS_DIR_LIN", "MODELS_DIR_WIN", "IMG_SIZE")), "Run earlier sections first."

# --- Configure the submission generation process ---
CROP_BORDER = 4     # Pixels to crop from each side before final resize.
N_OUT       = 10000 # 7000 original images + 3000 flipped images.
ZIP_BATCH   = 1
IMG_SIDE    = int(IMG_SIZE)

# --- Select the source images for generation ---
photo_dir = str(PHOTO_JPG_DIR)
jpgs = sorted(glob.glob(os.path.join(photo_dir, "*.jpg")))
if len(jpgs) < 7000:
    raise FileNotFoundError(f"Found {len(jpgs)} JPGs in {photo_dir}, need at least 7000.")
# Define the source images for the two passes.
src_originals = jpgs[:7000]
src_flips = jpgs[:3000]

# Define output paths for the new submission zip files.
zip_lin = os.path.join(MODELS_DIR_LIN, "images_epoch16-25.zip")
zip_win = os.path.join(MODELS_DIR_WIN, "images_epoch16-25.zip")

# --- Define Helper Functions ---
# Helper to load and normalize an image for inference.
def _load_and_norm(path):
    img = tf.io.read_file(path)
    img = tf.io.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [IMG_SIDE, IMG_SIDE], method="area")
    img = tf.cast(img, tf.float32) / 127.5 - 1.0
    return img
# Helper to de-normalize a generated image for saving.
def _denorm_to_u8(t):
    t = (tf.cast(t, tf.float32) + 1.0) * 127.5
    t = tf.clip_by_value(t, 0.0, 255.0)
    return tf.cast(t, tf.uint8)
# Helper to apply the border-cropping post-processing step.
def _maybe_crop(arr_u8):
    if CROP_BORDER <= 0: return arr_u8
    h, w = arr_u8.shape[:2]
    b = min(CROP_BORDER, h // 4, w // 4)
    if b <= 0: return arr_u8
    cropped = arr_u8[b:h-b, b:w-b, :]
    return np.array(Image.fromarray(cropped).resize((IMG_SIDE, IMG_SIDE), Image.BICUBIC))

# --- Build the Submission Zip File ---
t0 = time.perf_counter()
# The progress bar will now track all 10,000 images.
pbar = tqdm(total=N_OUT, desc="Building images_epoch16-25.zip", leave=True, mininterval=0.2)

with zipfile.ZipFile(zip_lin, "w", compression=zipfile.ZIP_DEFLATED) as z_lin, \
     zipfile.ZipFile(zip_win, "w", compression=zipfile.ZIP_DEFLATED) as z_win:

    # --- Pass 1: Generate from original images (0 to 6999) ---
    for start in range(0, len(src_originals), ZIP_BATCH):
        batch_paths = src_originals[start : start + ZIP_BATCH]
        # Load and normalize the batch of photos.
        bx = tf.stack([_load_and_norm(p) for p in batch_paths], axis=0)
        # Generate the Monet-style images on the GPU.
        with tf.device("/GPU:0"):
            y_fake = G_X2Y(bx, training=False)
        arrs = _denorm_to_u8(y_fake).numpy()
        # Post-process and save each image to the zip files.
        for j, arr in enumerate(arrs):
            arr_processed = _maybe_crop(arr)
            buf = io.BytesIO()
            Image.fromarray(arr_processed).save(buf, format="PNG")
            name = f"{start + j}.png"
            z_lin.writestr(name, buf.getvalue())
            z_win.writestr(name, buf.getvalue())
            pbar.update(1)
            pbar.set_postfix({"png": f"{pbar.n}/{N_OUT}"})
    

    # --- Pass 2: Generate from horizontally-flipped images (7000 to 9999) ---
    for start in range(0, len(src_flips), ZIP_BATCH):
        batch_paths = src_flips[start : start + ZIP_BATCH]
        # Load, normalize, AND FLIP the batch of photos.
        bx = tf.stack([_load_and_norm(p) for p in batch_paths], axis=0)
        bx = tf.image.flip_left_right(bx)
        # Generate the Monet-style images on the GPU.
        with tf.device("/GPU:0"):
            y_fake = G_X2Y(bx, training=False)
        arrs = _denorm_to_u8(y_fake).numpy()
        # Post-process and save each image to the zip files.
        for j, arr in enumerate(arrs):
            arr_processed = _maybe_crop(arr)
            buf = io.BytesIO()
            Image.fromarray(arr_processed).save(buf, format="PNG")
            name = f"{7000 + start + j}.png"
            z_lin.writestr(name, buf.getvalue())
            z_win.writestr(name, buf.getvalue())
            pbar.update(1)
            pbar.set_postfix({"png": f"{pbar.n}/{N_OUT}"})

pbar.close()
dt = time.perf_counter() - t0

# --- Print Final Paths and Performance ---
print("\nSaved paths - Linux")
print("  images_epoch16-25.zip :", zip_lin)
print("\nSaved paths - Windows")
print("  images_epoch16-25.zip :", zip_win)
print(f"\nimages_epoch16-25.zip build time: {dt/60:.1f} min  |  ~{N_OUT/dt:.1f} imgs/s  |  ~{dt/N_OUT:.3f} s/img")

Building images_epoch16-25.zip:   0%|          | 0/10000 [00:00<?, ?it/s]

Saved paths - Linux
  images_epoch16-25.zip : /mnt/c/Users/travi/AppData/Local/Programs/Microsoft VS Code/artifacts/models/images_epoch16-25.zip

Saved paths - Windows
  images_epoch16-25.zip : /mnt/c/Users/travi/Documents/Training/Colorado/MS-AI/Machine Learning Theory and Hands-on Practice with Python Specialization/Introduction to Deep Learning/Module 5/Week5_Kaggle_Monet_Competition/models/images_epoch16-25.zip

images_epoch16-25.zip build time: 11.9 min  |  ~14.0 imgs/s  |  ~0.072 s/img

Observation: The Push for a Better Score

My first submission score of 69.06 was a good for a first try, but for me, it was a disappointment. I had a personal goal to break the 60-point barrier, and after all the hours spent debugging, just getting the model to run was not enough. I needed to make it better. I went back and researched how to improve a converged CycleGAN without simply training for thousands more steps and risking overfitting on the 30,000 steps I had already completed.

This led to a new, multi-pronged battle plan. The core idea was a short, targeted fine-tuning run of 10 epochs at 250 steps each. This was enough for refinement but not enough to overfit. To make this work, I first halved the generator's learning rate to encourage smaller, more precise updates. To push for stronger stylization, I made the identity loss anneal more aggressive, dropping its minimum weight to 0.3. Crucially, I had to find a way to re-enable the score-boosting replay buffers.

The main battle was with the replay buffers. I knew they were key to a better score, but they were also the primary cause of my previous OOM errors. I started by enabling them with the default size of 50. Instant failure. I cut the size in half to 25. Still failed. I tried 15, and it ran for a while before blowing up mid-epoch, a particularly frustrating kind of failure. The final, stable configuration was a tiny buffer of just 10 images. It was not much, but it was the key. This small buffer, running on top of the global mixed precision policy that had been saving memory all along, finally gave me just enough VRAM headroom to survive.

With a fine-tuned model in hand, the final step was to squeeze every possible point out of the submission. I used Test-Time Augmentation, generating images from both the original 7,000 photos and horizontally flipped versions of the first 3,000. This increased the diversity of the final 10,000-image set. As a final touch, I applied a small 4-pixel border crop to every generated image to remove any subtle edge artifacts the GAN might have created. This was the final plan; all that was left was to run the pipeline and see if the hours of extra work had paid off.

---

Fine-Tuning Results

The carefully designed experiment was executed, and the results from the fine-tuning run confirmed the strategy's success.

• Successful and Stable Run: The model trained for the additional 10 epochs (2,500 steps) without any OOM errors. The re-introduction of the small ImagePool (max_size=10) was successful, allowing for more stable training without sacrificing the run's viability.

• Improved Model Performance (Key Finding): The primary training metric, G_total, improved as a result of the fine-tuning. After an initial, expected spike, the loss steadily decreased, reaching a new best average of 3.9714 in the final epoch. This is a clear quantitative improvement over the previous best of 4.0607 from the initial 15-epoch run.

• Confirmation of Training Dynamics: The logs show a healthy training process. The discriminator losses remained low and stable, and the identity loss lambda_id correctly annealed from a high value down to its new, lower floor, guiding the generator toward a more stylized output.

• Final Artifact Generation: The newly fine-tuned G_X2Y model was used to generate the final 10,000-image submission file. The process, which included TTA and border cropping, completed successfully in 12.3 minutes.

Conclusion: The fine-tuning experiment was a definitive success. The strategic changes to the training process not only ran stably but also produced a model with a quantitatively better loss score. The final submission file, enhanced with Test-Time Augmentation, represents the best possible output from this entire project.

Section 9: Conclusion & Final Analysis

After the initial 15-epoch training run resulted in a respectable but disappointing score of 69.06, the entire project came down to one final, high-stakes experiment. The plan was clear: a targeted fine-tuning run with a lower learning rate and re-enabled replay buffers, followed by a new submission generated with Test-Time Augmentation. After hours of debugging and careful tuning, the final 10,000-image submission file was ready. All that was left was the moment of truth: hitting 'Submit' on Kaggle and waiting for the score to come back.

The result was a breakthrough.

Final Score: 57.83429

After all the hours of debugging, the frustrating errors, and the targeted experiments, I not only met my goal of breaking 60, I smashed it. The final score was a massive 11.23-point improvement over the baseline, a result that propelled the project to rank #11 on the public leaderboard.

What Moved the Needle

• Methodical Debugging. The project's success was fundamentally dependent on solving a cascade of OOM errors, nan losses, and environment-specific bugs. The final, stable configuration was a direct result of this rigorous process.
• Targeted Fine-Tuning. Instead of simply training for longer, a short fine-tuning run with a halved learning rate allowed for careful refinement of the already-trained model.
• Re-enabling Replay Buffers. After extensive debugging, a small but effective image pool of 10 was successfully re-enabled, providing the necessary training stability for a higher-quality result.
• Test-Time Augmentation (TTA). Generating images from both original and **horizontally-flipped photos** created a more diverse 10,000-image submission set that better matched the true data distribution.
• Aggressive Stylization. A more aggressive **identity loss anneal** during fine-tuning encouraged the generator to produce more painterly, less photorealistic images, which was key to improving the final score.

What to Try Next

With a top-tier result secured, the next logical step would be to migrate the workflow to a more powerful, integrated platform to chase an even better score.

• TPU Training on Kaggle. The number one priority would be to adapt this notebook to run directly on Kaggle using their TPUs. The massive speed increase would allow for much larger-scale experiments.
• Full-Size Replay Buffers. With more available VRAM, the replay buffer size could be increased from 10 back to the paper's recommended value of 50 for maximum training stability.
• Architectural Experiments. A next step could be to increase the number of ResNet blocks in the generator or to experiment with a U-Net-based generator to see how its skip connections affect the final style transfer.

---

Final Reflection: Deep Learning, Art, and the Human in the Loop

At its core, this project was an attempt to quantify the unquantifiable: an artist's "style." The CycleGAN was not just learning to change colors; it was learning a complex, high-dimensional distribution of textures, lighting, composition, and brushstrokes that define Monet's work. The final generated images demonstrate that a deep learning model can indeed learn and replicate a style with stunning accuracy.

This raises the classic question: is the model being "creative," or is it just a highly sophisticated mimic? This project suggests the latter. The GAN is a tool that has learned the "rules" of Monet's visual language so well that it can apply them to new, unseen content. It is a master forger, not a new artist. Its success serves only to heighten the appreciation for the original genius of Claude Monet, who invented that language from scratch.

Finally, this project was a powerful reminder of the "human in the loop." The final, high-scoring model was not the result of a single, clean run. It was forged through hours of frustrating debugging, analyzing contradictory error messages, and making difficult, strategic trade-offs between performance and stability. The "art" of this project was as much in the methodical engineering and resilient problem-solving as it was in the final generated images. It's a testament to the fact that even with advanced deep learning, the most critical component is still the persistent, creative, and sometimes frustrated human guiding the process.

---

A Showcase of Final Generated Images

Below is a small, hand-picked gallery of the final 256x256 images produced by the fine-tuned model. These are examples of the images included in the final, high-scoring images.zip submission file, demonstrating the model's ability to translate landscape photographs into the style of Monet.

---

TRAVIS IS #1