Production-ready configurations for running pytest in continuous integration. Covers GitHub Actions matrix builds, coverage gates, parallel execution with xdist, flaky test detection, and caching strategies.
1. CI Philosophy for Test Suites
2. GitHub Actions Matrix Builds
4. Parallel Execution with xdist
5. Flaky Test Detection & Handling
Structure your CI pipeline as escalating gates:
Tier 1: Fast Feedback (< 2 min) ← Every push
- Linting (ruff/flake8)
- Type checking (mypy)
- Unit tests (no I/O, no network)
Tier 2: Thorough Validation (< 10 min) ← Every PR
- Integration tests
- Coverage report
- Property-based tests
Tier 3: Full Confidence (< 30 min) ← Before merge to main
- End-to-end tests
- Performance benchmarks
- Security scanning
- Multi-Python-version matrix
| Metric | Target | Action if Violated |
|---|---|---|
| Unit test pass rate | 100% | Block merge |
| Coverage (lines) | ≥ 80% | Warn, block if < 70% |
| Coverage (branches) | ≥ 75% | Warn |
| Test duration | < 5 min | Investigate slow tests |
| Flaky test rate | < 1% | Quarantine and fix |
See ci/github-actions-pytest.yml for the complete workflow file. Key design decisions:
# Run tests across multiple Python versions simultaneously
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
os: [ubuntu-latest]
fail-fast: false # Don't cancel other jobs if one failsWhy fail-fast: false? When debugging a version-specific bug, you want to see which versions pass and which fail. With fail-fast: true, a failure in 3.10 cancels the 3.11 and 3.12 runs before they complete.
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.10", "3.11", "3.12"]
exclude:
# Skip expensive combinations that rarely fail
- os: macos-latest
python-version: "3.10"
- os: windows-latest
python-version: "3.10"
include:
# Add a specific combination with extra settings
- os: ubuntu-latest
python-version: "3.12"
coverage: true # Only generate coverage on one comboIn pyproject.toml:
[tool.coverage.run]
# Measure branch coverage, not just line coverage
branch = true
# Only measure YOUR code, not installed packages
source = ["src"]
# Exclude test files from coverage measurement
omit = ["tests/*", "**/conftest.py"]
[tool.coverage.report]
# Fail the build if coverage drops below this threshold
fail_under = 80
# Show lines that weren't covered
show_missing = true
# Don't count these patterns as "missed" coverage
exclude_lines = [
"pragma: no cover",
"def __repr__",
"if __name__ == .__main__.",
"raise NotImplementedError",
"pass",
"\\.\\.\\.",
]
[tool.coverage.html]
directory = "htmlcov"# Generate coverage and fail if below threshold
pytest --cov=src --cov-report=xml --cov-report=term-missing --cov-fail-under=80
# For PR comments, generate JSON too
pytest --cov=src --cov-report=json --cov-report=xml# In CI: check coverage only for files changed in this PR
CHANGED_FILES=$(git diff --name-only origin/main...HEAD -- '*.py' | grep '^src/')
if [ -n "$CHANGED_FILES" ]; then
pytest --cov=src --cov-report=xml
# Use diff-cover to check only changed lines
# pip install diff-cover
diff-cover coverage.xml --compare-branch=origin/main --fail-under=90
fi# Use all available CPU cores
pytest -n auto
# Use specific number of workers
pytest -n 4
# Distribute by file (each worker gets whole test files)
pytest -n 4 --dist loadfile
# Distribute by test (each worker gets individual tests)
pytest -n 4 --dist load| Strategy | --dist flag | Best When |
|---|---|---|
| Load balancing | load | Tests have varying duration |
| By file | loadfile | Tests within a file share expensive fixtures |
| By group | loadgroup | You've marked test groups with @pytest.mark.xdist_group |
| Each | each | Every test needs to run on every worker (rare) |
Tests must be isolated to run in parallel. Common issues:
# PROBLEM: Shared file system state
def test_write_config():
with open("/tmp/config.json", "w") as f: # Two workers might collide!
json.dump({"key": "value"}, f)
# SOLUTION: Use tmp_path fixture (unique per test)
def test_write_config(tmp_path):
config_file = tmp_path / "config.json"
with open(config_file, "w") as f:
json.dump({"key": "value"}, f)
# PROBLEM: Shared database state
def test_create_user():
db.execute("INSERT INTO users (id) VALUES (1)") # Duplicate key error if parallel!
# SOLUTION: Use unique IDs or transactions
def test_create_user(db_transaction, fake):
user_id = fake.integer(min_val=10000, max_val=99999)
db_transaction.execute(f"INSERT INTO users (id) VALUES ({user_id})")
# PROBLEM: Fixed port binding
def test_server():
server = start_server(port=8080) # Port conflict with parallel workers!
# SOLUTION: Use dynamic port allocation
def test_server():
import socket
with socket.socket() as s:
s.bind(('', 0))
port = s.getsockname()[1]
server = start_server(port=port)# conftest.py
import pytest
@pytest.fixture(scope="session")
def worker_id(request):
"""Get the xdist worker ID (or 'master' if not running parallel)."""
if hasattr(request.config, "workerinput"):
return request.config.workerinput["workerid"]
return "master"
@pytest.fixture(scope="session")
def database_url(worker_id):
"""Each parallel worker gets its own database."""
return f"sqlite:///test_{worker_id}.db"| Cause | Solution |
|---|---|
| Time-dependent logic | Freeze time in tests |
| Network calls | Mock HTTP, use fakes |
| Race conditions | Add proper synchronization |
| Shared state | Isolate with fixtures |
| Random data | Use seeded generators |
| Order dependency | Run with --randomly-seed |
# Run tests multiple times to find flaky ones
pytest --count=5 -x # Requires pytest-repeat
# Run in random order to find order-dependent tests
pytest -p randomly # Requires pytest-randomly
# Report timing outliers
pytest --durations=20 # Show 20 slowest tests# conftest.py
import pytest
import os
# Marker for known-flaky tests
def pytest_configure(config):
config.addinivalue_line(
"markers", "flaky(reason): mark test as known-flaky with explanation"
)
def pytest_collection_modifyitems(config, items):
"""Handle flaky test quarantine based on environment."""
run_flaky = os.environ.get("RUN_FLAKY_TESTS", "false").lower() == "true"
if not run_flaky:
skip_flaky = pytest.mark.skip(reason="Flaky test quarantined (set RUN_FLAKY_TESTS=true to run)")
for item in items:
if item.get_closest_marker("flaky"):
item.add_marker(skip_flaky)
# Usage in tests:
@pytest.mark.flaky(reason="Intermittent timeout in CI, tracking in issue #234")
def test_webhook_delivery():
"""This test occasionally times out due to network jitter."""
response = send_webhook(url="https://api.example.com/hooks/test")
assert response.status_code == 200# conftest.py — simple retry mechanism without pytest-rerunfailures
import pytest
def pytest_addoption(parser):
parser.addoption("--retries", action="store", default="0",
help="Number of retries for failed tests")
@pytest.hookimpl(trylast=True)
def pytest_runtest_makereport(item, call):
"""Track test outcomes for retry logic."""
if call.when == "call" and call.excinfo is not None:
max_retries = int(item.config.getoption("--retries"))
retry_count = getattr(item, "_retry_count", 0)
if retry_count < max_retries:
item._retry_count = retry_count + 1
item._need_retry = True
def pytest_runtest_protocol(item, nextitem):
"""Re-run tests that were marked for retry."""
# Note: This is a simplified illustration. In production,
# use the pytest-rerunfailures plugin which handles this properly.
pass# In GitHub Actions:
- uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt', '**/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-# Cache Hypothesis examples between CI runs
- uses: actions/cache@v4
with:
path: .hypothesis
key: hypothesis-${{ github.ref }}-${{ github.sha }}
restore-keys: |
hypothesis-${{ github.ref }}-
hypothesis-main-- uses: actions/cache@v4
with:
path: ~/.cache/pre-commit
key: pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}Split tests across CI jobs based on historical run times:
# Record test times
pytest --durations-file=.test_durations --store-durations
# Split into N groups by time (requires pytest-split)
pytest --splits 4 --group 1 # In CI job 1
pytest --splits 4 --group 2 # In CI job 2
pytest --splits 4 --group 3 # In CI job 3
pytest --splits 4 --group 4 # In CI job 4# Job 1: Fast unit tests (every push)
pytest -m "not slow and not integration"
# Job 2: Slow + integration tests (PR only)
pytest -m "slow or integration"# Generate JUnit XML that CI systems understand
pytest --junitxml=test-results/junit.xml# In GitHub Actions — display test results in PR
- uses: dorny/test-reporter@v1
if: always()
with:
name: Test Results
path: test-results/junit.xml
reporter: java-junit# Generate coverage badge data
pytest --cov=src --cov-report=json
# Extract percentage for badge
python -c "
import json
with open('coverage.json') as f:
data = json.load(f)
pct = data['totals']['percent_covered']
print(f'Coverage: {pct:.1f}%')
"Track test duration over time to catch regressions:
# conftest.py — write timing data for trend analysis
import json
import time
from pathlib import Path
class TimingCollector:
def __init__(self):
self.timings = {}
def record(self, nodeid: str, duration: float):
self.timings[nodeid] = duration
def save(self, path: Path):
path.write_text(json.dumps(self.timings, indent=2))
_collector = TimingCollector()
def pytest_runtest_makereport(item, call):
if call.when == "call":
_collector.record(item.nodeid, call.duration)
def pytest_sessionfinish(session, exitstatus):
output = Path("test-results") / "timings.json"
output.parent.mkdir(exist_ok=True)
_collector.save(output)Patterns for creating test data that is deterministic, expressive, and maintainable. Covers fixture composition, factory functions, fake-data generators, and database/session fixture patterns.
2. Fixture Composition Patterns
5. Database & Session Fixtures
6. Builder Pattern for Complex Objects
Bad test data leads to:
user_id=42 mean? Why 42?Good test data follows these principles:
| Principle | Description |
|---|---|
| Minimal | Only specify fields relevant to the test assertion |
| Descriptive | Values tell you WHY they were chosen |
| Deterministic | Same input → same output, every time |
| Isolated | One test's data doesn't leak into another |
| Easy to create | One line to get a valid object |
Build complex test state by composing simple fixtures:
import pytest
from dataclasses import dataclass, field
from datetime import datetime, timezone
@dataclass
class User:
id: int
email: str
name: str
role: str = "viewer"
active: bool = True
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
@dataclass
class Organization:
id: int
name: str
plan: str = "free"
owner: User | None = None
@dataclass
class Project:
id: int
name: str
organization: Organization | None = None
members: list[User] = field(default_factory=list)
# Layer 1: Atomic fixtures
@pytest.fixture
def owner():
return User(id=1, email="owner@example.com", name="Org Owner", role="admin")
@pytest.fixture
def organization(owner):
return Organization(id=1, name="Acme Corp", plan="enterprise", owner=owner)
# Layer 2: Composite fixtures
@pytest.fixture
def project_with_team(organization):
members = [
User(id=2, email="dev@example.com", name="Dev User", role="developer"),
User(id=3, email="qa@example.com", name="QA User", role="tester"),
]
return Project(
id=1,
name="Backend Rewrite",
organization=organization,
members=members,
)
# Layer 3: Scenario fixtures
@pytest.fixture
def full_workspace(project_with_team):
"""A complete workspace scenario with org, project, and team."""
# Add tasks, settings, integrations, etc.
return {
"project": project_with_team,
"settings": {"notifications": True, "auto_assign": True},
}@pytest.fixture(params=[
pytest.param({"plan": "free", "seats": 5}, id="free-plan"),
pytest.param({"plan": "pro", "seats": 50}, id="pro-plan"),
pytest.param({"plan": "enterprise", "seats": 999}, id="enterprise-plan"),
])
def organization_with_plan(request, owner):
"""Test across different subscription tiers."""
return Organization(
id=1,
name="Acme Corp",
plan=request.param["plan"],
owner=owner,
)A factory is a callable that creates valid objects with sensible defaults, allowing you to override only what matters for your test:
import pytest
from typing import Any
def make_user(**overrides: Any) -> User:
"""Create a valid User with sensible defaults.
Override any field by passing it as a keyword argument.
Only specify what your test cares about — everything else
gets a reasonable default.
"""
defaults = {
"id": 1,
"email": "test@example.com",
"name": "Test User",
"role": "viewer",
"active": True,
}
defaults.update(overrides)
return User(**defaults)
# Usage in tests:
def test_admin_permissions():
admin = make_user(role="admin", name="Admin User")
assert admin.can_access("/admin-panel")
def test_deactivated_user():
inactive = make_user(active=False)
assert not inactive.can_login()When you need multiple unique instances:
class UserFactory:
"""Factory that generates unique users with sequential IDs."""
_counter: int = 0
@classmethod
def create(cls, **overrides) -> User:
cls._counter += 1
n = cls._counter
defaults = {
"id": n,
"email": f"user{n}@example.com",
"name": f"User {n}",
"role": "viewer",
"active": True,
}
defaults.update(overrides)
return User(**defaults)
@classmethod
def create_batch(cls, count: int, **overrides) -> list[User]:
"""Create multiple users at once."""
return [cls.create(**overrides) for _ in range(count)]
@classmethod
def reset(cls):
"""Reset counter between tests."""
cls._counter = 0
# As a fixture with automatic reset:
@pytest.fixture(autouse=True)
def reset_factories():
yield
UserFactory.reset()
def test_batch_creation():
users = UserFactory.create_batch(5, role="developer")
assert len(users) == 5
assert all(u.role == "developer" for u in users)
# Each has unique email: user1@example.com, user2@example.com, etc.
emails = [u.email for u in users]
assert len(set(emails)) == 5 # All uniqueCombine pre-defined traits to build specific scenarios:
class OrderFactory:
"""Factory with composable traits for common scenarios."""
_counter: int = 0
# Traits: pre-defined field combinations for common scenarios
TRAITS = {
"paid": {"status": "paid", "paid_at": datetime(2024, 1, 15, tzinfo=timezone.utc)},
"refunded": {"status": "refunded", "refunded_at": datetime(2024, 1, 20, tzinfo=timezone.utc)},
"large": {"total_cents": 99900, "item_count": 25},
"subscription": {"recurring": True, "interval": "monthly"},
"international": {"currency": "EUR", "shipping_country": "DE"},
}
@classmethod
def create(cls, *traits: str, **overrides) -> dict:
cls._counter += 1
defaults = {
"id": cls._counter,
"customer_email": f"customer{cls._counter}@example.com",
"status": "pending",
"total_cents": 2999,
"item_count": 1,
"currency": "USD",
"recurring": False,
"shipping_country": "US",
}
# Apply traits in order
for trait in traits:
if trait not in cls.TRAITS:
raise ValueError(f"Unknown trait: {trait}. Available: {list(cls.TRAITS.keys())}")
defaults.update(cls.TRAITS[trait])
# Explicit overrides win over traits
defaults.update(overrides)
return defaults
# Usage — readable and intention-revealing:
def test_refund_eligibility():
order = OrderFactory.create("paid", "large")
assert order["status"] == "paid"
assert order["total_cents"] == 99900
assert is_refund_eligible(order)
def test_international_tax():
order = OrderFactory.create("paid", "international", total_cents=5000)
tax = calculate_tax(order)
assert tax["rate"] == 0.19 # German VATWhen you need realistic-looking data but deterministic results:
import random
import hashlib
from datetime import datetime, timedelta, timezone
class DeterministicFaker:
"""Generate realistic fake data with deterministic output.
Uses a seed so the same test always gets the same data.
No external dependencies — pure stdlib.
"""
FIRST_NAMES = ["Alice", "Bob", "Charlie", "Diana", "Eve", "Frank", "Grace", "Henry"]
LAST_NAMES = ["Smith", "Johnson", "Williams", "Brown", "Jones", "Garcia", "Miller", "Davis"]
DOMAINS = ["example.com", "test.org", "acme-corp.example.com", "demo.example.net"]
CITIES = ["Portland", "Austin", "Denver", "Seattle", "Boston", "Chicago"]
def __init__(self, seed: int = 42):
self._rng = random.Random(seed)
def email(self) -> str:
first = self._rng.choice(self.FIRST_NAMES).lower()
last = self._rng.choice(self.LAST_NAMES).lower()
domain = self._rng.choice(self.DOMAINS)
return f"{first}.{last}@{domain}"
def name(self) -> str:
return f"{self._rng.choice(self.FIRST_NAMES)} {self._rng.choice(self.LAST_NAMES)}"
def uuid(self) -> str:
"""Deterministic UUID-like string."""
raw = self._rng.getrandbits(128).to_bytes(16, "big")
hex_str = raw.hex()
return f"{hex_str[:8]}-{hex_str[8:12]}-{hex_str[12:16]}-{hex_str[16:20]}-{hex_str[20:]}"
def timestamp(self, days_ago_max: int = 365) -> datetime:
days = self._rng.randint(0, days_ago_max)
hours = self._rng.randint(0, 23)
return datetime.now(timezone.utc) - timedelta(days=days, hours=hours)
def integer(self, min_val: int = 0, max_val: int = 1000) -> int:
return self._rng.randint(min_val, max_val)
def money_cents(self, min_val: int = 100, max_val: int = 100000) -> int:
"""Amount in cents to avoid floating-point issues."""
return self._rng.randint(min_val, max_val)
def choice(self, options: list) -> Any:
return self._rng.choice(options)
# Fixture that provides fresh, deterministic fake data per test
@pytest.fixture
def fake():
"""Deterministic fake data generator — same seed every test."""
return DeterministicFaker(seed=42)
def test_user_report(fake):
users = [
{"name": fake.name(), "email": fake.email(), "city": fake.choice(DeterministicFaker.CITIES)}
for _ in range(10)
]
report = generate_report(users)
# Always produces the same 10 users, so assertions are stable
assert len(report.rows) == 10When test data identity matters (e.g., cache keys):
def content_id(data: str) -> str:
"""Generate a deterministic ID from content — useful for cache-key tests."""
return hashlib.sha256(data.encode()).hexdigest()[:12]
def make_document(title: str, body: str = "Default body content") -> dict:
"""Document with a content-derived ID — same content = same ID."""
content = f"{title}:{body}"
return {
"id": content_id(content),
"title": title,
"body": body,
"word_count": len(body.split()),
}import sqlite3
import pytest
@pytest.fixture(scope="session")
def db_schema():
"""Create the database schema once for the entire test session."""
return [
"""CREATE TABLE users (
id INTEGER PRIMARY KEY,
email TEXT UNIQUE NOT NULL,
name TEXT NOT NULL,
role TEXT DEFAULT 'viewer',
active INTEGER DEFAULT 1,
created_at TEXT DEFAULT CURRENT_TIMESTAMP
)""",
"""CREATE TABLE orders (
id INTEGER PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id),
total_cents INTEGER NOT NULL,
status TEXT DEFAULT 'pending',
created_at TEXT DEFAULT CURRENT_TIMESTAMP
)""",
]
@pytest.fixture
def db(db_schema):
"""Fresh in-memory database per test with schema applied."""
conn = sqlite3.connect(":memory:")
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA foreign_keys = ON")
for statement in db_schema:
conn.execute(statement)
yield conn
conn.close()
@pytest.fixture
def db_with_users(db):
"""Database pre-populated with standard test users."""
users = [
(1, "admin@example.com", "Admin User", "admin", 1),
(2, "dev@example.com", "Dev User", "developer", 1),
(3, "inactive@example.com", "Inactive User", "viewer", 0),
]
db.executemany(
"INSERT INTO users (id, email, name, role, active) VALUES (?, ?, ?, ?, ?)",
users,
)
db.commit()
return dbFor tests that modify data — roll back after each test so tests don't affect each other:
@pytest.fixture
def transactional_db(db_with_users):
"""Wraps each test in a transaction that rolls back on completion.
This means tests can INSERT, UPDATE, DELETE freely — all changes
are reverted after the test finishes. Other tests see the original data.
"""
# SQLite doesn't support savepoints in the same way as Postgres,
# but this pattern demonstrates the concept:
db_with_users.execute("BEGIN")
yield db_with_users
db_with_users.execute("ROLLBACK")@pytest.fixture(scope="session")
def connection_pool():
"""Simulated connection pool shared across all tests.
In real code, this would be a SQLAlchemy engine or connection pool.
Here we demonstrate the lifecycle pattern.
"""
pool = {
"connections": [],
"max_size": 5,
"created": 0,
}
yield pool
# Cleanup: close all connections
for conn in pool["connections"]:
if hasattr(conn, "close"):
conn.close()
@pytest.fixture
def db_session(connection_pool):
"""Get a connection from the pool for one test."""
conn = sqlite3.connect(":memory:")
connection_pool["connections"].append(conn)
connection_pool["created"] += 1
yield conn
conn.close()
connection_pool["connections"].remove(conn)When factories aren't expressive enough — use fluent builders:
class RequestBuilder:
"""Fluent builder for constructing HTTP-like request objects for testing.
Usage:
request = (RequestBuilder()
.method("POST")
.path("/api/users")
.header("Authorization", "Bearer token123")
.json_body({"name": "Alice"})
.build())
"""
def __init__(self):
self._method = "GET"
self._path = "/"
self._headers: dict[str, str] = {}
self._query_params: dict[str, str] = {}
self._body: bytes | None = None
self._content_type: str = "application/json"
def method(self, m: str) -> "RequestBuilder":
self._method = m.upper()
return self
def path(self, p: str) -> "RequestBuilder":
self._path = p
return self
def header(self, key: str, value: str) -> "RequestBuilder":
self._headers[key] = value
return self
def query(self, key: str, value: str) -> "RequestBuilder":
self._query_params[key] = value
return self
def json_body(self, data: dict) -> "RequestBuilder":
import json
self._body = json.dumps(data).encode()
self._content_type = "application/json"
return self
def form_body(self, data: dict) -> "RequestBuilder":
from urllib.parse import urlencode
self._body = urlencode(data).encode()
self._content_type = "application/x-www-form-urlencoded"
return self
def authenticated(self, token: str = "test-token-123") -> "RequestBuilder":
"""Shorthand for adding auth header."""
return self.header("Authorization", f"Bearer {token}")
def build(self) -> dict:
self._headers["Content-Type"] = self._content_type
return {
"method": self._method,
"path": self._path,
"headers": dict(self._headers),
"query_params": dict(self._query_params),
"body": self._body,
}
# As a fixture:
@pytest.fixture
def request_builder():
return RequestBuilder()# BAD: One massive fixture that sets up everything
@pytest.fixture
def everything():
db = setup_database()
cache = setup_cache()
queue = setup_queue()
user = create_user()
org = create_org(user)
project = create_project(org)
# ... 50 more lines
return db, cache, queue, user, org, project
# GOOD: Small, composable fixtures
@pytest.fixture
def db(): ...
@pytest.fixture
def cache(): ...
@pytest.fixture
def user(db): ...# BAD: Why 42? Why this specific email? What does status=3 mean?
def test_order():
order = create_order(user_id=42, status=3, total=19.99)
assert order.is_valid()
# GOOD: Intention-revealing values
def test_paid_order_is_valid():
order = OrderFactory.create("paid", total_cents=1999)
assert order.is_valid()# BAD: Tests depend on execution order
_global_users = []
def test_create_user():
_global_users.append(User(name="Alice"))
assert len(_global_users) == 1
def test_user_count():
# Fails if test_create_user doesn't run first!
assert len(_global_users) == 1
# GOOD: Each test owns its data
def test_create_user(make_user):
user = make_user(name="Alice")
assert user.name == "Alice"# BAD: This test is about email validation but specifies everything
def test_email_validation():
user = User(
id=1, name="Test", email="invalid-email", role="admin",
active=True, created_at=datetime(2024, 1, 1), department="eng",
phone="+1234567890", avatar_url="https://example.com/img.png",
)
assert not is_valid_email(user.email)
# GOOD: Only specify what the test actually checks
def test_email_validation():
user = make_user(email="invalid-email")
assert not is_valid_email(user.email)Get the full Python Testing Toolkit and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.