Property-based API testing

There’s a great tool called schemathesis that can be used to test your API to match your OpenAPI spec.

Official docs: https://schemathesis.readthedocs.io

schemathesis is not bundled together with the django-modern-rest. You have to install it with:

uv poetry pip

uv add --group dev schemathesis

Now, let’s see how you can generate thousands of tests for your API with just several lines of python code:

tests/test_integration/test_openapi/test_schema.py

Show imports... 17 lines hidden

[source]

import logging
import os
from collections.abc import Iterator
from typing import TYPE_CHECKING, Any, Final

import pytest
import schemathesis as st
from django.conf import LazySettings
from django.contrib.auth.models import User
from django.urls import reverse
from hypothesis import settings as h_settings
from hypothesis import strategies
from schemathesis.specs.openapi.schemas import OpenApiSchema

from django_test_app.server.wsgi import application
from dmr.validation import ResponseValidator

_LOCAL_MAX_EXAMPLES: Final = 15
_MAX_EXAMPLES: Final = (
    50  # noqa: WPS432
    if os.environ.get('CI')
    else _LOCAL_MAX_EXAMPLES
)

if TYPE_CHECKING:
    import tracecov


@pytest.fixture(autouse=True)
def _patch_response_validation(monkeypatch: pytest.MonkeyPatch) -> None:
    # Patches the response validator class to never validate the responses,
    # despite their settings. This is needed to test schematesis's
    # response schema validation and compatibility between two. Not ours schema.
    # https://github.com/wemake-services/django-modern-rest/issues/776
    monkeypatch.setattr(
        ResponseValidator,
        '_should_validate_responses',
        lambda *args, **kwargs: False,
    )


@pytest.fixture(autouse=True)
def _disable_logging() -> Iterator[None]:
    # Logging has too much output with schemathesis:
    logging.disable(logging.CRITICAL)
    yield
    logging.disable(logging.NOTSET)


@pytest.fixture(autouse=True)
def _modify_integration_settings(settings: LazySettings) -> None:
    # Schemathesis tests only run meaningfully with DEBUG=False (the test
    # explicitly skips when DEBUG=True), so there is no value in running
    # the full suite twice via the parent conftest parametrisation.
    settings.DEBUG = False


# The `transactional_db` fixture is required to enable database access.
# When `st.openapi.from_wsgi()` makes a WSGI request, Django's request
# lifecycle triggers database operations.
# The `admin_user` fixture is required here so that auth can use
# its credentials (username and password) for authentication.
# This follows the `pytest-django` pattern for creating user fixtures:
# https://github.com/pytest-dev/pytest-django/blob/main/pytest_django/fixtures.py#L483
@pytest.fixture
def api_schema(transactional_db: None, admin_user: User) -> OpenApiSchema:
    """Load OpenAPI schema as a pytest fixture."""
    return st.openapi.from_wsgi(reverse('openapi_json'), application)


schema = st.pytest.from_fixture('api_schema')

# Register custom strategies:
st.openapi.format(
    'phone',
    strategies.from_regex(r'^\+7-495-[0-9]{3}-[0-9]{2}-[0-9]{2}$'),
)

# TODO: provide `Token` auth as well
# TODO: restore `django_session` auth, removed in #1550


@schema.parametrize()
@h_settings(max_examples=_MAX_EXAMPLES)
@pytest.mark.timeout(0)
def test_schemathesis(
    tracecov_map: 'tracecov.CoverageMap | None',
    *,
    case: st.Case[Any],
) -> None:
    """Ensure that API implementation matches the OpenAPI schema."""
    if tracecov_map is None:  # pragma: no cover
        pytest.skip(reason='missing `tracecov`')

    from tracecov.schemathesis import helpers  # noqa: PLC0415

    # `werkzeug` leaves `REMOTE_ADDR` out of the WSGI environ entirely,
    # while every real server sets it. Code that looks up the client IP
    # then behaves differently under test than in production: for example
    # `django-allauth` rate limiting answers `403` when it cannot find one.
    response = case.call_and_validate(
        environ_base={'REMOTE_ADDR': '127.0.0.1'},
    )
    # Record interaction for `tracecov` report:
    tracecov_map.record_schemathesis_interactions(
        case.method,
        case.operation.full_path,
        [helpers.from_response(case.method, response)],
    )

What will happen here?

1. schemathesis loads OpenAPI schema definition from the reverse('openapi') URL

2. Then we will create a top level schema object from the api_schema pytest fixture. It is needed to create a property-based test case

3. Lastly, we create a generated test case with the help of @schema.parametrize()

You can also provide settings, like the number of generated tests, enabled rules, auth, etc:

schemathesis.toml

[source]

[warnings]
# Fail on any warning:
fail-on = true

[auth.openapi.http_basic]
# Our own credentials from
# django_test_app/server/apps/controllers/auth.py
username = 'test'
password = 'pass'

[auth.dynamic.openapi.jwt]
path = '/api/jwt-auth/jwt-obtain-access-refresh-sync/'
# These are default user credentials from
# https://github.com/pytest-dev/pytest-django
payload = { username = 'admin', password = 'password' }
extract_selector = '/access_token'

[output.sanitization]
# We don't have any real data to hide, but it is easier to debug:
enabled = false

[generation]
# Don't generate `\x00` (NULL) bytes in strings:
# this thing fails tests against real databases,
# because \x00 is not accepted by databases by default.
allow-x00 = false

# Cursor-paginated endpoints answer `400`, when
[[operations]]
include-tag = 'model_cursor'

[operations.checks.positive_data_acceptance ]
expected-statuses = ['2xx', '400']

When running the test case with

pytest tests/test_integration/test_openapi/test_schema.py

it will cover all your API. In simple cases it might be enough tests. Yes, you heard right: in simple cases just using schemathesis can remove the need to write any other integration tests.

Important

Using schemathesis with django-modern-rest is very easy, because we offer state-of-the-art OpenAPI schema generation. It will be really hard to satisfy schemathesis with a different framework.

Validating responses

schemathesis can also be used in regular tests to validate the response schema. See https://schemathesis.readthedocs.io/en/stable/guides/schema-conformance/

Example:

from dmr.test import DMRClient

def test_with_conditional_logic(dmr_client: DMRClient) -> None:
    response = dmr_client.post(
       '/users',
       data={'name': 'Alice'},
   )

   assert schema['/users']['POST'].is_valid_response(response.json())