Devin Case Study: Automating Django Test Suite Migration from unittest to pytest for Mid-Size E-Commerce
Executive Summary
A mid-size e-commerce company with 1,200 Django test files faced a daunting migration from unittest to pytest. Manual estimates projected three months of developer time. By deploying Devin, Cognition’s autonomous AI software engineer, the entire migration—including dependency resolution, fixture generation, and CI pipeline updates—was completed in just ten days. This case study walks through the exact workflow, configuration, and results.
The Challenge
The engineering team at a growing e-commerce platform inherited a legacy Django test suite built entirely on Python’s unittest framework. The codebase had accumulated significant technical debt:
- 1,200 test files spanning 14 Django apps- Inconsistent setUp/tearDown patterns with deeply nested class hierarchies- Manual database fixture loading via JSON files and custom management commands- Tightly coupled test dependencies across modules- Jenkins CI pipelines hardcoded to unittest discovery and reportingThree senior engineers estimated the migration would take 12 weeks of focused effort. Context-switching costs and regression risk made it a project no one wanted to own.
The Solution: Devin-Driven Autonomous Migration
Step 1: Project Setup and Devin Session Initialization
The team connected Devin to their private GitHub repository and initialized a migration session with clear objectives:
# In Devin’s session prompt:
Migrate all 1,200 test files from unittest to pytest.
Requirements:
- Replace unittest.TestCase classes with plain pytest functions/classes
- Convert setUp/tearDown to pytest fixtures with appropriate scope
- Replace self.assertEqual, self.assertTrue, etc. with plain assert statements
- Generate conftest.py files with shared fixtures per Django app
- Update CI pipeline from Jenkins unittest runner to pytest with pytest-django
Maintain 100% test pass rate throughout migration
Step 2: Dependency Resolution and Configuration
Devin autonomously analyzed the project's dependency tree and updated the requirements:
# requirements/test.txt — Devin's automated changes
pytest==8.1.1
pytest-django==4.8.0
pytest-cov==5.0.0
pytest-xdist==3.5.0
pytest-mock==3.14.0
factory-boy==3.3.0 # replacing JSON fixture loading
Devin then generated the root pytest.ini configuration:
# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = config.settings.test
python_files = tests.py test_*.py *_tests.py
python_classes = Test*
python_functions = test_*
addopts =
--reuse-db
--no-migrations
-n auto
--cov=apps
--cov-report=html
--tb=short
### Step 3: Autonomous Test File Conversion
Devin processed files in batches of 50, applying consistent transformation patterns. Here is a representative before-and-after example:
**Before (unittest):**
from django.test import TestCase
from apps.orders.models import Order
class OrderTotalTestCase(TestCase):
def setUp(self):
self.user = User.objects.create_user(
username=‘testuser’, password=‘testpass123’
)
self.order = Order.objects.create(
user=self.user, status=‘pending’
)
def test_order_total_calculation(self):
self.order.add_item(product_id=1, quantity=2, price=29.99)
self.order.add_item(product_id=2, quantity=1, price=49.99)
self.assertEqual(self.order.calculate_total(), 109.97)
def test_empty_order_total(self):
self.assertEqual(self.order.calculate_total(), 0)
def tearDown(self):
Order.objects.all().delete()
User.objects.all().delete()</code></pre><p>**After (pytest):**
import pytest
from apps.orders.models import Order
@pytest.fixture
def user(db):
return User.objects.create_user(
username=‘testuser’, password=‘testpass123’
)
@pytest.fixture
def order(user):
return Order.objects.create(user=user, status=‘pending’)
def test_order_total_calculation(order):
order.add_item(product_id=1, quantity=2, price=29.99)
order.add_item(product_id=2, quantity=1, price=49.99)
assert order.calculate_total() == 109.97
def test_empty_order_total(order):
assert order.calculate_total() == 0
Step 4: Shared Fixture Generation
Devin identified repeated setup patterns across apps and generated conftest.py files with shared, scoped fixtures:
# apps/orders/conftest.py — auto-generated by Devin
import pytest
from factory import LazyAttribute, SubFactory
from apps.users.factories import UserFactory
from apps.orders.models import Order
@pytest.fixture
def customer(db):
return UserFactory(role=‘customer’)
@pytest.fixture
def pending_order(customer):
return Order.objects.create(user=customer, status=‘pending’)
@pytest.fixture
def completed_order(customer):
return Order.objects.create(user=customer, status=‘completed’)
@pytest.fixture(scope=‘session’)
def product_catalog(django_db_setup, django_db_blocker):
with django_db_blocker.unblock():
from django.core.management import call_command
call_command(‘loaddata’, ‘catalog_seed.json’)
Step 5: CI Pipeline Updates
Devin rewrote the Jenkins pipeline stage to use pytest natively:
# Jenkinsfile — updated test stage
stage('Test') {
steps {
sh '''
python -m pytest \
--junitxml=reports/junit.xml \
--cov-report=xml:reports/coverage.xml \
-n 4 \
--dist=loadscope
'''
}
post {
always {
junit 'reports/junit.xml'
cobertura coberturaReportFile: 'reports/coverage.xml'
}
}
}
## Results
Metric Before After Test framework unittest pytest 8.1 Migration duration ~12 weeks (estimated) 10 days Files converted 0 1,200 Generated conftest.py files 0 14 (one per app) Test execution time 47 minutes 11 minutes (with xdist) Test pass rate 100% 100% Developer hours spent N/A ~16 hours (review only)
## Pro Tips for Power Users
- **Batch size matters:** Instruct Devin to process 40–60 files per PR. Larger batches cause review fatigue; smaller batches create excessive PR overhead.- **Provide fixture scope rules upfront:** Tell Devin which fixtures should be session-scoped vs function-scoped to avoid database state leakage between tests.- **Use a canary app first:** Start Devin on your smallest Django app to validate the conversion pattern before scaling to the full codebase.- **Pin marker conventions:** Define your pytest.mark taxonomy (e.g., @pytest.mark.slow, @pytest.mark.integration) in the session prompt so Devin applies markers consistently.- **Leverage --lf during review:** Run pytest --lf (last-failed) after each Devin PR to quickly verify only the converted tests.
## Troubleshooting Common Issues
Database access errors after conversion
If you see Database access not allowed, ensure the test function or its fixture uses the db or transactional_db marker from pytest-django:
@pytest.fixture
def order(db): # 'db' marker grants database access
return Order.objects.create(status='pending')
### Fixture not found errors across apps
Pytest discovers conftest.py files based on directory hierarchy. If a fixture in apps/orders/conftest.py is needed in apps/payments/, move it to a parent-level conftest.py or use explicit imports via pytest_plugins:
# apps/payments/conftest.py
pytest_plugins = ['apps.orders.conftest']
### Parallel execution causing test interference
When using pytest-xdist, tests sharing mutable global state can fail intermittently. Instruct Devin to flag tests that modify settings or use @pytest.mark.forked:
@pytest.mark.forked
def test_payment_gateway_timeout(settings):
settings.PAYMENT_TIMEOUT = 1
# test runs in isolated subprocess
### CI pipeline not picking up pytest results
Ensure the --junitxml flag is set and the path matches your CI tool's expected report location. For GitHub Actions:
- name: Run Tests
run: pytest --junitxml=test-results/results.xml
- uses: actions/upload-artifact@v4
with:
name: test-results
path: test-results/
## Frequently Asked Questions
How does Devin handle complex unittest inheritance hierarchies during migration?
Devin traces the full class hierarchy of each TestCase subclass, identifying shared setUp and tearDown logic across parent and child classes. It flattens these into composable pytest fixtures with appropriate scope, placing shared fixtures in conftest.py files at the correct directory level. For mixins and multiple inheritance patterns, Devin creates separate fixture functions for each concern and composes them via fixture dependencies rather than class inheritance.
What level of human review is needed for Devin’s automated test conversions?
In this case study, two senior engineers spent approximately 16 hours total reviewing Devin’s pull requests over the ten-day migration period. The review focused on verifying fixture scope correctness, ensuring database isolation between tests, and confirming that assertion semantics were preserved. Devin’s conversion accuracy was above 97%, with manual corrections needed primarily for tests involving complex mock chains and custom test runners.
Can Devin handle migrations for test suites using third-party unittest extensions?
Yes. Devin can parse and convert tests that use extensions such as django-nose, unittest2, and custom assertion mixins. During the initial analysis phase, Devin catalogs all third-party test utilities in use and maps them to pytest equivalents. For example, nose.tools.assert_raises maps to pytest.raises, and custom assertion methods are converted to standalone helper functions or pytest plugins. You should include any project-specific testing conventions in the session prompt for best results.