Python developers face unique challenges when scheduling scripts with cron. From virtual environment activation to Django's specific requirements, this guide cuts through the confusion with precise, tested solutions for data scientists and web developers alike.
Running a Simple Python Script with Cron
Let's start with the basics: executing a standalone Python script on a schedule.
The Basic Python Cron Job
0 2 * * * /usr/bin/python3 /home/user/scripts/backup.py
This runs backup.py every day at 2:00 AM. But there's a better way.
The Shebang Approach (Recommended)
Add a shebang to the top of your Python script:
#!/usr/bin/env python3
# File: /home/user/scripts/data_scraper.py
import requests
from datetime import datetime
def main():
print(f"Script started at {datetime.now()}")
# Your code here
response = requests.get('https://api.example.com/data')
# Process data...
print("Script completed successfully")
if __name__ == "__main__":
main()
Make it executable:
chmod +x /home/user/scripts/data_scraper.py
Now your crontab entry is cleaner:
0 * * * * /home/user/scripts/data_scraper.py >> /var/log/scraper.log 2>&1
Why this is better:
- ✅ The script knows which Python interpreter to use
- ✅ Works across different systems
- ✅ Easier to read and maintain
- ✅ No hardcoded Python paths in crontab
Using Absolute Paths (Critical)
Cron runs with a minimal environment. Always use absolute paths:
# ❌ WRONG - Relative paths will fail
0 2 * * * python scripts/backup.py
# ✅ CORRECT - Absolute paths always work
0 2 * * * /usr/bin/python3 /home/user/scripts/backup.py
# ✅ EVEN BETTER - With shebang
0 2 * * * /home/user/scripts/backup.py
Find your Python path:
which python3
# Output: /usr/bin/python3
The Virtual Environment Problem (And The Solution)
Here's where 90% of Python developers hit a wall. Your script works perfectly when you run it manually, but fails silently in cron. The culprit? Virtual environments.
Why Virtual Environments Break in Cron
When you activate a virtual environment in your terminal, it modifies your PATH and sets environment variables. Cron doesn't know about any of that—it starts with a clean slate.
Solution 1: Activate the Virtual Environment in Cron
The most reliable approach is to activate the virtual environment before running your script:
0 2 * * * cd /home/user/projects/myproject && /home/user/projects/myproject/venv/bin/python /home/user/projects/myproject/scripts/process_data.py >> /var/log/process.log 2>&1
Breaking it down:
cd /home/user/projects/myproject- Navigate to project directory&&- Only continue if the previous command succeeded/home/user/projects/myproject/venv/bin/python- Use the Python from your venv/home/user/projects/myproject/scripts/process_data.py- Your script>> /var/log/process.log 2>&1- Log everything
Solution 2: Using a Wrapper Shell Script
For complex setups, create a wrapper script:
#!/bin/bash
# File: /home/user/scripts/run_scraper.sh
# Activate virtual environment
source /home/user/projects/scraper/venv/bin/activate
# Set working directory
cd /home/user/projects/scraper
# Run the Python script
python scraper/main.py
# Deactivate (optional, cron terminates anyway)
deactivate
Make it executable:
chmod +x /home/user/scripts/run_scraper.sh
Your crontab becomes simple:
0 */6 * * * /home/user/scripts/run_scraper.sh >> /var/log/scraper.log 2>&1
Solution 3: Direct venv Python Path (Simplest)
You don't actually need to "activate" the environment—just use its Python binary:
# Run every hour
0 * * * * /home/user/myproject/venv/bin/python /home/user/myproject/app.py
This works because the venv's Python knows where to find the installed packages.
The Django Way: Management Commands
Django developers have a powerful, elegant solution: custom management commands. This is the professional approach for scheduling Django tasks.
Creating a Django Management Command
Step 1: Create the command file
# Your Django project structure
myproject/
├── manage.py
└── myapp/
└── management/
└── commands/
└── send_daily_report.py
Create the directories if they don't exist:
mkdir -p myapp/management/commands
touch myapp/management/__init__.py
touch myapp/management/commands/__init__.py
Step 2: Write your command
# myapp/management/commands/send_daily_report.py
from django.core.management.base import BaseCommand
from django.utils import timezone
from myapp.models import User, Report
from django.core.mail import send_mail
from django.conf import settings
class Command(BaseCommand):
help = 'Send daily report emails to all active users'
def add_arguments(self, parser):
# Optional: Add command-line arguments
parser.add_argument(
'--dry-run',
action='store_true',
help='Run without actually sending emails',
)
def handle(self, *args, **options):
self.stdout.write(self.style.SUCCESS(
f'Starting daily report generation at {timezone.now()}'
))
users = User.objects.filter(is_active=True)
sent_count = 0
for user in users:
report_data = Report.objects.filter(
user=user,
created_at__date=timezone.now().date()
)
if options['dry_run']:
self.stdout.write(f'Would send to: {user.email}')
continue
# Send the actual email
send_mail(
subject='Your Daily Report',
message=f'Report data: {report_data.count()} items',
from_email=settings.DEFAULT_FROM_EMAIL,
recipient_list=[user.email],
fail_silently=False,
)
sent_count += 1
self.stdout.write(self.style.SUCCESS(
f'Successfully sent {sent_count} reports'
))
Step 3: Test your command manually
python manage.py send_daily_report --dry-run
python manage.py send_daily_report
Step 4: Schedule it with cron
0 8 * * * cd /var/www/myproject && /var/www/myproject/venv/bin/python manage.py send_daily_report >> /var/log/django-reports.log 2>&1
This runs every day at 8:00 AM.
Real-World Django Management Command Examples
Example 1: Clean Up Old Sessions
# cleanup_sessions.py
from django.core.management.base import BaseCommand
from django.contrib.sessions.models import Session
from django.utils import timezone
from datetime import timedelta
class Command(BaseCommand):
help = 'Delete sessions older than 30 days'
def handle(self, *args, **options):
cutoff_date = timezone.now() - timedelta(days=30)
old_sessions = Session.objects.filter(expire_date__lt=cutoff_date)
count = old_sessions.count()
old_sessions.delete()
self.stdout.write(self.style.SUCCESS(
f'Deleted {count} old sessions'
))
Cron entry (daily at 3 AM):
0 3 * * * cd /var/www/myproject && venv/bin/python manage.py cleanup_sessions
Example 2: Generate Weekly Analytics
# generate_analytics.py
from django.core.management.base import BaseCommand
from myapp.models import Order, User
from myapp.analytics import AnalyticsReport
import json
class Command(BaseCommand):
help = 'Generate weekly analytics report'
def handle(self, *args, **options):
report = AnalyticsReport()
report.calculate_weekly_metrics()
data = {
'total_orders': Order.objects.count(),
'total_users': User.objects.count(),
'revenue': report.get_weekly_revenue(),
}
with open('/var/reports/weekly_analytics.json', 'w') as f:
json.dump(data, f, indent=2)
self.stdout.write(self.style.SUCCESS('Analytics generated'))
Cron entry (every Monday at 9 AM):
0 9 * * 1 cd /var/www/myproject && venv/bin/python manage.py generate_analytics
Example 3: Database Maintenance
# database_maintenance.py
from django.core.management.base import BaseCommand
from django.db import connection
class Command(BaseCommand):
help = 'Optimize database tables'
def handle(self, *args, **options):
with connection.cursor() as cursor:
# PostgreSQL example
cursor.execute("VACUUM ANALYZE;")
self.stdout.write(self.style.SUCCESS(
'Database optimized successfully'
))
Python Cron Job Use Cases
Use Case 1: Web Scraping Script
#!/usr/bin/env python3
# scraper.py
import requests
from bs4 import BeautifulSoup
import json
from datetime import datetime
import logging
# Configure logging
logging.basicConfig(
filename='/var/log/scraper.log',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
def scrape_data():
try:
logging.info("Starting scrape job")
response = requests.get(
'https://example.com/data',
timeout=30
)
response.raise_for_status()
soup = BeautifulSoup(response.content, 'html.parser')
data = soup.find_all('div', class_='data-item')
results = [{
'title': item.get_text(),
'scraped_at': datetime.now().isoformat()
} for item in data]
with open('/var/data/scraped_data.json', 'w') as f:
json.dump(results, f, indent=2)
logging.info(f"Scraped {len(results)} items successfully")
except Exception as e:
logging.error(f"Scraping failed: {str(e)}")
raise
if __name__ == "__main__":
scrape_data()
Cron entry (every 4 hours):
0 */4 * * * /home/user/scraper/venv/bin/python /home/user/scraper/scraper.py
Use Case 2: Data Processing Pipeline
#!/usr/bin/env python3
# process_pipeline.py
import pandas as pd
import logging
from pathlib import Path
logging.basicConfig(level=logging.INFO)
def process_data():
logging.info("Starting data processing")
# Read raw data
df = pd.read_csv('/data/raw/input.csv')
# Process
df_clean = df.dropna()
df_clean['processed_at'] = pd.Timestamp.now()
# Save
output_path = Path('/data/processed')
output_path.mkdir(exist_ok=True)
df_clean.to_csv(output_path / 'output.csv', index=False)
logging.info(f"Processed {len(df_clean)} records")
if __name__ == "__main__":
process_data()
Cron entry (daily at midnight):
0 0 * * * /home/user/pipeline/venv/bin/python /home/user/pipeline/process_pipeline.py >> /var/log/pipeline.log 2>&1
Use Case 3: Database Backup Script
#!/usr/bin/env python3
# backup_database.py
import subprocess
from datetime import datetime
import os
import logging
logging.basicConfig(
filename='/var/log/backup.log',
level=logging.INFO
)
def backup_postgres():
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
backup_file = f'/backups/db_backup_{timestamp}.sql'
try:
# Run pg_dump
subprocess.run([
'pg_dump',
'-h', 'localhost',
'-U', 'dbuser',
'-d', 'mydatabase',
'-f', backup_file
], check=True, env={
'PGPASSWORD': os.environ.get('DB_PASSWORD', '')
})
logging.info(f"Backup created: {backup_file}")
# Compress
subprocess.run(['gzip', backup_file], check=True)
logging.info(f"Backup compressed: {backup_file}.gz")
except subprocess.CalledProcessError as e:
logging.error(f"Backup failed: {e}")
raise
if __name__ == "__main__":
backup_postgres()
Cron entry (daily at 2 AM):
0 2 * * * DB_PASSWORD=secret /home/user/backup/venv/bin/python /home/user/backup/backup_database.py
Use Case 4: Clean Up Old Files
#!/usr/bin/env python3
# cleanup_old_files.py
from pathlib import Path
from datetime import datetime, timedelta
import logging
logging.basicConfig(level=logging.INFO)
def cleanup_old_files(directory, days_old=30):
"""Delete files older than specified days"""
path = Path(directory)
cutoff = datetime.now() - timedelta(days=days_old)
deleted_count = 0
for file in path.glob('*'):
if file.is_file():
file_time = datetime.fromtimestamp(file.stat().st_mtime)
if file_time < cutoff:
file.unlink()
deleted_count += 1
logging.info(f"Deleted: {file.name}")
logging.info(f"Cleanup complete. Deleted {deleted_count} files")
if __name__ == "__main__":
cleanup_old_files('/var/tmp/uploads', days_old=7)
cleanup_old_files('/var/log/old', days_old=90)
Cron entry (weekly on Sunday at 4 AM):
0 4 * * 0 /usr/bin/python3 /home/user/scripts/cleanup_old_files.py
Advanced Techniques
Environment Variables in Cron
Set environment variables at the top of your crontab:
# Edit with: crontab -e
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=admin@example.com
DB_PASSWORD=secret123
0 2 * * * /home/user/venv/bin/python /home/user/backup.py
Or pass them inline:
0 2 * * * DB_PASSWORD=secret /home/user/venv/bin/python /home/user/backup.py
Using .env Files with python-dotenv
#!/usr/bin/env python3
# script_with_env.py
from dotenv import load_dotenv
import os
# Load environment variables from .env file
load_dotenv('/home/user/myproject/.env')
API_KEY = os.getenv('API_KEY')
DATABASE_URL = os.getenv('DATABASE_URL')
# Your script logic here
Parallel Execution with Multiple Workers
#!/usr/bin/env python3
# parallel_processor.py
from multiprocessing import Pool
import logging
def process_item(item):
# Process each item
return f"Processed: {item}"
def main():
items = range(100)
with Pool(processes=4) as pool:
results = pool.map(process_item, items)
logging.info(f"Processed {len(results)} items")
if __name__ == "__main__":
main()
Common Python Cron Pitfalls
Issue 1: Import Errors
Problem: Script can't find your modules
Solution: Add your project to PYTHONPATH
0 * * * * PYTHONPATH=/home/user/myproject /home/user/venv/bin/python /home/user/myproject/scripts/run.py
Issue 2: Timezone Issues
Problem: Script runs at wrong time
Solution: Set TZ variable
TZ=America/New_York
0 9 * * * /home/user/venv/bin/python /home/user/morning_report.py
Issue 3: Permission Denied on Log Files
Problem: Can't write to log file
Solution: Ensure correct permissions
touch /var/log/myscript.log
chmod 644 /var/log/myscript.log
chown username:username /var/log/myscript.log
Issue 4: Cron Can't Find System Commands
Problem: command not found errors
Solution: Use absolute paths for all commands
# Instead of:
subprocess.run(['psql', ...])
# Use:
subprocess.run(['/usr/bin/psql', ...])
Find command locations:
which psql
# /usr/bin/psql
Best Practices for Python Cron Jobs
✅ Always use absolute paths - for Python, scripts, and data files ✅ Use virtual environments - isolate dependencies ✅ Implement proper logging - don't rely on print statements ✅ Handle errors gracefully - use try/except blocks ✅ Set timeouts - prevent hanging scripts ✅ Test manually first - run your script outside of cron ✅ Use shebangs - makes scripts self-documenting ✅ Monitor execution - log start/end times and results ✅ Keep scripts idempotent - safe to run multiple times ✅ Document your crontab - add comments explaining each job
Testing Your Python Cron Jobs
Before adding to cron, test your setup:
# 1. Test the script directly
/home/user/venv/bin/python /home/user/script.py
# 2. Test with the exact cron environment
env -i /bin/bash -c 'cd /home/user && /home/user/venv/bin/python script.py'
# 3. Test as the cron user (if different)
sudo -u cronuser /home/user/venv/bin/python /home/user/script.py
Monitoring and Debugging
Check if Cron is Running Your Job
# View cron log (Ubuntu/Debian)
grep CRON /var/log/syslog
# Or (CentOS/RHEL)
grep CRON /var/log/cron
Add Debugging to Your Script
import sys
import logging
logging.basicConfig(
filename='/var/log/debug.log',
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logging.debug(f"Python version: {sys.version}")
logging.debug(f"Python path: {sys.path}")
logging.debug(f"Current directory: {os.getcwd()}")
Conclusion
Scheduling Python scripts with cron doesn't have to be painful. The keys to success are:
- Virtual Environments: Use the venv's Python binary directly
- Absolute Paths: Never rely on relative paths or PATH lookups
- Django Management Commands: The professional way for Django projects
- Proper Logging: Essential for debugging silent failures
- Test First: Always verify scripts work outside cron before scheduling
Whether you're automating data pipelines, web scraping, or Django maintenance tasks, these patterns will save you hours of debugging.
Ready to build your perfect cron expression? Use our Cron Expression Generator to create and test your schedules with confidence!
Related Articles
Language-Specific Guides:
- Node.js Cron Jobs: System vs node-cron - Node.js automation
- Mastering Cron Jobs in PHP & Laravel - PHP frameworks
Essential Cron Skills:
- The Ultimate Guide to Cron Jobs - Complete tutorial
- Environment Variables in Cron Jobs - Fix PATH and venv issues
- Why Your Cron Job Isn't Running - Troubleshooting guide
Best Practices:
- Where Did My Cron Job Output Go? - Logging guide
- Organize Your Crontab Like a Pro - Maintainability tips
- 5 Real-World Cron Job Examples - Production examples
Keywords: python cron job, run python script cron, django cron job, django management command cron, schedule python script, python virtual environment cron, python automation