The first aim of profiling is to test a representative system to identify what's slow, using too much RAM, causing too much disk I/O or network I/O. You should keep in mind that profiling typically adds an overhead to your code.
In this post, I will introduce tools you could use to profile your Python or Django projects, including: timer, pycallgraph, cProfile, line-profiler, memory-profiler.
ref:
https://stackoverflow.com/questions/582336/how-can-you-profile-a-script
https://www.airpair.com/python/posts/optimizing-python-code
timer
The simplest way to profile a piece of code.
ref:
https://docs.python.org/3/library/timeit.html
pycallgraph
pycallgraph is a Python module that creates call graph visualizations for Python applications.
ref:
https://pycallgraph.readthedocs.org/en/latest/
$ sudo apt-get install graphviz
$ pip install pycallgraph
# in your_app/middlewares.py
from pycallgraph import Config
from pycallgraph import PyCallGraph
from pycallgraph.globbing_filter import GlobbingFilter
from pycallgraph.output import GraphvizOutput
import time
class PyCallGraphMiddleware(object):
def process_view(self, request, callback, callback_args, callback_kwargs):
if 'graph' in request.GET:
config = Config()
config.trace_filter = GlobbingFilter(include=['rest_framework.*', 'api.*', 'music.*'])
graphviz = GraphvizOutput(output_file='pycallgraph-{}.png'.format(time.time()))
pycallgraph = PyCallGraph(output=graphviz, config=config)
pycallgraph.start()
self.pycallgraph = pycallgraph
def process_response(self, request, response):
if 'graph' in request.GET:
self.pycallgraph.done()
return response
# in settings.py
MIDDLEWARE_CLASSES = (
'your_app.middlewares.PyCallGraphMiddleware',
...
)
$ python manage.py runserver 0.0.0.0:8000
$ open http://127.0.0.1:8000/your_endpoint/?graph=true
cProfile
cProfile is a tool in Python's standard library to understand which functions in your code take the longest to run. It will give you a high-level view of the performance problem so you can direct your attention to the critical functions.
ref:
http://igor.kupczynski.info/2015/01/16/profiling-python-scripts.html
https://ymichael.com/2014/03/08/profiling-python-with-cprofile.html
$ python -m cProfile manage.py test member
$ python -m cProfile -o my-profile-data.out manage.py test --failtest
$ python -m cProfile -o my-profile-data.out manage.py runserver 0.0.0.0:8000
$ pip install cprofilev
$ cprofilev -f my-profile-data.out -a 0.0.0.0 -p 4000
$ open http://127.0.0.1:4000
cProfile with django-cprofile-middleware
$ pip install django-cprofile-middleware
# in settings.py
MIDDLEWARE_CLASSES = (
...
'django_cprofile_middleware.middleware.ProfilerMiddleware',
)
Open any url with a ?prof suffix to do the profiling, for instance, http://localhost:8000/foo/?prof
ref:
https://github.com/omarish/django-cprofile-middleware
cProfile with django-extension and kcachegrind
kcachegrind is a profiling data visualization tool, used to determine the most time consuming execution parts of a program.
ref:
http://django-extensions.readthedocs.org/en/latest/runprofileserver.html
$ pip install django-extensions
# in settings.py
INSTALLED_APPS += (
'django_extensions',
)
$ mkdir -p my-profile-data
$ python manage.py runprofileserver \
--noreload \
--nomedia \
--nostatic \
--kcachegrind \
--prof-path=my-profile-data \
0.0.0.0:8000
$ brew install qcachegrind --with-graphviz
$ qcachegrind my-profile-data/root.003563ms.1441992439.prof
# or
$ sudo apt-get install kcachegrind
$ kcachegrind my-profile-data/root.003563ms.1441992439.prof
cProfile with django-debug-toolbar
You're only able to use django-debug-toolbar if your view returns HTML, it needs a place to inject the debug panels into your DOM on the webpage.
ref:
https://github.com/django-debug-toolbar/django-debug-toolbar
$ pip install django-debug-toolbar
# in settiangs.py
INSTALLED_APPS += (
'debug_toolbar',
)
DEBUG_TOOLBAR_PANELS = [
...
'debug_toolbar.panels.profiling.ProfilingPanel',
...
]
line-profiler
line-profiler is a module for doing line-by-line profiling of functions. One of my favorite tools.
ref:
https://github.com/rkern/line_profiler
$ pip install line-profiler
# in your_app/views.py
def do_line_profiler(view=None, extra_view=None):
import line_profiler
def wrapper(view):
def wrapped(*args, **kwargs):
prof = line_profiler.LineProfiler()
prof.add_function(view)
if extra_view:
[prof.add_function(v) for v in extra_view]
with prof:
resp = view(*args, **kwargs)
prof.print_stats()
return resp
return wrapped
if view:
return wrapper(view)
return wrapper
@do_line_profiler
def your_view(request):
pass
ref:
https://djangosnippets.org/snippets/10483/
There is a pure Python alternative: pprofile.
https://github.com/vpelletier/pprofile
line-profiler with django-devserver
ref:
https://github.com/dcramer/django-devserver
$ pip install git+git://github.com/dcramer/django-devserver#egg=django-devserver
in settings.py
INSTALLED_APPS += (
'devserver',
)
DEVSERVER_MODULES = (
...
'devserver.modules.profile.LineProfilerModule',
...
)
DEVSERVER_AUTO_PROFILE = False
in your_app/views.py
from devserver.modules.profile import devserver_profile
@devserver_profile()
def your_view(request):
pass
line-profiler with django-debug-toolbar-line-profiler
ref:
http://django-debug-toolbar.readthedocs.org/en/latest/
https://github.com/dmclain/django-debug-toolbar-line-profiler
$ pip install django-debug-toolbar django-debug-toolbar-line-profiler
# in settings.py
INSTALLED_APPS += (
'debug_toolbar',
'debug_toolbar_line_profiler',
)
DEBUG_TOOLBAR_PANELS = [
...
'debug_toolbar_line_profiler.panel.ProfilingPanel',
...
]
memory-profiler
This is a Python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for Python programs.
ref:
https://pypi.python.org/pypi/memory_profiler
$ pip install memory-profiler psutil
# in your_app/views.py
from memory_profiler import profile
@profile(precision=4)
def your_view(request):
pass
There are other options:
http://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended
dogslow
ref:
https://bitbucket.org/evzijst/dogslow
django-slow-tests
ref:
https://github.com/realpython/django-slow-tests
