1---
2title: Profiling Python web applications with visual tools
3url: profiling-python-web-applications-with-visual-tools.html
4date: 2017-04-21T12:00:00+02:00
5type: post
6draft: false
7---
8
9I have been profiling my software with KCachegrind for a long time now and I was
10missing this option when I am developing API's or other web services. I always
11knew that this is possible but never really took the time and dive into it.
12
13Before we begin there are some requirements. We will need to:
14
15- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app,
16- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/),
17- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/).
18
19
20If you are using MacOS you should check out [Profiling
21Viewer](http://www.profilingviewer.com/) or
22[MacCallGrind](http://www.maccallgrind.com/).
23
24
25
26We will be dividing this post into two main categories:
27
28- writing simple web-service,
29- visualize profile of this web-service.
30
31## Simple web-service
32
33Let's use virtualenv so we won't pollute our base system. If you don't have
34virtualenv installed on your system you can install it with pip command.
35
36```bash
37# let's install virtualenv globally
38$ sudo pip install virtualenv
39
40# let's also install pyprof2calltree globally
41$ sudo pip install pyprof2calltree
42
43# now we create project
44$ mkdir demo-project
45$ cd demo-project/
46
47# now let's create folder where we will store profiles
48$ mkdir prof
49
50# now we create empty virtualenv in venv/ folder
51$ virtualenv --no-site-packages venv
52
53# we now need to activate virtualenv
54$ source venv/bin/activate
55
56# you can check if virtualenv was correctly initialized by
57# checking where your python interpreter is located
58# if command bellow points to your created directory and not some
59# system dir like /usr/bin/python then everything is fine
60$ which python
61
62# we can check now if all is good โ if ok couple of
63# lines will be displayed
64$ pip freeze
65# appdirs==1.4.3
66# packaging==16.8
67# pyparsing==2.2.0
68# six==1.10.0
69
70# now we are ready to install bottlepy โ web micro-framework
71$ pip install bottle
72
73# you can deactivate virtualenv but you will then go
74# under system domain โ for now don't deactivate
75$ deactivate
76```
77
78We are now ready to write simple web service. Let's create file app.py and paste
79code bellow in this newly created file.
80
81```python
82# -*- coding: utf-8 -*-
83
84import bottle
85import random
86import cProfile
87
88app = bottle.Bottle()
89
90# this function is a decorator and encapsulates function
91# and performs profiling and then saves it to subfolder
92# prof/function-name.prof
93# in our example only awesome_random_number function will
94# be profiled because it has do_cprofile defined
95def do_cprofile(func):
96 def profiled_func(*args, **kwargs):
97 profile = cProfile.Profile()
98 try:
99 profile.enable()
100 result = func(*args, **kwargs)
101 profile.disable()
102 return result
103 finally:
104 profile.dump_stats("prof/" + str(func.__name__) + ".prof")
105 return profiled_func
106
107
108# we use profiling over specific function with including
109# @do_cprofile above function declaration
110@app.route("/")
111@do_cprofile
112def awesome_random_number():
113 awesome_random_number = random.randint(0, 100)
114 return "awesome random number is " + str(awesome_random_number)
115
116@app.route("/test")
117def test():
118 return "dummy test"
119
120if __name__ == '__main__':
121 bottle.run(
122 app = app,
123 host = "0.0.0.0",
124 port = 4000
125 )
126
127# run with 'python app.py'
128# open browser 'http://0.0.0.0:4000'
129```
130
131When browser hits awesome\_random\_number() function profile is created in prof/
132subfolder.
133
134## Visualize profile
135
136Now let's create callgrind format from this cProfile output.
137
138```bash
139$ cd prof/
140$ pyprof2calltree -i awesome_random_number.prof
141# this creates 'awesome_random_number.prof.log' file in the same folder
142```
143
144This file can be opened with visualizing tools listed above. In this case we
145will be using Profilling Viewer under MacOS. You can open image in new tab. As
146you can see from this example there is hierarchy of execution order of your
147code.
148
149
150
151> Make sure you convert output of the cProfile output every time you want to
152refresh and take a look at your possible optimizations because cProfile updates
153.prof file every time browser hits the function.
154
155This is just a simple example but when you are developing real-life applications
156this can be very illuminating, especially to see which parts of your code are
157bottlenecks and need to be optimized.
158
159## Update 2017-04-22
160
161Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome
162web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/)
163that directly takes output from
164[cProfile](https://docs.python.org/2/library/profile.html#module-cProfile)
165module.
166
167<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="583880c1-002e-41ed-a373-020a0ef2cff9" data-embed-created="2017-04-22T19:46:54.810Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
168
169```bash
170# let's install it globally as well
171$ sudo pip install snakeviz
172
173# now let's visualize
174$ cd prof/
175$ snakeviz awesome_random_number.prof
176# this automatically opens browser window and
177# shows visualized profile
178```
179
180
181
182Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better
183way for installing pip software by targeting user level instead of using sudo.
184
185<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="f4f0459e-684d-441e-bebe-eb49b2f0a31d" data-embed-created="2017-04-22T19:46:10.874Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
186
187```bash
188# now we need to add this path to our $PATH variable
189# we do this my adding this line at the end of your
190# ~/.bashrc file
191PATH=$PATH:$HOME/.local/bin/
192
193# in order to use this new configuration you can close
194# and reopen terminal or reload .bashrc file
195$ source ~/.bashrc
196
197# now let's test if new directory is present in $PATH
198$ echo $PATH
199
200# now we can install on user level by adding --user
201# without use of sudo
202$ pip install snakeviz --user
203```
204
205Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can
206use [pipsi](https://github.com/mitsuhiko/pipsi).