aboutsummaryrefslogtreecommitdiff
path: root/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md
diff options
context:
space:
mode:
Diffstat (limited to 'content/2017-04-21-profiling-python-web-applications-with-visual-tools.md')
-rw-r--r--content/2017-04-21-profiling-python-web-applications-with-visual-tools.md193
1 files changed, 0 insertions, 193 deletions
diff --git a/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md
deleted file mode 100644
index 6e52ee0..0000000
--- a/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md
+++ /dev/null
@@ -1,193 +0,0 @@
1---
2layout: post
3title: Profiling Python web applications with visual tools
4description: Missing link when debugging and profiling python web applications
5slug: profiling-python-web-applications-with-visual-tools
6type: note
7date: 2017-04-21
8---
9
10**Table of contents**
11
121. [Simple web-service](#simple-web-service)
132. [Visualize profile](#visualize-profile)
143. [Update 2017-04-22](#update-2017-04-22)
15
16I have been profiling my software with KCachegrind for a long time now and I was missing this option when I am developing API's or other web services. I always knew that this is possible but never really took the time and dive into it.
17
18Before we begin there are some requirements. We will need to:
19
20- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app,
21- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/),
22- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/).
23
24
25If you are using MacOS you should check out [Profiling Viewer](http://www.profilingviewer.com/) or [MacCallGrind](http://www.maccallgrind.com/).
26
27![KCachegrind](/files/kcachegrind.png)
28
29We will be dividing this post into two main categories:
30
31- writing simple web-service,
32- visualize profile of this web-service.
33
34## Simple web-service
35
36Let's use virtualenv so we won't pollute our base system. If you don't have virtualenv installed on your system you can install it with pip command.
37
38```bash
39# let's install virtualenv globally
40$ sudo pip install virtualenv
41
42# let's also install pyprof2calltree globally
43$ sudo pip install pyprof2calltree
44
45# now we create project
46$ mkdir demo-project
47$ cd demo-project/
48
49# now let's create folder where we will store profiles
50$ mkdir prof
51
52# now we create empty virtualenv in venv/ folder
53$ virtualenv --no-site-packages venv
54
55# we now need to activate virtualenv
56$ source venv/bin/activate
57
58# you can check if virtualenv was correctly initialized by
59# checking where your python interpreter is located
60# if command bellow points to your created directory and not some
61# system dir like /usr/bin/python then everything is fine
62$ which python
63
64# we can check now if all is good ➜ if ok couple of
65# lines will be displayed
66$ pip freeze
67# appdirs==1.4.3
68# packaging==16.8
69# pyparsing==2.2.0
70# six==1.10.0
71
72# now we are ready to install bottlepy ➜ web micro-framework
73$ pip install bottle
74
75# you can deactivate virtualenv but you will then go
76# under system domain ➜ for now don't deactivate
77$ deactivate
78```
79
80We are now ready to write simple web service. Let's create file app.py and paste code bellow in this newly created file.
81
82```python
83# -*- coding: utf-8 -*-
84
85import bottle
86import random
87import cProfile
88
89app = bottle.Bottle()
90
91# this function is a decorator and encapsulates function
92# and performs profiling and then saves it to subfolder
93# prof/function-name.prof
94# in our example only awesome_random_number function will
95# be profiled because it has do_cprofile defined
96def do_cprofile(func):
97 def profiled_func(*args, **kwargs):
98 profile = cProfile.Profile()
99 try:
100 profile.enable()
101 result = func(*args, **kwargs)
102 profile.disable()
103 return result
104 finally:
105 profile.dump_stats("prof/" + str(func.__name__) + ".prof")
106 return profiled_func
107
108
109# we use profiling over specific function with including
110# @do_cprofile above function declaration
111@app.route("/")
112@do_cprofile
113def awesome_random_number():
114 awesome_random_number = random.randint(0, 100)
115 return "awesome random number is " + str(awesome_random_number)
116
117@app.route("/test")
118def test():
119 return "dummy test"
120
121if __name__ == '__main__':
122 bottle.run(
123 app = app,
124 host = "0.0.0.0",
125 port = 4000
126 )
127
128# run with 'python app.py'
129# open browser 'http://0.0.0.0:4000'
130```
131
132When browser hits awesome\_random\_number() function profile is created in prof/ subfolder.
133
134## Visualize profile
135
136Now let's create callgrind format from this cProfile output.
137
138```bash
139$ cd prof/
140$ pyprof2calltree -i awesome_random_number.prof
141# this creates 'awesome_random_number.prof.log' file in the same folder
142```
143
144This file can be opened with visualizing tools listed above. In this case we will be using Profilling Viewer under MacOS. You can open image in new tab. As you can see from this example there is hierarchy of execution order of your code.
145
146![Profilling Viewer](/files/profiling-viewer.png)
147
148> Make sure you convert output of the cProfile output every time you want to refresh and take a look at your possible optimizations because cProfile updates .prof file every time browser hits the function.
149
150This is just a simple example but when you are developing real-life applications this can be very illuminating, especially to see which parts of your code are bottlenecks and need to be optimized.
151
152## Update 2017-04-22
153
154Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) that directly takes output from [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) module.
155
156<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="583880c1-002e-41ed-a373-020a0ef2cff9" data-embed-created="2017-04-22T19:46:54.810Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
157
158```bash
159# let's install it globally as well
160$ sudo pip install snakeviz
161
162# now let's visualize
163$ cd prof/
164$ snakeviz awesome_random_number.prof
165# this automatically opens browser window and
166# shows visualized profile
167```
168
169![SnakeViz](/files/snakeviz.png)
170
171Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better way for installing pip software by targeting user level instead of using sudo.
172
173<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="f4f0459e-684d-441e-bebe-eb49b2f0a31d" data-embed-created="2017-04-22T19:46:10.874Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
174
175```bash
176# now we need to add this path to our $PATH variable
177# we do this my adding this line at the end of your
178# ~/.bashrc file
179PATH=$PATH:$HOME/.local/bin/
180
181# in order to use this new configuration you can close
182# and reopen terminal or reload .bashrc file
183$ source ~/.bashrc
184
185# now let's test if new directory is present in $PATH
186$ echo $PATH
187
188# now we can install on user level by adding --user
189# without use of sudo
190$ pip install snakeviz --user
191```
192
193Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can use [pipsi](https://github.com/mitsuhiko/pipsi).