aboutsummaryrefslogtreecommitdiff
path: root/_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
diff options
context:
space:
mode:
Diffstat (limited to '_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md')
-rw-r--r--_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md207
1 files changed, 0 insertions, 207 deletions
diff --git a/_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
deleted file mode 100644
index 2e2ec70..0000000
--- a/_posts/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
+++ /dev/null
@@ -1,207 +0,0 @@
1---
2title: Profiling Python web applications with visual tools
3permalink: /profiling-python-web-applications-with-visual-tools.html
4date: 2017-04-21T12:00:00+02:00
5layout: post
6type: post
7draft: false
8---
9
10I have been profiling my software with KCachegrind for a long time now and I was
11missing this option when I am developing API's or other web services. I always
12knew that this is possible but never really took the time and dive into it.
13
14Before we begin there are some requirements. We will need to:
15
16- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app,
17- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/),
18- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/).
19
20
21If you are using MacOS you should check out [Profiling
22Viewer](http://www.profilingviewer.com/) or
23[MacCallGrind](http://www.maccallgrind.com/).
24
25![KCachegrind](/assets/posts/python-profiling/kcachegrind.png){:loading="lazy"}
26
27We will be dividing this post into two main categories:
28
29- writing simple web-service,
30- visualize profile of this web-service.
31
32## Simple web-service
33
34Let's use virtualenv so we won't pollute our base system. If you don't have
35virtualenv installed on your system you can install it with pip command.
36
37```bash
38# let's install virtualenv globally
39$ sudo pip install virtualenv
40
41# let's also install pyprof2calltree globally
42$ sudo pip install pyprof2calltree
43
44# now we create project
45$ mkdir demo-project
46$ cd demo-project/
47
48# now let's create folder where we will store profiles
49$ mkdir prof
50
51# now we create empty virtualenv in venv/ folder
52$ virtualenv --no-site-packages venv
53
54# we now need to activate virtualenv
55$ source venv/bin/activate
56
57# you can check if virtualenv was correctly initialized by
58# checking where your python interpreter is located
59# if command bellow points to your created directory and not some
60# system dir like /usr/bin/python then everything is fine
61$ which python
62
63# we can check now if all is good ➜ if ok couple of
64# lines will be displayed
65$ pip freeze
66# appdirs==1.4.3
67# packaging==16.8
68# pyparsing==2.2.0
69# six==1.10.0
70
71# now we are ready to install bottlepy ➜ web micro-framework
72$ pip install bottle
73
74# you can deactivate virtualenv but you will then go
75# under system domain ➜ for now don't deactivate
76$ deactivate
77```
78
79We are now ready to write simple web service. Let's create file app.py and paste
80code bellow in this newly created file.
81
82```python
83# -*- coding: utf-8 -*-
84
85import bottle
86import random
87import cProfile
88
89app = bottle.Bottle()
90
91# this function is a decorator and encapsulates function
92# and performs profiling and then saves it to subfolder
93# prof/function-name.prof
94# in our example only awesome_random_number function will
95# be profiled because it has do_cprofile defined
96def do_cprofile(func):
97 def profiled_func(*args, **kwargs):
98 profile = cProfile.Profile()
99 try:
100 profile.enable()
101 result = func(*args, **kwargs)
102 profile.disable()
103 return result
104 finally:
105 profile.dump_stats("prof/" + str(func.__name__) + ".prof")
106 return profiled_func
107
108
109# we use profiling over specific function with including
110# @do_cprofile above function declaration
111@app.route("/")
112@do_cprofile
113def awesome_random_number():
114 awesome_random_number = random.randint(0, 100)
115 return "awesome random number is " + str(awesome_random_number)
116
117@app.route("/test")
118def test():
119 return "dummy test"
120
121if __name__ == '__main__':
122 bottle.run(
123 app = app,
124 host = "0.0.0.0",
125 port = 4000
126 )
127
128# run with 'python app.py'
129# open browser 'http://0.0.0.0:4000'
130```
131
132When browser hits awesome\_random\_number() function profile is created in prof/
133subfolder.
134
135## Visualize profile
136
137Now let's create callgrind format from this cProfile output.
138
139```bash
140$ cd prof/
141$ pyprof2calltree -i awesome_random_number.prof
142# this creates 'awesome_random_number.prof.log' file in the same folder
143```
144
145This file can be opened with visualizing tools listed above. In this case we
146will be using Profilling Viewer under MacOS. You can open image in new tab. As
147you can see from this example there is hierarchy of execution order of your
148code.
149
150![Profilling Viewer](/assets/posts/python-profiling/profiling-viewer.png){:loading="lazy"}
151
152> Make sure you convert output of the cProfile output every time you want to
153refresh and take a look at your possible optimizations because cProfile updates
154.prof file every time browser hits the function.
155
156This is just a simple example but when you are developing real-life applications
157this can be very illuminating, especially to see which parts of your code are
158bottlenecks and need to be optimized.
159
160## Update 2017-04-22
161
162Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome
163web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/)
164that directly takes output from
165[cProfile](https://docs.python.org/2/library/profile.html#module-cProfile)
166module.
167
168<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="583880c1-002e-41ed-a373-020a0ef2cff9" data-embed-created="2017-04-22T19:46:54.810Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
169
170```bash
171# let's install it globally as well
172$ sudo pip install snakeviz
173
174# now let's visualize
175$ cd prof/
176$ snakeviz awesome_random_number.prof
177# this automatically opens browser window and
178# shows visualized profile
179```
180
181![SnakeViz](/assets/posts/python-profiling/snakeviz.png){:loading="lazy"}
182
183Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better
184way for installing pip software by targeting user level instead of using sudo.
185
186<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="f4f0459e-684d-441e-bebe-eb49b2f0a31d" data-embed-created="2017-04-22T19:46:10.874Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
187
188```bash
189# now we need to add this path to our $PATH variable
190# we do this my adding this line at the end of your
191# ~/.bashrc file
192PATH=$PATH:$HOME/.local/bin/
193
194# in order to use this new configuration you can close
195# and reopen terminal or reload .bashrc file
196$ source ~/.bashrc
197
198# now let's test if new directory is present in $PATH
199$ echo $PATH
200
201# now we can install on user level by adding --user
202# without use of sudo
203$ pip install snakeviz --user
204```
205
206Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can
207use [pipsi](https://github.com/mitsuhiko/pipsi).