aboutsummaryrefslogtreecommitdiff
path: root/_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
diff options
context:
space:
mode:
Diffstat (limited to '_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md')
-rw-r--r--_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md192
1 files changed, 0 insertions, 192 deletions
diff --git a/_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
deleted file mode 100644
index dcd8ce1..0000000
--- a/_posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md
+++ /dev/null
@@ -1,192 +0,0 @@
1---
2
3layout: post
4title: Profiling Python web applications with visual tools
5description: Missing link when debugging and profiling python web applications
6
7---
8
9**Table of contents**
10
111. [Simple web-service](#simple-web-service)
122. [Visualize profile](#visualize-profile)
133. [Update 2017-04-22](#update-2017-04-22)
14
15I have been profiling my software with KCachegrind for a long time now and I was missing this option when I am developing API's or other web services. I always knew that this is possible but never really took the time and dive into it.
16
17Before we begin there are some requirements. We will need to:
18
19- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app,
20- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/),
21- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/).
22
23
24If you are using MacOS you should check out [Profiling Viewer](http://www.profilingviewer.com/) or [MacCallGrind](http://www.maccallgrind.com/).
25
26![KCachegrind](/files/kcachegrind.png)
27
28We will be dividing this post into two main categories:
29
30- writing simple web-service,
31- visualize profile of this web-service.
32
33## Simple web-service
34
35Let's use virtualenv so we won't pollute our base system. If you don't have virtualenv installed on your system you can install it with pip command.
36
37```bash
38# let's install virtualenv globally
39$ sudo pip install virtualenv
40
41# let's also install pyprof2calltree globally
42$ sudo pip install pyprof2calltree
43
44# now we create project
45$ mkdir demo-project
46$ cd demo-project/
47
48# now let's create folder where we will store profiles
49$ mkdir prof
50
51# now we create empty virtualenv in venv/ folder
52$ virtualenv --no-site-packages venv
53
54# we now need to activate virtualenv
55$ source venv/bin/activate
56
57# you can check if virtualenv was correctly initialized by
58# checking where your python interpreter is located
59# if command bellow points to your created directory and not some
60# system dir like /usr/bin/python then everything is fine
61$ which python
62
63# we can check now if all is good ➜ if ok couple of
64# lines will be displayed
65$ pip freeze
66# appdirs==1.4.3
67# packaging==16.8
68# pyparsing==2.2.0
69# six==1.10.0
70
71# now we are ready to install bottlepy ➜ web micro-framework
72$ pip install bottle
73
74# you can deactivate virtualenv but you will then go
75# under system domain ➜ for now don't deactivate
76$ deactivate
77```
78
79We are now ready to write simple web service. Let's create file app.py and paste code bellow in this newly created file.
80
81```python
82# -*- coding: utf-8 -*-
83
84import bottle
85import random
86import cProfile
87
88app = bottle.Bottle()
89
90# this function is a decorator and encapsulates function
91# and performs profiling and then saves it to subfolder
92# prof/function-name.prof
93# in our example only awesome_random_number function will
94# be profiled because it has do_cprofile defined
95def do_cprofile(func):
96 def profiled_func(*args, **kwargs):
97 profile = cProfile.Profile()
98 try:
99 profile.enable()
100 result = func(*args, **kwargs)
101 profile.disable()
102 return result
103 finally:
104 profile.dump_stats("prof/" + str(func.__name__) + ".prof")
105 return profiled_func
106
107
108# we use profiling over specific function with including
109# @do_cprofile above function declaration
110@app.route("/")
111@do_cprofile
112def awesome_random_number():
113 awesome_random_number = random.randint(0, 100)
114 return "awesome random number is " + str(awesome_random_number)
115
116@app.route("/test")
117def test():
118 return "dummy test"
119
120if __name__ == '__main__':
121 bottle.run(
122 app = app,
123 host = "0.0.0.0",
124 port = 4000
125 )
126
127# run with 'python app.py'
128# open browser 'http://0.0.0.0:4000'
129```
130
131When browser hits awesome\_random\_number() function profile is created in prof/ subfolder.
132
133## Visualize profile
134
135Now let's create callgrind format from this cProfile output.
136
137```bash
138$ cd prof/
139$ pyprof2calltree -i awesome_random_number.prof
140# this creates 'awesome_random_number.prof.log' file in the same folder
141```
142
143This file can be opened with visualizing tools listed above. In this case we will be using Profilling Viewer under MacOS. You can open image in new tab. As you can see from this example there is hierarchy of execution order of your code.
144
145![Profilling Viewer](/files/profiling-viewer.png)
146
147> Make sure you convert output of the cProfile output every time you want to refresh and take a look at your possible optimizations because cProfile updates .prof file every time browser hits the function.
148
149This is just a simple example but when you are developing real-life applications this can be very illuminating, especially to see which parts of your code are bottlenecks and need to be optimized.
150
151## Update 2017-04-22
152
153Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) that directly takes output from [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) module.
154
155<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="583880c1-002e-41ed-a373-020a0ef2cff9" data-embed-created="2017-04-22T19:46:54.810Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
156
157```bash
158# let's install it globally as well
159$ sudo pip install snakeviz
160
161# now let's visualize
162$ cd prof/
163$ snakeviz awesome_random_number.prof
164# this automatically opens browser window and
165# shows visualized profile
166```
167
168![SnakeViz](/files/snakeviz.png)
169
170Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better way for installing pip software by targeting user level instead of using sudo.
171
172<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="f4f0459e-684d-441e-bebe-eb49b2f0a31d" data-embed-created="2017-04-22T19:46:10.874Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
173
174```bash
175# now we need to add this path to our $PATH variable
176# we do this my adding this line at the end of your
177# ~/.bashrc file
178PATH=$PATH:$HOME/.local/bin/
179
180# in order to use this new configuration you can close
181# and reopen terminal or reload .bashrc file
182$ source ~/.bashrc
183
184# now let's test if new directory is present in $PATH
185$ echo $PATH
186
187# now we can install on user level by adding --user
188# without use of sudo
189$ pip install snakeviz --user
190```
191
192Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can use [pipsi](https://github.com/mitsuhiko/pipsi).