1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
|
<!doctype html><html lang=en-us><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><link href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL69vf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAv76+/8LBwQkAAAAAAAAAAAAAAAC+vb3/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL+9vf/Bv78JAAAAAAAAAAAAAAAAu7q6/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC7ubr/vr29CAAAAAAAAAAAy8nJAZ6foP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnqGj/6GipAoAAAAAHLjU/xcXHf/BwsL/I8XY/yPK3v8XGiD/IbjL/yPF2f8XGiD/Fxkf/yLF2f8gnK3/Fxog/62ztv8fwNf/FRcd/x271v8mz93/GRsi/xkXHf8p097/GiIp/xobIv8p0t3/KdPe/xocIv8fYmr/KNPe/xoZH/8aHCL/J87c/xy81/8VFxz/IsPZ/8zS0/8XGiD/Ir/R/yPH2/8XGiD/Fxkf/yPH2/8dd4T/GBog/yPJ3f8jyNr/uru9/xcUGv8cudb/EhITDKi5vRKlvMP/RUpOERwcHRAdOj4QHTk8EBwdHRAdNTgQHTo/EBwcHRAcHB0QSGduEKW4vf+koqQfHzg+EBqz0ewSFRv7EyMr/xq51vsTERb7ExUb+xq41fsau9j7ExUb+xiPp/sZudb7ExUb+xMVG/sZuNX/GKvI/BIUGfMdvdn/IrfL/xcaIP8n1eb/J9Dh/xkcIf8ZGR7/J8/f/xxCSv8ZGyH/J9Dg/ybQ4P8ZHCL/FSQs/yPK3/8UExj/GE1b/ybS5P8ZGB7/Ghwj/ynW5P8p2Ob/Ghwi/yWrtv8p1eH/Ghwi/xocIv8p1uT/J8XT/xkcIv8m1un/Hb7d/xUYH/8hzOr/HtHu/xcaIf8XGB//I8vi/xgxOv8XGSD/I8rg/yPK4P8XGiD/GUFL/yPP6f8SERj/Fhkh/x3A4f8AAAAAJ2f9/ydr//8mZPH/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlYu38J2v//ydo/f8AAAAAAAAAAAd8/fkFqf//Iob8sAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMY39awWr//8FfP3/AAAAAAAAAAAFm/7/SfD//wR+/f8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOB/f9B7v//BaX+/wAAAAAAAAAAQ878SAyZ/v9n1v4KAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADu9v8DDJb+/z3N/XgAAAAA3/sAAN/7AADf+wAA3/sAAAAAAAAAAAAAAAAAAN/7AAAAAAAAAAAAAAAAAAAAAAAAj/EAAI/5AACP8QAA3/sAAA==" rel=icon type=image/x-icon><title>Profiling Python web applications with visual tools</title><meta name=description content="I have been profiling my software with KCachegrind for a long time now and I wasmissing this option when I am developing API&#39;s or other web services."><link rel=alternate type=application/rss+xml title="Mitja Felicijan's posts" href=https://mitjafelicijan.com/index.xml><link rel=alternate type=application/rss+xml title="Mitja Felicijan's notes" href=https://mitjafelicijan.com/notes.xml><style>body{padding:1rem;max-width:760px;background:#fff;font-family:sans-serif;line-height:1.35rem;font-size:16px;margin:0 auto}hr{margin-block-start:1.5rem}h1,h2,h3{line-height:initial}h1{font-size:xx-large}footer{margin-block-start:2rem}cap{text-transform:capitalize}table{max-width:100%;width:100%;border-collapse:separate;border-spacing:2px;border:1px solid #000;border-left:1px solid #999;border-top:1px solid #999}blockquote{font-style:italic}table thead{background:#eee}ul.list li{padding:.2em 0}ul{line-height:1.4em}td,th{border:1px solid #000;padding:4px;border-right:1px solid #999;border-bottom:1px solid #999;text-align:left}pre{text-wrap:nowrap;overflow-x:auto;padding:0 1em;border:1px solid #dcdcdc}code{padding:0 3px;font-size:14px;border:0}pre code{line-height:1.3em}pre,code,pre *,code *{font-family:monospace}figure{margin-inline-start:0;margin-inline-end:0}figcaption{text-align:center}figcaption p{margin:.3em 0 0}img,video,audio{max-width:100%}header{display:flex;flex-direction:row;gap:3rem}nav{display:flex;gap:.75rem}nav.main{flex-grow:1}.pstatus-orange{background:gold}.pstatus-green{background:#9acd32}.pstatus-red{background:#cd5c5c}@media only screen and (max-width:600px){body{padding:15px}header{flex-direction:column;gap:1rem}a{word-wrap:break-word}}</style><header><nav class=main itemscope itemtype=http://schema.org/SiteNavigationElement role=toolbar><a href=/>Home</a>
<a href=https://github.com/mitjafelicijan target=_blank>Code</a>
<a href=/vault.html>Vault</a>
<a href=/mitjafelicijan.pgp.pub.txt target=_blank>PGP</a>
<a href=/curriculum-vitae.html>CV</a>
<a href=/index.xml target=_blank>RSS</a></nav></header><main role=main><article itemtype=http://schema.org/Article><h1 itemtype=headline>Profiling Python web applications with visual tools</h1><p><cap>post</cap>, Apr 21, 2017 on <a href=https://mitjafelicijan.com>Mitja Felicijan's blog</a><div><p>I have been profiling my software with KCachegrind for a long time now and I was
missing this option when I am developing API's or other web services. I always
knew that this is possible but never really took the time and dive into it.<p>Before we begin there are some requirements. We will need to:<ul><li>implement <a href=https://docs.python.org/2/library/profile.html#module-cProfile>cProfile</a> into our web app,<li>convert output to <a href=http://valgrind.org/docs/manual/cl-manual.html>callgrind</a> format with <a href=https://pypi.python.org/pypi/pyprof2calltree/>pyprof2calltree</a>,<li>visualize data with <a href=http://kcachegrind.sourceforge.net/html/Home.html>KCachegrind</a> or <a href=http://www.profilingviewer.com/>Profiling Viewer</a>.</ul><p>If you are using MacOS you should check out <a href=http://www.profilingviewer.com/>Profiling
Viewer</a> or
<a href=http://www.maccallgrind.com/>MacCallGrind</a>.<figure><img src=/posts/python-profiling/kcachegrind.png alt=KCachegrind></figure><p>We will be dividing this post into two main categories:<ul><li>writing simple web-service,<li>visualize profile of this web-service.</ul><h2 id=simple-web-service>Simple web-service</h2><p>Let's use virtualenv so we won't pollute our base system. If you don't have
virtualenv installed on your system you can install it with pip command.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># let's install virtualenv globally</span>
</span></span><span style=display:flex><span>$ sudo pip install virtualenv
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># let's also install pyprof2calltree globally</span>
</span></span><span style=display:flex><span>$ sudo pip install pyprof2calltree
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now we create project</span>
</span></span><span style=display:flex><span>$ mkdir demo-project
</span></span><span style=display:flex><span>$ cd demo-project/
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now let's create folder where we will store profiles</span>
</span></span><span style=display:flex><span>$ mkdir prof
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now we create empty virtualenv in venv/ folder</span>
</span></span><span style=display:flex><span>$ virtualenv --no-site-packages venv
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># we now need to activate virtualenv</span>
</span></span><span style=display:flex><span>$ source venv/bin/activate
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># you can check if virtualenv was correctly initialized by</span>
</span></span><span style=display:flex><span><span style=color:green># checking where your python interpreter is located</span>
</span></span><span style=display:flex><span><span style=color:green># if command bellow points to your created directory and not some</span>
</span></span><span style=display:flex><span><span style=color:green># system dir like /usr/bin/python then everything is fine</span>
</span></span><span style=display:flex><span>$ which python
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># we can check now if all is good ➜ if ok couple of</span>
</span></span><span style=display:flex><span><span style=color:green># lines will be displayed</span>
</span></span><span style=display:flex><span>$ pip freeze
</span></span><span style=display:flex><span><span style=color:green># appdirs==1.4.3</span>
</span></span><span style=display:flex><span><span style=color:green># packaging==16.8</span>
</span></span><span style=display:flex><span><span style=color:green># pyparsing==2.2.0</span>
</span></span><span style=display:flex><span><span style=color:green># six==1.10.0</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now we are ready to install bottlepy ➜ web micro-framework</span>
</span></span><span style=display:flex><span>$ pip install bottle
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># you can deactivate virtualenv but you will then go</span>
</span></span><span style=display:flex><span><span style=color:green># under system domain ➜ for now don't deactivate</span>
</span></span><span style=display:flex><span>$ deactivate
</span></span></code></pre><p>We are now ready to write simple web service. Let's create file app.py and paste
code bellow in this newly created file.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># -*- coding: utf-8 -*-</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#00f>import</span> bottle
</span></span><span style=display:flex><span><span style=color:#00f>import</span> random
</span></span><span style=display:flex><span><span style=color:#00f>import</span> cProfile
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>app = bottle.Bottle()
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># this function is a decorator and encapsulates function</span>
</span></span><span style=display:flex><span><span style=color:green># and performs profiling and then saves it to subfolder</span>
</span></span><span style=display:flex><span><span style=color:green># prof/function-name.prof</span>
</span></span><span style=display:flex><span><span style=color:green># in our example only awesome_random_number function will</span>
</span></span><span style=display:flex><span><span style=color:green># be profiled because it has do_cprofile defined</span>
</span></span><span style=display:flex><span><span style=color:#00f>def</span> do_cprofile(func):
</span></span><span style=display:flex><span> <span style=color:#00f>def</span> profiled_func(*args, **kwargs):
</span></span><span style=display:flex><span> profile = cProfile.Profile()
</span></span><span style=display:flex><span> <span style=color:#00f>try</span>:
</span></span><span style=display:flex><span> profile.enable()
</span></span><span style=display:flex><span> result = func(*args, **kwargs)
</span></span><span style=display:flex><span> profile.disable()
</span></span><span style=display:flex><span> <span style=color:#00f>return</span> result
</span></span><span style=display:flex><span> <span style=color:#00f>finally</span>:
</span></span><span style=display:flex><span> profile.dump_stats(<span style=color:#a31515>"prof/"</span> + str(func.__name__) + <span style=color:#a31515>".prof"</span>)
</span></span><span style=display:flex><span> <span style=color:#00f>return</span> profiled_func
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># we use profiling over specific function with including</span>
</span></span><span style=display:flex><span><span style=color:green># @do_cprofile above function declaration</span>
</span></span><span style=display:flex><span>@app.route(<span style=color:#a31515>"/"</span>)
</span></span><span style=display:flex><span>@do_cprofile
</span></span><span style=display:flex><span><span style=color:#00f>def</span> awesome_random_number():
</span></span><span style=display:flex><span> awesome_random_number = random.randint(0, 100)
</span></span><span style=display:flex><span> <span style=color:#00f>return</span> <span style=color:#a31515>"awesome random number is "</span> + str(awesome_random_number)
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>@app.route(<span style=color:#a31515>"/test"</span>)
</span></span><span style=display:flex><span><span style=color:#00f>def</span> test():
</span></span><span style=display:flex><span> <span style=color:#00f>return</span> <span style=color:#a31515>"dummy test"</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#00f>if</span> __name__ == <span style=color:#a31515>'__main__'</span>:
</span></span><span style=display:flex><span> bottle.run(
</span></span><span style=display:flex><span> app = app,
</span></span><span style=display:flex><span> host = <span style=color:#a31515>"0.0.0.0"</span>,
</span></span><span style=display:flex><span> port = 4000
</span></span><span style=display:flex><span> )
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># run with 'python app.py'</span>
</span></span><span style=display:flex><span><span style=color:green># open browser 'http://0.0.0.0:4000'</span>
</span></span></code></pre><p>When browser hits awesome_random_number() function profile is created in prof/
subfolder.<h2 id=visualize-profile>Visualize profile</h2><p>Now let's create callgrind format from this cProfile output.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span>$ cd prof/
</span></span><span style=display:flex><span>$ pyprof2calltree -i awesome_random_number.prof
</span></span><span style=display:flex><span><span style=color:green># this creates 'awesome_random_number.prof.log' file in the same folder</span>
</span></span></code></pre><p>This file can be opened with visualizing tools listed above. In this case we
will be using Profilling Viewer under MacOS. You can open image in new tab. As
you can see from this example there is hierarchy of execution order of your
code.<figure><img src=/posts/python-profiling/profiling-viewer.png alt="Profilling Viewer"></figure><blockquote><p>Make sure you convert output of the cProfile output every time you want to
refresh and take a look at your possible optimizations because cProfile updates
.prof file every time browser hits the function.</blockquote><p>This is just a simple example but when you are developing real-life applications
this can be very illuminating, especially to see which parts of your code are
bottlenecks and need to be optimized.<h2 id=update-2017-04-22>Update 2017-04-22</h2><p>Reddit user <a href=https://www.reddit.com/user/mvt>mvt</a> also recommended this awesome
web based profile visualizer <a href=https://jiffyclub.github.io/snakeviz/>SnakeViz</a>
that directly takes output from
<a href=https://docs.python.org/2/library/profile.html#module-cProfile>cProfile</a>
module.<div class=reddit-embed data-embed-media=www.redditmedia.com data-embed-parent=false data-embed-live=false data-embed-uuid=583880c1-002e-41ed-a373-020a0ef2cff9 data-embed-created=2017-04-22T19:46:54.810Z><a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/>Comment</a> from discussion <a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/>Profiling Python web applications with visual tools</a>.</div><script async src=https://www.redditstatic.com/comment-embed.js></script><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># let's install it globally as well</span>
</span></span><span style=display:flex><span>$ sudo pip install snakeviz
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now let's visualize</span>
</span></span><span style=display:flex><span>$ cd prof/
</span></span><span style=display:flex><span>$ snakeviz awesome_random_number.prof
</span></span><span style=display:flex><span><span style=color:green># this automatically opens browser window and</span>
</span></span><span style=display:flex><span><span style=color:green># shows visualized profile</span>
</span></span></code></pre><figure><img src=/posts/python-profiling/snakeviz.png alt=SnakeViz></figure><p>Reddit user <a href=https://www.reddit.com/user/ccharles>ccharles</a> suggested a better
way for installing pip software by targeting user level instead of using sudo.<div class=reddit-embed data-embed-media=www.redditmedia.com data-embed-parent=false data-embed-live=false data-embed-uuid=f4f0459e-684d-441e-bebe-eb49b2f0a31d data-embed-created=2017-04-22T19:46:10.874Z><a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/>Comment</a> from discussion <a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/>Profiling Python web applications with visual tools</a>.</div><script async src=https://www.redditstatic.com/comment-embed.js></script><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># now we need to add this path to our $PATH variable</span>
</span></span><span style=display:flex><span><span style=color:green># we do this my adding this line at the end of your</span>
</span></span><span style=display:flex><span><span style=color:green># ~/.bashrc file</span>
</span></span><span style=display:flex><span>PATH=$PATH:$HOME/.local/bin/
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># in order to use this new configuration you can close</span>
</span></span><span style=display:flex><span><span style=color:green># and reopen terminal or reload .bashrc file</span>
</span></span><span style=display:flex><span>$ source ~/.bashrc
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now let's test if new directory is present in $PATH</span>
</span></span><span style=display:flex><span>$ echo $PATH
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:green># now we can install on user level by adding --user</span>
</span></span><span style=display:flex><span><span style=color:green># without use of sudo</span>
</span></span><span style=display:flex><span>$ pip install snakeviz --user
</span></span></code></pre><p>Or as suggested by <a href=https://www.reddit.com/user/mvt>mvt</a> you can
use <a href=https://github.com/mitsuhiko/pipsi>pipsi</a>.</div></article></main><section><hr><h2>Posts from blogs I follow around the net</h2><ul><li><a href=https://chotrin.org/writing/2023-10-29.html target=_blank rel=noopener>even more bike things.</a><div>I wrote about all sorts of biking and not-biking things in my life this week. — <a href=https://chotrin.org>chötrin's wiki.</a><li><a href="http://offbeatpursuit.com:80/blog/?id=25" target=_blank rel=noopener>A fix by any other name</a><div>tags:
i2c, plan9
Another month, another file system.
Well, if you can’t fix it in software, fix it in hardware (looking at
you, bme680, we’re not
done yet). The show mu… — <a href=http://offbeatpursuit.com:80/blog/>WLOG - blog</a><li><a href=https://mirzapandzo.com/next-image-url-parameter-is-valid-but-upstream-response-is-invalid target=_blank rel=noopener>Next/Image "url" parameter is valid but upstream response is invalid</a><div>Getting "url" parameter is valid but upstream response is invalid error with Next/Image on WSL2 — <a href=https://mirzapandzo.com/>Mirza Pandzo's Blog</a><li><a href=https://drewdevault.com/2023/10/13/Going-off-script.html target=_blank rel=noopener>Going off-script</a><div>There is a phenomenon in society which I find quite bizarre. Upon our entry to
this mortal coil, we are endowed with self-awareness, agency, and free will.
Each of th… — <a href=https://drewdevault.com>Drew DeVault's blog</a><li><a href=https://solar.lowtechmagazine.com/2023/10/thematic-book-series-how-to-downsize-a-transport-network/ target=_blank rel=noopener>Thematic Book Series: How to Downsize a Transport Network?</a><div>Image: Book cover. [How to downsize a transport network?](https://www.lulu.com/shop/kris-de-decker/how-to-downsize-a-transport-network/paperback/product-42n4p7.htm… — <a href=https://solar.lowtechmagazine.com/posts/>LOW←TECH MAGAZINE English</a><li><a href=https://neil.computer/notes/chart-of-accounts-for-startups-and-saas-companies/ target=_blank rel=noopener>Chart of Accounts for Startups and SaaS Companies</a><div>Accounting is fundamental to starting a business. You need to have a basic understanding of accounting principles and essential bookkeeping. I had to learn it. Ther… — <a href=https://neil.computer/>Neil Panchal</a><li><a href=https://journal.valeriansaliou.name/deploy-a-nomad-cluster-on-alpine-linux-with-vultr/ target=_blank rel=noopener>Deploy a Nomad Cluster on Alpine Linux with Vultr</a><div>After spending countless hours trying to understand how to deploy my apps on Kubernetes for the first time to host Mirage, an AI API service that I run, I ended up … — <a href=https://journal.valeriansaliou.name/>Valerian Saliou</a><li><a href=https://jcs.org/2023/10/25/wifi_da target=_blank rel=noopener>BlueSCSI Wi-Fi Desk Accessory 1.0 Released</a><div>BlueSCSI Wi-Fi Desk Accessory
1.0 has been released:
wifi_da-1.0.sit
(StuffIt 3 archive)
SHA256: ccfc9d27dd5da7412d10cef73b81119a1fec3848e4d1d88ff652a07ffdc6a69aSHA1:… — <a href=https://jcs.org/>joshua stein</a></ul><p><a href=https://git.sr.ht/~sircmpwn/openring>Generated with openring.</a></section><footer><hr><p><big><strong>Want to comment or have something to add?</strong></big><p>You can write me an email
at <a href=mailto:mitja.felicijan@gmail.com>mitja.felicijan@gmail.com</a> or
catch up with me <a href=https://telegram.me/mitjafelicijan target=_blank>on Telegram</a>.<hr><p>This website does not track you. Content is made available under
the <a href=https://creativecommons.org/licenses/by/4.0/ target=_blank rel=noreferrer>CC BY 4.0 license</a> unless specified
otherwise. Blog is also available as <a href=/index.xml target=_blank>RSS feed</a>.</footer><script>
window.va = window.va || function () { (window.vaq = window.vaq || []).push(arguments); };
</script><script defer src=/_vercel/insights/script.js></script>
|
