aboutsummaryrefslogtreecommitdiff
path: root/public/profiling-python-web-applications-with-visual-tools.html
diff options
context:
space:
mode:
Diffstat (limited to 'public/profiling-python-web-applications-with-visual-tools.html')
-rwxr-xr-xpublic/profiling-python-web-applications-with-visual-tools.html143
1 files changed, 143 insertions, 0 deletions
diff --git a/public/profiling-python-web-applications-with-visual-tools.html b/public/profiling-python-web-applications-with-visual-tools.html
new file mode 100755
index 0000000..49cfffc
--- /dev/null
+++ b/public/profiling-python-web-applications-with-visual-tools.html
@@ -0,0 +1,143 @@
1<!doctype html><html lang=en-us><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><link href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL69vf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAv76+/8LBwQkAAAAAAAAAAAAAAAC+vb3/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL+9vf/Bv78JAAAAAAAAAAAAAAAAu7q6/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC7ubr/vr29CAAAAAAAAAAAy8nJAZ6foP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnqGj/6GipAoAAAAAHLjU/xcXHf/BwsL/I8XY/yPK3v8XGiD/IbjL/yPF2f8XGiD/Fxkf/yLF2f8gnK3/Fxog/62ztv8fwNf/FRcd/x271v8mz93/GRsi/xkXHf8p097/GiIp/xobIv8p0t3/KdPe/xocIv8fYmr/KNPe/xoZH/8aHCL/J87c/xy81/8VFxz/IsPZ/8zS0/8XGiD/Ir/R/yPH2/8XGiD/Fxkf/yPH2/8dd4T/GBog/yPJ3f8jyNr/uru9/xcUGv8cudb/EhITDKi5vRKlvMP/RUpOERwcHRAdOj4QHTk8EBwdHRAdNTgQHTo/EBwcHRAcHB0QSGduEKW4vf+koqQfHzg+EBqz0ewSFRv7EyMr/xq51vsTERb7ExUb+xq41fsau9j7ExUb+xiPp/sZudb7ExUb+xMVG/sZuNX/GKvI/BIUGfMdvdn/IrfL/xcaIP8n1eb/J9Dh/xkcIf8ZGR7/J8/f/xxCSv8ZGyH/J9Dg/ybQ4P8ZHCL/FSQs/yPK3/8UExj/GE1b/ybS5P8ZGB7/Ghwj/ynW5P8p2Ob/Ghwi/yWrtv8p1eH/Ghwi/xocIv8p1uT/J8XT/xkcIv8m1un/Hb7d/xUYH/8hzOr/HtHu/xcaIf8XGB//I8vi/xgxOv8XGSD/I8rg/yPK4P8XGiD/GUFL/yPP6f8SERj/Fhkh/x3A4f8AAAAAJ2f9/ydr//8mZPH/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlYu38J2v//ydo/f8AAAAAAAAAAAd8/fkFqf//Iob8sAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMY39awWr//8FfP3/AAAAAAAAAAAFm/7/SfD//wR+/f8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOB/f9B7v//BaX+/wAAAAAAAAAAQ878SAyZ/v9n1v4KAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADu9v8DDJb+/z3N/XgAAAAA3/sAAN/7AADf+wAA3/sAAAAAAAAAAAAAAAAAAN/7AAAAAAAAAAAAAAAAAAAAAAAAj/EAAI/5AACP8QAA3/sAAA==" rel=icon type=image/x-icon><title>Profiling Python web applications with visual tools</title><meta name=description content="I have been profiling my software with KCachegrind for a long time now and I wasmissing this option when I am developing API&amp;#39;s or other web services."><link rel=alternate type=application/rss+xml title="Mitja Felicijan's posts" href=https://mitjafelicijan.com/index.xml><link rel=alternate type=application/rss+xml title="Mitja Felicijan's notes" href=https://mitjafelicijan.com/notes.xml><style>body{padding:1rem;max-width:760px;background:#fff;font-family:times new roman,Times,serif;line-height:1.35rem}hr{margin-block-start:1.5rem}h1,h2,h3{line-height:initial}footer{margin-block-start:3rem}table{max-width:100%;border-collapse:separate;border-spacing:2px;border:1px solid #000;border-left:1px solid #999;border-top:1px solid #999}blockquote{font-style:italic}table thead{background:#eee}td,th{border:1px solid #000;padding:4px;border-right:1px solid #999;border-bottom:1px solid #999;text-align:left}pre{text-wrap:nowrap;overflow-x:auto;margin-block-start:1.5rem;margin-block-end:1.5rem;padding:.5rem 0;border-top:1px solid #000;border-bottom:1px solid #000}pre code{line-height:1.3em}pre,code,pre *,code *{font-family:monospace;font-size:initial!important}img,video,audio{max-width:100%}header{display:flex;flex-direction:row;gap:3rem}nav{display:flex;gap:.75rem}.pstatus-orange{background:gold}.pstatus-green{background:#9acd32}.pstatus-red{background:#cd5c5c}@media only screen and (max-width:600px){header{flex-direction:column;gap:1rem}a{word-wrap:break-word}}</style><header><nav class=main><a href=/>Home</a>
2<a href=https://git.mitjafelicijan.com/ target=_blank>Git</a>
3<a href=https://files.mitjafelicijan.com/ target=_blank>Files</a>
4<a href=/mitjafelicijan.pgp.pub.txt target=_blank>PGP</a>
5<a href=/curriculum-vitae.html>CV</a>
6<a href=/index.xml target=_blank>RSS</a></nav></header><main><div><h1>Profiling Python web applications with visual tools</h1><p>Apr 21, 2017<div><p>I have been profiling my software with KCachegrind for a long time now and I was
7missing this option when I am developing API's or other web services. I always
8knew that this is possible but never really took the time and dive into it.<p>Before we begin there are some requirements. We will need to:<ul><li>implement <a href=https://docs.python.org/2/library/profile.html#module-cProfile>cProfile</a> into our web app,<li>convert output to <a href=http://valgrind.org/docs/manual/cl-manual.html>callgrind</a> format with <a href=https://pypi.python.org/pypi/pyprof2calltree/>pyprof2calltree</a>,<li>visualize data with <a href=http://kcachegrind.sourceforge.net/html/Home.html>KCachegrind</a> or <a href=http://www.profilingviewer.com/>Profiling Viewer</a>.</ul><p>If you are using MacOS you should check out <a href=http://www.profilingviewer.com/>Profiling
9Viewer</a> or
10<a href=http://www.maccallgrind.com/>MacCallGrind</a>.<p><img src=/assets/python-profiling/kcachegrind.png alt=KCachegrind><p>We will be dividing this post into two main categories:<ul><li>writing simple web-service,<li>visualize profile of this web-service.</ul><h2 id=simple-web-service>Simple web-service</h2><p>Let's use virtualenv so we won't pollute our base system. If you don't have
11virtualenv installed on your system you can install it with pip command.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># let&#39;s install virtualenv globally</span>
12</span></span><span style=display:flex><span>$ sudo pip install virtualenv
13</span></span><span style=display:flex><span>
14</span></span><span style=display:flex><span><span style=color:green># let&#39;s also install pyprof2calltree globally</span>
15</span></span><span style=display:flex><span>$ sudo pip install pyprof2calltree
16</span></span><span style=display:flex><span>
17</span></span><span style=display:flex><span><span style=color:green># now we create project</span>
18</span></span><span style=display:flex><span>$ mkdir demo-project
19</span></span><span style=display:flex><span>$ cd demo-project/
20</span></span><span style=display:flex><span>
21</span></span><span style=display:flex><span><span style=color:green># now let&#39;s create folder where we will store profiles</span>
22</span></span><span style=display:flex><span>$ mkdir prof
23</span></span><span style=display:flex><span>
24</span></span><span style=display:flex><span><span style=color:green># now we create empty virtualenv in venv/ folder</span>
25</span></span><span style=display:flex><span>$ virtualenv --no-site-packages venv
26</span></span><span style=display:flex><span>
27</span></span><span style=display:flex><span><span style=color:green># we now need to activate virtualenv</span>
28</span></span><span style=display:flex><span>$ source venv/bin/activate
29</span></span><span style=display:flex><span>
30</span></span><span style=display:flex><span><span style=color:green># you can check if virtualenv was correctly initialized by</span>
31</span></span><span style=display:flex><span><span style=color:green># checking where your python interpreter is located</span>
32</span></span><span style=display:flex><span><span style=color:green># if command bellow points to your created directory and not some</span>
33</span></span><span style=display:flex><span><span style=color:green># system dir like /usr/bin/python then everything is fine</span>
34</span></span><span style=display:flex><span>$ which python
35</span></span><span style=display:flex><span>
36</span></span><span style=display:flex><span><span style=color:green># we can check now if all is good ➜ if ok couple of</span>
37</span></span><span style=display:flex><span><span style=color:green># lines will be displayed</span>
38</span></span><span style=display:flex><span>$ pip freeze
39</span></span><span style=display:flex><span><span style=color:green># appdirs==1.4.3</span>
40</span></span><span style=display:flex><span><span style=color:green># packaging==16.8</span>
41</span></span><span style=display:flex><span><span style=color:green># pyparsing==2.2.0</span>
42</span></span><span style=display:flex><span><span style=color:green># six==1.10.0</span>
43</span></span><span style=display:flex><span>
44</span></span><span style=display:flex><span><span style=color:green># now we are ready to install bottlepy ➜ web micro-framework</span>
45</span></span><span style=display:flex><span>$ pip install bottle
46</span></span><span style=display:flex><span>
47</span></span><span style=display:flex><span><span style=color:green># you can deactivate virtualenv but you will then go</span>
48</span></span><span style=display:flex><span><span style=color:green># under system domain ➜ for now don&#39;t deactivate</span>
49</span></span><span style=display:flex><span>$ deactivate
50</span></span></code></pre><p>We are now ready to write simple web service. Let's create file app.py and paste
51code bellow in this newly created file.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># -*- coding: utf-8 -*-</span>
52</span></span><span style=display:flex><span>
53</span></span><span style=display:flex><span><span style=color:#00f>import</span> bottle
54</span></span><span style=display:flex><span><span style=color:#00f>import</span> random
55</span></span><span style=display:flex><span><span style=color:#00f>import</span> cProfile
56</span></span><span style=display:flex><span>
57</span></span><span style=display:flex><span>app = bottle.Bottle()
58</span></span><span style=display:flex><span>
59</span></span><span style=display:flex><span><span style=color:green># this function is a decorator and encapsulates function</span>
60</span></span><span style=display:flex><span><span style=color:green># and performs profiling and then saves it to subfolder</span>
61</span></span><span style=display:flex><span><span style=color:green># prof/function-name.prof</span>
62</span></span><span style=display:flex><span><span style=color:green># in our example only awesome_random_number function will</span>
63</span></span><span style=display:flex><span><span style=color:green># be profiled because it has do_cprofile defined</span>
64</span></span><span style=display:flex><span><span style=color:#00f>def</span> do_cprofile(func):
65</span></span><span style=display:flex><span> <span style=color:#00f>def</span> profiled_func(*args, **kwargs):
66</span></span><span style=display:flex><span> profile = cProfile.Profile()
67</span></span><span style=display:flex><span> <span style=color:#00f>try</span>:
68</span></span><span style=display:flex><span> profile.enable()
69</span></span><span style=display:flex><span> result = func(*args, **kwargs)
70</span></span><span style=display:flex><span> profile.disable()
71</span></span><span style=display:flex><span> <span style=color:#00f>return</span> result
72</span></span><span style=display:flex><span> <span style=color:#00f>finally</span>:
73</span></span><span style=display:flex><span> profile.dump_stats(<span style=color:#a31515>&#34;prof/&#34;</span> + str(func.__name__) + <span style=color:#a31515>&#34;.prof&#34;</span>)
74</span></span><span style=display:flex><span> <span style=color:#00f>return</span> profiled_func
75</span></span><span style=display:flex><span>
76</span></span><span style=display:flex><span>
77</span></span><span style=display:flex><span><span style=color:green># we use profiling over specific function with including</span>
78</span></span><span style=display:flex><span><span style=color:green># @do_cprofile above function declaration</span>
79</span></span><span style=display:flex><span>@app.route(<span style=color:#a31515>&#34;/&#34;</span>)
80</span></span><span style=display:flex><span>@do_cprofile
81</span></span><span style=display:flex><span><span style=color:#00f>def</span> awesome_random_number():
82</span></span><span style=display:flex><span> awesome_random_number = random.randint(0, 100)
83</span></span><span style=display:flex><span> <span style=color:#00f>return</span> <span style=color:#a31515>&#34;awesome random number is &#34;</span> + str(awesome_random_number)
84</span></span><span style=display:flex><span>
85</span></span><span style=display:flex><span>@app.route(<span style=color:#a31515>&#34;/test&#34;</span>)
86</span></span><span style=display:flex><span><span style=color:#00f>def</span> test():
87</span></span><span style=display:flex><span> <span style=color:#00f>return</span> <span style=color:#a31515>&#34;dummy test&#34;</span>
88</span></span><span style=display:flex><span>
89</span></span><span style=display:flex><span><span style=color:#00f>if</span> __name__ == <span style=color:#a31515>&#39;__main__&#39;</span>:
90</span></span><span style=display:flex><span> bottle.run(
91</span></span><span style=display:flex><span> app = app,
92</span></span><span style=display:flex><span> host = <span style=color:#a31515>&#34;0.0.0.0&#34;</span>,
93</span></span><span style=display:flex><span> port = 4000
94</span></span><span style=display:flex><span> )
95</span></span><span style=display:flex><span>
96</span></span><span style=display:flex><span><span style=color:green># run with &#39;python app.py&#39;</span>
97</span></span><span style=display:flex><span><span style=color:green># open browser &#39;http://0.0.0.0:4000&#39;</span>
98</span></span></code></pre><p>When browser hits awesome_random_number() function profile is created in prof/
99subfolder.<h2 id=visualize-profile>Visualize profile</h2><p>Now let's create callgrind format from this cProfile output.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span>$ cd prof/
100</span></span><span style=display:flex><span>$ pyprof2calltree -i awesome_random_number.prof
101</span></span><span style=display:flex><span><span style=color:green># this creates &#39;awesome_random_number.prof.log&#39; file in the same folder</span>
102</span></span></code></pre><p>This file can be opened with visualizing tools listed above. In this case we
103will be using Profilling Viewer under MacOS. You can open image in new tab. As
104you can see from this example there is hierarchy of execution order of your
105code.<p><img src=/assets/python-profiling/profiling-viewer.png alt="Profilling Viewer"><blockquote><p>Make sure you convert output of the cProfile output every time you want to
106refresh and take a look at your possible optimizations because cProfile updates
107.prof file every time browser hits the function.</blockquote><p>This is just a simple example but when you are developing real-life applications
108this can be very illuminating, especially to see which parts of your code are
109bottlenecks and need to be optimized.<h2 id=update-2017-04-22>Update 2017-04-22</h2><p>Reddit user <a href=https://www.reddit.com/user/mvt>mvt</a> also recommended this awesome
110web based profile visualizer <a href=https://jiffyclub.github.io/snakeviz/>SnakeViz</a>
111that directly takes output from
112<a href=https://docs.python.org/2/library/profile.html#module-cProfile>cProfile</a>
113module.<div class=reddit-embed data-embed-media=www.redditmedia.com data-embed-parent=false data-embed-live=false data-embed-uuid=583880c1-002e-41ed-a373-020a0ef2cff9 data-embed-created=2017-04-22T19:46:54.810Z><a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/>Comment</a> from discussion <a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/>Profiling Python web applications with visual tools</a>.</div><script async src=https://www.redditstatic.com/comment-embed.js></script><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># let&#39;s install it globally as well</span>
114</span></span><span style=display:flex><span>$ sudo pip install snakeviz
115</span></span><span style=display:flex><span>
116</span></span><span style=display:flex><span><span style=color:green># now let&#39;s visualize</span>
117</span></span><span style=display:flex><span>$ cd prof/
118</span></span><span style=display:flex><span>$ snakeviz awesome_random_number.prof
119</span></span><span style=display:flex><span><span style=color:green># this automatically opens browser window and</span>
120</span></span><span style=display:flex><span><span style=color:green># shows visualized profile</span>
121</span></span></code></pre><p><img src=/assets/python-profiling/snakeviz.png alt=SnakeViz><p>Reddit user <a href=https://www.reddit.com/user/ccharles>ccharles</a> suggested a better
122way for installing pip software by targeting user level instead of using sudo.<div class=reddit-embed data-embed-media=www.redditmedia.com data-embed-parent=false data-embed-live=false data-embed-uuid=f4f0459e-684d-441e-bebe-eb49b2f0a31d data-embed-created=2017-04-22T19:46:10.874Z><a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/>Comment</a> from discussion <a href=https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/>Profiling Python web applications with visual tools</a>.</div><script async src=https://www.redditstatic.com/comment-embed.js></script><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># now we need to add this path to our $PATH variable</span>
123</span></span><span style=display:flex><span><span style=color:green># we do this my adding this line at the end of your</span>
124</span></span><span style=display:flex><span><span style=color:green># ~/.bashrc file</span>
125</span></span><span style=display:flex><span>PATH=$PATH:$HOME/.local/bin/
126</span></span><span style=display:flex><span>
127</span></span><span style=display:flex><span><span style=color:green># in order to use this new configuration you can close</span>
128</span></span><span style=display:flex><span><span style=color:green># and reopen terminal or reload .bashrc file</span>
129</span></span><span style=display:flex><span>$ source ~/.bashrc
130</span></span><span style=display:flex><span>
131</span></span><span style=display:flex><span><span style=color:green># now let&#39;s test if new directory is present in $PATH</span>
132</span></span><span style=display:flex><span>$ echo $PATH
133</span></span><span style=display:flex><span>
134</span></span><span style=display:flex><span><span style=color:green># now we can install on user level by adding --user</span>
135</span></span><span style=display:flex><span><span style=color:green># without use of sudo</span>
136</span></span><span style=display:flex><span>$ pip install snakeviz --user
137</span></span></code></pre><p>Or as suggested by <a href=https://www.reddit.com/user/mvt>mvt</a> you can
138use <a href=https://github.com/mitsuhiko/pipsi>pipsi</a>.</div></div></main><footer><hr><div><h3>Want to comment or have something to add?</h3>You can write me an email at
139<a href=mailto:m@mitjafelicijan.com>m@mitjafelicijan.com</a> or catch up
140with me
141<a href=https://telegram.me/mitjafelicijan target=_blank>on Telegram</a>.</div><hr><p>This website does not track you. Content is made available under
142the <a href=https://creativecommons.org/licenses/by/4.0/ target=_blank rel=noreferrer>CC BY 4.0 license</a> unless specified
143otherwise. Blog feed is available as <a href=/index.xml target=_blank>RSS feed</a>.</footer><script src=https://cdn.usefathom.com/script.js data-site=XHQARKXP defer></script> \ No newline at end of file
generated by cgit v1.2.3 (git 2.46.0) at 2026-04-30 23:10:45 +0000