From 23a56bd50b04211da3cab45f72c3390711b2416b Mon Sep 17 00:00:00 2001 From: Mitja Felicijan Date: Wed, 12 Jul 2023 18:35:08 +0200 Subject: Moved notes and posts into subfolders --- ...ng-python-web-applications-with-visual-tools.md | 206 --------------------- 1 file changed, 206 deletions(-) delete mode 100644 content/2017-04-21-profiling-python-web-applications-with-visual-tools.md (limited to 'content/2017-04-21-profiling-python-web-applications-with-visual-tools.md') diff --git a/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md deleted file mode 100644 index d1cea7c..0000000 --- a/content/2017-04-21-profiling-python-web-applications-with-visual-tools.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -title: Profiling Python web applications with visual tools -url: profiling-python-web-applications-with-visual-tools.html -date: 2017-04-21T12:00:00+02:00 -type: post -draft: false ---- - -I have been profiling my software with KCachegrind for a long time now and I was -missing this option when I am developing API's or other web services. I always -knew that this is possible but never really took the time and dive into it. - -Before we begin there are some requirements. We will need to: - -- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app, -- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/), -- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/). - - -If you are using MacOS you should check out [Profiling -Viewer](http://www.profilingviewer.com/) or -[MacCallGrind](http://www.maccallgrind.com/). - -![KCachegrind](/assets/python-profiling/kcachegrind.png) - -We will be dividing this post into two main categories: - -- writing simple web-service, -- visualize profile of this web-service. - -## Simple web-service - -Let's use virtualenv so we won't pollute our base system. If you don't have -virtualenv installed on your system you can install it with pip command. - -```bash -# let's install virtualenv globally -$ sudo pip install virtualenv - -# let's also install pyprof2calltree globally -$ sudo pip install pyprof2calltree - -# now we create project -$ mkdir demo-project -$ cd demo-project/ - -# now let's create folder where we will store profiles -$ mkdir prof - -# now we create empty virtualenv in venv/ folder -$ virtualenv --no-site-packages venv - -# we now need to activate virtualenv -$ source venv/bin/activate - -# you can check if virtualenv was correctly initialized by -# checking where your python interpreter is located -# if command bellow points to your created directory and not some -# system dir like /usr/bin/python then everything is fine -$ which python - -# we can check now if all is good ➜ if ok couple of -# lines will be displayed -$ pip freeze -# appdirs==1.4.3 -# packaging==16.8 -# pyparsing==2.2.0 -# six==1.10.0 - -# now we are ready to install bottlepy ➜ web micro-framework -$ pip install bottle - -# you can deactivate virtualenv but you will then go -# under system domain ➜ for now don't deactivate -$ deactivate -``` - -We are now ready to write simple web service. Let's create file app.py and paste -code bellow in this newly created file. - -```python -# -*- coding: utf-8 -*- - -import bottle -import random -import cProfile - -app = bottle.Bottle() - -# this function is a decorator and encapsulates function -# and performs profiling and then saves it to subfolder -# prof/function-name.prof -# in our example only awesome_random_number function will -# be profiled because it has do_cprofile defined -def do_cprofile(func): - def profiled_func(*args, **kwargs): - profile = cProfile.Profile() - try: - profile.enable() - result = func(*args, **kwargs) - profile.disable() - return result - finally: - profile.dump_stats("prof/" + str(func.__name__) + ".prof") - return profiled_func - - -# we use profiling over specific function with including -# @do_cprofile above function declaration -@app.route("/") -@do_cprofile -def awesome_random_number(): - awesome_random_number = random.randint(0, 100) - return "awesome random number is " + str(awesome_random_number) - -@app.route("/test") -def test(): - return "dummy test" - -if __name__ == '__main__': - bottle.run( - app = app, - host = "0.0.0.0", - port = 4000 - ) - -# run with 'python app.py' -# open browser 'http://0.0.0.0:4000' -``` - -When browser hits awesome\_random\_number() function profile is created in prof/ -subfolder. - -## Visualize profile - -Now let's create callgrind format from this cProfile output. - -```bash -$ cd prof/ -$ pyprof2calltree -i awesome_random_number.prof -# this creates 'awesome_random_number.prof.log' file in the same folder -``` - -This file can be opened with visualizing tools listed above. In this case we -will be using Profilling Viewer under MacOS. You can open image in new tab. As -you can see from this example there is hierarchy of execution order of your -code. - -![Profilling Viewer](/assets/python-profiling/profiling-viewer.png) - -> Make sure you convert output of the cProfile output every time you want to -refresh and take a look at your possible optimizations because cProfile updates -.prof file every time browser hits the function. - -This is just a simple example but when you are developing real-life applications -this can be very illuminating, especially to see which parts of your code are -bottlenecks and need to be optimized. - -## Update 2017-04-22 - -Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome -web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) -that directly takes output from -[cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) -module. - -
Comment from discussion Profiling Python web applications with visual tools.
- -```bash -# let's install it globally as well -$ sudo pip install snakeviz - -# now let's visualize -$ cd prof/ -$ snakeviz awesome_random_number.prof -# this automatically opens browser window and -# shows visualized profile -``` - -![SnakeViz](/assets/python-profiling/snakeviz.png) - -Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better -way for installing pip software by targeting user level instead of using sudo. - -
Comment from discussion Profiling Python web applications with visual tools.
- -```bash -# now we need to add this path to our $PATH variable -# we do this my adding this line at the end of your -# ~/.bashrc file -PATH=$PATH:$HOME/.local/bin/ - -# in order to use this new configuration you can close -# and reopen terminal or reload .bashrc file -$ source ~/.bashrc - -# now let's test if new directory is present in $PATH -$ echo $PATH - -# now we can install on user level by adding --user -# without use of sudo -$ pip install snakeviz --user -``` - -Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can -use [pipsi](https://github.com/mitsuhiko/pipsi). -- cgit v1.2.3