Ephes Blog

Miscellaneous things. Mostly Weeknotes and links I stumbled upon.


PyCon DE Day One

, Jochen
Didn't expect to attend PyCon DE 2022, but somehow it worked out nevertheless. I arrived in Berlin at sunday evening. It's really good to be able to meet people in person again.



On monday morning there was I long line before the conference venue and I only arrived at 09:30, but I had no trouble to get in on time. Florian Bruhin greeted me from the waiting line, but he confused me with someone else. I should have used the opportunity and ask him about recording an podcast episode about pytest, but I was too perplexed.



Some time ago I tried to join the Python Software Verband e.V, but didn't succeed. Filling out the form locally should work now :).

The first session I attended was Building an ORM from scratch by Patrick Schemitz and Jonathan Oberländer which was great. I had trouble following through because it was so fast paced, but I like to be challanged. Here's the code used in the presentation.



After lunch I listened to the keynote of the first day: Beyond the basics: Contributor experience, diversity and culture in open source projects, which was held online by Melissa Weber Mendonça from brasil. This is a really difficult topic, but there's also a lot of room for improvement.

The talk Python 3.10: Welcome to pattern matching by Laysa Uchoa was informative as well as funny.
 

Seeing the needle AND the haystack: single-datapoint selection for billion-point datasets was also interesting. Never thought about that it might be hard to visualize a dataset that has many more datapoints than the screen has pixels, but it's kind of obvious to me now :)

The last slot I attended on the first day was the lightning talks. There was a lot of interesting stuff, just to hightlight some: But the most valuable experience was of course running into people I haven't seen in a long time and learn about the interesting stuff they have been up to in the meantime.

Weeknotes 2022-04-04

, Jochen

Applied for Prototype Fund with django-cast, I would love to be able to improve documentation and usabiltiy.  Finished a command line client for syncing kptncook recipes with my self-hosted mealie instance. Revisited my old "will_it_saturate" project to compare caddy vs uvicorn speed. Good fun.

Things I Learned

  • You can have validators run befor assigning values in pydantic models, creating a previously non-existing directory, for example.

Articles

Talks

Youtube

Twitter

Podcasts


Benchmarking nginx vs caddy vs uvicorn for serving static files

, Jochen
Disclaimer: I have no clue what I'm doing here. If you do: pls halp.


Setup

Last year I did a talk about serving files with Django. For demonstration purposes I wrote a little benchmark tool called "will it saturate". Recently I played around with mealie a little bit and noticed that it uses an additional caddy to serve images for recipes. On discord I asked why and was told it's used because caddy is faster at serving images than uvicorn / starlette. So I wondered how much faster it might be and tried to get my old "will_it_saturate" project to test it.

My base assumption is that there should not be a big difference between different web servers serving static files, because there's not much those web servers do when serving files. They just orchestrate operating system syscalls that do the real work and whether you call those via c (nginx), go (caddy) or python (uvicorn) shouldn't matter that much. Well, turns out that this assumption might be wrong. Which is interesting. The webservers used are: nginx, caddy, uvicorn. I included nginx, because it represents the state of the art in serving static files when configured properly, but I'm not sure whether I managed to do that. Caddy and uvicorn are the two servers I wanted to benchmark against each other.

For the benchmark I created 12.5K files, each containing 100KB of random data (similar to a recipe image in mealie) so that downloading them would saturate a gigabit link for about ten seconds. I know it's possible to saturate a gigabit connection with concurrent file downloads serving those files from uvicorn so I didn't repeat that. The question I'm interested this time is: How much faster than uvicorn is caddy? The main metric is transferred bytes per second. The bigger, the better.


Results

Here's a first result running this notebook:
 
server client elapsed file_size_h bytes_per_second_h arch
nginx/minimal wrk 0.695651 97.66KB 1.67GB x86_64
nginx/sendfile wrk 0.718180 97.66KB 1.62GB x86_64
caddy wrk 0.880563 97.66KB 1.32GB x86_64
fastAPI/uvicorn wrk 6.153709 97.66KB 193.72MB x86_64

Ok, well. Seems like caddy is a lot faster than uvicorn, wow. The server is an old intel xeon running linux. The client is wrk opening up 20 connections concurrently (more than a browser usually does). Here are some points that surprised me:
 
  • Turning on sendfile didn't make nginx faster. I think this is because there's some hard limit on ssd bandwidth or something like that
  • Had to use multiple workers with uvicorn. Using a single worker yields only 50MB/s.
  • Caddy is not much slower than nginx despite nginx is using 4 workers and caddy only one. Probably another hint that there is some hardware bottleneck.
Let's see how the numbers change when I run the script on my macbook air running macOS:
 
server client elapsed file_size_h bytes_per_second_h arch
nginx/sendfile wrk 0.252434 97.66KB 4.61GB arm64
nginx/minimal wrk 0.287506 97.66KB 4.05GB arm64
caddy wrk 0.481639 97.66KB 2.42GB arm64
fastAPI/uvicorn wrk 4.228977 97.66KB 281.89MB arm64

Ok, also interesting. My macbook air (M1) does not even get warm running the benchmark. Surprising details:
 
  • Using nginx with multiple workers is much faster now (no hardware bottleneck?)
  • Using sendfile is faster - I didn't know macOS even had a sendfile syscall, weird


Conclusion

There's indeed a big difference between nginx and caddy on one side and uvicorn on the other. And I don't know why, so further research is needed :). The reason I tried to benchmark nginx with and without sendfile was to make sure I'm not measuring some form of kernel level io / zero copy tcp vs having to do all the work in userspace, because uvicorn lacks zerocopy send support atm. The results seem to suggest that it's possible to be very fast without sendfile. Which is good, because nowadays we usually serve files from some kind of object store and not from the file system and therefore it's not possible to benefit from sendfile anyway (please correct me if I'm wrong).

Still, those numbers won't make a big difference in practice, because most machines don't have network links faster than one gigabit. It will become much more relevant when this changes to 10Gb, because than uvicorn will be too slow to saturate it.

The next thing I'll be testing is to preload the files in memory to rule out aiofiles being the culprit for being slow. What I also find really interesting is that I had to use wrk as http client for the benchmarks because the python http clients I tried (httpx, aiohttp) were far too slow. With aiohttp being a little bit faster than httpx. They max out at about 80MB/s. Why is that?


Weeknotes 2022-03-28

, Jochen

Got my kptncook to mealie integration to work. My mealie instance is now running at mealie staging. If you'd like to have an account, drop my a line. Not much progress on other projects.

Things I Learned

  • You can record traffic with your fritzbox using this link - very nice for debugging

Articles

Youtube

Twitter

Software 

Podcasts


Weeknotes 2022-03-21

, Jochen
Catched a cold, not corona, but still bad. Spent most of last week refactoring fastdeploy. There are still two difficult to fix issues left but it's usable anyway. Wrote some release notes this time :).

The second thing I spent time on was mealie. It's a receipt manager implemented with fastAPI and vue which is a stack I use, too. So I played around with it a little bit and deployed it to my own infrastructure. This was easier than expected, but I found an a bug in fastdeploy (big task output breaks on `await proc.stdout.readline()`). Wonder whether people pay money for a hosted version of mealie?
 

Things I Learned

  • Usually I prefer to run my tests directly with pytest instead of vscode because my console output is nicely colorized and it's much easier to add options like '--full-trace' etc. but I just found out it's at least possible to view the test stdout in vscode using ⌘⇧U and then selecting "Python Test Log" from the dropdown on the upper right. So I might run tests from vscode from time to time now :).
  • You can install optional poetry dependencies with `poetry install -E optional_name`

Articles

Youtube

Twitter

Software 

Podcasts