Ephes Blog

Found a workaround for the fastdeploy / SQLAlchemy / asyncpg / Linux deadlock bug. I now instantiate a new SQLAlchemy engine on every request. That's probably not very efficient, since it would be better to use the engine's built-in ability to hold a connection pool, but having working code is a good start.

Progress on having a landing page where people can create podcasts/blogs:

• Finished Add remove domain / deployed fqdn #14
• Finished Switch between cast / wordpress #2 for deployments 🍾 - it was a complete surprise for me that this is even possible. But since I have now a working deployment system where all responsibilities are nicely separated, it's possible no not only to deploy Django, but anything including weird stuff like Wordpress without having to worry the rest of my system might suffer.
• Finished Fix pytest warnings + some cleanup stuff and coverage

Articles

• An SPA Alternative
• How Did REST Come To Mean The Opposite of REST?
• How To Test Django Models (With Examples)
• Building the blog while flying it Series' Articles

Videos

• Two Sigma Presents Pandas at a Crossroads the Past Present and Future with Jeff Reback

Twitter

• In Python, partials are an easy way to reuse function code. 🐍 🤩
• Don't move elements in a list while looping over it! You will get surprising (wrong!) results: 😳 🤨
• Since #Python 3.7, modules can define __getattr__ to provide dynamic attributes on modules. Why use two-argument arithmetic when you can do it with one!? 😎 🥳 😍

Software

• django-readers | A lightweight function-oriented toolkit for better organisation of business logic and efficient selection and projection of data in Django projects.
• django-sesame has a new tutorial | I'm searching for a solution to be able to log in users for a newly deployed site without having to store a password. This could be one.

Podcasts

• UKW100 Brexit: Kopf unter der Decke (Unsere kleine Welt) | Yup, brexit is still a trainwreck.

Out of Context Images

Fought with strange bugs last week, also some work and much outdoor stuff.

For example when using Jupyter Notebooks in a Django environment, the PYTHONPATH was always set to some old project and therefore imports from my current project didn't work. And it was simply not possible to get the PYTHONPATH right. I could not find the old projects name in any config or virtualenv. Weird. But then I had a closer look at the kernels using jupyter kernelspec list:
 

$ jupyter kernelspec list Available kernels: django_extensions /Users/jochen/Library/Jupyter/kernels/django_extensions python3 /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/python3

Ok ~/Library/Jupyter/kernels is weird. And there the name of the different project appeared in the kernel.json. Whoa. Somehow this old django_extensions kernel has overwritten my django_extensions kernel from the virtualenv. After deleting the old one, my kernelspec looked like this:

$ jupyter kernelspec list Available kernels: django_extensions /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/django_extensions python3 /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/python3

This was pretty hard to debug.

And then there was this strange fastdeploy bug forcing me to have a closer look at SQLAlchemy. I still don't know what the root cause of the problem is. I switched to using SQLAlchemy in combination with asyncpg a frew months ago and there were this "The garbage collector is trying to clean up connection <asyncpg connection>" in the logs right from the beginning. I didn't pay attention, because it all seemed to work fine. But during my deployment tests with cast_registry I finally saw some real errors. The tests caused a lot (well, not really) of read queries to the database while the deployment process is posting steps to fastdeploy causing writes. And now I saw 401 responses to some requests from cast_registry, because the access token could not be verified or 404 because some object was not found. Looking at the logs I found some messages from asyncpg like "asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress". Hmm, seems like two coroutines are trying to use the same database connection which should not happen. After some debugging I found out the reason for this was me using "uow=SqlAlchemyUnitOfWork()" holding the database connection as a default argument in a bootstrap function, which causes uow to be instanciated at import time and never be replaced by a new uow, ouch. But after fixing that, fastdeploy is now ending up in a deadlock after some time. But only on Linux.

And while I'm sure it's all my fault and I'm just holding SQLAlchemy wrong and being and early adopter using it in combination with asyncpg doesn't help - I just cannot wrap my head around the SQLAlchemy interface. Why are there engines, connections and sessions? When do I have to instanciate/open/close them?  Why do I have to close a connection, but dispose an engine? I often close a session, then a connection and then dispose the engine. If you should always use this combination, why isn't there a highlevel function doing just that? If you don't, why is there no documentation showing why and how? 😤

Progress on having a landing page where people can create podcasts/blogs:

And then I started an experiment trying to help someone getting started with Django and writing a series of blogposts about it.

• Merged the old work in progress on the "Deployed Services Configuration"-issue to be able to work on this database bug without conflicts getting out of hand 🫣

Out of Context Images

Articles

• Pydantic V2 Plan
• Interactively compare Hubble and Webb images
• Forms in Django 4.0+

Videos

• The Amplituhedron - Nima Arkani-Hamed | Hmm, didn't know about this one..

Twitter

• With the newest version of Git 2.37.0, you can run just "git push" to push new branches. No more "--set-upstream origin".
• ChainMap is in collections in the #Python stdlib: it's a list of dictionaries, searched in order for items. Handy for nested contexts like programming language scopes. Or pizza specials! 🍕🥤😎🍕
• I was recently looking for materials to learn causal inference. These lectures given by @david_sontag are really amazing.
• A Python pattern that I like to use is attaching exceptions to callables that use them. To me this is syntactical sugar on larger projects. #twitter4devs
• This is probably well-known in some circles but not everywhere. The most important skill for Research Scientists in AI (at least at @OpenAI ) is software engineering. Background in ML research is sometimes useful, but you can usually get away with a few landmark paper. | Software engenieering is the most useful skill in a lot of fields right now..

Software

• Community: Component Libraries | List of web component libraries for date pickers etc..
• htmx 1.8.0 Release
• Typer Release 0.6.1 | Integration with rich

Podcasts

• #799 (Fünfphasig) (Bits und so)
• #293 – Donald Hoffman: Reality is an Illusion – How Evolution Hid the Truth (Lex Fridman Podcast)
• WR1384 Zeitenwende (Zum Thema) | Great episode, would like to hear a lot more stuff like this
• Kommunismus - Ein schillerndes Gespenst (Sein und Streit) | Interesting topic, but the episode was rather boring (at least for me)
• Chip Kidd—award-winning designer, musician, author, and all around rock star—joins for his fifth time to talk about his recent projects and so much more (Design Matters)

Getting Started

Lately, I was asked about how to get started with web development and Python in general. I pointed out some books and tutorials but then thought about what could improve the probability of success over just pointing to a lot of information. And the only thing I could think of was: Implement a small project with Django step by step. Create a GitHub repository for the project. I will have a look at your progress and keep adding issues. And of course provide help when you get stuck.

If you have a better idea on how to help people get started, please tell me. I'm very interested to hear about it 🤓.

And then someone (maybe my wife) told me that writing down all those steps as a series of blogposts could be also helpful for other people. So here we are 😏. We had to choose a platform to use for this project and despite being a happy macOS user by myself we agreed upon using Debian / Ubuntu for this project.

Python Setup

Installing Python

The first rule of Python is: Don't use your preinstalled system Python!

This is especially true for Debian-based distributions since they brake the Python standard library up into smaller packages, which breaks all sorts of things in weird ways. Even really basic stuff like creating a virtual environment won't work anymore.

And you don't want to replace your system Python with a working one, because it's used by the distribution itself for internal purposes, which probably depends on a lot of bugs they introduced, so: No easy Python installation for you!

For now, I recommend using pyenv to manage your Python installations until something better comes along. Install it by cloning the git repository like this.

And then prepare your shell for using pyenv by using the appropriate snippet for your preferred shell.

Before being able to install a real Python version using pyenv, you have to install some packages that allow you to compile Python without errors.

And now the installation of Python should finally be possible.

Create a New Virtual Environment

Despite now having the standard libraries venv module available, I prefer using virtualenv to create virtual environments, because it's faster and automatically installs the latest versions of pip, wheel and setuptools in the virtual environments it creates, which means one line less to type.

Now just create a directory for your new project and build a new virtual environment for it.

The venv argument tells virtualenv which directory it should use to install the virtual environment. Using venv for that is a popular choice and IDEs like VSCode or PyCharm will automatically find and use it. The --prompt argument is used to configure what the activated virtual environment will add to your shell prompt. Using "." just uses the name of the current directory.

Now the last step is to activate your Python environment, which depends on your shell. Use venv/bin/activate for bash/zsh or venv/bin/activate.fish for the fish shell.

Congratulations, you have now set up Python for your new project. The next step will be to bootstrap Django 😎.

Progress on having a landing page where people can create podcasts/blogs:

• Wrote some tests for cast_registry
• Fixed some deprecation warnings
• Enabled a custom admin url for production
• I wanted to add a "remove this blog"-feature to cast_registry to make it easier for me to test deployments. My time estimate for this feature would have been "just a few hours". But this story kept spawning sub-issues and getting bigger and bigger. The initial estimation would have been far off. What would be more valuable: Stick to the original estimate and promise myself to fx it later or fix it now and forget about my estimation? My lesson from situations like this is that estimates are pretty worthless. I just try to maximize the value of the thing I'm working on and don't do any estimations. Below are some of the sub-issues 😅.
  • Just store the last seen deployment in the session and compare it with a newly fetched one to determine new steps to show to the client
  • Move deployment metadata into its own model | I tried to be more verbose on my thought process here, let's see whether it helps

Articles

• He Dropped Out to Become a Poet. Now He’s Won a Fields Medal
• Using GPT-3 to explain how code works

YouTube

• Hyperscript and HTMX - Form Examples with Django
• Lisa Eckhart – Ist das lustig? (Sternstunde Philosophie)

Websites

• StackExchange: Seasoned Advice | There's a stackoverflow site for cooking 🍲
• Use GPT-3 to turn english description of a regex into an actual regex

Twitter

• isinstance() can check for multiple types. Python 3.10 makes it more convenient with pipes for unions
• starting the day like
• Operators in #Python can be implemented with dunder methods. >> is __rshift__, << is __lshift__, etc. They can even do strange things like modify the object!
• An ancient and unspeakable horror from the dawn of OOP.
• Too many ppl seem to use Jira's breathtakingly incorrect definitions of story, epic, task—one way that using Jira distorts your process away from agility. What are they really? 1/7
• The point of agile ways of working is to deliver more value. Not more value faster. More value period.
• everything about this is so quotable
• A simple lexical analyer (tokenizer) with #Python regular expressions. Uses regex verbose mode "(?x)", and match.lastgroup to know what kind of token was found.
• When people say they can estimate accurately, here's what usually happens
• As you can guess, this #Python program prints "Hello, world!". But how!? 🤯 😎 Also: don't do this! 😡

Books

• Das Ultraschall-Handbuch

Podcasts

• Tetra: A Full Stack Web Framework That Doesn't Make You Write Everything Twice - Episode 369 (Podcast.__init__) | It's very closely related to laravels livewire, but for django
• One of the master artists of our time, Richard Tuttle joins to talk about his 60+ year career that has revolutionized the landscape of temporary art (Design Matters) | Interesting definition of art: It's the thing you are able to care the most about
• Episode 60: Lies, Damned Lies, and Statistics (Hotel Bar Sessions) | Good episode, but the hosts demonstate by themselves how hard it is to distinguish between mistakes and damned lies, because it contained two points which could be counted as both:
  • Leigh stated that we do not know about the long term effects of vaccination. This is wrong. We know that vaccinations do not have long term effects (how should this even be possible?). In past vaccination campaigns it took time to be able to see the short term effects, because the events were rare and sample size was small (only a fraction of people got vaccinated). In the covid case the sample size is huge and we should have seen the effects immediately if there were any.
  • Switching from transmission to hospitalization statistics was criticized. But you have to do it at some point. Transmission does not tell you much about risk if most of the people getting infected are vaccinated or were previously infected. The case fatality ratio has dropped about an order of magnitude. Investing in warning people about transmission now only pays off in moral appearance but does not help (thoughts and prayers).
• Wie bleiben Sie integer? (Das philosophische Radio) | Good content, but poor audio quality.
• Engineering Stack Overflow with Roberta Arcoverde (The Hanselminutes Podcast) | Strong recommendation. Stack overflow is a rare example of a good system built on top of the microsoft stack. Checks many points on my list of things that are actually the right thing to do, but most people wont believe if you told them: No cloud, no microservices, just 9 webservers + good networking and a database with 1.5TB of memory on prem.

Progress on having a landing page where people can create podcasts:

• Made cast_registry use postgres instead of sqlite, mainly because it's easier to integrate into existing backup / monitoring.
• Automated deployment / removal for cast_registry
• Created a service for fastdeploy to be able to deploy / remove single cast hosting instances (a podcast or blog)
• Being able to deploy a podcast / blog via a cast_registry instance running on my staging environment 🥳

Articles

• Der Juni 2022 im Chaosdorf | Kind of a monthlog of my local hackerspace
• Fossil Versus Git
• PHP The Right Way | Since I have to take a look at php anyway (deploying wordpress) it would be helpful to do it the right way, for sure 🤓

Twitter

• This is really smart, especially the second part which borders on genius. | Asking this question would have saved me from serious pain
• MINNEAPOLIS GOTHIC
• Python makes it easy to take a flat list of key/value data and turn them into a real dictionary / Personally prefer to make an iterator and zip over it twice, no need to have multiple lists
• It's summer and I want an icecream!
• List slices in #Python can have a step: a third number to include every Nth element. You can assign to these also. Let's find primes with them! 😎
• This is a mathematics joke
• A comic making fun of python using 'if __name__ == "main"'

Software

• PyWebIO | Hmm, is this an alternative to plotly dash?

Podcasts

• Peter Sloterdijk zum 75. Geburtstag „Es braucht eine neue Ethik für das Zusammenleben mit der Erde“ (Sein und Streit)
• Kunsthistoriker Matthias Müller - Frühe Neuzeit wie heute: Pracht und Luxus sozialer Aufsteiger (Hörsaal)
• attrs and dataclasses - Hynek Schlawack (Test & Code in Python)
• Joscha Bach on AI, Cosmology, Existence and the Bible (Singularity Podcast)
• WR1372 Pfützen für die Wissenschaft (WRINT: Wissenschaft)
• Read The Docs - Eric Holscher (Django Chat)
• Ansible: config management & deploying code with James Cammarata from Red Hat (The Syscast podcast)
• Episode #290: Sentient AI? If so, then what? (Python Bytes) | I like that someone else is reading all the python news and I just have to listen to a podcast to stay up to date. I also like the topics and even the hosts. But sometimes the content quality of this podcast is kind of bad (almost as bad as journalistic publications). This AI story is complete bullshit and it took me about 5 minutes of using google to find out.
• Revision 532: Infrastructure as Code (Working Draft)