Ephes Blog

Miscellaneous things. Mostly Weeknotes and links I stumbled upon.


Weeknotes 2022-07-18

, Jochen

Fought with strange bugs last week, also some work and much outdoor stuff.

For example when using Jupyter Notebooks in a Django environment, the PYTHONPATH was always set to some old project and therefore imports from my current project didn't work. And it was simply not possible to get the PYTHONPATH right. I could not find the old projects name in any config or virtualenv. Weird. But then I had a closer look at the kernels using jupyter kernelspec list:
 

$ jupyter kernelspec list Available kernels: django_extensions /Users/jochen/Library/Jupyter/kernels/django_extensions python3 /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/python3

Ok ~/Library/Jupyter/kernels is weird. And there the name of the different project appeared in the kernel.json. Whoa. Somehow this old django_extensions kernel has overwritten my django_extensions kernel from the virtualenv. After deleting the old one, my kernelspec looked like this:

$ jupyter kernelspec list Available kernels: django_extensions /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/django_extensions python3 /Users/jochen/.virtualenvs/foo/share/jupyter/kernels/python3

This was pretty hard to debug.

And then there was this strange fastdeploy bug forcing me to have a closer look at SQLAlchemy. I still don't know what the root cause of the problem is. I switched to using SQLAlchemy in combination with asyncpg a frew months ago and there were this "The garbage collector is trying to clean up connection <asyncpg connection>" in the logs right from the beginning. I didn't pay attention, because it all seemed to work fine. But during my deployment tests with cast_registry I finally saw some real errors. The tests caused a lot (well, not really) of read queries to the database while the deployment process is posting steps to fastdeploy causing writes. And now I saw 401 responses to some requests from cast_registry, because the access token could not be verified or 404 because some object was not found. Looking at the logs I found some messages from asyncpg like "asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress". Hmm, seems like two coroutines are trying to use the same database connection which should not happen. After some debugging I found out the reason for this was me using "uow=SqlAlchemyUnitOfWork()" holding the database connection as a default argument in a bootstrap function, which causes uow to be instanciated at import time and never be replaced by a new uow, ouch. But after fixing that, fastdeploy is now ending up in a deadlock after some time. But only on Linux.

And while I'm sure it's all my fault and I'm just holding SQLAlchemy wrong and being and early adopter using it in combination with asyncpg doesn't help - I just cannot wrap my head around the SQLAlchemy interface. Why are there engines, connections and sessions? When do I have to instanciate/open/close them?  Why do I have to close a connection, but dispose an engine? I often close a session, then a connection and then dispose the engine. If you should always use this combination, why isn't there a highlevel function doing just that? If you don't, why is there no documentation showing why and how? 😤

Progress on having a landing page where people can create podcasts/blogs:

And then I started an experiment trying to help someone getting started with Django and writing a series of blogposts about it.

  • Merged the old work in progress on the "Deployed Services Configuration"-issue to be able to work on this database bug without conflicts getting out of hand 🫣

Out of Context Images


Articles

Videos

Twitter

Software

Podcasts


Django Beginner Series: Setting up Python

, Jochen

Getting Started

Lately, I was asked about how to get started with web development and Python in general. I pointed out some books and tutorials but then thought about what could improve the probability of success over just pointing to a lot of information. And the only thing I could think of was: Implement a small project with Django step by step. Create a GitHub repository for the project. I will have a look at your progress and keep adding issues. And of course provide help when you get stuck.

If you have a better idea on how to help people get started, please tell me. I'm very interested to hear about it 🤓.

And then someone (maybe my wife) told me that writing down all those steps as a series of blogposts could be also helpful for other people. So here we are 😏. We had to choose a platform to use for this project and despite being a happy macOS user by myself we agreed upon using Debian / Ubuntu for this project.


Python Setup

Installing Python

The first rule of Python is: Don't use your preinstalled system Python!

This is especially true for Debian-based distributions since they brake the Python standard library up into smaller packages, which breaks all sorts of things in weird ways. Even really basic stuff like creating a virtual environment won't work anymore.

python3 -m venv venv

The virtual environment was not created successfully because ensurepip
is not available. On Debian/Ubuntu systems, you need to install the
python3-venv package using the following command.

  apt-get install python3-venv

And you don't want to replace your system Python with a working one, because it's used by the distribution itself for internal purposes, which probably depends on a lot of bugs they introduced, so: No easy Python installation for you!

For now, I recommend using pyenv to manage your Python installations until something better comes along. Install it by cloning the git repository like this.

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

And then prepare your shell for using pyenv by using the appropriate snippet for your preferred shell.

Before being able to install a real Python version using pyenv, you have to install some packages that allow you to compile Python without errors.

sudo apt install build-essential libssl-dev zlib1g-dev libgdbm-dev libnss3-dev \
libreadline-dev libbz2-dev libsqlite3-dev libncurses5-dev libffi-dev liblzma-dev

And now the installation of Python should finally be possible.

pyenv install 3.11.3 # compiles + installs Python
pyenv global 3.11.3  # instructs pyenv to use version 3.11.3 as your global Python version

Create a New Virtual Environment

Despite now having the standard libraries venv module available, I prefer using virtualenv to create virtual environments, because it's faster and automatically installs the latest versions of pip, wheel and setuptools in the virtual environments it creates, which means one line less to type.

python -m pip install virtualenv

Now just create a directory for your new project and build a new virtual environment for it.

mkdir -p ~/projects/foo
cd ~/projects/foo
python -m virtualenv --prompt . venv

The venv argument tells virtualenv which directory it should use to install the virtual environment. Using venv for that is a popular choice and IDEs like VSCode or PyCharm will automatically find and use it. The --prompt argument is used to configure what the activated virtual environment will add to your shell prompt. Using "." just uses the name of the current directory.

Now the last step is to activate your Python environment, which depends on your shell. Use venv/bin/activate for bash/zsh or venv/bin/activate.fish for the fish shell.

source venv/bin/activate

Congratulations, you have now set up Python for your new project. The next step will be to bootstrap Django 😎.


Weeknotes 2022-07-11

, Jochen

Progress on having a landing page where people can create podcasts/blogs:

  • Wrote some tests for cast_registry
  • Fixed some deprecation warnings
  • Enabled a custom admin url for production
  • I wanted to add a "remove this blog"-feature to cast_registry to make it easier for me to test deployments. My time estimate for this feature would have been "just a few hours". But this story kept spawning sub-issues and getting bigger and bigger. The initial estimation would have been far off. What would be more valuable: Stick to the original estimate and promise myself to fx it later or fix it now and forget about my estimation? My lesson from situations like this is that estimates are pretty worthless. I just try to maximize the value of the thing I'm working on and don't do any estimations. Below are some of the sub-issues 😅.

Articles

YouTube

Websites

Twitter

Books

Podcasts


Weeknotes 2022-07-04

, Jochen

Progress on having a landing page where people can create podcasts:

  • Made cast_registry use postgres instead of sqlite, mainly because it's easier to integrate into existing backup / monitoring.
  • Automated deployment / removal for cast_registry
  • Created a service for fastdeploy to be able to deploy / remove single cast hosting instances (a podcast or blog)
  • Being able to deploy a podcast / blog via a cast_registry instance running on my staging environment 🥳

Articles

Twitter

Software

Podcasts


Weeknotes 2022-06-27

, Jochen

Tried to use this new 9€ Ticket for a trip we would normally use a car for and it was possible, but not comfortable :).

Not much work this week due to covid. But I started to work on my own projects again and got a lot of deployment stuff done:

  • Automated the deployment of a JupyterHub server to be able to give attendees of a python training just a username/password-combination and an url. This caused more work than expected, but at least I'm pretty sure now covid didn't rob me of my admin-mojo. Thanks jupyterhub. Things that turned out kind of unexpected:
    • The jupyterhub package did not require pycurl, jupyter_server and jupyterlab, but wouldn't work correctly without them being installed (for example getting a 404 on starting a users jupyterlab server)
    • Adding system users via jupyterhub admin UI adds system users, but removing them via admin UI removes them only from the jupyterhub sqlite, not the system, whoa. It also adds a valid login-shell to added users, which is not needed for jupyterhub, but makes forgetting to remove deleted users especially dangerous (ok, skipped jupyerhub useradding, just using ansible)
    • Had to set c.Spawner.cmd = ['/path/to/venv/bin/jupyterhub-singleuser'] in the config, dunno why (probably because of the venv, but it's the one jupyterhub itself is running from, well)
    • There are two attempts to provide a simplified installation for jupyterhub: The Littlest JupyterHub and Zero to JupyterHub with Kubernetes - for me, they caused more confusion than help. Just installing jupyterhub via pip and playing around til it showed a notebook worked for me, finally.
  • Automated the deployment of WordPress
    • Automated the deployment of MariaDB. The trickiest part was how to set the initial root password.
    • After being able to deploy MariaDB, it was only a small step to be able to deploy WordPress on my own infrastructure. Never thought I would do something like this. The trickiest part was to get wordpress to run behind a ssl terminating reverse proxy (had to set some additional request headers at the loadbalancer + some variables in wp-config.php). Why is there this weird installation script greeting you after you installed wordpress? This whole wordpress environment feels really strange and quirky. Didn't managed to get podlove publisher to work, maybe next week :).

Articles

Twitter

Websites

Books

Podcasts