Heckinchonkeires' Blog

How Microsoft and Apple Pushed Me Into Using Linux

2024-11-16T00:00:00Z

Setting up open source software is worth the effort

This post is about what it took for me to figure out the best way of working from my phone, and how obstacles created by Microsoft and Apple influenced where I ended up.

I’ve had plenty of gripes with big tech companies over the many years I’ve used their products. I’m well into the process of extricating myself from their systems today. They aren't making it easy, but I've already found plenty of alternatives that work well for me. Over the past few years, I switched from using Twitter (it was still Twitter when I deleted my account) to Mastodon, I started transitioning my primary email away from Gmail, I switched my primary browser from Chrome to Firefox, and I switched my default search engine to DuckDuckGo. Thanks to Nebula, some of my favourite YouTube content is available outside of Google's ecosystem, but network effect still forces me to use the platform. I've been thinking about trying Ubuntu as an operating system for some time, but my primary use for a desktop computer historically has been gaming. Linux gaming has progressed a lot in large part thanks to SteamOS, but I still begrudgingly want to have access to a Windows installation. Apple hasn't yet managed to push me away from iOS, but I'm sure it's only a matter of time, so I'm thinking about making the jump myself first.

After some struggle, I no longer really need my PC for working on software projects. I can do almost everything from my iPhone. (Not being able to do one important thing seems to be entirely on Apple and I'll address it specifically later in this post.) I hope it will be interesting to detail the things I tried to get to this point that didn't work, and the things I had to learn along the way.

Giving Windows one last chance

At the time I published my first post here, I had only worked on this project using VS Code on Windows. Since then, circumstances have necessitated me working from bed most of the time. I don't mind working on my Windows laptop in bed, but being a chunky old gaming laptop makes it less than ideal for doing so for long periods of time. A more ideal device for coding in bed is outside my current means, so my best option was using my iPhone. I didn't relish the thought of transitioning from two moderately sized monitors to a screen that fits it my hand, but I was determined to make it work as well as it could.

The first iOS app I tried was iSH. It's totally free, and it emulates Alpine Linux with a file system that's accessible through the native Files app. It's limited by being on the App Store, but I think its developers have done admirably within those limitations. Apple's Files not showing hidden files was a significant one. See the scripts below for how I got around it.

The Bash scripts I wrote to deal with hidden files in iSH

make_hidden.sh: I wrote this first because I was working on files called bashrc and bash_profile, and I needed versions that started with . so Bash would recognize them.

# takes an arbitrary list of file name arguments and creates hidden versions of them

for arg in "$@"
do
	if [ -f $arg ]; then
		if [ -f .$arg ]; then
			read -p ".$arg already exists. Overwrite? (Y/n): " response
			if [[ $response =~ [Yy] ]]; then
				rm .$arg
				cp $arg .$arg
				echo ".$arg overwritten"
			else
				echo skipping $arg
			fi
			unset response
		else
			cp $arg .$arg
		fi
	else
		echo "$arg not found"
	fi
done

unset arg

unhide.sh: Later I needed this similar script so I could "unhide" copies of hidden files from my server for editing.

# takes an arbitrary list of hidden file name arguments and creates non-hidden versions of them

for arg in "$@"
do
	if [ -f $arg ]; then
		unhidden=${arg##*"."}
		if [ -f $unhidden ]; then
			read -p "$unhidden already exists. Overwrite? (Y/n): " response
			if [[ $response =~ [Yy] ]]; then
				rm $unhidden
				cp $arg $unhidden
				echo "$unhidden overwritten"
			else
				echo skipping $arg
			fi
			unset response
		else
			cp $arg $unhidden
			unset unhidden
		fi
	else
		echo "$arg not found"
	fi
done

unset arg

Thanks to my latest (hopefully final) setup I don't need these anymore.

My recently refreshed Git knowledge gave me an idea: I didn't want to store every single file necessary for my projects on my phone, so perhaps I could figure out how to use Git to work with only a subset of a repository. I was stymied by outdated information on workarounds for that use case, then I found the simpler, modern solution after thoroughly reading a thread on <well-known QA service>. git-sparse-checkout is still marked as experimental, so I won't claim it's the correct way to do things, just that it worked exactly as expected for the time I used it.

With git-sparse-checkout and iSH, I was able to download only the website content to my phone, modify it, and push it wherever I needed it. Though if all I needed to do was compose and upload posts, I would have found a much easier way to do it. I still needed my Windows machine to actually publish the site, as well as–critically–run the development server so I could test new pages in a browser. I briefly considered trying to automate things using Git hooks, but it seemed like too much work, so I decided I needed to get more software involved.

A component of one of my university courses was learning how to work with a Raspberry Pi Zero W over WiFi with OpenSSH. So I had a little experience with SSH, and I was aware of how ubiquitous it was. That course had only covered using SSH clients outside of Linux, so I needed a little help in the form of this tutorial by Scott Hanselman to set up the Windows OpenSSH server. After working through some additional Windows issues not covered by the tutorial, I had sshd running on my PC. Through iSH on my phone, I was then able to manage the Eleventy development server and publish the site. Manually using Git to sync files was still a major annoyance, but I got too sidetracked to consider Git hooks again.

Next, I took a significant detour trying to "improve" remote development on my PC. I dabbled with some Bash scripting, and I tried to make Windows run without a GUI, all the time feeling I had to wrestle Windows into doing what I wanted–if it let me do what I wanted at all. A couple of my "favourite" problems were having to remove /dev/null in a Bash script because Git Bash couldn't use the Windows equivalent, NUL, and having to resort to adding the location of bash.exe specifically to Windows' PATH environment variable to get Windows to find it, despite Windows having no trouble finding git.exe in the same folder. Eventually, though I hadn't exhausted my options, I had exhausted my patience for Microsoft's nonsense. I no longer cared if it was going to take more work to set up a Linux server to do the same things. I was finally ready to put the work in to embrace open source software.

The home Ubuntu server

If I were to make a recommendation to a friend who was also sick of Microsoft and Apple operating systems, I'd tell them to grab the latest LTS version of Ubuntu Desktop. I decided to make things a little more difficult for myself. When I was first considering installing Linux on my old PC, I thought I wanted to self-host a website. I had removed the graphics card and all the storage save the now pitifully small 128 gigabyte SSD, so installing an entire GUI seemed like a waste of resources. I installed Ubuntu Server and tinkered with it for a few days. I didn't get very far before I lost interest. (This was before I had ever used SSH, so constantly having to move between the server and my new PC didn't make for a great experience.) When Windows made me resolve to spend my effort on an operating system I could actually control, my old PC still had that older version of Ubuntu Server installed. I had a much better idea of what I wanted to do, and Ubuntu Server still looked like the operating system for the job.

The first hurdle I encountered was the original Ubuntu Server installation. When I tried to update to the latest version of the operating system, the update failed twice. On top of that, I had elected to fully encrypt the storage drive. I hadn't lost the encryption password, but I convinced myself that I would have to physically access the computer to enter the encryption password every time it restarted. (I now think there could be some way to decrypt a fully encrypted drive remotely, but finding out how hasn't been a priority.) Some cursory research indicated that removing the encryption was effectively impossible, so I decided I might as well pave the whole thing and start over. I hadn't left anything I cared about on that drive anyway.

Getting a fresh Ubuntu Server installation on that machine proved more complicated than I expected. I don't know if it was the encrypted drive, the older version of Balena Etcher I used, the wonky existing Ubuntu installation, or something else entirely, but the first fresh installation failed so spectacularly that it left the operating system inoperable and made a complete mess of the USB drive I used. Never before had I plugged in a USB drive that Windows couldn't even recognize well enough to demand it be formatted. Fortunately, Windows' Disk Management could still see it, so I hadn't managed to completely destroy it. To be safe, I gave it a full format and flashed it with the new Ubuntu Server image using the latest version of Rufus. The second installation worked, but I must have managed to type the same wrong character(s) twice while setting the administrator password, because I got locked out of the system without ever having logged in. I was too frustrated to consider whether waiting for the lock to expire would be faster than another full installation, so I did the thing I could do immediately and started installing Ubuntu for the third time. At least I didn't have to flash the USB drive again. That time I had no trouble logging in, so I started configuring things.

I wanted to get to the point where I could work on the server from my phone as soon as possible. Sitting on my couch hunched over a small old VGA monitor supported by a convenient folding chair was even worse than working at my desk. Unlike the first time I configured Ubuntu Server, I knew exactly what I needed to do. I updated the pre-installed software packages and assigned the server a static IP address using my router, since I didn't want to learn how to set a static IP on the command line just then. I configured the firewall to deny all incoming traffic except from my local network using ufw, and set up the SSH server so I could connect with iSH. Copying the public SSH encryption key from iSH to the server using OpenSSH's provided script worked the first time, unlike with Windows. Once I rebooted and tested the SSH connection one more time, I was confident I could do everything else from my phone.

Getting iOS and Ubuntu in `unison`

In less time than I would have anticipated a year prior, I was at the same place with that Ubuntu server I had gotten to with my PC. I could easily manage the Eleventy development server, upload and download files, and publish to my Codeberg Pages repository from anywhere in my WiFi network's range. I could have tolerated that situation, but manually syncing files between devices was still getting in the way. I wasn't sure what I needed specifically, so I did some searching for how to handle remote software development on an iPhone. I eventually found some useful tools to test, but I had to get pretty specific with my queries to not get pages of results about remote iOS developer jobs.

I gave Working Copy a try, but quickly determined it wasn't the right tool. It seemed great at streamlining working with Git on iOS, but it didn't seem to do anything I couldn't do using iSH with a little more effort. I came across two iOS code editor apps in the same listicle that sounded promising: Buffer and Textastic. They were both one time purchases with no additional monetization (which is always a good sign to me), and they happened to be the same price. Textastic was the #1 app in the developer tools category, with better reviews, better documentation, and more customization. Buffer, however, had a pretty slick website, and it listed a killer feature there that I couldn't find anywhere in Textastic's excellent manual: "Edit files directly on your server without download/upload." If Buffer could do that while being a reasonably useable code editor, nothing Textastic could offer could compare. That did not turn out to be the case. I'm not writing this post using Buffer.

I paid my $12.99 plus tax for Buffer, and after quick look at the settings menu, got straight to the point. Buffer instantly failed to connect to my server, and offered an unhelpfully vague error message. I double-checked the SSH settings and tested them again in iSH, and they worked perfectly there. I gave the connection a few more tries in Buffer, but kept getting the same vague error. Then I decided that if I was going to have to put in a lot more effort to get the app to do the thing I bought it for, I should find out how it was as an editor first. I think that decision ended up saving me some frustration, if not some time. Over the course of an hour trying to work on local files, Buffer froze and/or crashed at least five times. Immediately after the app recovered from one last crash, an iOS standard "If you're enjoying <app>, please rate it" pop up appeared. I normally dismiss those without much thought, but this time I had plenty to say about the app. (I do try to remember to write reviews for apps I really like, but I much prefer to do it on my own terms.) So I gave Buffer a one star rating, left a review explaining why, uninstalled it, and requested a refund–which I later received. A couple search queries later, I actually managed to find an article that discussed remote software development on an iPhone. I'm kicking myself for not bookmarking it at the time, because I can't find it again and I'd really like to credit the author. It goes over a few options, but I didn't spend much time thinking about most of them, because the first one sounded great. code-server: remote-hostable VS Code with a browser interface.

Despite my growing animosity toward Microsoft, I'm still reasonably happy with VS Code. It happened to be the IDE used in a tutorial I followed years ago, so I picked it up it without much thought, and kept using it as it handled everything I needed an IDE to do well. So I thought code-server would mean using the tool I was most comfortable with from my phone, as well as from any other device with a web browser. I'm sad to also report that I am not writing this post in a web browser. I can't recall much about what it took to get code-server to even render a web page. I probably had to forward a port or two, and I know I even tried building it from source code before giving up. The best I got out of code-server with my setup was even worse than Buffer. I actually edited and saved a file or two using Buffer. code-server's web interface was so unresponsive for me that I never even got a file open. Unlike with Buffer, I think the failing is more likely in myself or my setup than the software. I think either I didn't find the right way to configure things for my situation, or something about my hardware makes running a useable code-server instance impossible (the iPhone part if I had to guess).

So I searched yet again. I found this article by Willem Middelkoop, which would (eventually) lead me to the solution I'm using today. It describes using the cross-platform software Unison to synchronize files from a tablet to a remote server, then using the app Secure Shellfish to integrate that remote server into the native Files app on an iPhone. Unison being free and open source made it the obvious thing for me to try first. I thought I wouldn't even need another iOS app if I could just keep the files I was working on synchronized between iSH and my server.

Then I ran into a major limitation of iSH for the first time: its emulation is impressive, but it isn't capable of everything standard Alpine Linux can do. On top of that, the software package repositories available through iSH have to be custom ones approved by Apple. (I since found a cool workaround that lets you use the official Alpine repositories on the iSH wiki, but that only removes the Apple-imposed limitation.) So the software you can run in iSH is mostly limited to older versions. Unison was available, so I tried getting the default versions of Unison on Ubuntu and iSH to talk to each other. I had no luck with that straightforward approach. I noticed the Unison versions reported by the two systems were different, so even though I didn't yet know backwards compatibility was a well understood problem with older versions of Unison, it occurred to me to try matching the server version with the iSH version. I must have missed something easy and/or obvious, because I'm still not clear on the correct way to get an older version of a package, either from the default Ubuntu repository or from GitHub. What came closest to working was downloading the source code for the right Unison version and building it right on the server. (I think it's likely I made the mistake of using the wrong compiler version. My later reading told me that even two instances of the same older version of Unison compiled with two different compiler versions wouldn't work together. With the difficulty I had getting an old version of Unison, I suspect I couldn't be bothered to do the same thing for the compiler and just used whatever version Ubuntu gave me.) The freshly compiled Unison made it through the tutorial in Unison's manual without a hitch, and showed no signs of trouble as I configured it. Unfortunately when I tried to start syncing files I actually wanted to work on, I ran into error after error and eventually couldn't think of anything else to try.

Then I was back to DuckDuckGo, but with a more specific idea of what I wanted. Sure I couldn't get Unison working, but there had to be other file synchronization software out there, and maybe that would work in iSH. Syncthing sounded better than Unison, and I even found a version of it I could install in iSH. The fact that I found it in the Alpine repositories and not the iSH ones was a warning sign I ignored. The initial run of Syncthing caused a kernel panic for iSH, something I hadn't seen it do before that point. So I have to admit Apple is a little right in restricting iSH's available packages. Some guardrails are warranted, but I think a warning prompt before installing known unstable or untested packages would be enough. Having done what felt like the most I could do with the free tools available, it was time to just pay somebody to solve the problem for me.

The right tools

In the process of struggling with Unison, I installed and started using Shellfish. It was a significant terminal upgrade over iSH, and, most importantly, it had the aforementioned feature of integrating a remote server with the Files app. What kept me on the file synchronization path for so long was that full Files integration was a feature only available in the pro version of Shellfish. My time isn't worth much right now, so if I can save any amount of money by spending time, especially spending time on something fun and/or interesting, I'm going to do it. By the time Syncthing was causing a kernel panic in iSH, I was bored and frustrated. I wanted to get to work on my Anytype project (more on that soon, I promise), and all I had to do was cough up some cash.

So I bought the pro version of Shellfish. $50 for the lifetime pro unlock was a lot more than I'm comfortable spending on an iOS app, but the other option was a subscription, and I much prefer to pay for something once and get it over with when I can afford to. Also, I really like Shellfish. The ads in the free version can be disruptive, but they're tame by free app standards and I think I only saw ads for the pro version and the developer's other apps. It has significantly more extra keyboard keys than iSH, including some like delete, home, and end that I really feel the absence of in other apps now. Some other features it has that iSH doesn't include multiplexing support, password integration, and terminal persistence support. It even has a command snippets feature that I completely forgot about until going back into the settings looking for other things to praise while writing this. All of that had been very easy to use. Shellfish's development seems to be mostly a solo effort, so it even feels good to support it. With my server near seamlessly integrated with the Files app by Shellfish, I could treat files on my server just as though they were stored on my phone, without the hassle of manually copying them back and forth.

So now I can address the thing I've teased a few times: what I'm actually using to write this post. For my first two posts, I did the bulk of the composition in Anytype. I really like being able to put all the information I work with in one place. At least in theory, Anytype is that one place. Unfortunately, Anytype is in beta, so I had to pay an early adopter tax in the form of time spent working around underdeveloped features. Also, Anytype's mobile app is behind the desktop version in terms of features, so that further limits what Anytype can do on my phone. For my first post, I did most of the writing in Anytype and pasted the text into VS Code on my desktop to add markdown formatting. That made me comfortable enough with markdown to add formatting characters as I wrote, so I tried that when composing my second post. I quickly discovered that Anytype automatically processes markdown formatting characters as you input them and adds the respective formatting while removing the characters. That sounds helpful for somebody who's working on a more finished product, but it's the opposite for me trying to write actual markdown. Beta being beta, I couldn't turn that feature off, so I found a workaround: I used alternate characters that were available on the iPhone keyboard, that I was sure would never naturally appear in my writing, in place of the proper markdown formatting characters. Then I wrote a basic Python script to find and replace those characters in case I needed to do the same for more posts later.

My quick and dirty Python script

I put a little extra work in to encapsulate the character replacements so I could easily modify them later, but I doubt I'm coming back to this.

"""
    A script for cleaning up text copy-pasted from Anytype
"""

def add_backticks(atstr: str) -> str:
    return atstr.replace("•", "`")

def add_underscores(atstr: str) -> str:
    return atstr.replace("¥", "_")

if __name__ == "__main__":
    out_str = ""
    with open("sample_copy.txt", "r", encoding = "utf-8") as f:
        for line in f:
            line = add_backticks(line)
            line = add_underscores(line)
            out_str += line

    with open("atclean.txt", "w", encoding = "utf-8") as f:
        f.write(out_str)

The other text editor I used to use was Koder, since coding in Anytype is completely out of the question right now. It was the best-sounding free code editing app I found, even boasting remote connectivity features. I used it to test every option I found for working from my phone, and as of a few days ago, I thought it would continue to serve as my code editor. There had been plenty of small issues I had dismissed though. It wasn't nearly as bad as Buffer was for me, and it was free, so I think that made me cut it some slack. There were a few things I tried that it seemed incapable of doing, and it would crash or freeze instead of telling me no.

The first crash I remember happened when I was trying to sync Koder with a remote folder. It had no trouble connecting to my server and uploading and downloading, so when I found its "sync with ftp" option, I thought that was it. But it crashed every time I tried to sync, no matter the configuration. I had only thought of it as an editor, so I wasn't very disappointed. Some other things caused instability, but they never got in the way of editing. Then I realized that there was no way to manage the list of recent files Koder maintains, so that feature would become useless once that list was long enough. Those little annoyances kept piling up, but I could still edit code much more easily than in Apple's Notes or Anytype, so I didn't make a change. Finally, with remote development on my phone underway at last, I ran into something I needed a text editor to do that Koder couldn't do: I needed to edit a Nunjucks template file. I found out that Koder just refuses to open files with extensions it doesn't recognize, and even certain files with no extension at all, inexplicably.

So at last I'll reveal that I'm writing this post using Textastic. I started with Anytype out of habit, but the first time I tried to add markdown formatting characters, the familiar frustration reminded me that I had bought Textastic a few hours earlier to edit that Nunjucks template. Textastic sounded great when I was comparing it to Buffer, and I had solved being able to edit remote files. So far, using Textastic has been a breeze. I just added a couple of directories from my server as external folders, and everything has worked perfectly. It can do its own SFTP connections, but Shellfish makes that unnecessary for the most part. It's no substitute for multiple monitors, a mouse, and a full keyboard, but for working on an iPhone, I've found next to nothing to complain about. It seems to be #1 in developer tools for a reason, so sometimes you do get what you pay for. However, I don't regret the time I spent trying to find free alternatives. Sure I'd have gotten back down to business faster if I'd spent the money right away, but I'd have learned a lot less and I'd have significantly less to say.

Going forward with open source

For the time being, I don't need my desktop PC for anything. I still have to use my Windows laptop when I need Firefox's developer tools, since I haven't found a way to access something similar on my iPhone without connecting it to the actual Mac I don't have. I'm more free from Microsoft and Google than I've ever been since I started using their products, but I still have work to do on the Apple front. I'm glad I managed to make my humble iPhone SE work for me for the time being, but planned obsolescence will come for it eventually. When it does, I'm getting a phone that lets me choose its operating system. I came across Ubuntu Touch the other day, so maybe I'll try that.

I don't think I'm uninstalling Windows any time soon. I'm sure I'll want to play video games on a reasonably sized screen using a nice keyboard and mouse in the future, and I expect Windows will remain my best option for doing so for a while. I am planning to also install Ubuntu Desktop when I'm capable of working at a desk again. And then I want to try working on my various projects using only open source software. It might take more effort to set everything up, but now I think the freedom and control will be worth it.

Things I Learned About Bash Scripting Last Week

2024-11-15T00:00:00Z

Learning a lot through unnecessary automation

This post recounts how I learned enough about Bash scripting to automate activating and deactivating Python virtual environments created by the venv module.

As I’ve become more comfortable with the command line over the years, I’ve learned a little about shell scripting here and there. I was always reluctant to really get into it because, being too young to remember a time before GUIs, it seemed archaic to me. My earliest memories of playing video games involve launching them from a command line though. Lately, while I’m still a little reluctant, I’ve been finding more and more reasons to learn about it.

When I was trying to debug my modifications to the deploy-pages Bash script for this website, I had an incentive to try Git Bash and learn a little bit more about Bash scripting. The first thing I needed was a way to save the directory the script was executed from. I just went with the first search result for “bash save current directory.”

My thanks to <user> on <well-known QA service>

I wanted to show what I found, link to where I found it, and credit the person who posted it, but I’m concerned that doing so could be outside of my “personal, non-commercial use.” I expect the risk of legal action is quite low, but I prefer to avoid it even so.

# imagine a tidy one-liner that saves the path to the current working directory

Then I ran into a persistent Codeberg bug that frequently forces you to re-run git push. If I was writing a JavaScript or Python script, I would have known exactly how to catch the error and run git push again. So I went looking for the Bash equivalent of a try/catch block.

My error handling for the Codeberg authentication bug

git push "$remote" "$remote_branch"
if [ $? -ne 0 ]; then
    echo "Codeberg authentication failed. Retrying."
    git push "$remote" "$remote_branch"
    if [ $? -ne 0 ]; then
        echo "Error: Retry failed. Bother Codeberg about it."
        exit 1
    fi
fi

I picked up the structure of a Bash script if statement and that the $var syntax was used to access the value of a variable from reading the original deploy-pages script. What was new to me in the example I found was the [ $? -ne 0 ] part. I learned that ? is a special bash variable that holds the exit status of the last program, and that any value besides zero meant an error had occurred. -ne 0 meaning "not equal to zero" made sense, but the syntax still felt strange. (I have since learned that [ ] is an alias for the test program, so -ne looking like a command line option and not a typical operator makes sense.) That script necessitated me learning more Bash script syntax, but as usual I only learned enough to get the job done. It did get me to set Git Bash as my default terminal though, and that set me on the path to finally attaining a working knowledge of Bash scripting.

Last week I was setting up everything I needed to work on my Anytype Python module. (I’ll write a post specifically about that project once I have enough done to write about.) Since my goal with that project is to make something other people can use, I’m doing my best to learn and apply best practices for Python module development. Part of that is setting up a virtual environment for the project. The venv module bundled with the current version of Python makes doing so almost seamless, but I still had to type python -m pip every time I wanted to use pip. I also didn’t like that I had to run the venv activate script every time I wanted to work in the virtual environment. I was already using Git Bash, and my recent experience modifying a Bash script gave me the confidence that I could figure out how to automate away those annoyances.

Command aliasing was quick and easy. I just needed a .bash_profile file in my home directory with the line alias pip='python -m pip'. That worked as soon as I launched a new terminal, and it taught me that .bash_profile is where I should put things for setting up the shell. (At the time of editing this, I've improved my Unix knowledge substantially, and now know that .bashrc is the better place for aliases. That's a future post.) Automatically managing Python virtual environments ended up taking the rest of the day. Though I feel that particular diversion wasn’t a waste of time. At some point in the preceding couple days, I read that it was necessary to add export PROMPT_COMMAND='history -a' to a .bashrc file in your home directory so that VS Code could save the command history of Git Bash properly. In setting up my Python project, I then had cause to learn that PROMPT_COMMAND is a shell variable where you can set commands you want to run every time after the shell finishes executing the commands you entered. Then the solution for activating a Python virtual environment seemed straightforward: add an activate_pyvenv script to PROMPT_COMMAND to check for the venv activate script and run it if it’s there.

My first attempt at activate_pyvenv.sh

if [ -f ./Scripts/activate ]; then
    source ./Scripts/activate
fi

It seemed to work at first, but I'll get into the problems I encountered later in this post. After I tested it, some quirk of VS Code made it so that the nice colour coded Git Bash prompt that updated automatically with the current directory stopped updating. So I decided it was time to learn how to customize a Bash prompt myself. I skimmed a tutorial or two, but I wanted to really understand my options, so I ended up on the Bash Reference Manual page for controlling the prompt. I patterned my custom prompt on the original Git Bash one, with some additions that appealed to me. My first pass looked something like this:

A computer terminal prompt. White text on a black background across two lines. The text consists of, in order and separated by spaces, the shell name (bash) and host name (localhost) with an at symbol between them, the date 2024-11-09, the time 18:47, and the full directory path /home/heck/projects/website followed by a colon and an underline-style cursor. The text overflows onto a second line in the middle of the word projects.

The corresponding value of PS1

"\s@\h \D{%F} \D{%R} \w:"

I’m reconstructing these from memory in iSH as I didn’t have the foresight to document all my iterations.

That prompt had the information I wanted, but it was a little plain, and the full directory path felt like too much. Plus, the cursor being so close to the prompt bothered me. So over a few more iterations, I replaced the spaces with vertical lines, set PROMPT_DIRTRIM to 2 in .bash_profile, added a newline and the \$ special character, and went back to this tutorial I had skimmed earlier to figure out the colours. This is what my prompt looks like today:

A computer terminal prompt. Text in multiple colours on a black background across three lines. First, in order and separated by white vertical lines, the shell name (bash) and host name (localhost) with an at symbol between them in dark green, the date 2024-11-09 and the time 19:14 in purple, the partial directory path /projects/website abbreviated with an ellipsis in yellow. The yellow directory path overflows onto the second line part way through the word website. On the last line, a dollar sign and an underline-style cursor in white are separated by a space.

My current value of PS1

"\e[0;32;40m\s@\h\e[m|\e[7;30;45m\D{%F}\e[m|\e[7;30;45m\D{%R}\e[m|\e[7;30;43m\w\e[m\n\$ "

With a prompt I was happy with, I returned to figuring out how to manage Python virtual environments with a Bash script. I quickly realized that checking for the activate script itself was not the best way to detect a Python virtual environment directory. ./Scripts/activate seemed like a pretty generic sounding path that could easily exist outside the context of the venv module. Now pyvenv.cfg was nice and specific, and (I think) should exist in the root directory of every venv environment. I found a solid way of detecting when I was in the right place to run the activate script, but after some testing I realized I needed to do a whole lot more.

At that point, my activate_pyvenv script ran the corresponding activate script every time I moved to a venv root directory. So if I navigated to a subdirectory and back, it would run the script again. I could have lived with that being the only problem. But then I started installing modules and noticed they weren’t always ending up where I expected. I realized that, in addition to automatically activating the virtual environment, I needed to automatically deactivate it when I left the venv root directory. The first step I took towards solving that problem was writing a script that simply detected whether, after the shell finished executing, it had gone up, down, or stayed in the same place in the directory hierarchy. I found a common Bash pattern for detecting if a string was a prefix of another using a case statement. Then I tried comparing the shell environment variables PWD and OLDPWD, but OLDPWD wasn’t changing the way I expected it to, so that was insufficient. A new variable, VENV_PREV_DIR, that I updated to $PWD at the end of the script had the behaviour I was looking for.

My first step towards solving the problem: detect_cd.sh

case $PWD in
    "$VENV_PREV_DIR"*)
        if [[ $VENV_PREV_DIR != $PWD ]]; then
            echo down a level
        else
            echo same level
        fi
    ;;
    *)
        echo up a level
    ;;
esac

VENV_PREV_DIR=$PWD"

I was sure that the only time I would want to run the venv activate script was when I navigated down into a directory containing a pyvenv.cfg file and a virtual environment wasn’t already active. I decided not to waste time figuring out the “correct” way to tell if a virtual environment was already active, so I simply initialized a flag called PYTHON_VENV_ACTIVE to zero in .bash_profile and used that.

My first pass at activate_pyvenv.sh

#!/bin/bash

case $PWD in
    "$VENV_PREV_DIR"*)
			if [[ $VENV_PREV_DIR != $PWD ]]; then
				if [ -f ./pyvenv.cfg ] && ! (( "$PYTHON_VENV_ACTIVE" )); then
					PYTHON_VENV_ACTIVE=1
					echo activating python venv
					source ./Scripts/activate
				fi
			else
				# placeholder
            : # do nothing
			fi
    ;;
    *)
			PYTHON_VENV_ACTIVE=0
			echo deactivating python venv
			deactivate
    ;;
esac

VENV_PREV_DIR=$PWD

I was getting closer. The logic for determining when to activate worked well, but I was deactivating every time I went up in the directory hierarchy. I needed to detect when I left the actual venv directory. Saving the current directory to VENV_DIR, a new shell variable, when activating a virtual environment came to me right away. Yet I struggled to effectively determine when I had left that directory. I tried if [[ $VENV_PREV_DIR = $VENV_DIR ]] when going up, but that only worked when moving up out of exactly the venv root directory, and not when moving up out of that directory from a subdirectory. Then I came to an understanding: the core logic of the script was figuring out if a new directory was a subdirectory of the previous one, and I needed to know if a new directory wasn’t a subdirectory of the saved VENV_DIR. So I nested another case statement in the “moving up” condition and the script was done but for some minor polishing.

The current version of activate_pyvenv.sh

# manages automatically activating and deactivating Python virtual environments created by the venv module

# this case statement checks if the current directory is a subdirectory of
# the last one visited by the shell
case $PWD in
    "$VENV_PREV_DIR"*)
        if [[ $VENV_PREV_DIR != $PWD ]]; then
            # the shell just started or we've gone down in the directory hierarchy
            if [ -f ./pyvenv.cfg ] && ! (( "$PYTHON_VENV_ACTIVE" )); then
                # we're in a python virtual environment and it needs activating
                PYTHON_VENV_ACTIVE=1
                # save the venv directory so we can tell when we've left
                VENV_DIR=$PWD
                short_venv_dir=${VENV_DIR##*'/'}
                echo activating python venv $short_venv_dir
                source ./Scripts/activate
            fi
        else
            # placeholder in case I want to do something when I don't
            # change directories
            : # do nothing
        fi
    ;;
    *)
        # we've gone up in the directory hierarchy
        case $PWD in
            "$VENV_DIR"*)
                # we're still in a subdirectory of VENV_DIR
                : # so do nothing
            ;;
            *)
                # we've escaped the virtual environment
                PYTHON_VENV_ACTIVE=0
                short_venv_dir=${VENV_DIR##*'/'}
                unset VENV_DIR
                echo deactivating python venv $short_venv_dir
                deactivate
            ;;
        esac
    ;;
esac

unset working_dir
unset short_dir
unset short_venv_dir
VENV_PREV_DIR=$

I tried to save a couple lines of code by omitting the variables for the short directory names, but I quickly gave up on figuring out the correct syntax.

At the time I finished the script, I was sure there were some things I had done clumsily that could be improved by somebody with more Bash experience. I also acknowledged to myself that it wouldn’t work if I navigated straight into a subdirectory of a venv directory. But, as the saying goes: “perfect is the enemy of done.” I thought I would have to figure out how to check every directory above the current one for a pyvenv.cfg file, and that it would take some time. I added the script to my list of things to revisit if I run out of project ideas. It worked well enough, so I was happy to move on and get some “real work” done.

How I Built This Site

2024-11-06T00:00:00Z

From a domain and an idea to a functional personal blog

This is my first proper post on web.heckinchonkeires.me. It’s the story of how this site came to be, and the things I learned and mistakes I made along the way. I think it will serve as a helpful explanation of the hows and whys of this website, as well as a good example of the kind of writing I intend to post here.

I’ve had a personal website on my mind for a long time. I experimented with some free web application hosts in the past, but their offerings didn't seem appropriate for what I wanted to do here. I finally bought heckinchonkeires.me this year, and I started doing some work on turning an old PC into a web server. I still intend to finish that project, but it dawned on me that it was going to take a while. I’m trying to start a career as a software developer (hire me please), and while I expect knowing how to operate a small web server would be an asset, I need to prioritize. If learning all that is going to be of any use to my career, I need a way to show potential employers what I’ve learned. So that’s where this website is starting from; a place to publicly document my software projects. In addition to benefiting my career, this site serves as an incentive to document my work as I go. And turning my collection of comments, notes, and discarded test scripts into something comprehensible to somebody else helps me better understand and remember what I learned.

I had a domain and a decent idea of what I wanted to do with it. Before I found my current hosting solution, I had done some research into free blog options, but they all seemed to be aimed at non-coders. Wordpress.com and the like are fine if you just want an easy way to get words onto the internet, but they limit what somebody with web development skills can do with them. Last week I was thinking about changing git hosts, and Codeberg came to mind. I looked around their website and I came across codeberg.page. That page had three steps I understood, it was free, and they even allowed custom domains. I was already convinced to use Codeberg as a Git host, and Codeberg Pages also solved my web hosting problem. It only hosts static sites, so I can't do everything I want to do with a personal website, but it gives me more than enough control for my immediate goals. For the time being, I just want to host text content somewhere on the open web, and now I can do that and more.

Getting anything at all on the internet was solved, but I still needed a way to make something presentable in a timely manner. I came across this tutorial, and it was very helpful. Not only did following it get a custom web page online quickly, it made me realize that what I needed was a static site generator. I chose Eleventy because the author used it in the tutorial, and it turned out to be a great choice. I also really appreciate the included deploy-pages shell script, and I’m still using a modified version of it.

With a domain, a web host, and a site generator, I still needed to put a site together. I'm sure I could have built a whole blog from scratch using Eleventy, but it would have taken significantly longer and probably not looked great. So I went looking for a better starting point and quickly found the Eleventy Base Blog starter project linked in the Eleventy documentation. It was perfect. That project would give me a functional blog in little time, and from what I had learned about Eleventy, it seemed like it would be easy to build on that base. On top of that, the Base Blog repository is well documented, so I feel confident that figuring out what I need to change when the time comes won't be a problem.

So I learned my way around the starter project and got a copy of it online under my own domain. At that point, I decided I wanted to limit the Git repository for the actual website to the rendered content and keep the generator itself in a separate repository. Changing the output directory for Eleventy was easy. I also wanted to keep using the deploy-pages script to save some typing every time I updated the site, but it didn't quite work for my new use case. I didn’t know much about shell scripting at the time; I understood where I needed to make the script change directories and why that was necessary, but parts of it that were beyond me kept causing errors. I knew enough to tell which parts were essential, so I removed the error-prone sections until it worked. My major addition was some error handling logic for a Codeberg bug that requires re-running git push. I think the script is still overcomplicated for my use case, though it allows for some flexibility should I change things.

My addition to deploy-pages.sh

git push "$remote" "$remote_branch"
if [ $? -ne 0 ]; then
    echo "Codeberg authentication failed. Retrying."
    git push "$remote" "$remote_branch"
    if [ $? -ne 0 ]; then
        echo "Error: Retry failed. Bother Codeberg about it."
        exit 1
    fi
fi

Once I had things organized so that I could easily push any updates I made to the live site, I started working on building the starter project into something I would feel comfortable calling my own. I started in the root directory, deleting some files that were there to facilitate services I didn’t plan on using. I updated the package.json file with details about myself and the project. I made sure to write my own README, and then I turned my attention to the LICENSE file. I had done a good amount of reading about copyright and licenses and I settled on using CC0 for my writing. My research hadn’t focused on software, so I wasn’t sure what my options were regarding licensing my modifications to code with an existing license. I searched for information about this more specific case, though I could have just stopped at reading the MIT license included in the starter project. In short: I can do whatever I want as long as I include the same license in the project. That precluded me from using CC0 for the site generator project, but I found the MIT license permissive enough. So I’m going to license my modifications using MIT. Also, while not required by that license, I decided I would keep the original author’s copyright in the license file as a sign of my respect and appreciation for their work. I added the author of the deploy-pages script to the license file as well, and I intend to do the same for as much of the non-library code I use in my projects as is feasible.

Having made my version of the Eleventy Base Blog repository ready to publish under my own (pseudo)name, I started working on the content for the site. All that thinking about copyright made me prioritize adding a CC0 declaration to the site’s footer, and I added my explanation of CC0 to the about page. Creative Commons’ template for a CC0 footer included links to their CC0 page, and I elected to put one on my about page as well. Everything worked, but I didn’t like that the external links opened in the same tab. At the time I thought opening external links in a new tab by default was a good thing to do, but I'm reconsidering that now. I looked up the HTML attributes I needed to add, then immediately realized there was no way I was going to manually add those attributes to every external link on the site. I’m a programmer after all, so I’m always up for automating things, even if it might not be necessary. I was sure Eleventy would make applying the attributes I wanted easier, so I went back to its documentation. Transforms sounded like what I was looking for: I needed to automatically transform external links after all. At the time, I thought Eleventy made the content of each output page available as HTML text strings (or at least something similar) and I was already familiar with Fast HTML Parser, a JavaScript library for working with HTML. My misunderstanding was that the content Eleventy provides to transforms is the rendered output before it’s written to the output directory, or at least something much closer to that than what it actually is. I wrote my transform function to parse the content provided by Eleventy and add the necessary attributes to all external links. I got that function working relatively quickly, but it only worked on the links in the footer.

My first working transform function from eleventy.config.js

eleventyConfig.addTransform("external-link", function (content) {
    if ((this.page.outputPath || "").endsWith(".html")) {
        try {
            const html = parse(content);
            const links = html.getElementsByTagName("a");
            // make sure we found links to transform
            if (links && links.length > 0) {
                for (let link of links) {
                    // avoid an error if the link somehow doesn't have an href
                    const lhref = link.getAttribute("href") || "";
                    // all external links should start with http
                    if (lhref.startsWith("http")) {
                        // avoid overwriting links with existing attributes
                        if (!link.getAttribute("target"))
                            link.setAttribute("target", "_blank")
                        if (!link.getAttribute("rel"))
                            link.setAttribute("rel", "noopener noreferrer")
                    }
                };
                return html.toString();
            }
            // no links found, return original content
            return content;
        } catch(error) {
            // make sure to still return the original content if something goes wrong
            console.error(error);
            return content;
        }
    };
});

That was when I realized that what I was actually transforming was the content of the template and layout files before Eleventy starts rendering them. My transform only worked for the footer because it was defined in a Nunjucks template file, which partly used HTML syntax. It was getting late, so I left myself some sections in the transform function to fill in for handling external links in non-HTML formatted files. The next day I went completely off the rails. I spent most of it trying to write a general JavaScript function that could identify and modify links in arbitrary strings. The Linkify library helped, but the most it could do in this case was pick out every URL in a string. So Linkify saved me the identification part, but what it gave me was a list of URL strings and their start and end indices in the main content string. The problem perceived was that it found every URL in the main string. Including the ones inside comments. I now realize it would have been fine to skip filtering out URLs in comments. Comments don’t make it into Eleventy’s final render (by default at least), so there would be no harm in processing the links inside them. But. I identified a problem and I thought I knew how to solve it. I enjoy solving problems, especially when I don’t have a deadline, so I went for it. By the end of the day, I had written what I later realized was my own implementation of JavaScript’s String.lastIndexOf() method (despite consulting the full list of JavaScript string methods earlier). I could generate a list of all (non-commented) URL strings and I thought I had almost figured out how to find out if they were inside tags. I realized I was struggling, so I gave up and went to bed.

What's left of my first attempt to filter out commented strings

// Returns an array of the start and end indices of every instance of
// str in content that is not within a comment.
// Currently works for xml, nunjucks, and javascript comments.
function findUncommented(str, content){
    const regex = RegExp(str, 'gd');
    let found_array = [];
    let search_array;
    let last_index = 0;
    // Regex.exec() provides a handy way to find all the instances of str
    while((search_array = regex.exec(content)) !== null) {
        const start_index = search_array.indices[0][0];
        const end_index = search_array.indices[0][1];
        const before_url = content.slice(last_index, start_index);
        const after_url = content.slice(end_index);
        let first_delim = null;
        let first_delim_index = 0;
        let last_delim = null;
        let last_delim_index = after_url.length;
        // STARTDELIMS and ENDDELIMS are constant arrays containing comment
        // delimiters for xml, javascript, and nunjucks defined earlier in
        // the file. They keep breaking my markdown, so I'm not including them
        // here.
        for (let delim of STARTDELIMS) {
            const dl_index = before_url.indexOf(delim);
            // we want the index of the delimiter closest to str
            if (dl_index && dl_index > first_delim_index) {
                first_delim_index = dl_index;
                first_delim = delim;
            }
        };
        // special case of single-line comments delimited by newlines
        if (first_delim == "\\") {
            last_delim = "\n";
            // the first newline after str should always be the
            // last delimiter in this case
            last_delim_index = after_url.indexOf("\n");
        } else {
            // similar to start delimiters
            for (let delim of ENDDELIMS) {
                const dl_index = after_url.indexOf(delim);
                if (dl_index && dl_index < last_delim_index) {
                    last_delim_index = dl_index;
                    last_delim = delim;
                }
            };
        }
        if (!first_delim || !last_delim) {
            found_array.push(start_index);
            found_array.push(end_index);
        }
    };
    return found_array;
};

Based on what I learned to arrive at isCommented() below, this doesn't actually work. I thought it did at one point though.

The most functional thing I produced

I remembered String.lastIndexOf() existed for this one.

// Returns true if the substring of str starting at start_index and
// ending at end_index is within a pair of comment delimiters.
// Currently works for xml, nunjucks, and javascript comments.
function isCommented(start_index, end_index, str){
    const before = str.slice(0, start_index);
    const after = str.slice(end_index);
    let first_dl_index = 0;
    let first_delim = null;
    // Javascript has comments that are partial delimited by newlines,
    // so we have to keep a special index for those.
    let nl_dl_index = 0;
    // STARTDELIMS and ENDDELIMS are constant arrays containing comment
    // delimiters for xml, javascript, and nunjucks defined earlier in
    // the file. They keep breaking my markdown, so I'm not including them
    // here.
    for (let delim of STARTDELIMS) {
        const dl_index = before.lastIndexOf(delim);
        if (delim == "//" && dl_index > nl_dl_index) {
            nl_dl_index = dl_index;
        } else if (dl_index > first_dl_index) {
            first_dl_index = dl_index;
            first_delim = delim;
        }
    };
    let matched_delim;
    // Pretty hacky way of avoiding finding the "//" in a url
    if (nl_dl_index > first_dl_index && before[nl_dl_index - 1] != ":") {
        // I don't think this ever happens with the content I'm using this with.
        // To generalize this function further, I'd need to figure out how to
        // actually tell when "//" delimits a comment.
        first_delim = "//";
        matched_delim = "\n";
    } else {
        // Matching delimiters should always work at this point.
        // Better to be safe than sorry though.
        matched_delim = ENDDELIMS[STARTDELIMS.indexOf(first_delim)] || null;
    }

    let last_delim = null;
    if (after.indexOf(matched_delim) > 0)
        last_delim = matched_delim;

    if (first_delim && last_delim)
        return true;
    return false;
};

This worked great, but it still left me needing to find and modify the HTML tags

The next morning came with some critical clarity: I realized why transforms were the wrong tool for what I was trying to do. Over breakfast I returned to the Eleventy documentation, looking for some way to insert my code further into the rendering process. I found the events page and the eleventy.after event. It wasn't exactly what I wanted, but it did the job. (I was hoping to avoid re-writing to the output directory. eleventy.after feels like only a minor improvement over a standalone script that modifies the output content, since it makes accessing the necessary page attributes easier.) All I had to do was modify my original HTML-parsing function slightly and slip it into eleventy.after. I had that working less than 15 minutes after sitting down at my desk. I'm working on reining in this tendency towards creating XY problems for myself. That day spent barking up the wrong tree was fun, but I’d rather I got around to working on this post a day earlier.

The actual solution in eleventy.config.js

	// make all external links open in a new tab
	eleventyConfig.on(
		"eleventy.after",
		async ({ dir, results, runMode, outputMode }) => {
			for (let result of results) {
				if ((result.outputPath || "").endsWith(".html")) {
					const html = parse(result.content)
					const links = html.getElementsByTagName("a")
					// make sure we found links to modify
					if (links && links.length > 0) {
						for (let link of links) {
							// avoid an error if the link somehow doesn't have an href
							const lhref = link.getAttribute("href") || ""
							// by convention, all external links will start with http(s): or mailto:
                            // TODO: make a regex already
							if (lhref.startsWith("http:") || lhref.startsWith("https:") || lhref.startsWith("mailto:")) {
								// avoid overwriting links with existing attributes
								if (!link.getAttribute("target"))
									link.setAttribute("target", "_blank")
								if (!link.getAttribute("rel"))
									link.setAttribute("rel", "noopener noreferrer")
							}
						}
						fs.writeFile(result.outputPath, html.toString(), (err) => {
							if(err)
								console.error(err)
						})
					}
				}
			}
		}
	)

So I have a functional blog and I have a post on it. If you’ve made it this far, dear reader: sorry, first of all. Secondly: you might be wondering what else you can expect from this site. I have plenty of wild ideas for things to do with a personal website, but I’m prioritizing building a career, so I’m going to implement mostly less wild ones. I’m trying to specialize in tools programming, and I have a lot of learning to do in the process. (I hardly know any C++ as of posting this, for example.) So I’m going to document that process, both to aid in my understanding and create a public record of my skills for potential employers. Maybe some people on a similar path to me will also benefit from my work here, but that would be a bonus. Outside of my chosen career path, I’m still very interested in web design and passionate about digital accessibility. I want to explore the limits of what’s possible with a static site, and I want to make this website as accessible as I can. Memorizing all of WCAG wouldn’t be much help in making the web better for everyone if I didn‘t have the knowledge and experience to implement and go beyond those guidelines. So expect writing about things I learn and projects I work on related to tools programming and web design in the near future. I also have interests unrelated to my career that I want to write about, but such writing would belong elsewhere. If you're still here and interested, watch this space. If you're really interested in anything I write here, you're welcome to send me an email about it. And if you're looking to hire a software developer, I'm available.