[{"content":"When you need to compare string in Python, a lot of us reach for .strip().lower() by default to pre-normalize the strings. But it turns out that .lower() doesn\u0026rsquo;t really work when you start thinking about non-ASCII characters. For example, the German letter \u0026ldquo;ß\u0026rdquo; (Eszett) should probably be equivalent to \u0026ldquo;ss\u0026rdquo; in a case-insensitive comparison, but .lower() doesn\u0026rsquo;t handle that correctly. So how can we do better? Python has a method .casefold() that is specifically designed for caseless matching of strings. It takes .lower() a step further by handling more complex cases, including those involving special characters and different alphabets:\nwords = [ \u0026#34;HELLO\u0026#34;, \u0026#34;Straße\u0026#34;, # German ß \u0026#34;İstanbul\u0026#34;, # Turkish dotted capital I \u0026#34;ΜΆΪΟΣ\u0026#34;, # Greek with accents ] for word in words: print(f\u0026#34;Original: {word}\u0026#34;) print(f\u0026#34;lower(): {word.lower()}\u0026#34;) print(f\u0026#34;casefold(): {word.casefold()}\u0026#34;) print(\u0026#34;-\u0026#34; * 30) # Equality comparison example a = \u0026#34;Straße\u0026#34; b = \u0026#34;strasse\u0026#34; print(\u0026#34;Comparison example:\u0026#34;) print(\u0026#34;Using lower():\u0026#34;, a.lower() == b.lower()) print(\u0026#34;Using casefold():\u0026#34;, a.casefold() == b.casefold()) Gives:\nOriginal: HELLO lower(): hello casefold(): hello ------------------------------ Original: Straße lower(): straße casefold(): strasse ------------------------------ Original: İstanbul lower(): i̇stanbul casefold(): i̇stanbul ------------------------------ Original: ΜΆΪΟΣ lower(): μάϊος casefold(): μάϊοσ ------------------------------ Comparison example: Using lower(): False Using casefold(): True So now you know :)\n","permalink":"https://til.dchan.cc/posts/01-22-2026/","summary":"\u003cp\u003eWhen you need to compare string in Python, a lot of us reach for \u003ccode\u003e.strip().lower()\u003c/code\u003e by default to pre-normalize the strings. But it turns out that .lower() doesn\u0026rsquo;t really work when you start thinking about non-ASCII characters. For example, the German letter \u0026ldquo;ß\u0026rdquo; (Eszett) should probably be equivalent to \u0026ldquo;ss\u0026rdquo; in a case-insensitive comparison, but \u003ccode\u003e.lower()\u003c/code\u003e doesn\u0026rsquo;t handle that correctly. So how can we do better? Python has a method \u003ccode\u003e.casefold()\u003c/code\u003e that is specifically designed for caseless matching of strings. It takes .lower() a step further by handling more complex cases, including those involving special characters and different alphabets:\u003c/p\u003e","title":"January 22, 2026 - Using .casefold() for Case-Insensitive String Comparison in Python"},{"content":"Recently, I\u0026rsquo;ve been working to migrate my own personal infrastructure off of Kubernetes, given that it has become quite expensive to run for just a single hosted website. For example, on Linode, the smallest cluster (3 nodes), along with a load balancer, and several disks, was approaching 50$ a month. Indeed, what did I actually use my cluster for? I used it for hosting one-off websites for a couple of days, and then tearing them down, I used it for my own personal website, and I used it for a host of CRON jobs that have to run every hour or every couple of days. The one-off websites I could easily migrate to vercel, and my personal website I migrated to Cloudflare Pages + static site generation (11ty), but the CRON jobs were a bit trickier. Luckily, I found Modal, a service for serverless compute which allows running compute on a schedule (with a bit of configuration).\nAll of my cron jobs were python scripts - so they already fit with Modal\u0026rsquo;s supported languages. The first thing to do was to install the Modal pip package, which is as simple as running pip install modal. Then sign up for an account on the modal website: https://modal.com/. You can then log in using the CLI with modal setup.\nThe next step was to create a model app. In the example, I\u0026rsquo;ll run a simple script which makes a GET request to a URL every hour.\nimport modal import requests app = modal.App(name=\u0026#34;example-cron-job\u0026#34;) @app.function() def main(): requests.get(\u0026#34;https://example.com\u0026#34;) Unfortunately, that\u0026rsquo;s not enough, since \u0026ldquo;requests\u0026rdquo; is not a built-in python module. To fix this, we need to create an image on which Modal can run:\nimport modal import requests app = modal.App(name=\u0026#34;example-cron-job\u0026#34;) image = modal.Image.debian_slim(python_version=\u0026#34;3.10\u0026#34;).pip_install(\u0026#34;requests\u0026#34;) @app.function(image=image) def main(): requests.get(\u0026#34;https://example.com\u0026#34;) Now that we have the image, we can setup the schedule:\nimport modal import requests app = modal.App(name=\u0026#34;example-cron-job\u0026#34;) image = modal.Image.debian_slim(python_version=\u0026#34;3.10\u0026#34;).pip_install(\u0026#34;requests\u0026#34;) @app.function(image=image, schedule=modal.Cron(\u0026#34;0 * * * *\u0026#34;)) def main(): requests.get(\u0026#34;https://example.com\u0026#34;) Here, the schedule is set to run every hour. The final step is to deploy the app:\nmodal deploy example-cron-job.py This will deploy the app to Modal\u0026rsquo;s infrastructure, and it will run every hour. The pricing is a bit hard to estimate, since jobs are chaged by CPU/Memory-seconds, however it comes with a pretty generous free tier of 30$/month, and in the 20-30 odd jobs that I\u0026rsquo;ve run so far, I haven\u0026rsquo;t even used 0.01$, so it\u0026rsquo;s quite cheap.\n","permalink":"https://til.dchan.cc/posts/07-29-2024/","summary":"\u003cp\u003eRecently, I\u0026rsquo;ve been working to migrate my own personal infrastructure off of Kubernetes, given that it has become quite\nexpensive to run for just a single hosted website. For example, on Linode, the smallest cluster (3 nodes), along with a\nload balancer, and several disks, was approaching 50$ a month. Indeed, what did I actually use my cluster for? I used it\nfor hosting one-off websites for a couple of days, and then tearing them down, I used it for my own personal website,\nand I used it for a host of CRON jobs that have to run every hour or every couple of days. The one-off websites I could\neasily migrate to vercel, and my personal website I migrated to Cloudflare Pages + static site generation (11ty), but\nthe CRON jobs were a bit trickier. Luckily, I found \u003ca href=\"https://modal.com/\"\u003eModal\u003c/a\u003e, a service for serverless compute which\nallows running compute on a schedule (with a bit of configuration).\u003c/p\u003e","title":"July 29, 2024 - Running CRON Jobs on Modal"},{"content":"I use Kubernetes (K8s) to manage both my personal website, and some client-facing projects. Recently, however, I\u0026rsquo;ve discovered something that is making my life a whole lot easier: K9s - a terminal-based UI (think htop) for managing Kubernetes clusters.\nSome of the awesome features:\nI can actually see all of my pods running at once, their CPU/memory usage, and what their internal IPs are I can tail logs using a UI, and I don\u0026rsquo;t have to remember all of the kubectl commands to do so I can see if things are dead, or died, at a glance. If you use Kubernetes, I highly recommend checking out K9s. It\u0026rsquo;s made my life a lot easier.\n(Look at that, I\u0026rsquo;m using a screenshot from their website. I\u0026rsquo;m not even sorry.) (Check out the logs view!) ","permalink":"https://til.dchan.cc/posts/06-11-2024/","summary":"\u003cp\u003eI use Kubernetes (K8s) to manage both my personal website, and some client-facing projects. Recently, however, I\u0026rsquo;ve discovered something that is making my life a whole lot easier: \u003ca href=\"https://k9scli.io/\"\u003eK9s\u003c/a\u003e - a terminal-based UI (think htop) for managing Kubernetes clusters.\u003c/p\u003e\n\u003cp\u003eSome of the awesome features:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eI can actually see all of my pods running at once, their CPU/memory usage, and what their internal IPs are\u003c/li\u003e\n\u003cli\u003eI can tail logs using a UI, and I don\u0026rsquo;t have to remember all of the kubectl commands to do so\u003c/li\u003e\n\u003cli\u003eI can see if things are dead, or died, at a glance.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf you use Kubernetes, I highly recommend checking out K9s. It\u0026rsquo;s made my life a lot easier.\u003c/p\u003e","title":"June 11, 2024 - Using K9s to Manage Kubernetes Clusters"},{"content":"To add a tag to an older commit, you can use the following command:\ngit tag -a {{ Tag Name }} {{ Commit Hash }} -m \u0026#34;Message here\u0026#34; For example, if you wanted to add a tag to the commit with the hash a1b2c3d4, you would use the following command:\ngit tag -a v1.0 a1b2c3d4 -m \u0026#34;Version 1.0\u0026#34; This will add a tag called v1.0 to the commit with the hash a1b2c3d4, and the message \u0026ldquo;Version 1.0\u0026rdquo; will be associated with the tag.\nYou can then push the tag to your remote repository using the following command:\ngit push origin {{ Tag Name }} ","permalink":"https://til.dchan.cc/posts/03-18-2024/","summary":"\u003cp\u003eTo add a tag to an older commit, you can use the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003egit tag -a \u003cspan style=\"color:#f92672\"\u003e{{\u003c/span\u003e Tag Name \u003cspan style=\"color:#f92672\"\u003e}}\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e{{\u003c/span\u003e Commit Hash \u003cspan style=\"color:#f92672\"\u003e}}\u003c/span\u003e -m \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Message here\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFor example, if you wanted to add a tag to the commit with the hash \u003ccode\u003ea1b2c3d4\u003c/code\u003e, you would use the following command:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003egit tag -a v1.0 a1b2c3d4 -m \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;Version 1.0\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will add a tag called \u003ccode\u003ev1.0\u003c/code\u003e to the commit with the hash \u003ccode\u003ea1b2c3d4\u003c/code\u003e, and the message \u0026ldquo;Version 1.0\u0026rdquo; will be associated with the tag.\u003c/p\u003e","title":"March 18, 2024 - Adding a tag to an older commit"},{"content":"Sometimes, I want to visualize some of my experimental results, or share them with others. One of the easiest ways to do this is using notion: which is capable of handling a variety of data types, and can be published to a relatively nice looking website, with minimal effort.\nStep 1: Creating a notion integration To do this, you first have to create a notion integration, which you can do here. Creating an integration will give you access to an API token, which we can use in the python script to interact with our notion database.\nStep 2: Create a database using Notion\u0026rsquo;s web interface Next, you\u0026rsquo;ll want to create a database using Notion\u0026rsquo;s web interface. This is pretty straightforward, and you can find instructions on how to do this here. Next, we\u0026rsquo;ll need the ID of the database, which you can find by navigating to the database, and copying the URL. The ID is the last part of the URL, and will look something like this: https://dmchan.notion.site/January-23-January-26-Paper-Review-24cb0a2d2c74438495321a85a8080331.\nAt the same time, we\u0026rsquo;ll need to give our integration access to the database. You can do this by navigating to the database, clicking on the \u0026ldquo;\u0026hellip;\u0026rdquo; button, scrolling to \u0026ldquo;Connections\u0026rdquo; and clicking \u0026ldquo;Add connection\u0026rdquo;. Then, you can select the integration you created in step 1, and give it access to the database.\nStep 3: Writing the python script Now that we have our integration set up, and our database created, we can write a python script to append content to the database. Here\u0026rsquo;s an example of a script that appends a new row to the database:\n# pip install notion-client from notion_client import Client import os client = Client(auth=os.environ[\u0026#39;NOTION_TOKEN\u0026#39;]) database_id = \u0026#34;24cb0a2d2c74438495321a85a8080331\u0026#34; new_row = { \u0026#34;parent\u0026#34;: {\u0026#34;database_id\u0026#34;: database_id}, \u0026#34;properties\u0026#34;: { \u0026#34;Name\u0026#34;: {\u0026#34;title\u0026#34;: [{\u0026#34;text\u0026#34;: {\u0026#34;content\u0026#34;: \u0026#34;New row\u0026#34;}}]}, \u0026#34;Tags\u0026#34;: {\u0026#34;multi_select\u0026#34;: [{\u0026#34;name\u0026#34;: \u0026#34;New tag\u0026#34;}]}, \u0026#34;Description\u0026#34;: {\u0026#34;rich_text\u0026#34;: [{\u0026#34;text\u0026#34;: {\u0026#34;content\u0026#34;: \u0026#34;This is a new row\u0026#34;}}]} }, # The children field is used to add content to the page, you can add a lot of different types of content here. # For example, you can add a paragraph, a heading, a list, or an image. # Unfortunately, the API doesn\u0026#39;t support adding images directly, instead, you\u0026#39;ll have to upload them to S3 or another # service, and then add a link to the image in the page content. \u0026#34;children\u0026#34;: { \u0026#34;object\u0026#34;: \u0026#34;block\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;paragraph\u0026#34;, \u0026#34;paragraph\u0026#34;: { \u0026#34;text\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: { \u0026#34;content\u0026#34;: \u0026#34;This is some page content!\u0026#34; } } ] } } } client.pages.create(**new_row) In this script, we first import the Client class from the notion_client package, and then create a new Client object using our API token. We then define a dictionary new_row that contains the properties of the new row we want to append to the database. In this case, we\u0026rsquo;re adding a new row with a \u0026ldquo;Name\u0026rdquo; and \u0026ldquo;Description\u0026rdquo; field. Finally, we call client.pages.create with the new_row dictionary as an argument, which appends the new row to the database.\nConclusion In this post, we went over how to create a notion integration, create a database using Notion\u0026rsquo;s web interface, and write a python script to append content to the database. This is a simple example, but you can use the notion API to do more complex operations, such as updating existing rows, or querying the database. You can find more information about the notion API here.\n","permalink":"https://til.dchan.cc/posts/01-29-2024/","summary":"\u003cp\u003eSometimes, I want to visualize some of my experimental results, or share them with others. One of the easiest ways to do\nthis is using notion: which is capable of handling a variety of data types, and can be published to a relatively nice\nlooking website, with minimal effort.\u003c/p\u003e\n\u003ch1 id=\"step-1-creating-a-notion-integration\"\u003eStep 1: Creating a notion integration\u003c/h1\u003e\n\u003cp\u003eTo do this, you first have to create a notion integration, which you can do \u003ca href=\"https://www.notion.so/my-integrations\"\u003ehere\u003c/a\u003e.\nCreating an integration will give you access to an API token, which we can use in the python script to interact with\nour notion database.\u003c/p\u003e","title":"January 29, 2024 - Appending content to a database using Notion's API"},{"content":"In lots of situations, we may need a unique identifier for an object, for example, when running a database transformation, we may want to create a unique key for each record, or when creating a database, we might want a unique key for each object in the database. In these situations, I\u0026rsquo;ve seen a lot of people use some variant of the following code:\nimport random import string def generate_random_unique_identifier(length=10): return \u0026#39;\u0026#39;.join(random.choice(string.ascii_letters) for _ in range(length)) I\u0026rsquo;ve certainly been guilty of this myself. However, there are a few problems with this approach:\nIt\u0026rsquo;s not guaranteed to be unique. It\u0026rsquo;s possible that two objects will be assigned the same identifier. It\u0026rsquo;s not very efficient. If we\u0026rsquo;re generating a lot of identifiers, we\u0026rsquo;re going to be wasting a lot of CPU cycles generating random strings. It depends on the random seed. If we\u0026rsquo;re using the same random seed, we\u0026rsquo;re going to get the same identifiers. This is fine for a lot of situations, but sometimes we want an identifier that we know is unique. In these situations, we can use a UUID. UUID stands for Universally Unique Identifier. It\u0026rsquo;s a 128-bit number that is guaranteed to be unique across space and time. It\u0026rsquo;s a standard that was developed by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE).\nThere are several variants of UUIDs, which all alter different parts of the UUID (usually the node identifier):\nUUID(1): Generates a unique number based on the current date/time and the MAC address of the computer. UUID(2): A variant of UUID1 for DCE Security. UUID(3/5): Generates a unique number based on hashing a \u0026ldquo;namespace\u0026rdquo; identifer, and a \u0026ldquo;name\u0026rdquo;. Version 3 uses MD5, and version 5 uses SHA-1. UUID(4): Generates a unique number based on random numbers. Future versions of UUIDs include:\nUUID(6): a field-compatible version of UUIDv1, reordered for improved DB locality UUID(7): a time-sortable version of UUID4 UUID(8): an RFC compatible format for experimenal or vendor-specific use cases. In python, generating a UUID is as simple as:\nimport uuid unique_identifier = uuid.uuid4() Using UUIDs generates identifiers which are guaranteed to be globally unique, and are also very efficient to generate, not only this, but it\u0026rsquo;s a lot easier to generate, and doesn\u0026rsquo;t depend on a source of randomness!\n","permalink":"https://til.dchan.cc/posts/01-24-2024/","summary":"\u003cp\u003eIn lots of situations, we may need a unique identifier for an object, for example, when running a database transformation,\nwe may want to create a unique key for each record, or when creating a database, we might want a unique key for each\nobject in the database. In these situations, I\u0026rsquo;ve seen a lot of people use some variant of the following code:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e random\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e string\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003edef\u003c/span\u003e \u003cspan style=\"color:#a6e22e\"\u003egenerate_random_unique_identifier\u003c/span\u003e(length\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e10\u003c/span\u003e):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#66d9ef\"\u003ereturn\u003c/span\u003e \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;\u0026#39;\u003c/span\u003e\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003ejoin(random\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003echoice(string\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003eascii_letters) \u003cspan style=\"color:#66d9ef\"\u003efor\u003c/span\u003e _ \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e range(length))\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eI\u0026rsquo;ve certainly been guilty of this myself. However, there are a few problems with this approach:\u003c/p\u003e","title":"January 24, 2024 - Using UUID for Unique Identifiers in Python"},{"content":"Sometimes you need to run an object detector as part of a larger system. One of the best tools for bringing in a pre-trained detector may be the detectron2 framework from Meta.\nInstallation To install detectron2, it\u0026rsquo;s as easy as a pip install:\npython -m pip install git+https://github.com/facebookresearch/detectron2.git Detectron2 API It\u0026rsquo;s amazingly easy to being to extract objects from images using their pre-trained models. First, you get a model from the model zoo:\nfrom detectron2 import model_zoo # get a model from the model zoo, in this case a RegNetY-4GF model trained on COCO model = model_zoo.get(\u0026#34;new_baselines/mask_rcnn_regnety_4gf_dds_FPN_400ep_LSJ.py\u0026#34;, trained=True) model = model.eval().to(\u0026#34;cuda\u0026#34;) # or \u0026#34;cpu\u0026#34;, if you don\u0026#39;t have a GPU (More models can be found in the model zoo)\nThen, you can use the model to detect objects in an image:\nfrom PIL import Image import numpy as np import torch # Load the frame to a numpy array frame = np.array(Image.open(\u0026#34;input.jpg\u0026#34;).convert(\u0026#34;RGB\u0026#34;)) # Convert to a torch tensor, and transpose to channels-first (HWC -\u0026gt; CHW) frame = torch.from_numpy(frame.transpose(2, 0, 1)).float() # Run the model on the frame outputs = model( [{\u0026#34;image\u0026#34;: frame, \u0026#34;height\u0026#34;: frame.shape[1], \u0026#34;width\u0026#34;: frame.shape[2], \u0026#34;file_name\u0026#34;: \u0026#34;input.jpg\u0026#34;}] ) We can then see our results:\n# get the boxes, object-masks, classes, and scores from the model output boxes = outputs[0][\u0026#34;instances\u0026#34;].pred_boxes.cpu().numpy() masks = outputs[0][\u0026#34;instances\u0026#34;].pred_masks.cpu().numpy() class_ids = outputs[0][\u0026#34;instances\u0026#34;].pred_classes.cpu().numpy() scores = outputs[0][\u0026#34;instances\u0026#34;].scores.cpu().numpy() # We can decode the class IDs using the COCO dataset from detectron2.data import MetadataCatalog coco_metadata = MetadataCatalog.get(\u0026#34;coco_2017_val\u0026#34;) class_names = [coco_metadata.thing_classes[i] for i in class_ids] In this way, we can easily extract objects from images using the detectron2 API, and use them in our own systems.\n","permalink":"https://til.dchan.cc/posts/01-23-2024/","summary":"\u003cp\u003eSometimes you need to run an object detector as part of a larger system. One of the best tools for bringing in a\npre-trained detector may be the \u003ca href=\"https://github.com/facebookresearch/detectron2\"\u003edetectron2\u003c/a\u003e framework from Meta.\u003c/p\u003e\n\u003ch2 id=\"installation\"\u003eInstallation\u003c/h2\u003e\n\u003cp\u003eTo install detectron2, it\u0026rsquo;s as easy as a pip install:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003epython -m pip install git+https://github.com/facebookresearch/detectron2.git\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"detectron2-api\"\u003eDetectron2 API\u003c/h2\u003e\n\u003cp\u003eIt\u0026rsquo;s amazingly easy to being to extract objects from images using their pre-trained models. First, you get a model from\nthe model zoo:\u003c/p\u003e","title":"January 23, 2024 - Detecting COCO Objects with Detectron2 API"},{"content":"It turns out that hosting a next.js application in a subdirectory isn\u0026rsquo;t as easy as it should be. There are two major issues that I\u0026rsquo;ve run into when doing this: (1) Handling links to local pages and (2) Handling links to static assets.\nThe right way to handle this is to set the basePath in the next.config.js file. This will cause the next.js router to prepend the basePath to all links:\n/** @type {import(\u0026#39;next\u0026#39;).NextConfig} */ const nextConfig = { ..., basePath: \u0026#39;/~davidchan/bair_staging\u0026#39;, }; module.exports = nextConfig; Unfortunately, there\u0026rsquo;s still some work to do, as this will only handle the routing. For any links to pages, we need to replace the a tag with the Link tag, which will correctly apply the basePath to the link:\n// From: \u0026lt;a href=\u0026#34;/pages\u0026#34;\u0026gt;\u0026lt;/a\u0026gt;; // To: import Link from \u0026#39;next/link\u0026#39;; \u0026lt;Link href=\u0026#34;/pages\u0026#34;\u0026gt;\u0026lt;/Link\u0026gt;; The biggest issue, however, is static assets, where the base path isn\u0026rsquo;t applied (even to components such as the component). There are several cited options which I\u0026rsquo;ve seen online including assetPrefix (which is ignored in modern next.js), the getRouter().basePath method (which doesn\u0026rsquo;t work in non-router components), and the getRuntimeConfig().basePath approach (where you specify the basePath in the next.config.js file and then access it via getRuntimeConfig(), which doesn\u0026rsquo;t work on client-side components). Instead, The best thing that I\u0026rsquo;ve found is to use addBasePath from next/dist/client/add-base-path.js:\n// From: \u0026lt;img src=\u0026#34;something.png\u0026#34; /\u0026gt;; // To: import { addBasePath } from \u0026#39;next/dist/client/add-base-path.js\u0026#39;; \u0026lt;img src={addBasePath(\u0026#39;something.png\u0026#39;)} /\u0026gt;; This works in both client-side and server-side components, and it\u0026rsquo;s the only thing that I\u0026rsquo;ve found that works for static assets. It\u0026rsquo;s a bit of a hack, but it\u0026rsquo;s the best that I\u0026rsquo;ve found so far.\n","permalink":"https://til.dchan.cc/posts/01-22-2024/","summary":"\u003cp\u003eIt turns out that hosting a next.js application in a subdirectory isn\u0026rsquo;t as easy as it should be. There are two major\nissues that I\u0026rsquo;ve run into when doing this: (1) Handling links to local pages and (2) Handling links to static assets.\u003c/p\u003e\n\u003cp\u003eThe right way to handle this is to set the basePath in the next.config.js file. This will cause the next.js router to\nprepend the basePath to all links:\u003c/p\u003e","title":"January 22, 2024 - Handling basePath in next.js client components"},{"content":"This isn\u0026rsquo;t really a TIL, but it turns out that Amazon provides 1TB of free egress from S3 buckets with a CloudFront CDN front-end (instead of the standard 100GB available for free with S3 alone). To get started, I followed this tutorial \u0026ndash; but it assumes a pretty high level of prior understanding of AWS, so if you\u0026rsquo;ve never used CloudFront, AWS, or S3 before, it might be best to google around for another tutorial.\nSo, what can you use this for? Well, turns out that this makes hosting static websites (with 1TB of bandwidth) really cheap. Almost free, in fact, since S3 has 5GB of free storage. Combine this with a simple Lambda for a contact form, and a domain name (Route53 DNS is free for CloudFront distributions), and you\u0026rsquo;ve got a pretty cheap website running on AWS.\n","permalink":"https://til.dchan.cc/posts/01-19-2024/","summary":"\u003cp\u003eThis isn\u0026rsquo;t really a TIL, but it turns out that Amazon provides 1TB of free egress from S3 buckets with a CloudFront CDN\nfront-end (instead of the standard 100GB available for free with S3 alone). To get started, I followed \u003ca href=\"https://aws.amazon.com/cloudfront/getting-started/S3/\"\u003ethis tutorial\u003c/a\u003e \u0026ndash; but it assumes a pretty high level of prior understanding of AWS, so if you\u0026rsquo;ve never used CloudFront, AWS, or S3\nbefore, it might be best to google around for another tutorial.\u003c/p\u003e","title":"January 19, 2024 - Cloudfront Egress is pretty cheap!"},{"content":"So, for a long time, it wasn\u0026rsquo;t possible to configure the AWS CLI to use a custom endpoint url by default. Since I usually use Wasabi for my S3 storage, I had to use the --endpoint-url flag every time I wanted to use the CLI, pretty annoying, right? Well, turns out at some point Amazon added a set of useful configs to their CLI. To configure the CLI to use a custom endpoint url, you can add the following to your ~/.aws/config file:\n[default] # Profile -- you can use any profile you want ignore_configure_endpoint_urls = true endpoint_url = \u0026lt;your endpoint url\u0026gt; ","permalink":"https://til.dchan.cc/posts/01-11-2024/","summary":"\u003cp\u003eSo, for a long time, it wasn\u0026rsquo;t possible to configure the AWS CLI to use a custom endpoint url by default. Since I usually\nuse Wasabi for my S3 storage, I had to use the \u003ccode\u003e--endpoint-url\u003c/code\u003e flag every time I wanted to use the CLI, pretty annoying, right? Well,\nturns out at some point Amazon added a set of useful configs to their CLI. To configure the CLI to use a custom endpoint url, you can\nadd the following to your \u003ccode\u003e~/.aws/config\u003c/code\u003e file:\u003c/p\u003e","title":"January 11, 2024 - Configuring AWS CLI Endpoint URL"},{"content":"If you\u0026rsquo;re just interested in the code, it\u0026rsquo;s super easy to use pingouin to do this:\nimport numpy as np import pingouin as pg data = np.random.normal(size=(100, 3)) output = pg.multivariate_normality(data, alpha=.05) print(output.hz, output.pval, output.normal) Sometimes, it\u0026rsquo;s useful to know when a sample looks a lot like a normal distribution. In a single dimension, we can use the function from scipy scipy.stats.normaltest to test whether a sample is normal. This test combines tests from D\u0026rsquo;Agostino and Pearson which measure the skew and kurtosis of the sample, and report when those differ from a similar normal population. Unfortunately, this test isn\u0026rsquo;t immediately generalizable to multiple dimensions, which makes a new test necessary. One example is a Henze-Zirkler test, which is based on a non-negative functional \\(D\\) which measures the distance between two distribution functions, and has the property that \\(D(N_d(0, I_d), Q) = 0\\) if and only if \\(Q\\) is a multivariate normal distribution with identity covariance matrix. In practice, the Henze-zirkler test computes a weighted integral of the difference between the empirical characteristic function (ECF) and it\u0026rsquo;s pointwise normal approximation (in the limit).\nThe test statistic can be calculated as:\n$$ T_{\\alpha} = n\\left(4I_{S_{singular}} + D_{n,\\alpha}I_{S_{nonsingular}}\\right) $$\nwhere $S$ is the sample covariance matrix, I is an indicator function, and:\n$$ D_{n,\\alpha} = \\int_{R^d} \\left| \\psi_n(t) - \\exp(-\\frac{1}{2} ||t||^2) \\right|^2 \\phi_{\\alpha}(t) dt $$\nwith $\\psi_n$ being the empirical characteristic function, $\\phi_{\\alpha}$ being a weighting function with parameter $\\alpha$:\n$$ \\phi_{\\alpha}(t) = (2\\pi\\alpha^2)^{-m/2}\\exp(-\\frac{||t||^2}{2\\alpha^2}), t \\sim R^m $$\nNotice that because \\(D_{n,\\alpha}\\) is undefined with S is singular, we set it to 4 (the maximum value) in that case. Because \\(T_{\\alpha}\\) is approximately distributed as log-normal, we can use the log-normal distribution to compute the null hypothesis probability.\n","permalink":"https://til.dchan.cc/posts/05-12-2023/","summary":"\u003cp\u003eIf you\u0026rsquo;re just interested in the code, it\u0026rsquo;s super easy to use pingouin to do this:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e numpy \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e np\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e pingouin \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e pg\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003edata \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e np\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003erandom\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003enormal(size\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e(\u003cspan style=\"color:#ae81ff\"\u003e100\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003e))\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eoutput \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e pg\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003emultivariate_normality(data, alpha\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e.05\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eprint(output\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003ehz, output\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003epval, output\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003enormal)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSometimes, it\u0026rsquo;s useful to know when a sample looks a lot like a normal distribution. In a single dimension, we can use\nthe function from scipy \u003ccode\u003escipy.stats.normaltest\u003c/code\u003e to test whether a sample is normal. This test combines tests from\nD\u0026rsquo;Agostino and Pearson which measure the skew and kurtosis of the sample, and report when those differ from a similar\nnormal population. Unfortunately, this test isn\u0026rsquo;t immediately generalizable to multiple dimensions, which makes a new\ntest necessary. One example is a Henze-Zirkler test, which is based on a non-negative functional \\(D\\) which measures the\ndistance between two distribution functions, and has the property that \\(D(N_d(0, I_d), Q) = 0\\) if and only if \\(Q\\) is a\nmultivariate normal distribution with identity covariance matrix. In practice, the Henze-zirkler test computes a\nweighted integral of the difference between the empirical characteristic function (ECF) and it\u0026rsquo;s pointwise normal\napproximation (in the limit).\u003c/p\u003e","title":"May 12, 2023 - Testing for Multivariate Normality with a Henze-Zirkler Test (In Python)"},{"content":"I was working on the Advent of Code (day 7) today, which I\u0026rsquo;m using to learn a bit more about Rust, and while I was able to quickly solve the problem with python (thanks to the fact that python allows references to everything everywhere), I had a hard time solving the problem with Rust, thanks to the inability to keep around references to a \u0026ldquo;parent\u0026rdquo; in the tree. This inconvenience is by design: Rust tries to ensure memory safety by forbidding you from doing things that might potentially be unsafe.\nStonewalled by a lot of error[E0106]: missing lifetime specifier errors, I decided to dig a bit deeper into this problem \u0026ndash; could we adjust the code in such a way that we could keep track of all of the things that we needed, but still be \u0026ldquo;safe\u0026rdquo; from a Rust perspective.\nSo, what we\u0026rsquo;d like to have is the following:\nFirst, a filesystem, which contains other file-system objects, a list of files, and their sizes in the local filesystem, and a reference to their parent, so we can implement the cd .. command. We need to be able to add files while we\u0026rsquo;re traversing the tree (hence why everything here is mutable).\nstruct FileSystem { name: String, directories: Vec\u0026lt;\u0026amp;mut FileSystem\u0026gt;, files: Vec\u0026lt;(String, usize)\u0026gt;, parent: Option\u0026lt;\u0026amp;mut FileSystem\u0026gt;, } First, the constructor. It seemed initially like this was pretty easy. We just initialized the objects.\nimpl FileSystem { // Constructor pub fn new(name: String, parent: Option\u0026lt;\u0026amp;mut FileSystem\u0026gt;) -\u0026gt; FileSystem { return FileSystem { name: name, directories: Vec::new(), files: Vec::new(), parent: parent, }; } Next, several key commands:\nimpl FileSystem { // Compute the size of the current directory pub fn size(\u0026amp;self) -\u0026gt; usize { let mut size = 0; for (_name, s) in self.files.iter() { size += s; } for \u0026amp;mut directory in self.directories { size += directory.size(); } return size; } // Add a file pub fn add_file(\u0026amp;mut self, name: \u0026amp;str, size: usize) { self.files.push((file_name.to_string(), size)); } // Add a directory pub fn add_directory(\u0026amp;mut self, name: \u0026amp;str) { self.directories.push(FileSystem::new(file_name.to_string()), self); } // Get a directory pub fn get_directory(\u0026amp;mut self, name: \u0026amp;str) -\u0026gt; Option\u0026lt;\u0026amp;mut FileSystem\u0026gt; { for \u0026amp;mut directory in self.directories { if directory.name == name { return Some(\u0026amp;mut directory); } } return None; } } If we stitch this together into a simple program:\nfn main() { // Create a root file system let mut root = FileSystem::new(\u0026#34;/\u0026#34;.to_string(), None); // Add a file root.add_file(\u0026#34;README.md\u0026#34;, 1024); // Add a directory root.add_directory(\u0026#34;src\u0026#34;); // Add a file to the src directory let mut src = root.get_directory(\u0026#34;src\u0026#34;).unwrap(); src.add_file(\u0026#34;main.rs\u0026#34;, 2048); // Compute the size of the root directory println!(\u0026#34;Size of root directory: {}\u0026#34;, root.size()); } We can compile this, and get our first set of errors!\nerror[E0106]: missing lifetime specifier --\u0026gt; ./part-1-tree.rs:8:22 | 8 | directories: Vec\u0026lt;\u0026amp;mut FileSystem\u0026gt;, | ^ expected named lifetime parameter | help: consider introducing a named lifetime parameter | 6 ~ struct FileSystem\u0026lt;\u0026#39;a\u0026gt; { 7 | name: String, 8 ~ directories: Vec\u0026lt;\u0026amp;\u0026#39;a mut FileSystem\u0026gt;, | error[E0106]: missing lifetime specifier --\u0026gt; ./part-1-tree.rs:10:20 | 10 | parent: Option\u0026lt;\u0026amp;mut FileSystem\u0026gt;, | ^ expected named lifetime parameter | help: consider introducing a named lifetime parameter | 6 ~ struct FileSystem\u0026lt;\u0026#39;a\u0026gt; { 7 | name: String, 8 | directories: Vec\u0026lt;\u0026amp;mut FileSystem\u0026gt;, 9 | files: Vec\u0026lt;(String, usize)\u0026gt;, 10 ~ parent: Option\u0026lt;\u0026amp;\u0026#39;a mut FileSystem\u0026gt;, | error: aborting due to 2 previous errors This is rust telling us that it\u0026rsquo;s possible that the references we\u0026rsquo;re keeping to the directories within a FileSystem could go out of scope without us knowing about it. Similarly, the references to the parents could go out of scope as well.\nThe directories are the easy part of this. Since we don\u0026rsquo;t necessarily care about symlinks (i.e. linking to other directories in the filesystem), we can update the code so that each FileSystem owns it\u0026rsquo;s own directories. To do this, we can update the code to go from directories: Vec\u0026lt;\u0026amp;mut FileSystem\u0026gt; to directories: Vec\u0026lt;FileSystem\u0026gt; and update the code accordingly:\nstruct FileSystem { name: String, directories: Vec\u0026lt;FileSystem\u0026gt;, files: Vec\u0026lt;(String, usize)\u0026gt;, } impl FileSystem { pub fn new(name: String) -\u0026gt; FileSystem { return FileSystem { name: name, directories: Vec::new(), files: Vec::new(), }; } // Add a directory pub fn add_directory(\u0026amp;mut self, name: \u0026amp;str) { self.directories.push(FileSystem::new(name.to_string())); } // Get a directory pub fn get_directory(\u0026amp;mut self, name: \u0026amp;str) -\u0026gt; Option\u0026lt;\u0026amp;mut FileSystem\u0026gt; { for directory in self.directories.iter_mut() { if directory.name == name { return Some(directory); } } return None; } } This gives us a working filesystem object, but we\u0026rsquo;ve lost two key components. The parent pointer, and the ability to link two directories together. Unfortunately, the best option in rust is to keep track of the parent pointer in any traversal we\u0026rsquo;re doing. For example, we could keep track of a stack of directories, and each time we do a traversal, we could push and pop elements onto the stack. A simple implementation of this might look like the following:\nfn main() { // Read the input file from the command line let filename = env::args().nth(1).expect(\u0026#34;No filename given\u0026#34;); let contents = fs::read_to_string(filename).expect(\u0026#34;Could not read file\u0026#34;); // Create a root file system let mut root = FileSystem::new(\u0026#34;/\u0026#34;.to_string()); let mut traversal_stack: Vec\u0026lt;\u0026amp;mut FileSystem\u0026gt; = Vec::new(); traversal_stack.push(\u0026amp;mut root); // Parse the input file for line in contents.lines() { print_path(\u0026amp;traversal_stack); // If the line is a command, execute it (Commands are of the form $ \u0026lt;command\u0026gt; args) if line.starts_with(\u0026#34;$\u0026#34;) { let command = line.split_whitespace().nth(1).unwrap(); // Print the command println!(\u0026#34;Command: {}\u0026#34;, command); match command { \u0026#34;mkdir\u0026#34; =\u0026gt; { let name = line.split_whitespace().nth(2).unwrap(); traversal_stack.last_mut().unwrap().add_directory(name); } \u0026#34;cd\u0026#34; =\u0026gt; { let name = line.split_whitespace().nth(2).unwrap(); if name == \u0026#34;..\u0026#34; { if traversal_stack.len() \u0026gt; 1 { traversal_stack.pop(); } } else if name == \u0026#34;/\u0026#34; { // Unwind the traversal stack to the root while traversal_stack.len() \u0026gt; 1 { traversal_stack.pop(); } } else { // Oh No! let new_dir = traversal_stack.last_mut().unwrap().get_directory(name); if let Some(new_dir) = new_dir { traversal_stack.push(new_dir); } } } \u0026#34;touch\u0026#34; =\u0026gt; { let name = line.split_whitespace().nth(2).unwrap(); let size = line .split_whitespace() .nth(3) .unwrap() .parse::\u0026lt;usize\u0026gt;() .unwrap(); traversal_stack.last_mut().unwrap().add_file(name, size); } \u0026#34;ls\u0026#34; =\u0026gt; { let directory = traversal_stack.last_mut().unwrap(); for directory in directory.directories.iter() { println!(\u0026#34;{} {}\u0026#34;, directory.name, directory.size()); } for (name, size) in directory.files.iter() { println!(\u0026#34;{} {}\u0026#34;, name, size); } } _ =\u0026gt; { println!(\u0026#34;Unknown command: {}\u0026#34;, command); } } } } // Print the size of the root directory println!(\u0026#34;Size of root directory: {}\u0026#34;, root.size()); } But wait! We get a new error!!!\nerror[E0502]: cannot borrow `traversal_stack` as mutable because it is also borrowed as immutable --\u0026gt; ./part-1-tree.rs:127:37 | 105 | let new_dir = traversal_stack | --------------- immutable borrow occurs here ... 127 | let directory = traversal_stack.last_mut().unwrap(); | ^^^^^^^^^^^^^^^ | | | mutable borrow occurs here | immutable borrow later used here Without the changing directory (i.e. everything the traversal stack is useful for\u0026hellip;), this code works, but the error illustrates a key problem with how Rust handles graph structures. In order to update the structure, you have to borrow the whole traversal stack (as either mutable or immutable), so we borrow it as mutable, but we can\u0026rsquo;t release it, since the mutable pointer to the new directory has the same lifecycle as the traversal stack! Even though these two objects aren\u0026rsquo;t related, Rust doesn\u0026rsquo;t know that the traversal stack and the directory are completely unrelated. Thus, we need to think of a new way to solve this problem.\nHow I originally solved this problem for the day of the advent of code was by using integer indices into arrays to keep track of the objects. This solves the problem, since we\u0026rsquo;ve basically implemented pointers on our own, instead of using Rust\u0026rsquo;s borrowing mechanisms, but it goes completely against the ethos of Rust, since we\u0026rsquo;ve just created a situation where unsafe dereferences of integers (our makeshift pointers), could introduce unknown effects. Yes, for advent of code this doesn\u0026rsquo;t really matter\u0026hellip; but what if we were working on something more complicated?\n","permalink":"https://til.dchan.cc/posts/12-07-2022/","summary":"\u003cp\u003eI was working on the Advent of Code (day 7) today, which I\u0026rsquo;m using to learn a bit more about Rust, and while I was able\nto quickly solve the problem with python (thanks to the fact that python allows references to everything everywhere), I\nhad a hard time solving the problem with Rust, thanks to the inability to keep around references to a \u0026ldquo;parent\u0026rdquo; in the\ntree. This inconvenience is by design: Rust tries to ensure memory safety by forbidding you from doing things that might\npotentially be unsafe.\u003c/p\u003e","title":"December 07, 2022 - How to make a tree with a parent pointer in Rust"},{"content":"Kubernetes uses base64 encoded secrets, but it\u0026rsquo;s important to make sure that when encoding the secrets, you don\u0026rsquo;t include any extra newlines - tldr; use echo -n instead of echo\n# Works \u0026gt; echo -n \u0026#34;my secret\u0026#34; | base64 bXkgc2VjcmV0 # Doesn\u0026#39;t work \u0026gt; echo \u0026#34;my secret\u0026#34; | base64 bXkgc2VjcmV0Cg== Otherwise, you could be in for a world of pain :)\n","permalink":"https://til.dchan.cc/posts/12-02-2022-0/","summary":"\u003cp\u003eKubernetes uses base64 encoded secrets, but it\u0026rsquo;s important to make sure that when encoding the secrets, you don\u0026rsquo;t\ninclude any extra newlines - tldr; \u003cstrong\u003euse echo -n instead of echo\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Works\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u0026gt; echo -n \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;my secret\u0026#34;\u003c/span\u003e | base64\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ebXkgc2VjcmV0\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# Doesn\u0026#39;t work\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u0026gt; echo \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;my secret\u0026#34;\u003c/span\u003e | base64\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ebXkgc2VjcmV0Cg\u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOtherwise, you could be in for a world of pain :)\u003c/p\u003e","title":"December 02, 2022 (2) - Encoding secrets for Kubernetes"},{"content":"So, I had a project recently where I have a bunch of API keys, but I didn\u0026rsquo;t want to accidentally commit them to a repo (since that would be bad\u0026hellip;). But I also wanted the convencience of just in-lining the code, and not have to worry about sourcing environment variables every time I ran the code (since this would inevitably cause annoyances). To solve this, I found a handy tool - python-dotenv, a package which handles all of the complicated bits.\nTo use python-dotenv you only need to import the library:\nimport os from dotenv import load_dotenv load_dotenv() ... my_api_key = os.getenv(\u0026#39;MY_COOL_API_KEY\u0026#39;) The library will automatically load any variables from a .env file present in the current directory if that file exits, otherwise it will fall back to real environment variables, meaning that when you put your app into production, you\u0026rsquo;ll still be able to configure your code (with 0 code changes).\n","permalink":"https://til.dchan.cc/posts/12-02-2022/","summary":"\u003cp\u003eSo, I had a project recently where I have a bunch of API keys, but I didn\u0026rsquo;t want to accidentally commit them to a repo\n(since that would be bad\u0026hellip;). But I also wanted the convencience of just in-lining the code, and not have to worry about\nsourcing environment variables every time I ran the code (since this would inevitably cause annoyances). To solve this,\nI found a handy tool - \u003cstrong\u003epython-dotenv\u003c/strong\u003e, a package which handles all of the complicated bits.\u003c/p\u003e","title":"December 02, 2022 - How to use python-dotenv to easily manage env vars"},{"content":"So, we have a package, VDTK which we\u0026rsquo;re planning to release in a new major version - there\u0026rsquo;s only one problem\u0026hellip; We depend on several packages which do not publish builds to the PyPi repository. In our pyproject.toml file, they are specified as:\ndependencies = [ \u0026#34;...\u0026#34;, \u0026#34;clip @ git+https://github.com/openai/CLIP.git\u0026#34;, \u0026#34;mauve-text @ git+https://github.com/krishnap25/mauve.git\u0026#34;, \u0026#34;en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.4.1/en_core_web_lg-3.4.1-py3-none-any.whl\u0026#34;, ] Unfortunately, when you upload a package built with dependencies like this, you get the error:\nERROR HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/ Invalid value for requires_dist. Error: Can\u0026#39;t have direct dependency: \u0026#39;clip @ git+https://github.com/openai/CLIP.git\u0026#39; This is rather unfortunate, but there are several possible options to get around this.\nOption 1: Vendor the code\nLicense permitting, we could copy the code into our own repository, update the pointers in our repo, and import the code that way. Unfortunately, this is time consuming, and difficult to maintain and update, since we become responsible for merging any upstream changes directly into our codebase and pushing a new release for any patches released in the upstream. Additionally, it means that we take on the security burden of any upstream code, which is usually untenable for small-scale projects.\nOption 2: Break the UX\nAnother option is to remove the ability for users to install our package from pypi, and force users to install the package with pip install vdtk @ git+https://github.com/cannylab/vdtk. This isn\u0026rsquo;t a bad option, but it makes the package less discoverable for other users, and forces any packages which are built on our code to have the same breaking change long-term.\nOption 3: Publish new pypi packages with the code\nThe final option is, license permitting, to publish our own pypi versions of the packages. This is almost as bad as option 1, but it makes life a little bit easier, since we\u0026rsquo;re mostly just running the build scripts on behalf of the package maintainers. It\u0026rsquo;s a bit of a PITA, but is probably one of the cleanst options. That being said, it\u0026rsquo;s not exactly polite to publish the code of something that you don\u0026rsquo;t maintain - but hey, if the package maintainer won\u0026rsquo;t do it themselves, then it would be better if somebody does it.\n","permalink":"https://til.dchan.cc/posts/11-29-2022/","summary":"\u003cp\u003eSo, we have a package, \u003ca href=\"https://github.com/CannyLab/vdtk\"\u003eVDTK\u003c/a\u003e which we\u0026rsquo;re planning to release in a new major version -\nthere\u0026rsquo;s only one problem\u0026hellip; We depend on several packages which do not publish builds to the PyPi repository. In our\n\u003ccode\u003epyproject.toml\u003c/code\u003e file, they are specified as:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-toml\" data-lang=\"toml\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#a6e22e\"\u003edependencies\u003c/span\u003e = [\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;...\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;clip @ git+https://github.com/openai/CLIP.git\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;mauve-text @ git+https://github.com/krishnap25/mauve.git\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e    \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.4.1/en_core_web_lg-3.4.1-py3-none-any.whl\u0026#34;\u003c/span\u003e,\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e]\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eUnfortunately, when you upload a package built with dependencies like this, you get the error:\u003c/p\u003e","title":"November 29, 2022 - PyPI Doesn't Allow Git Repo Dependencies"},{"content":"This TIL site is powered by Hugo, but it\u0026rsquo;s currently a lot of effort to create the title and name for one of these posts\u0026hellip; It would be great if we could have a single command line tool which does this! Each post right now has the following header:\n# post-header.yaml title: \u0026#39;November 18, 2022 - Creating the perfect archetype in Hugo\u0026#39; date: 2022-11-18T16:17:02-08:00 showToc: true TocOpen: false draft: false hidemeta: false comments: false disableShare: false disableHLJS: false hideSummary: false searchHidden: false ShowReadingTime: true ShowBreadCrumbs: true ShowPostNavLinks: true ShowWordCount: true ShowRssButtonInSectionTermList: true UseHugoToc: true Most of this information is super easy to deal with - everything after showToc is static, so our template can just write those directly. This means we have only to deal with the title and the date.\nThe date is also pretty simple. Hugo provides a built-in short code for managing the date: {{ .Date }}, so we can easily add that.\nThe difficult detail is the title. We want it to be today\u0026rsquo;s date, followed by the text, and ideally, we create this with as few commands as possible. Turns out that Hugo just isn\u0026rsquo;t powerful enough to do this. So we\u0026rsquo;ll turn to bash. We can create a zsh function:\n# .zshrc til () { # Change the working directory to this project cd ~/Projects/til # Get the current date in MM-DD-YYYY form date=$(date +\u0026#34;%m-%d-%Y\u0026#34;) # Get the current date in plain text date_long=$(date +\u0026#34;%B %d, %Y\u0026#34;) # Get the post name post_name=$1 # Check if the file exists, and while they exist, add a postfix # to the file name index=0 title_date=$date while [ -f \u0026#34;content/posts/$title_date.md\u0026#34; ]; do title_date=\u0026#34;$date-$index\u0026#34; index=$((index + 1)) done hugo new -k posts posts/${title_date}.md # Edit the title line to add the post title # If the index is 0, then we don\u0026#39;t need to add the postfix if [ $index -eq 0 ]; then index=$((index + 1)) sed -i \u0026#39;\u0026#39; \u0026#34;s/TITLE_TEMPLATE/$date_long - ${post_name}/g\u0026#34; \u0026#34;content/posts/$title_date.md\u0026#34; else sed -i \u0026#39;\u0026#39; \u0026#34;s/TITLE_TEMPLATE/$date_long ($index) - ${post_name}/g\u0026#34; \u0026#34;content/posts/$title_date.md\u0026#34; fi # Open the TIL document in VSCODE code content/posts/${title_date}.md } The final archetype looks like:\n\u0026lt;!-- posts.md --\u0026gt; --- title: \u0026#39;$date_long ($index) - ${post_name}\u0026#39; date: { { .Date } } showToc: true TocOpen: false draft: false hidemeta: false comments: false disableShare: false disableHLJS: false hideSummary: false searchHidden: false ShowReadingTime: true ShowBreadCrumbs: true ShowPostNavLinks: true ShowWordCount: true ShowRssButtonInSectionTermList: true UseHugoToc: true --- And running `til \u0026ldquo;My awesome title\u0026rdquo; from anywhere on my laptop publishes the TIL!\n","permalink":"https://til.dchan.cc/posts/11-18-2022/","summary":"\u003cp\u003eThis TIL site is powered by Hugo, but it\u0026rsquo;s currently a lot of effort to create the title and name for one of these\nposts\u0026hellip; It would be great if we could have a single command line tool which does this! Each post right now has the\nfollowing header:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e# post-header.yaml\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003etitle\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;November 18, 2022 - Creating the perfect archetype in Hugo\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003edate\u003c/span\u003e: \u003cspan style=\"color:#e6db74\"\u003e2022-11-18T16:17:02\u003c/span\u003e\u003cspan style=\"color:#ae81ff\"\u003e-08\u003c/span\u003e:\u003cspan style=\"color:#ae81ff\"\u003e00\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eshowToc\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eTocOpen\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003edraft\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003ehidemeta\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003ecomments\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003edisableShare\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003edisableHLJS\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003ehideSummary\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003esearchHidden\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eShowReadingTime\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eShowBreadCrumbs\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eShowPostNavLinks\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eShowWordCount\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eShowRssButtonInSectionTermList\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eUseHugoToc\u003c/span\u003e: \u003cspan style=\"color:#66d9ef\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eMost of this information is super easy to deal with - everything after showToc is static, so our template can just\nwrite those directly. This means we have only to deal with the title and the date.\u003c/p\u003e","title":"November 18, 2022 - Creating the perfect archetype in Hugo"},{"content":"Getting anything running on Kubernetes is a bit of a challenge, but today I was working on deploying MySQL so I could migrate my Ghost blog from v4.x to v5.x. This means creating a mysql instance, a user (for ghost) and any other data that we need to run the deployment. The first thing that we need to do is add a secret which will define the root user password and the password for ghost user:\n# secrets.yaml apiVersion: v1 kind: Secret metadata: name: mysql-secrets type: Opaque data: database__connection__password: {{ openssl rand -hex 20 | base64 }} database__user__password: {{ openssl rand -hex 20 | base64 }} Next, we need a persistent volume store which will retain the databse data if the pod dies. Since I\u0026rsquo;m using Linode\u0026rsquo;s LKE, we can use linode\u0026rsquo;s block storage:\n# volume.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pv-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: linode-block-storage-retain Note above that we generate a random password for both of the users. Next, we need to create the deployment and service for the mysql instance. We choose mysql version 8 (since it\u0026rsquo;s the latest), and we attach the persistent volume store here.\n# deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: selector: matchLabels: app: mysql strategy: type: Recreate replicas: 1 template: metadata: labels: app: mysql spec: containers: - image: mysql:8 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secrets key: database__connection__password - name: MYSQL_USER_PASSWORD valueFrom: secretKeyRef: name: mysql-secrets key: database__user__password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pv-claim Finally, we create the service:\n# service.yaml apiVersion: v1 kind: Service metadata: name: mysql spec: ports: - port: 3306 selector: app: mysql Assuming we put all of these yaml files in the same director (which I called mysql), we can now deploy the pods with:\nkubectl apply -f ./mysql Finally, we\u0026rsquo;ll add the user to the database. We can SSH to the created pods by using:\nkubectl get pods kubectl exec --stdin --tty $POD_NAME -- /bin/bash We can load up mysql and create the user with:\nmysql -p\u0026#34;$MYSQL_ROOT_PASSWORD\u0026#34; -e \u0026#34;CREATE USER \u0026#39;user\u0026#39;@\u0026#39;%\u0026#39; IDENTIFIED BY \u0026#39;${MYSQL_USER_PASSWORD}\u0026#39;;\u0026#34; We\u0026rsquo;d also want to run any commands here to set up tables, or grant user access.\nAnd now we\u0026rsquo;re done! We have a user of name user with password stored in the kubernetes secret database__user__password and the mysql database running at host: mysql.default.svc.cluster.local.\n","permalink":"https://til.dchan.cc/posts/11-17-2022/","summary":"\u003cp\u003eGetting anything running on Kubernetes is a bit of a challenge, but today I was working on deploying MySQL so I could\nmigrate my Ghost blog from v4.x to v5.x. This means creating a mysql instance, a user (for ghost) and any other data\nthat we need to run the deployment. The first thing that we need to do is add a secret which will define the root user\npassword and the password for ghost user:\u003c/p\u003e","title":"November 17, 2022 - Running Single-Server MySQL on Kubernetes"}]