[{"content":"Learning to Learn #Learning is a life long process, and learning computer science is no different. This set of articles serve to be an overview to some of the areas of academics and life that I get the most questions about. Think of them as something between a frequently asked questions and an introduction to a topic.\nWhat order should I tackle these? The articles are grouped by topic, so you can start reading through them according to the topics you are interested in. However, if you have no idea of where to start, consider reading the articles in \u0026ldquo;Meta\u0026rdquo; first. These articles will better equip you to read and get something out of the other articles\nAlong the way you will find activities added to each post formatted like this:\nThis is an activity designed to help you master the content in this section Lastly, if you find the articles insufficient in some way, please email me, and I\u0026rsquo;ll try to make updates to fill out the missing information\nMeta # Journaling, Task Management, and Time Tracking: three foundational skills for everything else Reading: how to get the most out of technical documents Studying: how to go about learning new topics Intake: how to stay up to date on current topics Writing: how to summarize topics for others to learn Traveling: how to prepare to travel Process # Linux how to get started with learning and using Linux Software Development how to approach writing a software project using Linux Software Patterns how to approach writing a software project using Linux HPC how to write software for High Performance Computing Experiments how to write empirical computer science experiments Software Teams: how to lead and be an effective member of a software team Programming Languages # C++ how to learn C++, a low-level systems language that helps you get the most out of your hardware Python how to learn Python, a high-level programming language aimed at developer productivity Tools # CMake the defacto C++ build system GDB the most commonly used open source debugger for natively compiled languages. MPI the defacto parallel processing framework in C++ for HPC Most Recent Updates #","date":null,"permalink":"https://robertu94.github.io/learning/","section":"","summary":"\u003ch1 id=\"learning-to-learn\" class=\"relative group\"\u003eLearning to Learn \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#learning-to-learn\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h1\u003e\u003cp\u003eLearning is a life long process, and learning computer science is no different.  This set of articles serve to be an overview to some of the areas of academics and life that I get the most questions about. Think of them as something between a frequently asked questions and an introduction to a topic.\u003c/p\u003e","title":""},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/learning-to-learn/","section":"Tags","summary":"","title":"Learning to Learn"},{"content":"Travel can be an opportunity to see and explore new cultures. I\u0026rsquo;ve had the chance to do a lot of traveling with work. Here are some of the things I\u0026rsquo;ve learned along the way.\nPacking #What to Pack #What you bring on a trip can meaningfully affect the comfort of your trip. If you are anything like me, the default thing to do is to bring more than you need, resulting in additional hassle and weight. However, I\u0026rsquo;ve also forgotten important items that I later regretted not having. The tips in this area reflect how I pack for all kinds of trips.\nSometimes you want slightly different items depending on the context. For example:\nWater bottle for flight travel (optimizing for temperature) vs backpacking (optimizing for weight and capacity). iPad vs Laptop for personal/work travel and weight/capability/battery life. Cold Weather Gear, depending on the expected low and weather, trade off with weight and bulk Shoes: dress, rain/water, workout, running, walking. I seldom pack more than 2 types of shoes, given the space they take up Bag Size according to what you need to carry. I have (3L, 20L, 28L, and 45L) bags. Sometimes I carry a smaller bag in a larger bag. When purchasing a bag, consider the sizes of carry-on and personal items allowed by the airlines you most frequently fly. For toiletries, prefer re-packable toiletries containers to both reduce waste and better choose products that work for you regardless of size. Sometimes there are special tools for this. For example toothpaste, there are refill valves.\nI recommend using a master packing list and update it regularly. As you consider the items on your list, measure and weigh your items to understand what size backpack you need and where you can optimize weight and bulk. I also Include in the packing list a to-do list for critical leaving items, such as taking out the trash, feeding pets, etc. Here is a copy of my packing list:\nHere are a few of the things that have strong opinions on\nBackpack: Aer Travel Pack 3 Small. It fits most personal item size capacities. I like its admin panel, hip stack and harness system, large clam shell compartment, and water bottle compartment. Sling: Tomtoc Aviator-T33 3.5L. It is perfectly sized to carry all the things I need quick access to while on a plane, and most of my every day carry items when I\u0026rsquo;m at my destination. Toiletries: Aer Dopp Kit 3. sits up on its own, and has clear places for everything I need. I have a small plastic bag for when they need to be remove to go through TSA. Notebook: Moleskin Medium Hardcover Dotted. Works great a bullet journal and the hardcover protects the pages. It fits perfectly in the sling. Pens: 0.3mm Pilot G-2. Smooth writing experience in a fairly cheap pen. I carry a few different colors. Pencil: 0.5mm Pentel Orenz - a very smooth automatic pencil with great lead protection. Headphones: Airpods Pro 2. Great noise cancellation, tiny form factor, find my support. Water Bottle: 1.4L Nalgene, Grayl Purifier Bottle, or Yeti 20oz Tumbler depending on if I need capacity, purification or insulation. Smartwatch: Garmin Forerunner 55. I prefer its hardware controls, long battery life, and fitness tracking features. Entertainment: Steamdeck vs XBox Controller. Steamdeck is better for longer trips, but an iPad+Xbox controller is perfectly suitable for shorter trips. The XBox controller has better 3rd party support than the Steamcontroller. Deck of Cards: requires no power, limited space, and supports a ton of games Sweeter a Moreno Wool Sweater. Surprisingly warm for its space and weight. My current one we got used. In slightly colder weather, I have a Port Authority light jacket which works well and is fairly breathable. Pants. I prefer pants with good pockets and a professional look. I like the BSA polyester microfiber uniform switchback pants for travel as they provide a good mix of comfort, breathability, and pockets. They also double as workout shorts/pants. I also like the Columbia Sportswear Men\u0026rsquo;s Vast Canyon Omni-Heat Infinity Softshell Pants as a good business casual trouser that packs down well. For colder weather, I like Duluth Firehose Cargo Pants. Shoes: Reebok Nano Shoes great for functional fitness and general use or Solomon X Ultra 5 Mid GORETEX boots waterproof and great support for longer walks. Charging USB-C based cables for everything charging related. This allows de-duplicating between devices. Packing for Weather #When choosing clothing, I recommend:\nTraveling in layers, including those for cold, wind, and water resistance. For example a long underwear, scarf, hat, and gloves can really add some temperature resistance in cold weather. Plan for worse weather than expected. I typically plan for comfort ±10ºF from the forecasted temperatures Plan for getting soaked because being soaking wet can be highly uncomfortable. I bring a spare pair of dry socks and a base layer in case of heavy rain For example, Moreno wool and materials matter for temperature regulation and Oder suppression, but modern synthetics can be unparalleled in terms of both water resistance and breathability. Packing to stay healthy #Exercise can be harder to work in depending on your routine and needed resources. Calisthenics exercises can be a good but time-consuming option. Also, some exercise programs, such as CrossFit, have options for drop-in passes to use a different gym while traveling. I personally carry a heavy resistance band as a way to get a bit more out of my calisthenics while not taking too much space when Gyms are not available.\nAnother harder aspect when traveling is maintaining a healthy food/diet. Eating healthy while traveling is extremely difficult. Restaurants (at least in the US) tend to serve fattier foods than if you carefully controlled the preparation of these foods at home. Additionally, you may not have access to the same selection of foods you\u0026rsquo;d have at home, making certain choices harder. While traveling, I often find that a larger breakfast is a good decision because healthier versions of these foods can more easily be found while traveling. I also prefer to get certain items from a grocery store over restaurants that can be prepared using at most hot water, which can be produced using the coffee machine in most hotel rooms by omitting the coffee.\nPacking Your Bag #A few tips regarding the practical packing of the bag itself. I Prefer a large single compartment combined with packing cubes to provide organization. They also pack more quickly as a “modular” go-bag system. I have specialized pouches for tech, admin (e.g., notepad, pens), weather (e.g., rain, cold), and toiletries, which can be useful to locate and pack frequently used items quickly. Within these cubes, learning to do a proper Ranger Roll inside the packing cube can increase capacity. Additionally, when you have large hollow things like shoes pack inside of them to save volume.\nUnderstand how the loading of the pack also affects comfort. Keep heavy items near your hips and back, where they are most supported can make carrying them all day easier.\nFor any bag over a few liters, a hip belt, chest strap, and load lifter straps can substantially improve the comfort of a backpack\nSingle shoulder straps are a recipe for shoulder pain with a long day if they have more than a few pounds.\nThe \u0026ldquo;10 essentials\u0026rdquo; from scouts when camping are still good advice when packing for the outdoors:\npocket knife sun protection Flashlight first aid kit rain gear Matches or fire starters food water bottle map and compass Extra clothing Offline Digital Packing #Internet on planes and in some remote areas can be expensive or limited. Sometimes it is helpful to disconnect entirely, but when you still need or want to do something, having a plan to do work or relax without the internet can be done with some planning. Here are some of the programs I use offline:\nSyncing Files: Git for code, Drafts for notes, Box/Google Drive for certain other files Google Docs, Slides, and Sheets have an offline mode that can be accessed and edited when using Google Chrome. Documentation: Zotero for academic papers and references, and Zeal for software documentation. Music and Entertainment: FinAmp is offline streaming for Jellyfin, Steaming Apps offer offline storage at premium tiers, podcast apps have had download features forever, RetroArch, even on iOS, provides an easy way to play classic games in an offline way. A tablet + an Xbox controller is a compact way to game on the go, especially if you already bring the iPad for something else. Odd Files Maintain a “scratch” folder distinct from /tmp, where scratch persists across reboots In addition to preparing to be offline, using a VPN like Tailscale to give access to files and resources in your home network rather than carrying everything with you.\nWhile not a way to prepare for being offline per se, preparing your device with full disk encryption is an important way to protect files if a device is lost or stolen. Likewise, if using a VPN, you should configure key expiration for devices that travel with you.\nAir Travel #If you don\u0026rsquo;t do it a lot, Air Travel can be stressful. Here are a few aspects that made it easier for me.\nPre Air Travel #There is a lot you can do to make your trip easier by preparing in advance:\nSome things you can do signifigantly in advance of travel. This begins with how you pick your flights. Unless you travel a lot (12+ times a year), points and status don\u0026rsquo;t matter, but perks like free checked bags, free rebooking within 24 hours, and priority service are handy if you do. Next, services like pre-check and Global Entry save time, but not tons, depending on your airport, mostly from not having to remove shoes and large electronics, and maybe slightly shorter lines. I found it saved the most time at airports with notorious security lines like Chicago and Atlanta. On a related note, don\u0026rsquo;t underestimate the utility of a passport as it is required for international travel and take significant time to obtain, but even domestically as it proves both citizenship and identity providing utility outside the travel context.\nWhen it comes time to checkin 24 hours before the flight, always check in where possible with the app ahead of time, and avoid lines at the airport\nWhen packing, I prefer single (personal item only) bag travel for air travel. I like a backpack because it\u0026rsquo;s less likely to be taken from you if overhead bin space is short; it also doesn\u0026rsquo;t get stuck in the sand/mud\nDay of Travel #I Always assume that international flights will be slightly cold, and plan accordingly with a lightweight sweater and pants. Likewise you want easy access to your pockets, so travel with pants with thigh pockets, as these can be easier to access than waist pockets while seated. From a digital perspective, having a travel focus mode for your phone that makes the apps, tickets, navigation tools, and itineraries you need quick at hand.\nWhen going through security, having an empty bag compartment/sling for your pocket contents makes going through security easier. Likewise, non-moist foods can be brought through domestic airport security and can be cheaper and tastier than airport food. However, If you need to buy food because you couldn\u0026rsquo;t pack it, mobile order food in airports for shorter lines because the checkout is a significant bottleneck.\nOnce you are on the flight there are a few things you can do to make the travel nicer:\nWork? This requires some planning in advance, but flights are a good time to work on heads down tasks that don\u0026rsquo;t require online research. See the section on digital packing for advice. Sleep? Depending on the timezone difference, sleep is a good options, but practically for me, mostly a lost cause, but resting is possible. Noise cancellation headphones and an eye mask help me to relax on flights Entertainment? Considering downloading books, movies, and podcasts in advance. Native phone games or even emulators such as RetroArch for games you have a legal right to own can also be a way to pass the time. Lastly, if or when your flight gets canceled or substantially delayed, gate agents can typically do more than you can on an app. Specifically, even if your airline doesn\u0026rsquo;t have a flight going to your destination, if another airline does, you may be able to get the gate agent for an endorsed ticket on another airline at no cost to you.\nRail and Bus Travel #While often not practical for inter-city travel in the United States, major metropolitan areas often have metro systems, which can be an efficient form of transport to common destinations especially if you traveled to the location by air and do not have a car. When considering intercity rail, consider the time of day. In the US, some metro systems become less safe at night. As for how to pay for the train, increasingly, metro systems have a pay-by-phone or credit card option over traditional card or ticket-based systems. These can be more reliable and convenient. Lastly, while traditional map apps like Google Maps increasingly have reliable schedules, apps like Citymapper do a better job of helping navigate intracity trains and multimodal connections.\nBackpacking #For me, backpacking is a relaxing way to disconnect, but because it puts you out in the wilderness, it requires appropriate preparation. Some unique or special considerations when backpacking include:\nOf utmost importance is to manage your water supply and have a way to obtain more purified water. You can go days without food, but only at most three days without water depending on conditions. For purifying water, purification tablets are lightweight, but taste bad. I prefer a pump or a sawyer squeeze. There is also a Squeeze Mini, but its back pressure is much higher, making it more painful to use. Weight matters of second importance for comfort when you carry everything on your back. Carefully weigh the weight vs. the value of each item. Lighter gear can dramatically improve your experience carrying if you backpack often, but it can be very expensive and often less comfortable when used than heavier options. Staying dry matters of third importance. Moisture decreases the effectiveness of thermal properties of equipment and increases the odds of blisters. Some nice to haves:\nGood shoes/boots to prevent blisters and provide support to prevent sprains/injuries cannot be understated. A good pair of boots will protect your arches with a shank and will protect your ankles from twisting on uneven ground. However, they need to be broken in by wearing them for 20-30 miles before being relied on for extensive trips. Full leather boots take longer to break in. Consider waterproof or resistant boots for wet environments. Consider packing a spare set of lighter “camp” shoes if practical to have a more comfortable way to relax in camp and allow your feet to breathe. Hiking poles take a lot of pressure off your knees when going up or down hill, and can be reused as part of your shelter. Plan as needed for wildlife, including things like bear bags for food and toiletries. For cooking, freezer bag cooking, which can be prepared in a single bag using only hot water, yields less mess and cleanup. For sleeping, consider hammock camping where allowed and practical as a lightweight and comfortable way to sleep. Road Travel #Road trips can be significantly cheaper than flights if time allows.\nFor the driver Plan for hands-free access to maps, media, and phone where needed. Operating a phone while driving is nearly as dangerous as driving intoxicated so avoid it as much as possible. Part of avoiding needing access to maps is understanding the road system. In the United States, understand the typical numbering of interstates for navigation. There are patterns in the numbers eg two digit primary interstates ending in a 5 run north-south, and 0 run east-west, and increase in value to the north-east. Three-digit interstates have numbers that indicate which interstate this interstate connects. Apps for travel. In addition to Google Maps, I also like the app iExit, which shows amenities for each exit on an interstate in the order of travel.\nAdditionally, pack the car to have easy access to food (Bowls, Paper Towels, Cooler, and Dry Goods) and admin items (e.g., Maps, Passports, Reservations, etc.) that can reached by a passenger can reduce the need for extended stops.\nHosting #While not about traveling yourself, it\u0026rsquo;s worth considering the experience of people traveling to you. This list is largely adapted from Deviant Ollam\u0026rsquo;s insightful video on the subject.\nBlackout curtains and access to light controls can make it easier to sleep when needed, which might be midday after a very long international trip. Power outlets near the bed to charge electronic devices. Wi-Fi connection instructions that are easy to find. Perhaps in the form of a QR code, but text is also important because not all devices have cameras. Toiletries basket and towels in case they forgot theirs Additional blankets in case they get cold. A place to put a suitcase at waist level improves ergonomics I\u0026rsquo;ll add to this a place to work:\na (ideally sit/stand) desk to work with an appropriate task lighting A monitor with HDMI or USB-C with power delivery connection A comfortable office chair To provide a place to work and be organized when needed.\nWhere to learn more #A few other general resources that I find generally helpful in travel:\nGear Recommendations: Packhacker General Travel Tips: Away Together w/ Nik and Allie Changelog # 3 May 2026 initial version ","date":"3 May 2026","permalink":"https://robertu94.github.io/learning/travel/","section":"","summary":"\u003cp\u003eTravel can be an opportunity to see and explore new cultures. I\u0026rsquo;ve had the chance to do a lot of traveling with work. Here are some of the things I\u0026rsquo;ve learned along the way.\u003c/p\u003e","title":"Learning to Learn: Travel"},{"content":"","date":null,"permalink":"https://robertu94.github.io/","section":"Systems++","summary":"","title":"Systems++"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/travel/","section":"Tags","summary":"","title":"Travel"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/display/","section":"Tags","summary":"","title":"Display"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/ipad/","section":"Tags","summary":"","title":"Ipad"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/plasma/","section":"Tags","summary":"","title":"Plasma"},{"content":"","date":null,"permalink":"https://robertu94.github.io/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"When traveling it is sometimes helpful to have access to a second monitor. An iPad does a passable job for this when your traveling and just need a little more screen real-estate.\nkrfb-virtualmonitor --name WirelessDisplay --resolution 2420x1668 --password \u0026#34;$PASSWORD\u0026#34; --port 5900 Then connect to the VNC server using an iPad app client.\n","date":"30 October 2025","permalink":"https://robertu94.github.io/posts/2025-10-30-ipad-monitor/","section":"Posts","summary":"\u003cp\u003eWhen traveling it is sometimes helpful to have access to a second monitor.\nAn iPad does a passable job for this when your traveling and just need a little more screen real-estate.\u003c/p\u003e","title":"Using my iPad as a Second Monitor with KDE Plasma"},{"content":"I am an Assistant Computer Scientist at Argonne National Laboratory under Franck Cappello. My particular research interests are lossy compression, science data and evaluation for AI, distributed systems, and systems programming.\nSome specific topics I have worked on include: AI for Science focusing on identifying, leveraging, and evaluating the semantic content of scientific datasets. Lossy compression for scientific data with an interest in interface design, optimization to preserve user qualities of interest, accelerator implementations, the use of compilers in compression pipelines, prediction and modeling of compression ratio performance. Data management including novel uses of checkpointing and lineages of data states beyond fault recovery to accelerate computation as applied to AI inference, model training, and other areas.\nAcademics #You can find my full CV here.\nI completed a postdoc at Argonne National Laboratory in October 2024 focusing on AI, lossy compression, and data management. I earned my PhD at Clemson University studying Computer Science in December 2021. I am previously worked with the Data Intensive Computing Environments (DICE) Lab and Fault Tolerant HPC Lab (FTHPC) studying error bounded lossy compression under my adviors Dr. Amy Apon, Dr. Jon C. Calhoun, and Dr. Franck Cappello.\nMy publications can be found on my Google Scholar page here.\nSome highlights of my career include:\n(2024-Ongoing) I accepted a position as Assistant Computer Scientist at Argonne National Laboratory (2024-Ongoing) I am the co-Lead for the Data team in AuroraGPT. Argonne\u0026rsquo;s effort to develop a LLM specialized for scientific use cases (2019-Ongoing) I developed LibPressio a library to abstract the differences between lossless and lossy compressors and their configurations. (2024) I was awarded the Postdoctoral Performance Award in Basic Research (2023) I was awarded an Laboratory Directed Research and Development grant along with Viktor Niktin on the topic of streaming lossy compression for tomographic data. (2022) I started a postdoctoral position at Argonne National Laboratory under Franck Cappello, Sheng Di, and Bogdan Nicolae (2019) I was selected for the US Department of Energy Office of Science Graduate Student Research Program award to study at Argonne National Laboratory under Dr. Franck Cappello. (2018) I taught a Fall section of Introduction to Operating Systems (CPSC 3220). (2017 - 2020) I was awarded the National Research Traineeship in Resilient Infrastructure and Environmental Systems fellowship. (2015 - 2018) I was competed in Clemson\u0026rsquo;s CU Cyber cyber security team. (2014 - 2016) I was the Vice President of the Clemson Chapter of the Association for Computing Machinery. I also completed on the programming team from 2013-2016. Outside Computer Science #I am a Christian. I am an Eagle Scout. I recently started doing crossfit. I enjoy spending time outdoors; I especially enjoy biking and backpacking. I also enjoy playing strategic board games with friends.\nContact Me #The best way to reach me is via email\n","date":"17 June 2025","permalink":"https://robertu94.github.io/about/","section":"Systems++","summary":"\u003cp\u003eI am an Assistant Computer Scientist at Argonne National Laboratory under Franck Cappello.\nMy particular \u003cstrong\u003eresearch interests\u003c/strong\u003e are lossy compression, science data and evaluation for AI, distributed systems, and systems programming.\u003c/p\u003e","title":"About Me"},{"content":"Guides #A set of \u0026ldquo;how to\u0026rdquo; rather than \u0026ldquo;how to think about\u0026rdquo;\nspack setup guides C++ Programming Other Guides #","date":null,"permalink":"https://robertu94.github.io/guides/","section":"","summary":"\u003ch1 id=\"guides\" class=\"relative group\"\u003eGuides \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#guides\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h1\u003e\u003cp\u003eA set of \u0026ldquo;how to\u0026rdquo; rather than \u0026ldquo;how to think about\u0026rdquo;\u003c/p\u003e","title":""},{"content":"Spack # Updating LibPressio with Spack General Spack Installation Guide ","date":null,"permalink":"https://robertu94.github.io/guides/spack/","section":"","summary":"\u003ch1 id=\"spack\" class=\"relative group\"\u003eSpack \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#spack\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h1\u003e\u003cul\u003e\n\u003cli\u003e\n      \n    \u003ca href=\"https://robertu94.github.io/guides/spack/updating_libpressio_with_spack/\"\u003eUpdating LibPressio with Spack\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\n      \n    \u003ca href=\"https://robertu94.github.io/guides/spack/spack/\"\u003eGeneral Spack Installation Guide\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":""},{"content":"tl;dr #module load spack git clone https://github.com/robertu94/spack_packages robertu94_packages spack repo add ./robertu94_packages What makes this machine special? # the team at NERSC works hard to maintain an up to date and customized spack environment. Unless you have a need for something special use theirs For the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nChangelog # 2024-07-12 created this document ","date":"11 July 2024","permalink":"https://robertu94.github.io/guides/spack/perlmutter/","section":"","summary":"configure spack for NERSC Perlmutter","title":"Spack on Perlmutter"},{"content":"For Windows #choco.exe install podman podman machine init podman machine start For MacOS ## install brew /bin/bash -c \u0026#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\u0026#34; # use brew to install podman and qemu brew install podman qemu # configure podman to have a linux vm podman machine init podman machine start You may also want to run an x86_64 container in which case you will need the --platform linux/amd64 flag\nFor Linux #Ubuntu #apt update apt install podman Fedora #dnf install podman ","date":"1 January 2024","permalink":"https://robertu94.github.io/guides/podman/","section":"","summary":"how to install the podman container engine on various operating systems","title":"Install podman on Windows, MacOS, and Linux"},{"content":"I run a small tower server in my house to provide services that I use for software development, recently CentOS-8-Stream reached end of life. Since this server distribution doesn\u0026rsquo;t support in-place updates, updating requires reinstalling with a new version of the CentOS-9-Stream. This post documents the challenges that I hit along the way.\nChallenge 1: My server is older than all of my monitors #I purchased my server used during the pandemic. While my server has serial line and VGA outputs, however after my wife and I moved, none of our remaining monitors support either of these inputs. So, how am I going to connect to the server to see the settings in the BIOS in order to perform the installation. IPMI to the rescue. For those who don\u0026rsquo;t know, IPMI is a remote management protocol that connects to a special piece of hardware installed in many enterprise grade servers. In my Dell server, IPMI is banded iDRAC, but works the same way.\nFirst we need to install and configure ipmi on the server\n# interactive shell for IPMI sudo dnf install ipmitool openipmi # interactive shell for IPMI sudo ipmitool shell Next we need to configure ipmi for remote access on the sever itself\nipmi_channelid=1 ipmi_userid=1 ipmi_ipaddr=192.168.1.120 sudo ipmitool lan set $ipmi_channelid ipsrc static sudo ipmitool lan set $ipmi_channelid ipaddr $ipmi_ipaddr sudo ipmitool lan set $ipmi_channelid netmask 255.255.255.0 sudo ipmitool lan set $ipmi_channelid defgw 192.168.1.1 sudo ipmitool lan set $ipmi_channelid arp respond on sudo ipmitool lan set $ipmi_channelid access on sudo ipmitool lan print 1 sudo ipmitool user set name $ipmi_userid root sudo ipmitool user set password $ipmi_userid #privilege 4 is admin privileges need for things like power control and console access sudo ipmitool channel setaccess $ipmi_channelid $ipmi_userid link=on ipmi=on callin=on privilege=4 sudo ipmitool user enable $ipmi_userid sudo ipmitool user list sudo ipmitool lan set $ipmi_channelid auth USER MD5 Now we can acesss ipmi from the over network:\nipmitool -I lanplus -H $ipmi_ipaddr -U $ipmiuser shell # floppy is the boot device used for any USB on this system chassis bootdev floppy chassis power cycle sol activate However, my linux ISO doesn\u0026rsquo;t configure the serial port for installation so we need to modify grub so that the serial console is configured for the install, and we can then access the actual installation media over VNC\n#Installing from a serial console, start a VNC server to run the grahical install # add this to the end of the kernel command line console=tty0 console=ttyS0,115200 inst.vnc And finally we can install the server over VNC which works as normal.\nChallenge 2: Re-configuring Tailscale #I use tailscale for remote management of my server and so things like backups work outside of the house. When I did this, I set this up last, but in hindsight it should have been one of the first ones to redo because I want all the services to be accessible over tailscale which only happens if the tailscale0 tunnel exists when the services bind. However, I don\u0026rsquo;t want to connect yet until I re-setup the home directories when I configure ZFS\ndnf config-manager --add-repo https://pkgs.tailscale.com/stable/centos/9/tailscale.repo dnf install tailscale sysetmctl enable --now tailscaled tailscale up # edit machine name in tailscale so \u0026#34;magic-dns\u0026#34; works firewall-cmd --zone=public --add-interface=tailscale0 systemctl restart sshd Challenge 3: Re-configuring Samba #Samba comes next after Tailscale because the permissions and packages for samba need to be installed so that the ZFS user shares are made public when ZFS comes back on line.\ndnf install -y samba samba-client systemctl enable --now smb nmb #add and enable the samba user (different from the Linux user) firewall-cmd --permanent --zone=public --add-service=samba firewall-cmd --zone=public --add-service=samba setsebool -P samba_enable_home_dirs # setup of users appears below Challenge 4: Reconnecting my ZFS array and recreating users #ZFS is where all the user data is stored. Now I just need to reload it.\ndnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval \u0026#34;%{dist}\u0026#34;).noarcupdate dnf install -y epel-release dnf install -y kernel-devel dnf install -y zfs modprobe zfs zpool import tank # edit /etc/passwd to point the user home directories on /tank usermod $USER -d /tank/$USER # re-add the other service account users with the old UID/GIDs groupadd plex -g 1003 useradd plex -u 1003 -g plex #finish setting up SAMBA users because I think this needs the home directories smbpasswd -a $USER smbpasswd -e $USER Challenge 5: Reconfiguring LibVirt #I have a few odd VMs on the server and libvirt/cockpit machine is how I manage them.\ndnf install -y libvirt cockpit-machines systemctl enable --now libvirt.socket Challenge 6: Restart services #The remaining services I use rely on podman. Because I am lazy, I have a folder called docker in my home directory. In this are sub directories that contain scripts called up.sh that download the container and re-setup the service and set appropriate firewall rules.\ndnf install vim podman-docker systemctl start podman.socket for i in docker/* do cd $i ./up.sh cd ../.. done Hope you learned something!\n","date":"23 November 2023","permalink":"https://robertu94.github.io/posts/2023-11-23-ipmi-install/","section":"Posts","summary":"\u003cp\u003eI run a small tower server in my house to provide services that I use for\nsoftware development, recently CentOS-8-Stream reached end of life.\nSince this server distribution doesn\u0026rsquo;t support in-place updates, updating\nrequires reinstalling with a new version of the CentOS-9-Stream.  This post\ndocuments the challenges that I hit along the way.\u003c/p\u003e","title":"Installing My Server Fully Remotely"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/server-administration/","section":"Tags","summary":"","title":"Server Administration"},{"content":"Dependencies matter a lot in terms of who can easily install and use your software. However there are trade-offs in what features are provided by each version and the availability of these versions. This presents an opinionated take on what these trade-offs are so one can know what can be widely supported.\nMajor Linux Distros # Distro Standard EoL Extended CentOS-8 compatible December 2021* 2029 CentOS-9 stream May 2027* 2032 CentOS-10 stream December 2030* 2038 Ubuntu 22.04 April 2027 Ubuntu 24.04 April 2029 Ubuntu LTS releases generally get 5 years of standard support Fedora releases generally get ~1 year of support * Third party vendors such as AlmaLinux support these for much longer 2029 for CentOS-8, 2032 for CentOS-9, 2038 for Cento-10. Tooling Versions # Tool Spack 1.1 Minimum Sandard EoL Ubuntu 22.04 Ubuntu 24.04 CentOS 9 Stream CentOS 10 Stream Fedora CentOS 8 EOL Current Extended gcc 15.2.0 11.3 11.4.0 13.2.0 11.3 14.2.1 15.0.1 8.5.0 clang 21.1.4 16.0 14.0.0 18.1.3 16.0 20.1.2 20.1.2 15.0.0 cmake 3.31.9@, 4.1.2 3.20 3.22.1 3.28.3 3.20 3.30.5 3.31.6 3.20 python3 3.14.0 3.9 3.10.0 3.12.3 3.9,3.11 3.12.10 3.13.3 3.6-3.9 julia 1.11.7 n/a n/a n/a n/a n/a 1.11.0-rc3 n/a cargo 1.86.0 1.66.1 1.66.1 1.75.0 1.61.1 1.85.0 1.86.0 1.66.1 swig 4.1.1 3.0.12 4.0 4.2.0 3.0.12 4.3.0 $ 4.3.0 3.0.12 nvcc * 13.0.2 11.5 11.5.0 12.0.140 n/a * n/a * n/a * n/a * numpy 2.3.4 1.21.5 1.21.5 1.26.4 1.20.1 1.26.4 2.2.4 1.14.3 pytorch 2.9.0 2.5.1 n/a n/a n/a 2.8.0 2.5.1 n/a mpich 4.3.2 4.1.1 n/a n/a 4.1.1 4.1.2 4.2.2 4.1.1 openmpi 5.0.8 4.1.1 n/a n/a 4.1.1 5.0.2 5.0.8 4.1.1 mpi4py 4.1.1 3.1.3 n/a n/a 3.1.3 4.0.1 4.0.3 3.0.3 #1 has known issues and upstream recommends avoiding using this version * CentOS, and Fedora do not package CUDA themselves, but instead rely on Nvidia to provide the package which provides the newest version. $ CentOS provides some packages in the code-ready builder or EPEL repositories @ Spack defaults to preferred versions\nLanguage Features #C++ #For more information consult cppreference.com C++ is divided into language and library features. Generally Language features are implemented before library features. While Clang originally led for compliance, increasingly GCC is getting newer features sooner.\nC++14 #You can safely assume that C++14 is supported on all major distributions.\nCompiler C++14 (language full) C++14 (language 90%) Missing C++14 (library full) C++14 (library 90%) Missing GCC/libstdc++ 5 5 N/A 10 5 partial support for null forward iterators (N3644) Clang/libc++ 3.4 3.4 N/A 3.4 3.4 N/A C++17 #C++17 language features are supported on all major distributions.\nC++17 library features require very new compilers to implement fully and are not widely available on LTS systems. The most common things to lag behind being parallel parallel algorithms, so-called special math functions used in statistics, and OS assisted features like hardware interference size. In many cases these can be \u0026ldquo;poly filled\u0026rdquo;\nCompiler C++17 (language full) C++17 (language 90%) Missing C++17 (library full) C++17 (library 90%) Missing GCC/libstdc++ 7 7 N/A 12 9 \u0026ldquo;elementary string conversions\u0026rdquo; (P0067R5), Clang/libc++ 4 4 N/A No 17 parallel algoirthms, hardware interference size, special math functions C++20 #C++20 language features are not not fully implemented in even the newest compilers. The biggest holdout is modules and features like consteval, but compilers that implement 90% of the features are present in recent LTSes.\nC++20 library features are more sparsely implemented, but are now fully implemented in GCC 14 and 90% implemented as of clang 18 starting to be available in \u0026ldquo;cutting edge distros\u0026rdquo; such as Fedora and some LTSes.\nCompiler C++20 (language full) C++20 (language 90%) Missing C++20 (library full) C++20 (library 90%) Missing GCC/libstdc++ 11* 10 Only partial support for Modules add in 11, using enum, operator\u0026lt;=\u0026gt;(...)=default, consteval 14 11 calendar, text formatting, atomics Clang/libc++ No 17 Modules, Coroutines, Non-type template parameters No 18 atomics, source location, operator\u0026lt;=\u0026gt; * full support for modules is the lone holdout.\nC++23 #Bleeding edge compilers now have 90% support for C++23 language features.\nLibrary features are not widely implemented in compilers, however some major constexpr features (e.g. constexpr unique_ptr, optional, variant) as well as some new vocabulary types like std::expected now have support.\nCompiler C++23 (language full) C++23 (language 90%) Missing C++23 (library full) C++23 (library 90%) Missing GCC/libstdc++ No 15 lifetime extensions for range for loops, scope for training lambda return types No No lots Clang/libc++ No 19 sized float types, CTAD from inherited constructors, pointers in costexprs, No No lots C++26 #It is too early to start looking at C++26 compiler conformance. GCC is roughly 50% implemented for language support as of GCC15 and Clang21, but most library feature remain unimplemented.\nCMake #Every major non-EoL distribution supports at least CMake 3.16. If you need to do things with CUDA try to stick to CMake 3.20 or newer which has much more robust support for GPU programs which is available on all distributions except CentOS7. 3.25 is needed for cuFILE APIs which is only available on more cutting edge distros.\n3.10 Added flang,ccachefor Ninja, GoogleTestgtest_discover_tests()` 3.11 Added add_library without sources, FindDoxygen, ` 3.12 Added cmake --build, \u0026lt;PackageName\u0026gt;_ROOT for find_package, many improvements to FindSwig 3.13 Added cmake -S ... -B ... to set source and build dirs, target_link_libraries can now be called on targets from different directories., more improvements to Swig 3.14 Added get_filename_component(), install(FILES)/install(DIRECTORIES) now uses GNUInstallDirs by default, FtechContent_MakeAvailable(), numpy support in FindPython 3.15 Improved Python lookups and Python3::Module, $\u0026lt;COMPILE_LANGUAGE:\u0026gt; with a single language, --install 3.16 Added support for unity builds 3.17 Added Ninja multi-config, CMAKE_FIND_DEBUG_MODE, FindCUDAToolkit, 3.18 add_library can now create Alias targets (useful for FetchContent), CMAKE_CUDA_ARCHITECTURES, FindLapack imported target, improvements to GoogleTest test discovery 3.19 Apple Silicon, CheckCompilerFlag generalizes C/C++ specific versions 3.20 CUDAARCHES environment variable added, Nvidia HPC SDK, OneAPI compilers, improved FindCudaToolkit with ccache, better implicit dependencies. 3.21 cmake-presets, Fujitsu compiler 3.22 CMAKE_BULID_TYPE default variable 3.23 improvements to presets, HEADER_SETS for IDE integration, many improvements to CUDA seperate compilation 3.24 improvements to presets, CMAKE_CUDA_ARCHITECTURES=native, fetch content can now try find_package first 3.25 improvements to presets, block scoping, try_compile doesn\u0026rsquo;t need a binary directory, cufile 3.26 imagemagick imported targets, Python \u0026ldquo;Stable ABI\u0026rdquo; targets 3.27 Cuda Object libraries, FindDoxygen config file support, FindCUDA (old and deprecated) removed 3.28 Cmake modules support is stable but not supported compilers until GCC 14 and LLVM 16 and using the Ninja generator, CMAKE_HIP_PLATFORM to compile hip code using Nvidia GPUs, CrayClang, many commands became JOB_SEVER_AWARE to better support nested builds. 3.29 CMAKE_INSTALL_PREFIX env variable, ctest jobserver support 3.30 c++26 support, Backtrace module support, free threaded python support 3.31 OpenMP CUDA support, cmake_pkg_config to avoid native pkg-config dependency 4.0 likely breaking due to before deprecated functions since 3.5 removed. cmake --project-file to allow maintaining a second cmake build system to facilitate staged updates. Swig #Swig 3 is widely available, but Swig 4 is not \u0026ndash; try to avoid C++14 in swig wrapped interfaces.\n4.3 Improvements for std::filesystem, std::unique_ptr, std::string_view, fold expressions and training return types 4.2 Many improvements for std::array, std::map, std::string_view, python3.12 4.1 Added move semantics support improved, many new languages supported (e.g. Node 18, Php 8, Python 3.11) 4.0 Added C++11 STL container support to Python, and better support for C++14 code 3.0 Added C++11 language support Cuda #CUDA places requirements on your GCC/clang version, but also places requirements on supported hardware. This note has much more detail.\nWhen using NVCC:\ncuda version max gcc sm versions 8.1 5.3 2-6.x 9.1 6 4-7.2 11.5 11 3.5-8.6 12.0 12.1 4-9 12.1-3 12.2 4-9 12.1-3 12.2 4-9 12.4-5 13.2 4-9 12.4-6 13.2 4-9 12.8 14.0 4-12 It is also possible to use clang++ to compile cuda\nclang version cuda release sm versions 6 7-9 3-7.0 10 7-10.1 3-7.5 14 7-11.0 3-8.0 15 7-11.5 3-8.6 16 7-11.8 3-9.0 18 7-12.2 3-9.0a 18 7-12.2 3-9.0a 19 7-12.5 3-9.0a 20 7-12.6 3-10.0 In newer versions of cuda, this command outputs your compute version.\nnvidia-smi --query-gpu=compute_cap --format=csv Python #For widest compatibility, avoid features newer than 3.6, however when CentOS7 is EoL, 3.8 is the next lowest common denominator.\n3.6 Added fstrings, types for variables, async generators, PYTHONMALLOC and more. 3.7 Added breakpoint(), @dataclass, time.perf_counter_ns() and more 3.8 Added := operator, position only parameters, fstring {var=} syntax, and more 3.9 Added | for dict, list instead of List in types, and is much faster and more 3.10 Added match pattern matching, parenthesized context managers, type | operator, and more 3.11 Added exception groups, tomllib, variatic generics, Self type, string literal type, is much faster and more 3.12 Added Path.walk, improved f-strings, type alias, sys.monitoring, collections.abc.Buffer 3.13 Added improved interpreter and error messages, jit bytecode interpreter for faster hot functions, copy.replace. Experimental support for noGIL python 3.14 Deferred evolution of annotations, t-strings, improved python debugging, noGIL out of experimental but still optional Manylinux #Python\u0026rsquo;s pip uses manylinux containers to provide broadly compatible binaries for use with python.\nVersion GCC Python Base Status manylinux_2_34 14 3.8.10+, 3.9.5+, 3.10.0+ Almalinux 9 Alpha manylinux_2_28 12 3.8.10+, 3.9.5+, 3.10.0+ Almalinux 8 Prod PEP 600 defines manylinux_x_y where x==glibc_major version, y==glibc_minor_version. There are docker containers that provide build envionments for these packages that should be preferred. One should also check the auditwheel command to ensure that the compiled library does not link to a disallowed library.\nNumpy # version eol features 1.17 eol* __array_function__ support, random module made more modular 1.18 eol 64 bit BLAS/LAPACK support 1.19 eol dropped support for python \u0026lt; 3.6 1.20 eol* Numpy added typing support, wider use of SIMD, start of dtype refactor 1.21 eol* more type annotations, more SIMD 1.22 eol most of main numpy is typed, array api and C support for dlpack supported 1.23 eol python support for dlpack 1.26 Febuary 2025 support array_api v0.2022.12, but fft not supported for now; new build flags 2.0 June 2026 many changes to the public/private api, changes to C functions, many performance improvements 2.1 August 2026 support for array_api v2023.12, prelimiary support for GIL free python 2.2 December 2026 improved support for GIL free python, improved use of BLAS for matrix vector products 2.3 June 2027 openmp acceleration for some functions, free threaded pythhon Items with * are out of upstream support but may be supported by a LTS distribution\nDLPack #DLPack is a zero-copy data exchange protocol used in many python libraries for CPUs and GPUs. The latest versions all major data libraries support version dlpack 1.0. Given many users install python libraries with pip this make it widely available across platforms.\nHowever, Numpy versions provided in many package managers do not support it at all. Given the numpy versions included with all packages it is important to also support NumPy\u0026rsquo;s __array_interface__ in the mean time. Additionally, For other libraries some distributions package libraries that support only 0.6 or newer versions. For example of how to support multiple versions simultaneously consider the implementation in Pytorch for __dlpack__ and from_dlpack.\ndlpack version library library version 0.1 Torch 0.3.0 0.5 Numpy 1.23 0.6 CuPy 10.0.0 0.6 MPI4PY 3.1.0 0.8 CuPy 12.0.0 0.8 Torch 2.5.1 1.0 CuPy 13.4.0 1.0 Torch 2.9.0 1.0 Numpy 2.1 1.0 MPI4PY 4.0.3 Julia #Generally Julia installations will be relatively new since they are often not provided by the package manager. There have been significant reductions in time to first plot in recent versions\n1.5 threading, @ccall 1.6 various quality of life improvements 1.7 property destructuring, @atomic, reproducible RNG, libblastrampoline 1.8 const on fields in mutable structs, SIGUSR1 profiling 1.9 :interactive threads, jl_adopt_thread, Iterators.flatmap, Package Extensions 1.10 CartesianIndex can now broadcast, vastly improved package compilation times 1.11 new Memory type, public keyword, @main for an opt-in entrypoint, greedy Threads.@threads 1.12 @atomic :monotonic, --task-metrics Hope this helps!\nChangelog # 2025-04-17: comprehensive updates 2024-08-28: comprehensive updates, addeed numpy to tracking 2024-02-22: added python and cuda and updated cmake 2023-09-12: Created ","date":"12 September 2023","permalink":"https://robertu94.github.io/guides/dependencies/","section":"","summary":"\u003cp\u003eDependencies matter a lot in terms of who can easily install and use your software.\nHowever there are trade-offs in what features are provided by each version and the availability of these versions.\nThis presents an opinionated take on what these trade-offs are so one can know what can be widely supported.\u003c/p\u003e","title":"What to support if you are supporting"},{"content":"As an academic, you write a lot of papers. If you contribute to the disciplines of Math, Computer Science, or some domains of physical sciences, you\u0026rsquo;ve probably used LaTeX \u0026ndash; a typesetting language \u0026ndash; through a tool like Overleaf. What\u0026rsquo;s nice about LaTeX is that it attempts to separate concerns: Authors with minimal effort can port the text and structure of a document from one style to another. It generally produces much more ascetically pleasing output by default than what most authors would do on their own. It handles formatting of three things that generally are extremely frustrating in other tools: citations/bibliography, cross-references, footnotes/endnotes, equations, and embedded code listings. When used with a source control like Git it provides one of the best change review and tracking mechanisms of any document platform.\nBut LaTeX is not perfect. It can be downright frustrating if the defaults don\u0026rsquo;t quite work correctly. LaTeX does a poor job with orphan control (creating large spaces in the document to keep a short part of a paragraph from starting or ending a column). This is especially poor in the presence of multiple figures. Providing manual hints like vspace are more likely to make the problem worse than solve the issue. The ability to be able to change styles depends significantly on the quality of the template author. Writing these templates is not trivial and not well documented. The style templates that are commonly used in LaTeX are by default black and white and often use the same font. Customizing these is harder than one might expect. LaTeX\u0026rsquo;s error messages are notoriously obtuse and difficult to debug. Overleaf – the Google Docs of LaTeX has long compilation cycles \u0026ndash; especially with longer documents when many authors are editing simultaneously. With the primary output being PDFs, adding features like interactivity, video elements, audio elements, and accessibility features like alternative text are non-trivial.\nThere is an opportunity for improvement. Google Docs has three key limitations to be used for next-generation scientific writing: its inability to handle cross-references, its lack of sophisticated equation or source code embedding, and its ease in accidentally overriding a style in a way that makes it hard to port from one style to another. others might advocate Jupyter Notebooks, which while not a document preparation platform, provides a mechanism to easily embed figures, interactive elements, and more in between cells. It however lacks features like cross-references and biography support.\nThere is however one format that could truly support these features: the modern web. The web has interactivity through JavaScript. It has had excellent support for cross-references since the introduction of the anchor tag. It even has an approach to citations through DOI and services like ArXiV. There are excellent web editing tools ranging from highly textual to highly visual, and the appearance is easily customizable with modern CSS features like CSS grid. There are even collaborative multi-player editing environments for the web with editors like Zed and VSCode LiveShare. Even equations, diagrams, and source code can be addressed with libraries like MathJax, d2/Mermaid.js, and highlight.js. It\u0026rsquo;s even more accessible, debuggable, and automatable than LaTeX.\nNow I can almost hear my colleagues now, “You can be serious!! No one likes HTML!” That criticism is likely somewhat fair, but is LaTeX actually better? You still have “begin” and “end” tags for most environments. Objections of this form seem to be largely syntactic and based on the preference for lack of angle brackets. And in many cases, you can use Markdown like the academic optimized zettlr and render to HTML which is even more syntactically light than HTML or LaTeX. You might say, “LaTeX produces documents that are so pretty!”To which I respond, most of the work that goes into making the documents so the beauty comes into the generating of templates, not the authors, and when the templates fail you, debugging them is a nightmare in LaTeX but is possible because of excellent developer tools in modern web browsers. Still, others might say “But LaTeX generates documents that are PDF/A compliant and can be more easily archived and more consistently viewed across systems!”. You can fairly easily get around this restriction by incorporating print styles into the journal/conference-wide style sheet and browser and producing a PDF where required.\nIt’s about time we move academic writing into the 20th century and adopt the modern web as our typesetting platform.\n","date":"2 July 2023","permalink":"https://robertu94.github.io/posts/2023-07-02-dethrone-latex/","section":"Posts","summary":"\u003cp\u003eAs an academic, you write a lot of papers. If you contribute to the disciplines of Math, Computer Science, or some domains of physical sciences, you\u0026rsquo;ve probably used LaTeX \u0026ndash; a typesetting language \u0026ndash; through a tool like \u003ca href=\"https://overleaf.com/\" target=\"_blank\" rel=\"noreferrer\"\u003eOverleaf\u003c/a\u003e. What\u0026rsquo;s nice about LaTeX is that it attempts to separate concerns:  Authors with minimal effort can port the text and structure of a document from one style to another. It generally produces much more ascetically pleasing output by default than what most authors would do on their own.  It handles formatting of three things that generally are extremely frustrating in other tools: citations/bibliography, cross-references, footnotes/endnotes, equations, and embedded code listings. When used with a source control like Git it provides one of the best change review and tracking mechanisms of any document platform.\u003c/p\u003e","title":"Dethroning LaTeX: What Would It Take?"},{"content":"Spack installations can be really slow on some platforms, however, there are pre-built packages for ubuntu 20.04\nsudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt update apt install gcc-11 spack mirror add spack-build-cache-develop https://binaries.spack.io/develop spack buildcache keys --install --trust And now you have spack pre-compiled goodness for your packages!\nYou can see the spack cache index for details for what packages are pre-built for which platforms.\n","date":"25 May 2023","permalink":"https://robertu94.github.io/guides/spack/ubuntu/","section":"","summary":"general guidance for using spack on Ubuntu 20.04","title":"Spack on Ubuntu 20.04"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/ci/cd/","section":"Tags","summary":"","title":"CI/CD"},{"content":"Most successful langauges have a \u0026ldquo;killer\u0026rdquo; features that motivates\nIn my previous post, I commented on the tools that I use for software development, but I didn\u0026rsquo;t talk about either the process of choosing a language or the libraries within a language that I use most frequently. This post expands on how I work within a language and specific \u0026ldquo;killer\u0026rdquo; libraries that I use most and how they compare to facililties that I know from other languages.\nLangauges I use reguarly #C++ #C++ is probably the langauge that I use most often. This is driven by a couple key factors:\nits expressive type system \u0026ndash; With classes, templates, template templates, concepts (and perhaps in the future meta-classes), C++\u0026rsquo;s type system is very expressive allowing encoding sophisticated abtractions concisely. In my mind, only Haskell and its decendents with true higher kinded types and linear types truely exceed it in its expressiveness, and only Rust comes close to matching it, but lacking a few key notions (e.g. variadics). \u0026ldquo;zero-cost\u0026rdquo; abstractions \u0026ndash; a key motivating principle in C++ is that using an abstraction in the language has as close to the same overhead as what the equivelent C would have if you wrote it from scratch, and you do not pay for what you do not use meaning that you can use many of C++\u0026rsquo;s abstractions with little concern about how it will effect performance. 1. Rust does compete in this aspect, but few higher level langauges do. C interopablility and FFI \u0026ndash; nearly every language that exists can interoperate with C through an FFI mechanism. Being able to write code that exposes a C ABI that can be used in most higher level languges is a key feature. Only Rust and C itself complete in this particular factet I and many of my colleagues in HPC can at least consume C++ libraries even if they may not know how to fully write it themselves. Here are things that I frequently reach for C++ for:\ncode that I know I want to use from many langauges performance critical code that will run on a super computer code that needs to access some hardware device (a GPU, a network card) to use acceleration code that I share with my colleagues in HPC A few of the libraries that I frequenly reach for:\nMPI \u0026ndash; distribute jobs in parallel across a computing cluster efficently. I\u0026rsquo;ve written extensively elsewhere on my usage of MPI. fmt \u0026ndash; this might become less important when std::print get\u0026rsquo;s widely implemented in standard libraries, but this vastly improves the default experience with C++ when printing and formatting messages LibPressio and various compression libraries \u0026ndash; many compression libraries are written in either C, C++, Cuda, SYCL, or HIP \u0026ndash; C++ is the lowest common denominator for working with these. OpenMP \u0026ndash; few libraries beat the compiler extension OpenMP for a consise way to write imparitive thread parallel code \u0026ndash; Rust probably has a more ergonomic tasking system in tokio, but for other programming paradigms, OpenMP get the job done quickly and relatively performantly. Mochi/Thallium \u0026ndash; a high level abstraction for writing RPC services in C++ exposing features of high perfomrance network interconnects that appear on clusters. sol2 \u0026ndash; one of the best embedded intepreters out there. Rust #Rust for me is an up and coming language.\nI find it\u0026rsquo;s dependency management and ecosystem story vastly superior to C++, but inferior to Julia. Rust frequently has dependencies that help writing and interacting with Web services and low level system aspects, but it\u0026rsquo;s ecosystem especially around AI/HPC to be lacking (e.g. the MPI wrapper is in-complete, very little GP-GPU support, limited AI libraries that would complete with Tensorflow or Pytorch). Rust\u0026rsquo;s async runtimes and strong correctness garuntees give me confidence that I did it right the first time, and debugging tools when to fix it when I didn\u0026rsquo;t. Many of my collagues don\u0026rsquo;t know rust, and it takes a long time time to begin being productive with Rust. Even knowing C++ and Haskell \u0026ndash; two languages on which Rust is obstensibly inspired \u0026ndash;, it took me close a month to become even mildly productive with Rust because of its expansive number of methods on its ubiquitous data structures like Map or Vec and unique ownership model and borrow checker. I reach for Rust when:\nWriting small unix-like utilties or web-services that need to interact with a 3rd party service because of its robust third party ecosystem. Any time I need to spawn and interact with multiple processes. When I want the protection of Rust\u0026rsquo;s advanced type system and production garuntees to prevent memory leaks, overflows, and other kinds of subtle hard to diagnose bugs. Writing non-HPC async code. If I need to interact with too many HPC libraries I\u0026rsquo;ll fall back to C++. Graphical User Interfaces \u0026ndash; Tauri is one of the best GUI libraries that I\u0026rsquo;ve encoutered in any langauge producing web powered abstractions, easy developer experience, and a small application size. Some Libraries that I find are awesome from Rust:\nserde \u0026ndash; one of the most sophisticated and elegant serialization libraries that I\u0026rsquo;ve ever encoutered. bindgen and cbindgen \u0026ndash; makes it really easy to import C code into Rust and visa-versa. tauri \u0026ndash; I firmly believe that the web is one of the best GUI platforms from a developer experience perspective. Tauri provides the ability to easily combine a platform native web-view with a rust backend and a web-frontend in a small bundle. Super cool IMHO, and probably my go-to GUI library. tokio \u0026ndash; one of the most sophisticated and complete async task libraries in a native compiled langauge wasmer/wasmtime and web-sys \u0026ndash; While I am concerned about the implications for the openness of the web, I genuinely think WebAssembly is a extreemly promising technology not just for the web, but for sandbox-able plugin systems everywhere \u0026ndash; I eagerly await when I can just run 6 python interpreters in a single process each with completely independent state and clone them initialized with just a memcpy using webassembly, but we are not quite there yet. Rust\u0026rsquo;s webassembly capabilities are first class with a promising future. Julia #Julia is another up and coming language concieved for productive HPC. This is influenced by:\nIt\u0026rsquo;s best in class dependency management system. Not only is it easy to package and install dependencies including native dependencies, but it is super simple to ensure that you have reproducable envionment (C++, Python cough, cough) that don\u0026rsquo;t install mulitple copies of every package on your system (Rust, Python cough, cough). It even uses a solver to properly resolve dependencies as opposed to python\u0026rsquo;s indeterministic mess where which packages are installed depends in large part on the order that they were installed or uninstalled over time. The sheer number of common HPC operations that are concise and easy to interface with. Want to solve a linear system A\\b? Want to do elementwise multiple A.*B, matrix multiplcation A*B. apply a function over a vector foo.(A). Learning these functions will require some doing unless you have frequenly used Matlab, but once you have its straightforward. Interfacing with schedulers and threading are pretty simple too. Several best in class packages for HPC tasks that work together because of multiple dispatch and what appears to be a set of well defined standards for several key functions. I reach for Julia when:\nI use it whenever I want to prototype some thing in HPC, but don\u0026rsquo;t need to handle memory management yet. Anything dealing with tabular data manipulatpion and plotting Some killer libraries in Julia:\nMKL.jl and OpenBLAS.jl \u0026ndash; best in class linear algebra primitives make it effortless to get good performance with easily readable code that closely resembles the math you are trying to implement. Cuda.jl \u0026ndash; one of the best and most productive GPU programming frameworks that I\u0026rsquo;ve encoutered for the GPU. Not only can you write kernels yourself, but high level routines pretty much just work as well. Sure port to the native langauge if you need that last 5% performance, but Julia gets you more than half way there. DataFrames.jl \u0026ndash; I love Pandas from Python, but for whatever reason the way that it handles joins and group operations has never felt intuitive for me. DataFrames.jl does, and it is substantially faster in many cases. Plots.jl and Makie.jl \u0026ndash; While I love Pandas, matplotlib is one of the most frustrating aspects of the python ecosystem. I can almost never seem to get it to do what I want outside of the most basic of plots. I was productively making advanced plots with Plots.jl within an hour. Makie takes that a step further and made making interactive plots easy too. Distributed \u0026ndash; makes it easy to distribute a Julia code across a cluster with limited direct interaction with the scheduler Debugger.jl \u0026ndash; one of the best and most sophisticated debuggers in any interpreted langauge. It rivals GDB for its capabilities. Revise.jl \u0026ndash; While I often use Julia as a REPL for short functions, when interating on a package Revise is the way to go. It automates the python equivelent of mylib = importlib.reload(mylib) making experimentation quick and easy JuMP \u0026ndash; JuMP is one of the best optimization libraries in any langauge. For me it is both intuitive and fast allowing me to quickly express different optimization tasks, and use different solvers and get good performance. PyCall, RCall, CXX, Clang.jl, etc\u0026hellip; \u0026ndash; Julia has great facilities to call functions from other programming languages from Julia making it easier to work around that say 10% of your problem that may not have a great native julia experience. Python #Another language that I reach for frequently, but is slowly being overcome by other languages. I does however lead in a few areas:\nEase of learning. Learning the basics of python can be done in an afternoon which is a super power not easily overstated. It\u0026rsquo;s incredibly vast library ecosystem. The a reason why ProgrammingHumor jokes that python imports its homework. This includes many best in class data science and AI packages. Ubiquity \u0026ndash; many tools that I frequently use allow python extensions: GDB, LLDB, Ansible, Dagger, m, etc\u0026hellip; Python is nearly everywhere so knowing a little goes a long way. I reach for python when:\nI am integrating into a tool that uses it I am doing machine learning or AI I am doing something short that would otherwise need to be a bash script pytorch/tensorflow \u0026ndash; the two leading AI libraries that everyone builds with. mpi4py \u0026ndash; one of the easiest ways to write a task pool with MPI. scikit-learn \u0026ndash; for classical machine learning, few libraries beat the breadth and API consistency of scikit-learn which makes working with it simple and productive. scipy \u0026ndash; scipy what seems like 1000s of scientific APIs that cover many uses cases. With it\u0026rsquo;s partners in crime cupy and numpy, it is the backbone of scientific computing in python argparse \u0026ndash; one of the best argument parsing libraries in any language. Often imitated, seldom replicated. Languages that I use infrequently #Lua #Lua\u0026rsquo;s super power is that multiple independent copies of lua\u0026rsquo;s interpreter can be used in a single process. Couple that with it\u0026rsquo;s relatively simple to learn syntax, and it is used for embedded scripting all over the place.\nHaskell #Haskell\u0026rsquo;s super power is its incredibly sophisticated type system that allows writing extreemly elegent poems for programs. One of the most frequently cited typeclasses is the Monad which while often misunderstood is the \u0026ldquo;programmable semi-colon\u0026rdquo; of the language giving extreem powers that are difficult to replicate in other lanaguges. Most other non-functional languages fail to model concepts like Monads despite them slowly creeping into other langauges as they are better understood by the larger programming community.\nLaTeX #LaTeX\u0026rsquo;s super power is typesettings. While I use LaTeX nearly weekly, I certainly do not use its more programatic elements like functions, counters, or conditionals directly.\nFlex/Bison/Antlr #These related languages super power is writing lexers and parsers for other lanaguges. While often a regex will do, when it won\u0026rsquo;t these languages are some of the best for parsing context free grammers \u0026ndash; Haskell is a close second.\nC #It\u0026rsquo;s super power is that it is the foundation of most modern programs. It\u0026rsquo;s simple grammer and semantics make it the portable assembly of the world. For me C is nearly completely replaced by C++, but there are a few embedded envionments and codes where you need to use proper C (often due to the lack of a C++ compiler or a policy that forbids them).\nGo #Go\u0026rsquo;s super power is its deliberate simplicity, writing concurrent network services, fast build times, and easy to distribute binaries. I find it\u0026rsquo;s lack of a more sophisticated type system to be limiting, but I also haven\u0026rsquo;t used the recently added Generics very much. It\u0026rsquo;s used extensively across the cloud native world.\nJavascript/Typescript #Javascript and Typescripts\u0026rsquo;s super powers are there ubuiquity on the web platform. I find myself frustrated by the loose language semantics, but you have to know it if you want to program client side on the web. Typescript largely replaces Javascript for me because I prefer types.\nSQL #SQL dialects like those in PostgresQL and SQlite super power is consise expressions for retrieving data from tabular databases. I personally don\u0026rsquo;t use relational databases extensively, but if you need a tranactional datastore that can survive a power outage, SQL and databases can vastly simplify things.\nbash #I use bash regularly, but at the point that I am doing anything more than something trivial, I want to switch to Python.\nNon-entries that deserve a quick mention #HTML and Markdown are not programming langauges in that they are not turing complete2, but these are the formatting languages of the web, and are worth some time familarizing yourself with as one of the best ways to create graphical interfaces.\nChangelog # 2023-06-23 created There are a few notable exceptions (e.g. \u0026lt;regex\u0026gt;, moving a std::unique_ptr into a function call in hot path extreemly performance critical code where inlining can\u0026rsquo;t happen), but largely you can trust that the C++ will do higher level things efficently where possible.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nOk, HTML might be in the same way that powerpoint is, but please no.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"24 April 2023","permalink":"https://robertu94.github.io/posts/2023-06-03-languages/","section":"Posts","summary":"\u003cp\u003eMost successful langauges have a \u0026ldquo;killer\u0026rdquo; features that motivates\u003c/p\u003e\n\u003cp\u003eIn my previous post, I commented on the tools that I use for software development, but I didn\u0026rsquo;t talk about either the process of choosing a language or the libraries within a language that I use most frequently.  This post expands on how I work within a language and specific \u0026ldquo;killer\u0026rdquo; libraries that I use most and how they compare to facililties that I know from other languages.\u003c/p\u003e","title":"How I Work: Killer Libraries"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/programming/","section":"Tags","summary":"","title":"Programming"},{"content":"CI/CD is a critical, but difficult to get right part of software engineering. You often want to test multiple distributions, multiple compilers on each commit, and you want that to be as fast as reasonably possible. This gets more complicated when you have large dependency trees that you want to remain consistent. Recently, I adapted the CI/CD system for a project that I maintain LibPressio to use Dagger \u0026ndash; a programmatic way to do CI/CD portable-ally across runner environments which made it easier to run our tests and verify correctness.\nWhat is Dagger and why do I care? #Fundamentally, Dagger is a really awesome set of language bindings on top of buildkit \u0026ndash; the library that tools like docker and buildah use to build container images. What is this means is that there is an easy way to build code in an isolated fashion with great caching, easy to write parallelism, remote execution, ability to spin up on-demand side car services, and portable abstractions.\nBefore you say, \u0026ldquo;but who cares? I have [bazel/pants/make/ninja/cmake/\u0026hellip;]!\u0026rdquo;. If bazel or one of these other tools works for you, great! Use it. And sure bazel \u0026ndash; for example \u0026ndash; provides great caching and parallelism, but how is its error handling when you need to interact with a flaky remote service? What about if you need to run a database as side car when testing your code? What if part of your stack uses a language without bazel rules modules like julia? In short, bazel does not give you the whole story while Dagger being built on buildkit gives you the flexibility you need. What makes it even better is that the way that you interact with Dagger is through client SDKs that are generated automatically. This means that you can often just use the language of your choice (nodejs, python, rust, and Go) and the various clients generally have feature parity with each other so you aren\u0026rsquo;t left out if you don\u0026rsquo;t use Go.\nWhy I really care is that it gives me flexibility to respond to changing CI environments. A few years ago, TravisCI drastically changed the availability of free CI/CD services and disrupted a lot of those writing open source software and depending on Travis for free CI including several that I work with. Since then Github Actions has taken off as the generous free for open source alternative, but I question how long ultimately how long any of these offerings can last a free service \u0026ndash; its costs money to run CI/CD for everything from power and cooling to bandwidth and compute; money doe not grow on trees. Ultimately as a user you want to write your CI/CD pipeline in a way so it\u0026rsquo;s easy to port to new platforms and to be able to reproduce issues seen in CI locally so that you aren\u0026rsquo;t beholden to whims and pricing of each CI/CD provider. Dagger gives you that.\nApplying Dagger to LibPressio #LibPressio is a C++ library with bindings in C/Python/Julia/Rust/etc\u0026hellip; It abstracts away the details of compression libraries like BLOSC or SZ3 so users can focus on their applications, and compression libraries can be more easily adopted by applications. We have a moderate set of things that we do in CI/CD:\nverify things build and pass tests on the latest Fedora, CentOS, and Ubuntu releases verify we can build a many_linux container for python installers verify that we can build with spack build our pre-built container image using spack Integrating this into Dagger using the Python SDK was relatively straightforward, and only took about 238 lines of code.\nCaching, Caching, Caching #The core of LibPressio has about 25,000 lines of C and C++ code, and there are about another 20,000-40,000 lines of code in other extension packages in the LibPressio ecosystem. LibPressio supports over 30 different compression schemes, and many of which as optional dependencies that require installing additional dependencies. Without any form of caching, building a full build of LibPressio with all of its optional direct dependencies (and their transitive dependencies) from source can take over 8 hours. That isn\u0026rsquo;t a great developer experience, and would be an awful CI/CD experience where jobs simply cannot run that long on the free tiers. Caching is absolutely key so I pay careful attention to leverage caches in LibPressio using Dagger. There are a few key types of caches we integrate with Dagger:\nccache/sccache \u0026ndash; allows fast incremental compilation when source files are not changed, this really accelerates the implement fix, retry CI loop. caching system packages (i.e. dnf/apt) \u0026ndash; not everyone has fast internet, caching downloads for dnf and apt can really speedup downloading system packages. spack buildcache \u0026ndash; spack is a source-based package manager for high performance computing software. In 2022 it added a public binary cache which can drastically speed up installs. For each, we leverage Dagger\u0026rsquo;s with_mounted_cache function to mount directories to contain these caches into the build environment. Additionally, we get to leverage the buildkit cache\nFinally Dagger gives us fine control over cache invalidation. When we re-build each week, we want to invalid the caches for system packages and spackto test with new updates that may have changed, but if iterating on a fix locally, it\u0026rsquo;s helpful to have these caches remain in tact. It\u0026rsquo;s easy to use the \u0026ldquo;cache-bursting\u0026rdquo; techniques which use an environmental variable with a computed value in the build pipeline to decide when to invalidate a cache layer, and doing so is easier than the alternatives you would otherwise have to do in a Dockerfile.\nParallel Execution #LibPressio has a lot of code that needs to get build on every commit. Building in parallel enables getting the best use of the available hardware. There are two key levels of parallelism and coordination to consider. First is the parallel building of separable build pipelines. Second is de-conflicting execution between pipelines.\nThe former is simple with Dagger, simply use your languages underlying async/await system to await key tasks. Dagger will handle executing the directed acyclic graph of your program with as much parallelism as possible. You should even be able to distribute builds with a distributed buildkit in Kubernetes \u0026ndash; something I hope to try soon.\nThe second requires a bit more work, but is possible. Yes, you can pass -j flags to make, ninja, or spack to run multiple tasks in these system in parallel \u0026ndash; that part is easy. What makes this harder is when you want to run multiple of these concurrently without them all stomping on one another. For this I turn to flags like -l in make and ninja that instruct the system to wait until the system load average is below a certain level before starting new tasks. This isn\u0026rsquo;t ideal because it causes bubbles in the execution pipeline, but also isn\u0026rsquo;t dagger\u0026rsquo;s fault. What would be more ideal would be to expose something like a POSIX jobserver in to the container through a mounted pipe to coordinate job slots among all of the separate instances, which should be possible, but not where I started with my implementation because I would need to figure out how to pipe it through to tools like spack.\nWhat I would like to automate next #Dagger is already providing great benefit with what I\u0026rsquo;ve automated right now, but there are a few more things that I hope to do in the near future:\nDagger also provides mechanisms to handle secrets (i.e. GitHub tokens, slack tokens) in a portable way. I hope to use this soon to handle things like automatically updating the docs hosted on github pages, or pinging a slack channel when a build fails. build our tutorial container image from our container image \u0026ndash; this should not be hard, just something else to do. I\u0026rsquo;d really like to automate the opening PRs to spackto bump versions of the various packages in the libpressio ecosystem as they are updated and dependencies are changed. This is a tedious, but relatively easy task making it prime for future automation. This part will require a little thinking both to do the CI but also to think about how to best collect the various changes across repos into a single PR. Where are the gaps? #There are a few areas that I think where Dagger could improve as project to enable some more complex workflows.\nsupport GPU software \u0026ndash; while in principle running code that requires GPU to execute in container is possible, there are some nuances about doing this in dagger that will require some sorting out. MacOS and Windows \u0026ndash; fundamentally Dagger is a CI/CD tool for containers. Containers share the OS kernel making doing cross platform development where supporting multiple OSes out of reach of the current implementation. However it shouldn\u0026rsquo;t be completely impossible. There have for years been tools like docker-osx that allow running a VM in a container. I would love to see this kind of feature integrated into Dagger\u0026rsquo;s platform \u0026ndash; even if only as a extension or library with the same syntax \u0026ndash; to implement CI/CD for Windows and MacOS with the same great UX. I hope this help!\nChangelog # 2023-04-24 created post Acknowledgement #LibPressio is supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative.\n","date":"24 April 2023","permalink":"https://robertu94.github.io/posts/2023-03-29-dagger/","section":"Posts","summary":"\u003cp\u003eCI/CD is a critical, but difficult to get right part of software engineering.\nYou often want to test multiple distributions, multiple compilers on each commit, and you want that to be as fast as reasonably possible.\nThis gets more complicated when you have large dependency trees that you want to remain consistent.\nRecently, I adapted the CI/CD system for a project that I maintain LibPressio to use Dagger \u0026ndash; a programmatic way to do CI/CD portable-ally across runner environments which made it easier to run our tests and verify correctness.\u003c/p\u003e","title":"Refactoring CI/CD for a Moderately Large C++ Code Base"},{"content":"From time to time I get a question of what tools I use for what jobs. Here the tools I use to get things done:\nLanguage Specific # Purpose C++ Python Julia Rust Compiler/Interpeter gcc/clang python:quick,ipython:long julia rustc,cargo Profiler perf/FlameGraphs + perftrace cprofile:function, line_profiler:line BenchmarkTools.jl cargo Debugger GDB + mpigdb *c++, pdb *c++ Debugger.jl:default,Cathulu.jl:metaprogramming *c++ rust-gdb Build System CMake,(meson, bazel) + m ?? + m Pkg.jl, BinaryBuilder.jl + m cargo + m Sanitizers asan, msan, tsan, ubsan *c++ *c++ *c++, cargo Cache sccache sccache LSP clangd pyright LanguageServer.jl rust-analyzer Refactoring clang-tools-extra +custom libtooling/clang-tidy (comby) ?? (comby) ?? (comby) ?? Testing Google Test unittest/(py-test) Test.jl cargo Docs Doxygen sphinx (Documenter.jl) cargo Distribution Container,spack Container,spack Container,spack Container,crates.io,spack Language-Agnostic # Purpose Tool Server Distro CentOS Workstation Distro Fedora Filesystem BTRFS:Fedora, ZFS:Centos Desktop KDE Plasma Screenshots Spectacle Container Engine Docker/Podman Configuration Management Ansible Development Package Manager spack Desktop Software flatpak:desktop-apps, dnf:system-software Source Control git Source Code Splunking sourcetrails:exploration??, ag/rip-grep:search, LSP:editor, ?? (sourcegraph) PDF Viewer Okular Web Browser FireFox File Browser Dolphin File Transfer Globus Text Editor neovim Terminal konsole Shell bash Prompt starship diff/git diff Tool difftastic find Tool fd-find grep Tool ag/rip-grep Disk Usage du dust cat bat cd zoxide Multiplexer neovim Backup restic:non-zfs, zfs:default VPN tailscale CI Jenkins/Gitlab:hosting, Dagger:abstraction Text Collaboration Slack:work,Discord:home,Email:other Video Collaboration Zoom Email GMail/Outlook Slides Google Slides, Beamer:math-heavy Documents Google Docs:Default, Overleaf:Publications Diagram Editing Draw.io:refine, d2/graphviz:quick Plotting Plots.jl:static, Makie.jl:interactive Bitmap Image Editing Gimp Vector Image Editing Inkscape Print Layout Editing (e.g. Posters) Scribus Video Editing Kdenlive Audio Editing Ardour:edit, pipewire:foundation, Carla:live Task Management Whiteboard:home, a paper journal:travel, Reminders:notification Time Management Activity Watch:Desktop, Screentime:Mobile Calendaring Google Calendar Journaling Vimwiki:default, a paper journal:travel, Good Notes:sketching, Drafts:mobile Passwords KeepassXC Music Plex Citations Zotero RSS Feedly Podcasts Overcast Blog Hugo Gaps #There are a few places where I see gaps in the current ecosystem of development tools\nA build system for Python with good support for native extensions that produces pip install-able packages. There is not an easy way to build native extensions for python with complex C/C++ dependencies that is scalable and consistent. See also pypackaging-native. Python would learn a lot from Julia here in terms of ease of use. I personally don\u0026rsquo;t find Anaconda to be the answer here because you can\u0026rsquo;t use pip the blessed installer for python to install packages from it you have to use conda (which is/was very slow) or better yet the faster compatible mamba but both still behaves poorly once conda-forge is used. Refactoring tools for any language that isn\u0026rsquo;t Java or C++. Comby may be that tool, but I am also hopeful for tools built on top of treesitter. Sourcetrails was one of my favorite tools for exploring large software projects giving me ability to quickly visualize system dependencies. Sourcetrails has been abandoned because it was A) too hard to maintain and B) for whatever reason couldn\u0026rsquo;t find a good funding model. I hope that a tool that could fit some of these needs could be built from either LSP servers generically. Monorepo/Package management tools for maintaining large multi-language applications. The ideal tool would be able to\nsupport C++, Python, Julia, Rust, and other languages well isolate builds from system dependencies by building in a sandbox for reproducible builds An escape hatch to import system dependencies help with coordinated package releases (bumping versions, refactors, rebuilds) execute builds in parallel at least locally, but preferably distributed as well and coordinate between packages (i.e. make\u0026rsquo;s jobserver) rebuild things only when needed Bazel, Pants, and Spack are the tools that are closest, but each lacks a key feature to really solve this for me.\nI\u0026rsquo;ve admired tools like warp.dev from afar because they are MacOS only. I think there is room for an improved terminal like software with improved auto-completion, contextual documentation, intelligent folding/scroll-back, collaborative tools, and more sophisticated graphical capabilites has a place. Tools like Jupiter notebooks address some of these, but are really heavy weight but also don\u0026rsquo;t embed things like vim easily. Terminal\u0026rsquo;s like kitty are closer, but feel like they lack some polish compared to Warp or my current terminal editor Konsole.\nComments # I find I use the C++ tools even for languages like Python and Julia because they allow me to get down to the native code that is being run and understand the stack issues there. I avoid ZFS on Fedora because Fedora runs too new a kernel for ZFS with Fedora to be really reliable. CMake especially its complex scoping rules and string based typing leave things to be desired. Meson is pleasant, but since CMake is ubiquitous and at least last time I tried, meson\u0026rsquo;s CMake integration was imperfect for complex dependencies such as LLVM. Bazel WORKSPACE files are verbose even for simple multi-language processes but is getting better with bzlmod. Bazel also makes it hard to use system provided packages or to install things using UNIX package conventions. Spack requires the package definition to be seperate and has to be used with a language specific build tool. Changelog # 2023-03-14 initial version Key # / can be used interchangeable , used in addition to for a specific sub purpose + tool I developed on top of these () runner up/under evaluation/contextual ?? looking for a replacement no tool needed *c++ I also use the C++ tools here : context ","date":"14 March 2023","permalink":"https://robertu94.github.io/posts/2023-03-14-how-i-work/","section":"Posts","summary":"\u003cp\u003eFrom time to time I get a question of what tools I use for what jobs.  Here the tools I use to get things done:\u003c/p\u003e","title":"How I work: The Tools I Use"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/clang-tidy/","section":"Tags","summary":"","title":"Clang-Tidy"},{"content":"Clang style refactoring has been something I\u0026rsquo;ve been admiring from afar for quite a while now. However for a user to actually use it, it has previously required forking llvm to be able to use it in a reasonable fashion because of things like the hack in clang used to locate the resource directory or other fragile hacks like LD_PRELOAD.\nRecently, the Clang/LLVM developers vastly improved the situation by allowing loadable clang-tidy modules and by installing all of the headers that you actually need to do something with clang tidy, and they are finally packaged on both Fedora and Ubuntu!\nWhile you may think that clang-tidy modules are just for linting, they are actually a super helpful simple way to build clang based tooling that have access to most if not all of the features that you would like to have to do refactoring including\ndiagnostics ast matchers fix-it hints including simple source replacement with clangTransformer compile_commands.json support and much much more To show this off, I created this example repo which roughly follows the Clang-Tidy documentation to create the check. However, I needed to do a few small things to get things working externally:\nFirst you need to do a dance with CMake to add the appropriate build commands:\nfind_package(Clang REQUIRED) list(APPEND CMAKE_MODULE_PATH \u0026#34;${CLANG_CMAKE_DIR}\u0026#34;) list(APPEND CMAKE_MODULE_PATH \u0026#34;${LLVM_CMAKE_DIR}\u0026#34;) include(AddLLVM) include(AddClang) set( LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD} Core Option Support ) Which makes the cmake commands add_clang_library and clang_target_link_libraries which understand the specific linking syntax used for clang and LLVM.\nNext, you need to use these commands to create your check module:\nadd_clang_library(myclang_tidy SHARED ./src/clangtidy_libpressio.cc ./include/clangtidy_libpressio.h ) clang_target_link_libraries(myclang_tidy PRIVATE clangAnalysis clangAST clangASTMatchers clangBasic clangFormat clangFrontend clangLex clangRewrite clangSema clangSerialization clangTooling clangToolingCore clangTidy clangTidyModule ) I\u0026rsquo;m not sure if all of these modules are needed, but this is what clang-tidy used at the time of writing.\nThat\u0026rsquo;s it. add_clang_library adds install rules for you to install it in a reasonable place. So you are ready to write your clang-tidy check.\nHope this help!\n","date":"10 March 2023","permalink":"https://robertu94.github.io/posts/2023-03-10-clang-tidy/","section":"Posts","summary":"\u003cp\u003eClang style refactoring has been something I\u0026rsquo;ve been admiring from afar for\nquite a while now.  However for a user to actually use it, it has previously\nrequired forking llvm to be able to use it in a reasonable fashion because of\nthings like the hack in clang used to locate the resource directory or other\nfragile hacks like \u003ccode\u003eLD_PRELOAD\u003c/code\u003e.\u003c/p\u003e","title":"External Clang-Tidy Modules! C++ Refactoring for the Common Project"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/libtooling/","section":"Tags","summary":"","title":"Libtooling"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/refactoring/","section":"Tags","summary":"","title":"Refactoring"},{"content":"Outside of school very seldom in software development will you completely work alone on a project. Leading and being an effective member of a software team will be critical to your success in this field. However each team and project is different. Some projects have a high iteration cost because testing could cause people to die (e.g. bugs in rocket guidance systems in manned space flight), testing could be lengthy (e.g. if you need to manufacture a physical thing), or expensive resources are consumed (e.g. issues that only manifest when using 1000 nodes) and have to be more planning focused. Other line of business or research applications can afford to be more iterative. Knowing which situation you are in is key to crafting your process to be more effective. Regardless, there are some timeless principles of teams which are important to consider when working in or leading a team. In this post, I will highlight what research says makes effective teams and highlight how one popular approach implements these principles.\nThe Principles #Industrial organizational psychology is the disciple that studies how individuals work in teams. While there are many specific debates within this discipline on specific approaches that work the best, Dr. Fred Switzer argues that there are roughly 3 attributes that are indicative of high performing teams:\nBackup behavior triggered by mutual performance monitoring and enabled by trust. Coordinated effort enabled by closed loop communication. Adaptability between team members There is a lot here:\nBackup behavior is a term that describes where team member(s) proactively assist other members of the team that are facing challenges in their part of the task. For example, if I am struggling, my team will notice and help me to face the task ahead and I do the same for them. This is triggered by mutual performance monitoring which means that I am aware of how other members of my team are preforming, and I of them. This is not suppose to be some sort of Orwellian surveillance state, but an openness and honestest around progress on towards the teams shared goals which is only possible when team members trust one another.\nCoordinated effort means that teams both have a shared vision of the work to be done and are working cooperatively to achieve that vision. The shared vision describes clearly where the team is hoping to go, and the rough goals and plans that will take them there. Working coordinately means that team members each do their tasks, that each task is done once, and that proper synchronization occurs between team members when necessary to complete tasks. This is made possible by \u0026ldquo;closed loop\u0026rdquo; communication which means that each party acknowledges receipt and the their understanding of the message. You might have seen this in air plane where pilots say \u0026ldquo;I have control\u0026rdquo;, and the co-pilot responds \u0026ldquo;you have control\u0026rdquo; indicating that he or she is now the person responsible for flying the plane. A similar concept exists in the TCP/IP protocol where packets are acknowledged by SYN, ACK, and SYN+ACK flags to ensure that the message was received.\nAdaptability between team members means that various members of the team know and can assist each other with tasks. I often think of the \u0026ldquo;bus factor\u0026rdquo; of a project. It\u0026rsquo;s a tad morbid, but \u0026ldquo;how many people would need to be hit by a bus simultaneously for the project to die or be significantly delayed?\u0026rdquo; I suspect that many people have been on projects where a specific task or group of tasks could only be completed by a single team member, and they became a bottleneck for the progress of the team. Such a project likely has bus factor of 1. Ideally each member of the team can fill all roles required which requires careful attention to cross training and knowledge sharing.\nIn leading or working in a software project, you have a responsibility to help strengthen these aspects of your team. How you will implement these principles will very. Every team is different. Every project is different. The goal is to adapt these principles to each team.\nImplementation #In software development on popular methodology is Scrum which a form of iterative or agile software development practice. In a podcast that I thought described it well entitled Product Development Structures as Systems:\nSo Scrum - I see it as kind of like a management training wheels, because it gives you good practices out of the box. You know, just follow the good practices, and it works. But if you understand the principles of why it works, you can make Kanban work in the same way. As long as you don’t create a longer queue, and you’re always revising what you’re going to put into the queue, maybe you can have even faster feedback \u0026ndash; Lucas Fernandes Da Costa\nMy goal here is to illustrate how the principles above can be implemented in Scrum as kind of an \u0026ldquo;easy default\u0026rdquo; for teams to start with and evolve to their specific needs.\nIn the scrum process, you have a few key \u0026ldquo;roles\u0026rdquo; and a few key \u0026ldquo;rituals\u0026rdquo;. The key roles are:\nthe \u0026ldquo;scrum master\u0026rdquo; who leads the team and helps ensure that the process is followed the \u0026ldquo;product owner\u0026rdquo; who represents the interests of the stakeholders/clients/users and the cross-functional team members Amazon famously tries to make these \u0026ldquo;pizza box\u0026rdquo; sized teams \u0026ndash; no bigger than can share a box of pizza without getting hungry. The idea is to give each team as much autonomy as possible by giving them authority to make decisions and all the pessary skills to accomplish the project within the team so that it doesn\u0026rsquo;t get stuck waiting on things from other teams which allows it to move quickly.\nThe key rituals are designed to be lightweight processes that enable the team to accomplish tasks and improve their performance over time.\n\u0026ldquo;sprint planning\u0026rdquo; \u0026ndash; the team and the product owner negotiate on what work will be preformed over the next development cycle (sprint). \u0026ldquo;sprint\u0026rdquo; \u0026ndash; the team works together to accomplish the work usually in a short time boxed window of 1-4 weeks. \u0026ldquo;sprint retrospectives\u0026rdquo; \u0026ndash; the team celebrates what was accomplished over the last sprint and discusses what went well and what did not. \u0026ldquo;standups\u0026rdquo; \u0026ndash; the team meets for less than 15 minutes each day to share what they accomplished, what they plan to do next, and what is blocking them. \u0026ldquo;ad-hoc meetings\u0026rdquo; \u0026ndash; Anything that can\u0026rsquo;t be resolved in the standup or sprint planning or retrospective is done separately with only the team members responsible for a specific task. During Sprint planning the team reviews their backlog of tasks \u0026ndash; the list of all of the tasks that they would like to attempt at sometime in near future. They decide if tasks need to be broken up into smaller tasks to be more reasonably sized for each person to do or to merge tasks that may be too small. They decide what to include in the sprint often using some sort of a \u0026ldquo;points\u0026rdquo; system based on triangular numbers (e.g. 1 trivial, 3 a few hours, 6 a day, 10 several days, 15 several weeks). The team members each individually estimate their perceived effort for a task and then share them together. Large differences in the score of a task could prompt discussion on the approach or the idea person to perform a task. Large scores may also prompt a task to be broken up into smaller tasks. The product owner and team then collaborate on what tasks to attempt based on the perceived effort within an allotted budget of points. Over time and based on team member availability the budget is adjusted for each sprint to better reflect what is feasible. Ideally, once this list is agreed upon it is not modified until the end of the sprint which is ideally short enough that this is not challenging restriction. In practice however, if a team member has many \u0026ldquo;on-demand\u0026rdquo; tasks which are urgent and important, their availability is reduced to make time for them.\nDuring the sprint itself, the team works hard and focuses on the tasks at hand. They meet daily for a standup to quickly share progress and determine what meeting need to happen that day. I\u0026rsquo;ve frequently seen this done it \u0026ldquo;kanban\u0026rdquo; board which shows the progress of the tasks and the team as things progress. They often will work through the scrum master to clarify the tasks with the product owner as needed. I\u0026rsquo;ve often seem people do this part wrong by having stand ups which take hours which generally wastes a lot of peoples time. If longer discussions are need they should be handled separately with only the people who need to handle them.\nAt the end of the sprint, the retrospective both celebrates what was accomplished and tries to identify where improvements can be made. The team creates and adds new items to the backlog that they discovered thought out the sprint.\nNow how does scrum act as training wheels? Scrum supports the principles of effective teams by:\nBackup behavior triggered by mutual performance monitoring and enabled by trust. \u0026ndash; Scrum accomplishes this with the daily standup and \u0026ldquo;no-blame\u0026rdquo; post-postmortems during the retrospective Coordinated effort enabled by closed loop communication. \u0026ndash; Scrum closes the loop with the sprint retrospective and synchronizes with the daily stand-up. Adaptability between team members \u0026ndash; Scrum accomplishes this through it\u0026rsquo;s sprint planning and retrospective mechanisms Are these the only ways to accomplish these goals? no. But they are relatively straight forward and give you a solid foundation for more developed practices later.\nWhat are your current team practices? How do they enable backup behavior, coordinated effort, and adaptability? Task specific resources #Now that you have some general knowledge on work planning, there are a few topics specific to software engineering:\nGoogle produced a book on software engineering Chapters 13-20 focus on engineering process useful for filling our your knowledge on specific topics on testing, documentation, and code review. It also covers some long term technical leadership culture things (i.e. blame free postmortems), and how to be a good team member. However, it focuses much less on the work planning of software development life cycle stuff. A related book also by Google is on Site Reliability Engineering which is the team in Google that develops and maintains their critical infrastructure. Specifically Parts 2 and 3 on design and implementation are also valuable for understanding current practices they use to make their systems reliable. \u0026ldquo;The Phoenix Project\u0026rdquo; by Gene Kim et al \u0026ndash; is software engineering folk tale that shows how teams can work together on software projects and common challenges a light hearted quick read. \u0026ldquo;The Unicorn Project\u0026rdquo; by Gene Kim \u0026ndash; the follow up to the Phoenix project from the software engineering organization\u0026rsquo;s perspective. I hope this helps!\nChangelog # 2023-02-09 created ","date":"9 February 2023","permalink":"https://robertu94.github.io/learning/software_for_teams/","section":"","summary":"\u003cp\u003eOutside of school very seldom in software development will you completely work alone on a project.\nLeading and being an effective member of a software team will be critical to your success in this field.\nHowever each team and project is different.\nSome projects have a high iteration cost because testing could cause people to die (e.g. bugs in rocket guidance systems in manned space flight), testing could be lengthy (e.g. if you need to manufacture a physical thing), or expensive resources are consumed (e.g. issues that only manifest when using 1000 nodes) and have to be more planning focused.\nOther line of business or research applications can afford to be more iterative.\nKnowing which situation you are in is key to crafting your process to be more effective.\nRegardless, there are some timeless principles of teams which are important to consider when working in or leading a team.\nIn this post, I will highlight what research says makes effective teams and highlight how one popular approach implements these principles.\u003c/p\u003e","title":"Learning to Learn: Software Teams"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/software-engineering/","section":"Tags","summary":"","title":"Software Engineering"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/teams/","section":"Tags","summary":"","title":"Teams"},{"content":"What is software development? At a most basic level, it is the activity of using a programming language to achieve some set of goals over time. It includes everything from scripts that a graduate student might write to analyze some data to massive systems that control aircraft. As our world continues to progress technically, software development will likely become even more commonplace than it is now. In this post, I aim to provide a comprehensive overview of how one can develop software efficiently using free and open source tools on Linux.\nWith the rise of software development has come the discipline of software engineering. Software engineering is the study, generalization, and application of the practice of software development. It often concerns it self with how to most efficiently use time and developer resources to achieve the goals of a particular project. Within software engineering are methodologies which provide templates that have been shown to consistently produce results.\nSoftware engineering methodologies (Waterfall, Agile, Kanban, etc\u0026hellip;) differ over the exact components and ordering of the processes, but there are some overarching themes that are consistent. They begin by engaging stakeholders to gather precise, accurate, and prioritized requirements. Then these requirements are synthesized into a design which is evaluated and refined through communication with stakeholders. As the design becomes clearer, engineers begin to develop individual components that represent high value returns to the stakeholders. As components become available, their designs are tested individually and as part of the integrated whole. Finally, once the design is considered finished, it is released to the world.\nHowever if all we study is software engineering, we perhaps have missed the point. We lose sight of the development of the individual over time, their maturation in the discipline, and their self awareness of their abilities. This is what I call software craftsmanship.\nSoftware craftsmanship highlights the need for the development of the individual developer over time. As I use it, software craftsmanship is the inherently personal processes of software engineering that transcend a project boundary. Software engineering more often focuses on a topic like task management as something that is shared among a team. Whereas, software craftsmanship is the self awareness of how an individual will progress through the task. Software engineering often looks at measures of performance across an organization. Whereas, software craftsmanship focuses inward as to how one can grow for the long term.\nThat is not to say that software craftsman are alone. A true craftsman seeks out mentor-ship and the experience of others as they grow in their trade, and joyfully shares their knowledge with others. Their passion for their craft excites and inspires others to go forth in their example. This post aims explain how I have learned the craft of software engineering using open source technologies.\nThis post is the second in a series focused on using Linux to get work done. The previous post serves as an introduction to Linux for general use. If you have never used Linux before, you should read that article first. While this article is primarily aimed at new developers, I hope that some of the insights that I offer here will empower more experienced developers to get more done as well.\nHow to read this post #There are three ways not to read this article:\nFirst, do not try to read and apply this article all at once. Software development is a skill that requires practice; so I recommend practicing before making a decision on whether something is useful or not. Instead, first skim the entire article; then pick a section or subsection that you would like to focus on improving, read it carefully, and integrate it into your work-flow. Software development is a creative process, and like other creative processes different approaches will work better for some than others. That is not to say that there are not norms or best practices that on average show improvements, but how evidence-backed concepts are implemented will very from person to person.\nThe first time that I was aware of doing this with software development when reading Drew Neil\u0026rsquo;s excellent book \u0026ldquo;Practical Vim\u0026rdquo;. The book is a collection of more than 50 tips about how to use the Vim text editor. Yes, reading the book through passively was fascinating. It gave me several insights into the elegance of using Vim that stick with me to this day. However, when it really became powerful to me is when I applied what I was reading to the tasks that I had at work. When I put them into practice, I started to see other areas and applications where I could use my new found skill. Reading actively is essential.\nSecond, these practices are not dogma, you will not be excommunicated for following or not following them. From my experience, you it will not be too long before you will hear comments like these:\nIf you are not using a $TEXT_EDITOR, you are an inferior software developer. $LANGUAGE_1 is always a better language than $LANGUAGE_2 be cause it has $FEATURE $ENGINEERING_METHODOLOGY_1 is strictly better than $ENGINEERING_METHODOLOGY_2, why are you stuck in the past? Often upon further extermination, these kinds of claims can be shown to logically false and are more an expression of the ignorance or preference person who said them. That is not to say that they may not defect some truth, but software development is a study of trade-offs. Lets look at a some specific examples:\nVim is a better text editor than Nano\nThis is an incredibly vague claim. There are may ways in which text editors may be better than one another: simplicity of implementation, number of lines of efficient, correct code written per hour by an new or experienced user, flexibility to new programming languages, etc. And in some ways, yes Vim is better than Nano: it offers features such as build system integration, code completion, and code navigation that have been shown to increase productivity for the average developer. However, Vim has a steep learning curve requiring weeks to become proficient and Nano can be used by many users in a manner of minutes. Over time, it can be shown that Vim offers better productivity, but that doesn\u0026rsquo;t make it strictly better than Nano in all cases.\nJava is a better language than C because it does not use pointers\nThis claim is also vague. But oftentimes when you push the people who make it, they mean that Java is better than C because it is garbage collected and is immune to some classes of memory related errors such as segmentation faults or use after free errors. However in exchange for the immunity to segmentation faults or use after free errors, Java now has to preform garbage collection. In a \u0026ldquo;real-time\u0026rdquo; context, your you need the code to have a precise, consistent runtime, garbage collection can be a huge source of variation. This means that move from something that will be hard to get working to something that will likely never work. Beyond that, a pedantic C developer might point out that Java has the concept of a NullPointerException which often operates similarly to a segmentation fault as far as program execution goes.\nUltimately, in both of these cases, you need to refine the question and determine objective means to come to a conclusion. Software engineering the is study of techniques that make software development more repeatable and effective. Software engineering gives us tools to answer questions such as: Do I favor performance over readability? Resilience over performance? Portability over Implement-ability? These are all choices that skilled software developers make and one a careful application of process can help answer.\nFinally, these practices are best learned in community. Often when we are alone, we loose perspective that can be gained from others. That is not to say that internal reflection is not important, but that others can provide keen insight into ourselves that we might otherwise be unwilling or unable to see.\nI can think of several more seasoned craftsman who have shaped my development practice: From one of my first technical mentors Irish who inspired me to get things right and to care about my tools. To Austin, one of the most gifted engineers that I know who taught me the importance of making a design that communicates. Marshall who showed me observant technical leadership and a quiet curiosity. My colleagues at Boeing who showed me what it means to fit into the bigger picture and make a difference for customers. My adviser Dr. Apon who showed me the power of empowering your team and taught me about the science of computing. Dr. Malloy who taught me to prototype early and don\u0026rsquo;t be afraid to fail. To Dr. McGregor who showed me the importance of history and trade-offs in design.\nWe learn best in community, be aware of who you can learn from and who you can best teach. We all have something to offer.\nGathering Good Requirements #What are requirements? #It may seem odd that I don\u0026rsquo;t begin with a discussion of languages and tools. Remember my discussions of trade-offs in the previous section? You can\u0026rsquo;t really begin to discuss the way you are going to implement a design until you truly understand the constraints involved. In most software engineering methodologies, this is called requirements analysis or requirements gathering.\nSo what does this do with the craftsmanship of software? Writing good requirements is difficult. Almost no one knows exactly what they want from the beginning and can correctly anticipate how that will shift over time. A software craftsman can tell the difference between certain and uncertain requirements and design in such a way as to obviate those concerns. However, a good software craftsman can also identify when the stakeholders are asking for something impossible balances of conflicting concerns. My graduate software engineering professor put it this way:\nSo what are requirements? Requirements are not just what the software is supposed to do (functional requirements), but also qualities of the system (non-functional requirements). Functional attributes are conceptually simpler. You can often perform a test, does it do what is required or not. This then implies the responsibility to actually test those requirements, but I will address testing later.\nHowever non-functional requirements can be just as important if not more so. Non functional requirements are those requirements are attributes of the system. It\u0026rsquo;s hard for these requirements to run an explicit test. Example of non-functional requirements may be it must be maintainable or modifiable. For these kinds of requirements different kinds of processes are needed to access if you are actually meeting your requirements; I\u0026rsquo;ll discuss that in the section on verification.\nThe goal of considering requirements and stakeholders is to build realistic expectations for yourself and others. Seldom if ever are you going to write the next big application that is going to skyrocket to a million users. Rarely are things going to take the time or cost that you think they are going to, especially if you don\u0026rsquo;t think carefully about what is involved. Setting unrealistic expectations ruins credibility, destroys moral, and wastes time and effort. You are not going to do this perfectly; the goal is to do better than you would if you did not try.\nWhat are good requirements? # A good requirement is: correct, unambiguous, complete, consistent, prioritized, verifiable, modifiable, traceable, necessary ~ John McGreggor\nAt first this list seems very obvious, but there is often more nuance to this, so lets break it down.\nCorrect means that the system does exactly what it is supposed to do. For example, people seldom need 100% accuracy, and can tolerate 99% accuracy if the system runs an order of magnitude faster. In economics, these trade offs are measured in marginal costs \u0026ndash; how much would you give up for a small gain in something else. Now people may not know at first what they are willing to trade, but a good requirement seeks out how to measure the user\u0026rsquo;s marginal cost structure.\nUnambiguous indicates that the requirement can be interpreted in only one way. However, unambiguous requirements are really challenging to write. For example, consider the requirement that a function return a python dict. On one level the requirement is unambiguous \u0026ndash; it names the specific type. However, often what is actually required is a dict containing certain keys which in turn refer to values of a specific type.\nComplete requirement describe the totality of the requirements. For example consider this definition of a find operation in a dictionary: it returns a exactly the value associated with a key. However, this leaves out a key part of the requirement, what should be done if the key cannot be found? This doesn\u0026rsquo;t mean that the behavior should always be strictly and formally dictated. A good example of this is undefined behavior in C/C++. Compilers rely on allowing certain outcomes to produce unknowable effects to optimize the generated code for heterogeneous hardware and simplify implementation. The point of having complete requirements is to specify what behaviors are well-defined and which are not; not to specify which behavior to take in every case.\nConsistent means that it is possible that all requirements can be implemented at the same time. On one level, this means that functional requirements should be non-contradictory. On a deeper level it also means that non-functional requirements should be non-contradictory. For example, consider the requirement that a system be highly usable. This may be in conflict with requirements such as high security or extensive modify-ability which tend to increase complexity.\nFew if any systems have ever implemented every feature considered. In this case, budget concerns (both cost and schedule) can limit the extent of the implementation of the system. Priorities are how we make these decisions. When considering priorities, it is key to consider the dependencies of a requirement and what would be lost if the feature is not delivered. It is also important to consider a project with multiple stakeholders is a form of multi-objective optimization. In this case, there may not be one globally optimal point, but several Pareto optimal points. Priorities need to reflect both kinds of concerns.\nVerifiable means that it is possible to know if a requirement has been completed. For things like functional requirements, this can be easier since the system either does or doesn\u0026rsquo;t meet its requirements. This is not always true, consider massive computation problems. You may not be able to afford to run an experiment twice. In this case you need to develop proxy requirements that approximate your actual needs. For non-functional requirements, it can be essential to develop metrics that accurately reflect the extent to which the requirement is met. For example usability, usability could be assessed by having potential users use an application and track what they can do without help and how quickly they can do it.\nModifiable indicates that a requirement can be changed without necessitating making substantial changes to other requirements. That is not to say that lock in and hard requirements should be avoided in all circumstances. Rather, hard requirements and lock-in should be considered carefully. A more complete treatment of modifiable requirements has been written by Gregor Hohpe. A clear personal example of this is when a team that I was working with required support of a particular protocol. Requiring this protocol essentially limited their design space to one very expensive option because the protocol was proprietary to one vendor. Did they actually need this? Perhaps they did. In reality, they could have considered other and possibly better designs if they relaxed this requirement.\nTrace-ability is not so much a question of the content of a requirement but rather the process that created it. The idea is that you should be able to give a rational basis for why the requirement was introduced in the first place. It can be \u0026ldquo;traced\u0026rdquo; to its cause. Trace-ability becomes increasingly important for long standing projects where the understanding design decisions of years past can become a act of archaeological excavation and avoid repeating the mistakes of the past.\nLastly the principle of necessity is the Occam\u0026rsquo;s Razor of software architecture. Systems that are less constrained are easier to build. I remember countless times where I later relaxed the original design constraints of my code after learning that I had made an unnecessary restriction.\nStakeholders: Who makes the decisions? #So who makes these requirements? That would be stakeholders; those who have an interest in the outcome. Often they are the persons funding the effort, will be using the results of the effort, but also those who will be leading the effort. At this point one may say, \u0026ldquo;but this is a personal project\u0026rdquo;, or \u0026ldquo;no one outside of my team will use or care about this\u0026rdquo; to argue that they don\u0026rsquo;t need to think about stakeholders. These views are misguided.\nEven for personal projects you have yourself as a stakeholder. Both yourself now and in the future. You never know when you are going to need to understand, adapt, or reuse code that you are writing now. It is important to consider what you may need for this task in the next month, year, or decade.\nOften the process of thinking about who the stakeholders are can be illuminating on just how your experiment, tool, or library will be used. You could discover that you may have other users in the future. You could uncover that other uses of your design. You might design things differently.\nDifferent stakeholders have different priorities and requirements. Too many times, I have seen systems that were never designed for anyone else to use them. They were undocumented, untested, and in many cases had to be substantially rewritten. Get their feedback early and often.\nWho are the stakeholders and requirements for your system? Crafting a Design: Converting Requirements to an Architecture #How do I get better at design? Study existing designs #One on level, this suggestion goes without saying. Of course, you should learn from the successes mistakes of others. The is true regardless of what field or task you are facing.\nYou may be tempted to think that you are the first person to face a particular challenge. This is likely not true. Maybe you have different tools to work with, maybe you even have some unique problem that truly prohibits the most common solution, but that doesn\u0026rsquo;t mean you are the first to face such a challenge. The \u0026ldquo;Teacher\u0026rdquo; or \u0026ldquo;Preacher\u0026rdquo; or Ecclesiastes said it best, \u0026ldquo;There is nothing new under the sun.\u0026rdquo; You may have to look to completely different disciplines than the ones you exercise regularly, but more than likely you are not alone.\nI am constantly amazed how many \u0026ldquo;new\u0026rdquo; innovations are simply either better tooling or better marketing for a solution that was developed in the early days of UNIX. Take containers for example. These concepts were first introduced in FreeBSD as \u0026ldquo;jails\u0026rdquo;, and improved in Solaris as \u0026ldquo;crossbow\u0026rdquo;. Yes, docker provides a substantially better interface for building and distributing containers than these existing technologies, but that doesn\u0026rsquo;t mean it was first.\nNow how can one learn the most from source code or architectures written by someone else? I suggest that you read with a purpose; I\u0026rsquo;ve recorded some thoughts on this in my post entitled \u0026ldquo;Learning to Learn: Reading\u0026rdquo; Here are some common questions that I ask when I am reading to understand a code base:\nWhat are the major parts of the code base/architecture related to what I do? Why did each one need to be included? Understanding the role a component fills can provide guidance about what kinds of roles or information you may need to implement your system. How are the major components named and why? Naming is powerful in that shapes the way that we think about the system. A good name maps uniquely and naturally to the purpose of the thing that is named. What interface or interfaces did they use? A good interface does not \u0026ldquo;leak\u0026rdquo; to reveal information about the information. Often you don\u0026rsquo;t have access to more than the interface of an object or function. Interfaces are the connective tissue of your application and how they are designed matters. How do they accomplish their task? Implementation details matter. Exactly how processing and memory resources are utilized matters. Sometimes you just need the structure of a solution. What impacts would alternative designs possibly have on the usage of the interface? Understanding the trade-offs that exist within an architecture will guide you to better weigh which trade-offs that you should make yourself. What could I reuse design ideas from this work? Legally (is it allowed by the license) then either actually (copy/paste/#include/import), pragmatically (implement a analogous interface, or create an \u0026ldquo;wrapper\u0026rdquo;/\u0026ldquo;adapter\u0026rdquo;), abstractly (use the idea, but depart substantially from the interface)? Why or why not? Reusing work saves time and effort if possible. Keep in mind you don\u0026rsquo;t have to reuse the implementation to learn from it. How does this design compare/contrast to other designs preforming similar tasks? Why might these changes exist? Before you make your own trade-off decisions, understanding how others have made them can help you make them yourself. What are the advantages and disadvantages of this design? Once you have considered all of the trade-offs, it\u0026rsquo;s time to make finally decide what is good and bad about a design. The ordering here is important. First you need to understand what a design is on its own before you can consider using it, then comparing it, and finally evaluating it.\nIn the remainder of this section, I want to highlight a series of books, software, and other resources that have shaped my thoughts on software design.\n\u0026ldquo;Object Oriented Design Patterns\u0026rdquo; by Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides. This book while also about specific patterns on how to construct software, really introduced me to the concept of how to consider a broader class patterns than simply function prototypes. \u0026ldquo;Modern C++ Design: Generic Programming and Design Patterns Applied\u0026rdquo; by Alexander Alexandrescu. This book is dense and introduces the concepts of using generic and meta programming to implement common software patterns to build higher levels of abstraction. \u0026ldquo;Functional Design Patterns in F#\u0026rdquo; by Scott Wlashchin. This talk introduced me the concept that design patterns could be implemented in abstract ways to solve the same problems. Linux Kernel. The Linux kernel showed me how much could truly be done with a comparatively simple language like C. Clang/LLVM. LLVM has a host of advanced programming methods and structures used throughout its design. It is a veritable menagerie of cool data structures and algorithms. I\u0026rsquo;d also like to caution against using Stack Overflow, random GitHub repos, and other Internet sources unquestioningly. The quality of information you get depends greatly on how carefully it was constructed. While there is doubtlessly many examples of good design and best practices, there are [countless studies that record poor quality answers](https://scholar.google.com/scholar?q=stack overflow quality) on platforms such as these.\nWhat Lessons Have I Learned? #Software craftsmanship is not learned overnight. It isn\u0026rsquo;t even necessarily learned reading posts or books such as these. It is learned by doing and by learning from the doing of others. In this section, I want to highlight some of the most important lessons that I have learned about software engineering.\nQuality Attributes: A powerful way to think about non-functional requirements #One final tool that I think has been very powerful in how I think about software craftsmanship is quality attributes. Quality attributes are a way to think about non-functional requirements of a system. Here is a listing of Quality Attributes that I learned about in my graduate software engineering course:\nEfficiency Time Economy - does it run quickly? Can we implement it quickly? Resource Economy - does it efficiently use hardware/human resources? Functionality Completeness - does it do everything that we want it to? Correctness - does it handle everything that it does handle correctly? Security - how robust is the security model, and how resilient to the modeled attacks it the system? Do we care? Compatibility - does it fit into our existing work-flow? Interoperability - does it inter-operate with our our other systems? Maintainability Correct-ability - how easy is it to fix bugs when they occur? Analyze-ability - how easy is it to figure out what is going on with the system? Modify-ability - how easy is it to modify the functionality of the system? Test-ability - how easy is it to test that the system is operating as expected? Portability Hardware Independence - can the system run on multiple kinds of hardware? Software Independence - can we change the underlying software dependencies (the operating system, standard library, other library components, etc\u0026hellip;) Adaptability - how can the system adapt to new circumstances without being modified? Install-ability - how easy is it to install on the target platforms? Co-existence - can the system be installed or used where other systems are installed or used? Replace-ability - how difficult is it to repair the system after it has been installed? Reliability Maturity - how much testing and use has this system had before it was deployed? Fault Tolerance - how many and which kinds of faults can the system experience before it stops offering meaningful service? Recover-ability - how quickly do we recover from a fault after it has occurred? What do we have to do in order to recover? Usability Understand-ability - how easy is it to reason about the system to new users and developers? Learn-ability - how quickly can users learn the new system given their current knowledge? Operate-ability - how quickly or easily can users operate the system? The important thing to recognize about quality attributes is that they represent trade-offs. To make something more secure, you often make it less usable and adaptable. To make something more adaptable or modifiable, you often make it less understandable. So when thinking about quality attributes in a design, it is important to think about which attributes are most important, and what is the lowest and highest level of each that is required.\nSo how can one design for quality attributes, let\u0026rsquo;s consider a few examples:\nDesign for \u0026ldquo;native paradigms\u0026rdquo; \u0026ndash; this requirement emphasizes the learn ability of the system for developers. By using software patterns that they are used to, they will likely be able to learn and modify the system more easily. However, using native paradigms make the system less portable to systems that don\u0026rsquo;t share these paradigms. I tend favor native paradigms things that are unlikely to change. You probably won\u0026rsquo;t rewrite the entire application in a different language, so using a language paradigm is probably ok. You probably will change the database system or UI, so organizing your data to optimize for a particular database/graphical toolkit probably isn\u0026rsquo;t worth it. Favor small modules and functions with a single task \u0026ndash; this requirement favors modify-ability and understand-ability of individual component over the time/resource economy of the whole system. From personal experience, the cost in time/resource economy is overstated; optimizing compilers can often remove the layers of abstraction introduced completely, and the improved readability of the system is very often worth it. Additionally, when you isolate concerns like computation from IO, you have to change less if you want to introduce different IO or computational patterns later. Prefer self-describing IO formats (protocol buffers, HDF5, JSON, CSV, ORMs). Using these formats makes your system more inter-operable, but will likely increase your resource usage over a custom wire/disk protocol. Personally, I tend to prefer Interoperability and switch to a custom protocol later if I have to for resource constraints. Diagramming: A Quick Picture is Worth a Hundred Hours #One of the best examples of lessons that I didn\u0026rsquo;t learn by reading, but by doing is the importance of prototyping and diagramming. In every software engineering class that I have ever taken, you would hear about something called the verification and validation curve as a model of the process for design software. You can see an example of it below:\nTasks at the top of the V have high impact, and tasks at the bottom have less impact on the final design. However there are a two major problems: At the beginning of the V, you have very little information about the design and what impact particular design decisions may have. At the end of the V, you have a lot of information about the impact of design decisions, but less ability to change them without massive cost. Therefore, it is desirable to be able to get as much information as possible to make the right decisions earlier in the process. This is where diagramming and prototyping can be enormously helpful.\nDiagramming makes a pictorial representation of one or more aspects of a system and possibly how they interact. The major benefit of a good diagram is that it can take fraction of the time to create a diagram useful diagram than it would take to implement the system and has the side benefit of often more understandable to clients and colleagues. I have found that I can create diagrams for a half dozen different candidate designs in the time it would take to code one component of a larger system.\nA good diagram often doesn\u0026rsquo;t display every aspect of the system. Rather it highlights whatever aspect of the system that is currently important to the viewer. If you need to consider a different aspect of the system, make another diagram. It won\u0026rsquo;t take you too long.\nHere is an example from a paper that I published for FRaZ \u0026ndash; a compressor framework that I developed while at Argonne National Laboratory:\nDoes this diagram show everything we did in the system? No. This diagram was intended to show the major elements of the system, how they interact, and which elements we wrote vs got from others. Now this is a publication quality figure, like what one might put in a presentation or a paper, but not all diagrams need to be this refined. The original version of this diagram was just a sketch on a sheet of paper that took me about 10 minutes to make. The publication quality version took about an hour, but the whole system took me weeks if not months to write.\nNow, what and how can I diagram software systems? You could potentially diagram any aspect of a system where you want to consider multiple possible implementations. A book that I found helpful on subject is \u0026ldquo;UML Distilled\u0026rdquo; by Martin Fowler. Two notations that I have found helpful are UML and AADL. UML (unified markup language) is a notation that provides a common design language for expressing system interactions. A vast majority of the time I use a subset of UML. Another notation that I have used is AADL (architecture analysis and design language). I think it does a better job of modeling information flow than UML by forcing you to focus on inputs, outputs, and their formats. I don\u0026rsquo;t use these notations strictly, but where I think they are helpful.\nLastly, what tools should I use to create diagrams? Most of the time, I use a white board or pen and paper. The goal is to reduce the effort of brainstorming interactions. However, when I need something that looks more \u0026ldquo;professional\u0026rdquo;, I use either Inkscape, draw.io or Dia to make a publication quality version.\nBlank Baling: The Art and Value of Prototyping #One of the lectures that has stuck with me the most in my academic career was by Dr. Malloy about the value of prototyping. He would explain that archers, in order to practice their stance and release, would often stand blind-folded inches away from the target to practice shooting. By closing their eyes and focusing on their form, they learn to over come target anxiety. He analogized that prototyping software is very similar to blank-bailing. He described how by focusing on the smallest aspects of our system, we can try and learn by practice what design decisions do in the small scale without worrying how things will fit into a larger architecture. He encouraged us to create a \u0026ldquo;play\u0026rdquo; directory in our computer where we can feel free to try any odd thing and not have to worry about what it would do to the rest of our system or projects. Prototyping often and early has been one piece of advice has been one of the most transformative practices in becoming a software craftsman. In the remainder of this section, I want to highlight a few things about prototyping that I have learned since I started regularly practicing it a few years ago.\nThe first question that I had when I started prototyping regularly was what should I prototype? There are two cases in which I often create prototypes.\nI often create a prototype for each new API that I want to use. Want to learn about MPI\u0026rsquo;s type system, make a prototype. Want to play with OpenMP tasking or nested parallelism, make a prototype. These prototypes were nothing spectacular, just enough code to do something useful or prove that something could work in a vacuum.\nI also from time to time create prototypes of client and library code when I am creating a new interface. The goal with this kind of prototype is to determine what kind of information needs to pass across the interface boundary and how \u0026ldquo;ergonomic\u0026rdquo; an interface will be. By creating several examples of client code, I get a better idea of what will be efficient or easy to do and what will be a pain. It can also help me identify where I don\u0026rsquo;t have all of the information that I need, and I need to add an method to the interface.\nThe second major question that I face with prototyping is how can I avoid this code \u0026ldquo;ending up in production\u0026rdquo;? Prototype code often lacks important details (i.e. security, logging, documentation, and others) that would be in a \u0026ldquo;production-ready\u0026rdquo; code. So you often want to be careful about using it in a way that is likely to end up being used directly for someone else.\nThis is where the \u0026ldquo;play\u0026rdquo; directory comes into play. The \u0026ldquo;play\u0026rdquo; directory is not checked into the same source code repository that the \u0026ldquo;production code\u0026rdquo; would be. Additionally, I never share code in prototype form with others. If someone wants code that I have used for prototyping (it doesn\u0026rsquo;t happen often, but occasionally it does), I typically go through a short process of making it \u0026ldquo;production-ready\u0026rdquo; which involves adding a license file, commenting the interfaces, ensuring \u0026ldquo;secure\u0026rdquo; and \u0026ldquo;efficient\u0026rdquo; APIs have been used where appropriate, and adding some tests cases and assertions to ensure correctness. About 80% of my prototypes are fewer than 100 lines of code, with the average being about 61 lines of code including comments. The person you are giving the code will probably appreciate it the higher quality code and for 60 lines of code it shouldn\u0026rsquo;t take too long to make these changes.\nAnother major tool when building prototypes is mocks. Mocks are components that rather than interacting with a more complex aspect of your system, returns some predefined data. These can be really powerful in writing short code that can model some more complicated aspect of your system without bringing in the entire system as a dependency. Most languages have a library or two that make it easier to quickly build mocks from existing classes. Learning to use tools like these, can really cut down your time in writing prototypes\nThe last topic that I want to address here is when are higher level languages useful in prototyping? C++ is a fantastic language if what you want is the some of the absolute highest level of control about what code gets generated, but it isn\u0026rsquo;t always the best language for writing something quickly. Sometimes, you want the more robust library support, or don\u0026rsquo;t want to deal with questions like object lifetime. For cases such as these, I have found Python \u0026ndash; and more recently Julia \u0026ndash; to be great languages to prototype in. These languages come \u0026ldquo;batteries included\u0026rdquo; with language features that make writing quick and dirty code easy. However, some times what you want to prototype is how to do something specific in the language in which you are going to finally implement everything. In that case, you probably do want to use whatever lower-level language you will ultimately be using.\nFocus on the interface: API Design Reviews #Another transformative talk that informed how I thought about software design was Titus Winter\u0026rsquo;s 2018 CppCon talk entitled \u0026ldquo;Modern C++ API Design\u0026rdquo;. While this talk is focused on C++, it really got me thinking about what constitutes a good interface. As I mentioned above, for anyone who has used Python is dict really the interface you want, or is it really a dict or is a Dict[str,int]? Or perhaps Dict[AnyStr, float] or even Dict[AnyStr, List[float]] Likewise do you really want a dict or are you looking for something closer to a Java interface or a Rust trait? Or does extension not matter, and what you really want is a struct with fixed names and types? The interface you provide is essential. It makes the difference between what is possible to achieve with a design and what is not.\nSo how can you ensure that you have the right design? Winter\u0026rsquo;s suggests, as others have, that there should be a design review for important interfaces to ensure they can be use efficiently and give the users what they need. Here are some questions that I think about when designing types:\nIs the usage of the class \u0026ldquo;intuitive\u0026rdquo;? Do I have to make several API calls to accomplish common tasks, or will a few calls suffice? Are there more than one way to solve a problem, and if so is it clear when to use each? Does each call do a minimal number meaningful of things? Does it fit within the larger software design? Does it use consistent conventions? Naming? Parameter ordering? Return values? Are their other existing classes with do similar things? If so why do both need to exist? Does the code use the same paradigms as the rest of the code? Does this design meet the functional and quality attributes goals of the architecture? What are the invariants that the class/function upholds? Some invariants I consider are: Ownership \u0026ndash; who creates and then later who owns the object? Does it have static or dynamic lifetime? Uniqueness \u0026ndash; how many copies of the object need to exist for its intended purpose? Constancy/Purity \u0026ndash; what parameters are held constant vs. Mutable? How does it interact with its environment if at all? Thread Safety \u0026ndash; what level of thread safety is provided? Exception/Error behavior \u0026ndash; can the invariant be maintained in the case of errors/warnings? How well does this class/function follow the robustness principle (be liberal in what you accept, and conservative in what you return)? Am I really specifying the broadest interface that I can still consistently use correctly? You can find some my thoughts about liberal vs conservative interfaces here Does the function/class appropriately propagate errors at the correct level of abstraction? Reviewing your plan after diagramming and prototyping some of the important APIs can really go along way into developing a use-able interface.\nGood Requirements: Problems with Not Invented Here and Never Invent Here #As mentioned earlier, it is important to write good requirements. However, from my experience two of the most wrongly imposed requirements are that the code either be written by someone else (also called never invent here) or that it must be developed in house (so called not invented here). That is not to say that there isn\u0026rsquo;t a trade-off that must be weighed between the two, but it is seldom a hard requirement that must be carefully decided between. The reason I think these requirements are often over imposed is that they intuitively seem self-supporting.\nNot invented here has been often driven by the engineering teams desire to create, the marketing teams desire to differentiate the product, or by the argument that \u0026ldquo;our case is different\u0026rdquo;. If you are the only user, and you are doing this to learn, the first can be a sufficient reason. However, the argument that your specific use-case is so different that you have to write everything or even most things from scratch is likely false. You may not be using the same words, you may not think about it the same way, you may have to change the default configuration or re-write modular component, but in all likelihood someone has done something very similar to what you are doing now.\nNever invent here has often been driven either by the desire to reduce cost or effort. Components off the shelf (COTS) have many benefits, you don\u0026rsquo;t have to maintain them, you might not have to support them, and they likely are already finished. All of these properties are nice to have. However, if you never allow the possibility of building what you need, you may end up with an experience which is more generic than you might hope.\nUltimately, you have to weigh the trade-offs between these two cases and decide which more closely matches with what you are doing. Would writing this component provide sufficient value to overcome the additional maintenance and development costs?\nSo where can you find existing components? Today, there is a tremendous volume of code that has been open sourced. Provided it meets your license requirements and desired quality attributes, you can likely get it to work. However, you should also consider proprietary solutions.\nRational Expectations: Preparing for the Future versus YAGNI (You Ain\u0026rsquo;t Gonna Need It) #A similar conundrum that people get trapped in is preparing for the future vs YAGNI. This conundrum deals with how modifiable and modular should you make the code vs how simple the implementation should be. On one extreme you have \u0026ldquo;Enterprise Software\u0026rdquo; memes where any and everything can be reconfigured with a change to an XML file. On the other extreme you have code which is so tied to a specific implementation, you can\u0026rsquo;t change anything without re-writing from scratch. Over the years, I have been susceptible to both.\nSo how can you avoid the traps of either extreme? It really boils down to thinking carefully about what are rational expectations given your conversations with your stakeholders and your well-grounded beliefs about their future needs. It is crucial to ask not just might I use this in the future, but do I clearly foresee doing this soon? Additionally, it is important ask what will I have to change if I need to change this in the future, and how disruptive will that be? Between these two factors you can do a cost-benefit analysis as to which situation you find yourself.\nRefactoring: Improving the Design of Existing Software #There is a reason the Martin Fowler\u0026rsquo;s signature book is the only book in this document to get its own subheading. This book really changed my expectations about what could be do quickly to change source code and my understanding about how it would effect users. You should definitely read it.\nTwo quick notes about refactoring. Since the development of Fowler\u0026rsquo;s book, a number of tool have been developed to preform basic refactoring. Which tools are best, and which tools work for which languages shift over time, but I strongly advise learning these tools. They take a lot of grunt work out of programming. In fact, I regularly use two different development environments for several languages because one of them provides better code writing/exploration facilities and the other provides better refactoring support.\nThe other key part of refactoring is how to quickly test if a change broke something. Of course if the code you are using has extensive unit-tests that have high coverage, you can just use that. If they do not, Approval tests can be a quick way to write tests for unfamiliar code. Essentially, they take an application and assert that the output hasn\u0026rsquo;t changed. Additionally there are a number of tools that can help you write Approval Tests quickly regardless of what language you use frequently.\nSecurity and Threat Modeling: thinking about confidentiality, integrity, and availability, #It is often very hard to secure a system after it has been designed and widely deployed. Therefore you should think about the security of your system upfront. The process of planning for security often involves threat modeling. Consider each of the valuable or private aspects of data that your system may interact with. This can be personal information, expensive computing or storage resources, proprietary information, or a service that you provide. For each ask:\nConfidentiality: how can private information be inadvertently shared by someone without authorization? What would someone need to do to access this information? Integrity: how could private information be forged or modified without authorization? What would someone need to know to access this? Availability: How could private information be made inaccessible? What could someone do to prevent users or systems with appropriate access from accessing the information? Now, just because there is a threat doesn\u0026rsquo;t mean that you will implement a mitigation. Some mitigation are prohibitively expensive to implement relative to their cost to remediate. Balancing these concerns is key to implementing a secure system.\nConsider the techniques (quality attributes, prototyping, diagramming, api reviews, rational expectations, refactoring, etc\u0026hellip;) listed above to craft requirements into a design. Which ones have you used? How have they effected they ways that you design software? What language/library should I use? #This is probably the question that several of you had when you first started reading this post. There is a reason that I waited until now to cover it: design is often more important than implementation. That doesn\u0026rsquo;t mean that you can\u0026rsquo;t have a bad implementation ruin a good design, but a bad design will ruin every implementation. The choice of language is an implementation detail that has trade-offs. In this sub-section, I present some things to think about when choosing languages.\nFirst consider what existing work has been done. Are you starting on a \u0026ldquo;green-field\u0026rdquo; project where you are writing the first line of code, or are you adding to an existing code-base? Don\u0026rsquo;t just think about the code that you are going to be writing when you are making this decision. Think about code that you colleagues have or will write, think about what libraries and frameworks exist to do the task you are attempting. You don\u0026rsquo;t want to impose undue costs on yourself or your team by choosing something that no one else knows or uses for your task. That doesn\u0026rsquo;t mean that existing work should be dispositive (you eventually need to move on from Fortran-77 if only to a more modern version of Fortran), but it should weigh heavily on your decision.\nYou should also consider what language(s) you already know. Learning a language takes time. Is your goal to learn a new language, then great, ignore this advise. However, if your goal is to be productive, you may be better off sticking to what you already know.\nYou should also consider how the language will effect quality attributes. Two that are often important are stability and performance. Some languages are relatively young and need frequent changes to their syntax and standard library. They probably aren\u0026rsquo;t a good choice for a system that needs to last 10 years. Secondly, there is often a trade-off between compiled and interpreted languages in terms of runtime performance and development time. Consider carefully if machine time is more important than human time for your particular task.\nSo lastly, I want to close out this section with a brief table of what languages I generally recommend for which purposes:\nPurpose Languages Reason New programmer Python Super easy to learn New computer scientist C++ Exposes the how almost everything is written \u0026ldquo;Enterprise Software\u0026rdquo; Java The language serves no other purpose in life Containers Go Go produces tiny static binaries by default; great for containers Interoperability C/C++ Almost everything has a C foreign function interface Operating Systems C By the time you use only the features in C++ that work without an OS, you have C; Rust is getting close Performance Critical C/C++ Nothing beats C/C++ for performance critical tasks Prototyping Python Python is very concise, forgiving, and expressive Safety Critical Rust Rust\u0026rsquo;s borrow checker and bounds checking are awesome tools for ensuring you have reliable behavior Scientific Julia, R, Python Most code you commonly would write is already written for you Web Back-end Python, Go, Javascript Javascript can also be used on the front-end, Go works well in containers, Python has great tooling Web Front-end HTML, CSS, Javascript These are the standards, and you don\u0026rsquo;t have many other choices Implementation #Now that we have clearly thought about what design we want and which tools we are going to use, it comes time to finally implement the design In this section, I want talk about how to actually implement the software.\nLearning A Language #Learning a specific language is out of scope for this article, but I recommend that you check out my posts on learning languages:\nC++ Python Tactics vs Strategy #The difference between tactics and strategy is the time-frame on which they play out. Strategy is the plan writ-large; tactics are the day to day, moment to moment decisions. Almost everything that I have talked about so far could be call strategy. Now I want to turn to focus on tactics.\nTactics are small units of change that you can introduce to your design to fix a particular problem. Here are some tactics we discussed in my course work:\nAny System Splitting \u0026ndash; decompose a monolithic system or module into two or more modules; reducing cost of modifying a single responsibility. Substitution \u0026ndash; one module is replaced by another with equivalent behavior but a different implementation Modular Systems Augmenting \u0026ndash; an additional module is added to the system. Excluding \u0026ndash; a module is removed from the system. Inversion \u0026ndash; two or more modules are modified to create a third module that capture common behavior Porting \u0026ndash; A module is divided into a into a module that is coupled to the system and another that is free from a single system. Layered Systems Maintain Semantic Coherence \u0026ndash; ensure layers do make undue access to other layers Raise the Abstraction \u0026ndash; create a new layer to encapsulate common work Abstract Common Services \u0026ndash; group responsibilities within a layer into a service. Use Layered Encapsulation \u0026ndash; layers either: + provide a facade to lower layers + provide an interface to the current layers new functionality Restrict Communication Paths \u0026ndash; defines an ordering of layers such that layer N only access layer N-1 Use an Intermediary \u0026ndash; have one layer act on behalf of another layer Relax Layered System \u0026ndash; allow a layer to access a deeper layer directly (for performance or simplicity) Layering through inheritance \u0026ndash; a pattern that binds relationships between the layers at compile time Other tactics: General Encapsulation \u0026ndash; putting a module behind an interface so that it can be replaced. Intermediary \u0026ndash; introduce a module between two components to perform some extra work. Proxy \u0026ndash; decouples the components of a system from being on the same system/process. Reflection \u0026ndash; allow the system to inspect the state of services at run time to select an implementation These aren\u0026rsquo;t the only tactics that exist. Martin Fowler\u0026rsquo;s book \u0026ldquo;Refactoring: Improving the Design of Existing Software\u0026rdquo; has a another large list of tactics that can be applied to a software system. They can be found his catalog of refactoring techniques online.\nLastly in addition to the question of what to do, there is the tactical question of what order to implement the system. In general you should start with the part of the system which provides the highest value at the lowest cost. However within that, there has been research on the ordering use to construct the various modules of a system (search for \u0026ldquo;Integration Test Order Strategies\u0026rdquo;). In general, these approaches create a dependency graph of the system then order the implementation of the software components in topological order of the resulting directed acyclic graph. They may have some weighting assigned based on the importance of the system, but that roughly the all work the same.\nComments #Nearly every programming language has a feature to include comments within the body of the source code. Additionally, tools like Git allow developers to associate comments with changes that they introduced to their source code. However, just because it is possible to comment, does not mean that a comment is the most appropriate way to communicate the message contained in the comment.\nComments are most appropriate when:\nThe code would otherwise be completely opaque — for example most “lock free” using weakly consistent atomic featuring atom is or distributed synchronization code where the use of locks is distributed to several functions To document interfaces especially when it documents pre or post conditions that are not easy to express in code with functions like assert. For example that the code requires a pointer to be “registered” with some other call first, or that the function “frees” the memory associated with its input To provide macro level context of why code was written and what makes it different from other code that came before To mark TODOs for the code When the code serves as a personal reference or a teaching tool when it makes the code self-contained or to emphasize a subtle point. I tend to favor a lighter comment style. A comment can be seen in some cases as a code smell indicating that the APIs and functions involved dose not reflect the intent of the code. A developer favoring this style of comments would use variable and function names to show intent rather than comments. Note that this works best when functions are small and describe a single intent.\nMaybe that changes if the code is truly arcane (a regex, shell code, forth, awk, Perl, some Haskell for a C programmer, and TeX all come to mind) maybe this changes, and documentation is sparse to non-existent.\nHowever it’s worth knowing in every windows IDE I have ever used you just hover unfamiliar APIs like VirtualAllocEX and the IDE summarizes either the call and it’s arguments. If your comment shows up on hover, you don’t need to add it.\nI write comments while learning new APIs. I keep a running journal of code like this, and even have a directory on my machine for code like this called play (Thank you Dr. Malloy) for this code. It also has a place in teaching examples. When I was first learning Perl and Haskell, it annoyed me how terse some of the code was without any comments, and it was not until I found a book that had comments like this that I finally got it.\nWhen thinking about whether and how to comment consider:\nTools can do a lot for you (even tools like Vim), and you should think about if you are duplicating their effort. Good comments like code need to be updated and maintained with the code around them; is this comment worth the cost to maintain it in addition to the code?. Saying close closes a file in a comment is probably obvious and updating the comment is probably more effort than it’s worth, but I would not have guessed that VirtualAllocEx could allocate memory in another process. As a primarily a linux/unix developer, I appreciated this comment pointing out this huge possible foot-gun in the Windows API. Git commit messages can not only serve as a place to keep important context about why a change was made, but also a high level summary of what is going on within a change. Reading through a large set of small changes is still challenging without some high level context of what was changed and why. In the note taking world this is called progressive summarization and is powerful in helping you review critical information quickly. Tools #A key part of learning to be a software craftsman is learning how to make the most out of your tools. In is subsection, I discuss what kinds of tools to add to your belt and which ones that I use.\nText Editors #Another incredibly common question is what text editor/ IDE (integrated development environment) should I use? This is a highly personal choice which will differ from person to person. So while I don\u0026rsquo;t want to proscribe a particular editor, I do want to caution against what I see as two common foibles.\nFirst, don\u0026rsquo;t shoot yourself in the foot by using a tool that provides so few features that you are going to be productive relative to more feature-full tools. In this category, I place editors like \u0026ldquo;pico\u0026rdquo;, \u0026ldquo;nano\u0026rdquo;, \u0026ldquo;gedit\u0026rdquo; on Linux, and tools like \u0026ldquo;notepad\u0026rdquo; on windows. These tools don\u0026rsquo;t provide useful features like auto-completion, auto-indentation, and in some cases even basics like syntax highlighting. There are many better tools that will help you write code better and faster. Use them, they don\u0026rsquo;t have that much higher of a learning curve.\nMy second \u0026ndash; perhaps more controversial take \u0026ndash; some IDEs are also not good choices. Some IDEs have high monetary cost, are tied to a particular language/tool-chain, and require you to learn a completely new interface for each language that you use. For these reasons, using proprietary IDEs may be not a good choice. That doesn\u0026rsquo;t mean that IDEs are not helpful, I use a proprietary IDE as a refactoring tool, but they shouldn\u0026rsquo;t be the primary tool in your tool-belt.\nWhen you want these rich features, I would recommend instead using a tool like the Language Server Protocol. The Language Server Protocol provides an API to many of the features commonly implemented by IDEs in a way that many text editors can use them. This makes these features much more portable to different text editors if you ever need or want to switch. Likewise, I also automate the things that Language Server does not with a small tool that I wrote called m\nI\u0026rsquo;ve said quite a bit about what text editors what I wouldn\u0026rsquo;t use, so which ones do I recommend. I currently recommend:\nVim \u0026ndash; my daily driver emacs \u0026ndash; another capable editor vscode \u0026ndash; a light-weigh graphical editor Editor Vim Emacs Visual Stdio Code Pros Ergonomic keyboard controls Installed and runs almost everywhere Emacs can be your entire workflow Elisp is fully featured language Familiar Graphical Environment for new users while not pidgin holing you like an IDE Cons The key bindings can be a nightmare to learn at first. Vimscript is an awful language. Emacs's learning curve is almost as steep as vim Emacs constantly swaps files which is really frustrating on slow filesystems Requires a graphical console Uses the most memory and resources of the bunch Extensions are written in JavaScript/TypeScript Common Misconception Vim doesn't have advanced features like macros, autocompletion, syntax highlighting, code-formatting. In reality these features are hidden behind obscure keyboard shortcuts Emacs key combos will hurt your hand. You can rebind almost any key, and plugin \"EVIL mode\" makes emacs much more ergonomic You can't use VS Code on remote machines. In reality, it has built-in remote editing support that can edit files on other machines. Getting Started vimtutor the build-in vim getting started exercise spacemacs a emacs distribution with good defaults the emacs tutorial, press control+h followed by t in emacs Introductory videos on the help page Next Steps VimCasts - short videos on using vim Practical Vim - a comprehensive book on vim tricks learn elisp control+hi then read choose elisp Read the more extensive product documentatoin from the welcome screeen Debuggers #Debuggers are invaluable tools for programmers. They allow you to step through a program at run time and interrogate the state of the program in a way that would otherwise be impossible or incredibly tedious. Unfortunately debuggers are less universal than text editors. I cannot just suggest one debugger that will work for every language even though gdb and lldb get close. As such, I intend to point you to some features that are important and useful to have in your debugger.\nRun configurations/scripts \u0026ndash; good debuggers allow you to allow you to save interesting sets of breakpoints/watchpoints and other settings in configuration files per project. You should use the features to be able to quickly run repeatable tests with your debugger. Core Dumps \u0026ndash; While not as emphasized today, core dumps are a dump of memory from the execution of a process that is created when a process is killed by the operating system. It is designed in such as way that all the key state is contained within the core-dump file making them invaluable for debugging problems from users. Stop-hooks \u0026ndash; stop hooks are a powerful feature when combined with break points. Stop hooks are arbitrary code that gets run when a breakpoint or watch point is triggered. You can use it to quickly print out useful state when the debugger stops or to make decisions about whether or not to a particular stop is interesting. Watch points and conditional breakpoints \u0026ndash; Watchpoints allow you to pause execution when a particular region of memory is modified. Conditional breakpoints are breakpoints that only actually stop when some condition is true. Together, these tools give a lot of power to control when to stop execution. Writing plug-ins \u0026ndash; Some of the most powerful debuggers like GDB and LLDB allow you to write extensions in a higher level language. They are several sets of the extensions that have been written and are used frequently such as chisel and pwndbg. Additionally, using a debugger is not a substitute for documenting and verifying your invariants (statements that are always true in your program) in your program. Using language features or function like, C\u0026rsquo;s assert macro can save a lot of time of determining when your state has become invalid. There is an argument to be made that assert shouldn\u0026rsquo;t be in production code; I\u0026rsquo;m ambivalent on the question. On the one hand, yes invalid state is bad, but is a crash worse? At the minimum, it is better when debugging your code; use asserts at testing time. If you have to take them out during runtime, use a feature like C\u0026rsquo;s assert macro which remove debugs when the -DNDEBUG command line argument is passed.\nNow some comments on how to use a debugger. As I have suggested elsewhere, debugging is a scientific exercise. You are creating a hypothesis that explains the behavior of the system, and you are using the debugger as tool to test that hypothesis. This has some implications for how you use a debugger. You aren\u0026rsquo;t going to just step through the program line by line. That is not a efficient use of your time, and it isn\u0026rsquo;t a efficient use of the debugger. Instead, postulate where you think the problem is, and stop there. If you don\u0026rsquo;t know where the problem is, use watchpoints or back tracing to find the suspect state. You\u0026rsquo;ll thank me later.\nYou can find more about GDB here.\nProfilers #Profilers are typically \u0026ldquo;lightweight\u0026rdquo; tools that measure where the execution time of a program is being spent. They save you the effort of having to manually instrument your code at all points where you would like to gather timings. Some profilers can also incorporate low level system information from the kernel to give a more complete picture of performance.\nOne common problem across these tools is how to get a meaningful stack-traces. Oftentimes, this comes down to two issues: including debugging symbols and not clobbering the frame pointer. For gcc and clang with C/C++ programs, the flags you need are -g -fno-omit-frame-pointer, but there likely are similar flags for your language.\nThere are many different kinds of profilers. Each has its own advantages and disadvantages. Here are the ones that I use most often:\nTool Type Usecase Perf sampling profiler get instruction level usage information with low overhead llvm-xray call-sled profiler get function call level overhead information with low overhead; Requires recompilation callgrind linker-based profiler get function call level overhead information with moderate to high overhead; can simulate cache sizes nvprof/nsight Nvidia GPU profiling You are profiling a GPU program on a Nvidia GPU dtrace/ftrace/eBPF Kernel call tracing You want to profile time spent in the kernel You may also see suggesting to use gprof, In my opinion, this advise is largely out of date. The above tools are either higher performance, easier to use, or both.\nOnce you have a trace of the execution of your program, you will probably want to visualize it. There are a number of tools for this, the ones I use are:\nTool What it does FlameGraphs A set of Perl scripts that create SVG flame graphs which show hot spots in the application KCacheGrind An iterative tool that shows annotated call graphs Google Chrome\u0026rsquo;s chorme://tracing Interactive tool that can zoom in and out of complex and parallel traces To find more information, I would highly recommend Brandon Greg\u0026rsquo;s page on Linux introspection which lists a host of other tools to get the information you need during runtime.\nBuild Systems #Another key tool to become familiar with is your build system. Build systems as you expect allow you to build your project. Yes, you could write a shell script or a program to build your program, but a proper build system will do a better job than you can quickly do without it:\nparallelize your build according to dependencies handle caching and incremental compilation between builds download and include dependencies handle differences between compilers provide a system to cross compile for a different native architecture automatically provide hooks to customize installation, and debug builds provide reasonable defaults. Every language seems to rely on its own build system and there are relatively few of them that work across languages.\nSo what should you learn to do with your build system? At the risk of being obvious, you should learn to automate your entire build process using your build system. This includes generating files when that is required, locating or downloading dependencies, or generating the source documentation. This may not seem like a big deal, but by integrating with a build system for your language, you often get a series of knock-on effects like tooling that can use the information provided by your build system.\nAnother key thing to automate with the build system is the deployment of your application, but more on that in a later section.\nWhy do this, it ultimately saves you time and effort for almost everything else you want to do.\nVersion control #Lastly, the final tool that you should learn is a version control system. Without a version control system, you can still track changes yourself: You\u0026rsquo;ve probably had files on your computer called thing_v1.txt, thing_v2.txt, etc. But then the question quickly becomes what if multiple team mates are making changes simultaneously, what was changed? Everything merged correctly? Why were things changed? Version control systems solve the problems of\ntracking changes over time making it easier to understand why a change was made make it easier to share those changes consistently with others. At this point in history, that tool used most often is git. Git currently is prevalent because it scales well to extremely large code bases and is flexible enough to support a number of different work flows. To learn Git, I recommend the git book\nHowever, beyond the question of mechanics, there are questions of policy. When should you commit and why? If you commit, should you make one commit or a series of commits? What constitutes a good commit message? If you use branching, when do you use it and why? If you use branching, what will your strategy be around handling conflicts when code diverges? These are all questions that a seasoned craftsman should be able to answer.\nSo when should you commit? There are two answers to this question. On the one hand you should commit only when the code cleanly compiles and passes the test (a so-called atomic commit). On the other hand, you might need multiple commits to have reasonable reversion points if you are making a big change. So how do you resolve this conflict? Different projects have different strategies, but I favor squashing the reasonable reversion points on development branches while avoiding rewriting history on the primary branch (typically called master or develop). Other reasons to split commits would include to wall-off controversial changes from less controversial changes so they can be separately reviewed and committed.\nSecond, what constitutes a good commit message and why? Here is a post that I think thoughtfully addresses the issue. Ultimately it should be consistently formatted and spelled correctly, have a short descriptive title, a body that explains what was done, why it was done, and what other impacts it has, and cross references to issues databases if applicable.\nFinally come the questions of branching. There are several different philosophies about this. There are naive answers like \u0026ldquo;don\u0026rsquo;t\u0026rdquo; but such answers don\u0026rsquo;t comprehend that in any distributed version control system, conflicts are inevitable on multi-person projects. More nuanced answers say things like, \u0026ldquo;one branch per feature\u0026rdquo;, \u0026ldquo;one branch per person\u0026rdquo;, or \u0026ldquo;one branch per release\u0026rdquo;. I\u0026rsquo;ve worked with each, and don\u0026rsquo;t have a strong opinion on which is correct. It is more important to be consistent.\nLibraries #Also important to any development effort is what you won\u0026rsquo;t develop yourself. Libraries are a form of dependency that provide standard functionality that you can adopt into your code leaving these dependencies to others. While I talk about dependencies more fully in another post I wanted to briefly list here the more kinds of things that you typically want to bring in as dependencies, and some of the key trade-offs\nLogging #Logging is a deceptively simple problem. Yes, the absolute simplest approach of printing to stdout or console.log is really easy to do, but very quickly, you have a much more complicated problem.\nHere are some of the things that send people looking for more complex logging tools:\nautomatic insertion of context (i.e. stack traces, hostname, time, etc\u0026hellip;) distributed logging across several machines log immutability and tamper dectection filtering messages from a particular subsystem, severity, or host extremely high reliability requirements \u0026ndash; if logging does not work, you can not debug anything extremely high bandwidth requirements \u0026ndash; logging should not slow down the system extremely constrained resource requirements \u0026ndash; for example printk in the kernel needs to work before memory allocators are initialized. human and machine readability Appropriate logging frameworks differ for each language, but projects like open telemetry are trying to provide standard approaches to this that work across languages.\nConfiguration Files #Almost every program over a certain size has configuration files of some sort. When considering how to design your configuration files, the real question is what is your goal, and what will your users need and expect?\nHere are some of the things that send people looking for more complex configuration tools:\nhuman readable error messages when an error is encountered parsing the file types other than strings fast parsing performance human and machine read-abilty machine and machine edit-ablity Here are a few possible choices:\nLanguages like like CUE or dHall provide advanced features (data validation, schema definition and error reporting, code generation, scripting tools) but may not serve your particular language well or provide more features than you may need.\nYAML is another popular choice, giving a lot of flexibility, but features aspects like type autodetection which are misfavored by some for the same reasons that implicit casts are disfavored by some in C++. Let’s be fair here: most the reasons people hate yaml are not yaml\u0026rsquo;s fault. They often boil down to the person who implemented yaml config files, essentially wrote a Turing complete programming language in them (cough Kubernetes), or to use it as a relational database and it is not the right tool for that.\nTOML has many of the niceties of yaml but in my opinion provides less pain in the edge cases and stripping out some of the unneeded features.\nJSON is ubiquitous and simple, but some would argue is really easy to typo. Classic ini files are also really simple, but lack schema validation.\nXML is a very verbose (and often much maligned), but enables very sophisticated manipulation and expression which may be appropriate given the complexity of a particular use case \u0026ndash; for example libVirt uses XML for virtual machine definition and the web uses HTML which is similar in several respects.\nSeveral projects that use embedded python or lua as the config file language. This requires some setup, but is incredibly powerful granting functions, looping, variables, string manipulation, and object orientation. However all of this flexibility makes validation harder.\nWhat tools and libraries are part of your toolchain? If you do not use one of these tools why not? What would make your tool use more effective? Testing #One of the most expensive aspects of software development is when software either doesn\u0026rsquo;t do what is supposed to. Even innocuous seeming changes can sometimes have massive unintented effects, being able to detect these quickly massively increases productivity. For this reason as soon as you know what you want to develop, testing code should follow swiftly after.\nSo now you have some code that is ready to test, how do you test it? Scholars divide testing into two categories: verification and validation.\nVerification - does the system do what the requirement says the system does? Validation - does the system do what we (the stakeholders) want it to? As suggested earlier in the sections on diagramming and prototyping, verification and validation should occur at several points along the development process in order to ensure the software works as expected and to minimize development efforts. As such verification and validation occur in several forms. I don\u0026rsquo;t have room to say everything that could be said about testing.\nYou may be surprised that this section is far shorter than the others. On the one level, I\u0026rsquo;ve touched on verification and validation throughout the document. On the another level, writing good tests is just something that comes with practice In my opinion, writing good tests is one of the hardest skills to develop as a software craftsman. One the one level, its easy to write tests. You just do it using some library that makes it relatively easy for your language. On another level, ensuring that you have the right tests is far harder.\nSo what are the right tests? Perhaps obviously, you should conduct sufficient tests to ensure that the requirements are obeyed for all inputs. Does that mean you should test every integer between INT_MIN and INT_MAX? No. But you should test enough of them to know that you didn\u0026rsquo;t make a mistake. So how do you know what that is? In practice, tests cases are chosen by:\nChoosing cases that execute a given function Choosing a few arbitrary cases Choosing boundary conditions (i.e. INT_MIN, -1, 0, 1, and INT_MAX, powers of 2 ± 1, etc for a routine that takes an integer) in addition to the arbitrary cases. Choosing cases that execute every branch of a given function at least once Choosing cases that execute every path through a given function As you could imagine, as you go down the list the difficulty of creating the test cases becomes sizable, but we have more confidence that the implementation is correct. This has led to efforts to find ways to otherwise verify the correctness of an implementation.\nSome more esoteric options that have been used to build a list of test cases before are:\nUse fuzz testing which uses random inputs for a given length of time. Use some generator which creates a provably sufficient set of test cases from the state machine that the program describes. Write two (or more) separate implementations that use different algorithms/designs and compare their answers for a number of inputs. When you do your tests also matters. Tests can be ordered in terms of the amount of the system that they test. At one extreme, you have unit tests which test a single function or small set of functions. Slightly larger are integration tests which test interface boundaries between modules. Finally there are acceptance tests which validate the entire system (one example is approval test mentioned earlier).\nLastly what about testing without code? One of the most common form of testing without code is the review. If you have access to it, I would read the paper \u0026ldquo;Design and Code Inspections to Reduce Errors in Program Development\u0026rdquo;. While it is an older paper, the techniques outlined it seem to be rediscovered every few years. The paper shows that reviews are most effective when the reviewers and the person putting their code up for review are given a checklist which indicates areas that should be considered during the review in advance. During the review, the reviewers bring up the aspects they found during their private review which they consider most important. The checklist serves about which sections of the system are prone to error. There are several of these lists that exist, but ultimately for the greatest utility, you will want to modify these lists to include issues which frequently are represented amongst you and your team.\nThe other major form of testing without code involves modeling. In modeling, you construct an accurate mathematical model or simulation of the system to emulate the real world testing. If you can prove some form of correspondence between the code and the model, and the model produces correct results, then you can conclude that the design is \u0026ldquo;correct\u0026rdquo;. While the ultimate form of this where all software is verifiable mathematically is provably a pipe dream 1, short of that there are extensive areas where these problems can be solved allowing for so-called \u0026ldquo;static analysis\u0026rdquo;. Static analysis has and continues to be powerful tool in proving the correctness of software.\nWhat is the current state of tests and reviews in the system you use? Do they catch bugs before you can? Why or why not? Deployment #The last major step of software development is to release the software to the world. Now how you release it may differ: you might release the software strictly in binary form in exchange for monetary compensation (proprietary software), you might release the software under an open source license, you might publish a paper about the concepts revealed or discovered while writing the software. Software is more useful while it is shared.\nI do want to pause for a moment to remark that even those who aren\u0026rsquo;t typically taught of as software engineers such as scientists and analysts need to think about deployment. I think this is true for two reasons. First, many conferences and journals now have reproducibility requirements in order to publish. Second, even if you don\u0026rsquo;t publish the work to the broader world, you almost certainly have other people on your team. They need to be able to run your software too. If they can, they can help you do your research and provide insights that they couldn\u0026rsquo;t do if they can\u0026rsquo;t reproduce your work.\nSo what does it mean to release software? There is some debate about this question. Fundamentally, there are a few axis on which teams develop their release process: timing or release, scope of release, and support of a release.\nFirst comes the timing of a release. Should you use a staged release process where you have a feature freeze, a time of bug fixing, and then a coordinated release? Several projects do this: LLVM, the Fedora project, and the Linux kernel are all examples. That doesn\u0026rsquo;t mean that you couldn\u0026rsquo;t download the pre-release code for any of these. All of these have publicly available pre-release versions of their software, but that is not what they recommend that the general population use. The alternative is rolling releases. In a rolling release model, software is release continuously. There are no \u0026ldquo;releases\u0026rdquo; that are to be considered more stable than the others. Everything goes on the top of the source development tree and keeps going. Projects like ArchLinux, OpenSUSE LEAP, Google\u0026rsquo;s Abseil all do this. The benefits of a staged release process is that it gives an opportunity to stabilize the code base as a formal part of the development process. The benefits of a rolling release process is that you don\u0026rsquo;t have to wait for a release to deliver a fix or new feature.\nSecond is the scope of the release. What is true at the time of a release? Is there documentation and test cases for all the code as their is with many pieces of proprietary software? Is it coordinated with the release of other software like the sub-projects of LLVM? When you release code, what promises are you making to your users? This is the scope of the release.\nFinally what is the support of the release? Are users allowed to expect that they can use the software for 1 year, 2 years, 5 years, 10 years or more without changes? What compilers and language version are you going to support? What dependencies are you expecting that the users provide and which versions? Writing software is important, maintaining it so that others can rely upon it is almost more important.\nIn the remainder of this section, I want to briefly address some specific concerns about how to release software for Linux and how to use Container technology which have largely shaped how software has been written for Linux in the last few years.\nTargeting Linux #Linux is not just one platform. Yes it is one kernel, but that most software uses more than just the kernel. Therefore, understanding the broader ecosystem is important.\nStandards #Writing portable software for Linux is difficult. Every distribution seems to put files in different places, depend on different fundamental libraries, and make other slight differences. One way help sort out the chaos is to take advantage of standards.\nSome of the most important standards are POSIX and the XDG-Desktop standards. These standards specify how you can do some basic things on a Linux system in portable way. Things like reading files, starting processes, inter-process communication, are all covered by the POSIX standards. The XDG-Desktop standards cover things like where to put configuration files for user facing programs, how to specify icons and launcher files, where to put documentation. If there is a POSIX or XDG Desktop way to do things, you should probably prefer it.\nNext, you should be aware of the services that are provided by systemd. I\u0026rsquo;ll talk about systemd more in a later section, but on most Linux distributions (with a few narrow exceptions), if you write your software to use Systemd APIs your program is likely to be portable across many different Linux distributions. Systemd provides mechanisms to create users, monitor files, start daemons, control the network state, and much more. I would prefer systemd mechanisms second.\nFinally, I would look at specific desktop specific design guidelines such as those put out by Gnome Ubuntu KDE Elementary OS. These guidelines can give lot of insight into how to make your application fit graphically into the desktop but often tie you to a specific desktop environment. I use these methods with caution.\nSecurity Model #Another important thing to think about when writing software for Linux is how you want to secure your program. Linux provides a host of facilities to control what resources that particular programs have access to. Most of these techniques are centered around creating a service user/group which has permissions to access particular resources in particular ways.\nYou should also be aware that there are more advanced mechanisms that implement mandatory role based access control schemes enforced by the kernel. Two such examples are SELinux and AppArmor. If you need more than what permissions and groups can offer you should consider using these tools.\nWriting Daemons #What is a daemon? A daemon is an old word for long-running software that runs in the background to provide some service. There are many daemons that run on a typical machine, but examples include httpd (the web server), vsftpd (a ftp server), and even systemd (the system services daemon).\nOn Linux, there are two common types of daemons, new-style daemons and old-style daemons. New style daemons work on any system running systemd (most of them). Old style daemons work on more systems, but have far fewer default services available to them. How to write each of them is out of scope for this article, but I recommend that you read the distinction between them on Systemd\u0026rsquo;s page.\nAnother key piece of software to know about for writing user-facing daemons on Linux is DBus \u0026ndash; the daemon bus. It provides an interprocess communication mechanism that allows for services to register at both \u0026ldquo;well-known\u0026rdquo; names, but also at specific names allowing different implementations of the same services. Many of these well-known interfaces are standardized under XDG-Desktop services.\nAvailable Components #Linux has a number of infrastructure services that are available to use to build your software. A great list of services and tools is available on the Arch Linux Viki. There are probably several implementations of any particular supporting service that you need. If you need help choosing between the services I recommend you checkout the corresponding section on choosing a Linux distribution of my learning to learn Linux post since there are many similarities between choosing a distribution and choosing a particular implementation of a service.\nTraditional Package Formats #So great, you\u0026rsquo;ve written your software and now you are trying to figure out how to get it installed on someone else\u0026rsquo;s computer. That is where package management comes in. I covered the major different kinds of packages in the learning to learn Linux post. If your language provides a package management mechanism (most do), I recommend using it. Most of the major packaging systems used by each distribution have ways to convert these specific package types to the one native for each system.\nContainer Technology #The other major alternative traditional package management is container technology. On the one hand they aren\u0026rsquo;t that different. If you couldn\u0026rsquo;t get your code to install consistently without a package manager, containers aren\u0026rsquo;t going to magically fix it. What they do fix is the distribution of the dependencies that you might need.\nHere are a few key pieces of advice for building containers:\nUse a build script \u0026ndash; the more autonomous you can make the build process, the less work it will be for you in the long run. Use a container optimized distribution like \u0026ldquo;Alpine Linux\u0026rdquo;. Alpine Linux choose a great set of defaults for making minimal container images. They build libraries that you will typically need statically allowing you to install only what you absolutely need in the container to build a minimal image. Restrict the permissions of the container as much as possible. This gives you a secure by default behavior which makes you containers easier to adopt and use. Likewise, don\u0026rsquo;t run as root in the container. Use tools like .dockerignore to avoid adding unnecessary files to the container images. When building container build scripts, install the dependencies first as their own layer. This will make rebuilding the containers much faster Either use a container build system that allows you to manually specify when layers are created (buildah, moby, etc\u0026hellip;), or use scripting tools like \u0026amp;\u0026amp; to minimize the number of layers that are created. Consider setting up a caching proxy if you are going to be building containers frequently to speed up the download process. If you need to start several processes in the container, then write an entry point script that will start them or use a container management system like supervisord If at all reasonable, try to separate different daemons into separate containers. This will make it easier to update them and restrict the permissions on a single container. Consider using a management tool like docker-compose, dagger, or kubernetes when you wan to run more than a few containers on a regular basis. This will help you ensure that containers are spawned in a consistent fashion. Continuous Integration and Continuous Deployment #The two final topics related to deployment regularly talked about are continuous integration and continuous deployment (CI/CD). These deal with automating the process of building the application, running a series of tests against it, and releasing it to users. This is desirable because release management efforts are time consuming, tedious, and error prone. Automating the process means that your computer can work along side you to ensure that you are doing things correctly.\nThere are a bunch of tools for doing CI/CD. The most well known are probably Jenkins, TravisCI, AppVeyor, and GitLab. I\u0026rsquo;ve recently come to appreicate tools like dagger which enable portable CI workflows. Each offers their own distinct advantages and disadvantages that you should consider carefully when choosing which to use. Almost all of them have templates for most common languages that you can copy and paste for your applications. Setting up CI/CD often isn\u0026rsquo;t hard, but you will likely be glad that you did.\nConclusion #Becoming a software craftsman is not something that happens overnight. It takes consistent practice and effort to better your skills. I hope this article was helpful in helping you consider where you can improve and showing some ways to improve. Please let me know if you have any feedback.\nChange Log # February 2023 - Added section on security, fixed table January 2023 - Added section on libraries, updated sections on tools to explain why to use them. October 2022 - Added section on comments August 2020 - Added links to flame graphs. March 2020 - Initial Version Software that could prove that software always terminates with the correct result would require the prover to first determine that the software would terminate which has been proved in to be impossible in a computationally efficient way. See the discussion in Cook, Stephen A. \u0026ldquo;The complexity of theorem-proving procedures.\u0026rdquo; Proceedings of the third annual ACM symposium on Theory of computing. 1971.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"22 January 2023","permalink":"https://robertu94.github.io/learning/development/","section":"","summary":"\u003cp\u003eWhat is software development?\nAt a most basic level, it is the activity of using a programming language to achieve some set of goals over time.\nIt includes everything from scripts that a graduate student might write to analyze some data to massive systems that control aircraft.\nAs our world continues to progress technically, software development will likely become even more commonplace than it is now.\nIn this post, I aim to provide a comprehensive overview of how one can develop software efficiently using free and open source tools on Linux.\u003c/p\u003e","title":"Learning to Learn: Software Development on Linux"},{"content":"A few of the key things that you will need as you need as a professional are the ability to manage tasks, understand how you spend your time, and improve your own processes. For me and many others, journaling, task managers, and time trackers help you answer these important questions.\nWhat is Journaling? #Journaling is a reflective process which aims to capture tasks, observations, ideas, and lessons in order to be able to recall and learn from them. Tasks something actionable that needs to be done. Observations describe what happened \u0026ndash; akin to a lab journal. Ideas are more unspecific to be processed later. Lastly I record Lessons. Lessons record what was needed to accomplish something hard that I want to share with others.\nJournaling allows you to process the world at a slower pace. Processing the world slower helps you to make more deliberate decisions, and to recognize patterns so you can act differently into the future.\nMy current approach to journaling is based on two key sets of ideas:\nBullet Journaling Building A Second Brain What Are The Key Components? #What you journal and how you journal are deeply personal. Here are a few of the high level takeaways for me:\nFirst, treat your journal as an append only log. This is a key idea from bullet journaling. For me, journaling is most useful when I can record ideas without having to devote substantial amounts of thought to it. The more friction that there is to journaling, the less that I do it, and the less that I derive value from it. Now you may ask, \u0026ldquo;if the journal is append only, and everything is jumbled together, how can you ever find anything?\u0026rdquo; the answer is the second key take-away.\nSecond, periodically distill and reflect on your journal. In these reflections, you sort through your notes and tasks to organize what is actually important into more permanent homes. To emphasize this, bullet journaling recommends doing this with paper journals, because it requires effort to physically copy things from one journal to another, and you have limited space rather than mere copy and paste. Bullet journaling recommends a daily, weekly, and monthly reflection. For me, I do a daily reflection, and other reflections as needed. From Building a Second Brain, I learned the trick of progressive summarization which is to first go back and underline things that where important, and when you do you next \u0026ldquo;level\u0026rdquo; of reviews, review everything that was underlined and highlight the most important subset. This key practice of reviewing things periodically helps you actual derive value from your notes.\nThe larger the period of the reflection, the more substantial the reflection:\ndaily: what did I do today, what should I do tomorrow, what connections should I make? weekly: what is important this week? What was important from last week? What should I schedule? monthly: what should I stop, start, and continue doing for next month? Are my habits helpful and effective? seasonally: what are my vision/theme/aspirations for this season? How well did I live up to the aspirations for last season? What from this journal is worth migrating to the next? Types Of Journal Entries #Once you get accustomed to journaling, here are some of the other ways that I use my journal (several are taken from bullet journaling).\nDaily Log \u0026ndash; the append only log that I mentioned earlier Topic Spreads \u0026ndash; list of related things currated during my monthly review. I have ones for packing list, recipes, etc\u0026hellip; ; The key thing is to create these as needed, and not in anticipation. Mind Maps \u0026ndash; a spatial way to organize information which can be thought of as an alternative to an outline. I prefer it to brain storming because it doesn\u0026rsquo;t require me to fill the page in order, allowing me to revisit topics as I think of them. Habit Tracking \u0026ndash; a simple set of checklists for each day that I use to track key habits that I want to keep: exercise, bible reading, and journaling. You could of course make your own list. Kanban List \u0026ndash; a collection of tasks organized into blocked, new, in-progress, done, and backlog. Commonly used in software development to keep track of projects. What Is Task Management? #In addition to journaling, I also use a task management system. Tasks need to be managed or they often are left undone or done without respectfully to priority which can hurt your longer term goals. The task management system is a consistent view of all of the tasks that I need to do rather than the journal which is a strict super-set of this information. In fact, when I travel, there is a page in my journal that I create before the trip with all of the tasks to be done \u0026ndash; this is my task manager. When I am at home, I keep this information on a large whiteboard near my desk \u0026ndash; this is my task manager. Call me old school, but I like the tactile feel of moving tasks across my board or checking them off my list.\nWhat goes in the task manager? The so-called \u0026ldquo;Eisenhower Square\u0026rdquo; named for the former US president and general is a good guide. The Eisenhower square divides tasks along two axis: \u0026ldquo;urgent and not urgent\u0026rdquo; and \u0026ldquo;important and not important\u0026rdquo;. Tasks that are urgent must be done immediately or as soon as possible. Tasks that are important help you meet your broader goals and objectives. His advice then goes: A task that is urgent and important should be done immediately, A task that is urgent, but not important should be delegated if possible, A task that is not urgent, but important should be scheduled, and A task that is neither urgent nor important should be discarded.\nWhen recording a task you need to record enough to make it SMART \u0026ndash; specific, measurable, attainable, relevant, and timely.\nSpecific \u0026ndash; can you remember what it is unambiguously? Measurable \u0026ndash; how will you know when it is done? Attainable \u0026ndash; can you can actually possibly accomplish this? You won\u0026rsquo;t always know before hand but try to think this part through beforehand. Relevant \u0026ndash; how is the task related to your broader goals and themes? Timely \u0026ndash; what is the deadline or deadlines needed for this task if any? This information can help you determine know how a task fits into your larger goals as well as when and how it should be done.\nWithin my task manager, I have 5 key statuses:\nBlocked \u0026ndash; I am waiting on something or someone to do something New \u0026ndash; a task that needs to be done soon, but I haven\u0026rsquo;t started yet In Progress \u0026ndash; a task that I have started. I try to limit this to 1-3 tasks to keep focused and avoid context switching. Done \u0026ndash; tasks completed recently so I can see my progress each week. Backlog \u0026ndash; things that I would like to do someday, maybe. Together journaling and task management provide a structure that enables good task management. The journal helps you reflect on the why of your tasks and what you learned along the way. The task manager gives you a single place to look at for all that you need to do.\nTime tracking #The last key part of this trio of tools is time tracking. This is how you can measure what you actually did which enables meaningful retrospective. When I review my time spent, I try to think where was the time spent, spent where you would have liked to have spent it? Why or why not did that happened? What could be automated/delegated?\nThere are various tools and ways to do this. The tools can be automatic or manual. Manual time tracking is straightforward \u0026ndash; just log it in your journal (or in another tool) as you do things through out the day. Automatic time tracking tools use things like which applications were on screen and/or focused at through out the day. These tools often require a set of rules to tell the tracking program what each task actually was. You maybe surprised where you actually spend your time each week. The best tools give you good ways to visualize how your time was spent as a summary. I encourage you to try it at least for one whole week and see where you spend your time. Where you spend your time reveals a lot about what you really value.\nOne closing thought here on both time tracking and task management. In the words of David Sparks and Mike Schmitz a pair of podcasters who I thought put it well, \u0026ldquo;life is about more than cranking widgets\u0026rdquo;. The goal of these tools is not to enable you to do more \u0026ndash; they probably will \u0026ndash;, but to better steward the gift of the time that you have been given. Working on an endless productivity grind while initially satisfying, can never, and will never truly satisfy you. Work, stuff, accomplishments, pleasure, knowledge, wisdom \u0026ndash; all of this will eventually pass away and you will be forgotten. Use these tools to make time for what is truly important.\nTry tracking your time for a week to the nearest 15 minutes. What does it reveal about your preferences and focuses? Does it match your goals for where you spend your time? What granularity of time tracking is useful to you? Consider adopting a journaling system such as \u0026ldquo;building a second brain\u0026rdquo; or \u0026ldquo;bullet journaling\u0026rdquo; for 2 weeks. Consider the tasks in your task manager, are the tasks SMART and if not, what details do you tend to not include? why? Tools #People often ask me what tools they should use for this. I suggest that people start with paper and pen/pencil \u0026ndash; I did for years and often still do. It\u0026rsquo;s cheap and infinitely flexible. The important thing about these tools is that they help you and don\u0026rsquo;t get in the way.\nI do however now use some software and tools for specific things:\nDrafts - iOS quick notes when I am away from my desk Reminders - iOS location based reminders, time based reminders, mostly my \u0026ldquo;personal\u0026rdquo; backlog GoodNotes - iOS used and like, just had problems syncing large journal files Vim + VimWiki \u0026ndash; My favorite text editor, I sync it using a cloud file syncing service Tasks \u0026ndash; I currently use a whiteboard Calendar \u0026ndash; I use Google Calendar for home, and Microsoft 365 Calendar for work. ActivityWatch \u0026ndash; Cross platform automatic time tracking tool I hope this helps!\nChangelog # 2023-01-22 \u0026ndash; Created ","date":"22 January 2023","permalink":"https://robertu94.github.io/learning/time/","section":"","summary":"\u003cp\u003eA few of the key things that you will need as you need as a professional are the ability to manage tasks,\nunderstand how you spend your time, and improve your own processes.  For me and many others, journaling,\ntask managers, and time trackers help you answer these important questions.\u003c/p\u003e","title":"Learning to Learn: Task Management, Time Tracking, and Journaling"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/linux/","section":"Tags","summary":"","title":"Linux"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/c++/","section":"Tags","summary":"","title":"C++"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/gdb/","section":"Tags","summary":"","title":"GDB"},{"content":"GDB is a powerful tool that is underutilized by most programmers that I\u0026rsquo;ve met. It can tell you the state of one or more running or crashed programs, and even manipulate the memory of a running process. It is an invaluable tool for understanding what is going wrong with your programs.\nHow to get started #To get the most out of GDB you need to compile your program with debug information \u0026ndash; metadata that describes the program source. If writing proprietary software, be careful who gets access to debug builds of your software, it embeds the entire source code of your application into the debugging information.\nIf using gcc or clang, you can get debugging information by simply adding -g to the compilation flags and recompiling. If using CMake, you can enable debugging information by using either CMAKE_BUILD_TYPE=Debug or CMAKE_BUILD_TYPE=RelWithDebInfo \u0026ndash; similar options exist for other buildsystems and compilers.\nOnce you have a debug build you can run your program as gdb --args ./path/to/program example arguments. When the program crashes, type bt at the prompt, and you will get a trace of the stack at the time the program crashed, you can then type info locals to get a printout of all the local variables, and you can then type dis to see the assembely where the program crashed. If this was all GDB did it would be enormously helpful.\nCore functionality #Here is some of the most valuable capabilities provided by GDB:\nPrinting local variables you can print a local variable by calling p variablename. You can also print simple expressions such as p var1+var2 or p some_func(var1). You can print an array a like so p a[0]@length. C++ objects in the standard library have pretty printers so you get sane output for things like std::vector by calling p v.\nBreakpoints allow you stop execution of a program at a certain location. You can call break some_func to break on a function name and complete names with tab. For C++ which includes function overloads, you can break on specific template and overloads by tabbing out the specific implementation. You can also break at file locations break foo.cc:30. You can run until a breakpoint is hit by running continue or run to start for the first time. Breakpoints can also be conditional on boolean expressions break foo.cc:30 if a==3 to break at line 30 of file foo.cc if and only if a is 3. Lastly, you can break at an individual instruction which is powerful when you don\u0026rsquo;t have symbols\nYou can also break on C++ exceptions with catch throw and C signals with catch signal.\nRunning commands at breakpoints After defining a breakpoint, you can provide a set of commands that will run after the breakpoint is hit. For example:\nbreak foo.cc:30 commands print a contine end Would print a and then continue automatically.\nWatch points in addition to breaking on a function location, you can also break when a variable is modified. When this has hardware support it is reasonably fast, but only supports a small number of bytes. You can watch expressions such as watch left.\nIterative debugging (make) GDB allows you run make followed by run to rebuild and re-run. If you use cmake you can run !cmake --build builddir -j 8 to run builds too.\nStepping, navigating the stack GDB also allows you to single step and move up and down the stack with commands like s step into a function call, n go to the next program line, ni go to the next instruction, up go up the stack, down go down the stack, and fin to run until a function returns. However, these functions exist to supplement the functions described above instead of the primary means of using GDB.\nExamining Memory #Printing raw memory GDB allows you to print raw memory using commands like `x/8xg to print the next 64 bytes as hex. This can be useful to debug when something was overwritten or to debug without debug symbols.\nSource and where to find it #Types of debuginfo there are many kinds of debug info. On linux systems, the most common kind is dwarf (what you get with -g). Dwarf verbosity can be increased with GCC by using -g3 which includes for example macro definitions in the source code.\nStripping in addition to compiling with debug info the strip command allow removing this after the fact, this allows you to produce debug information and then re attach it later without having to ship it with every copy of the binary.\nSource directory (re-pathed) if you still have the source directory arround, you can tell GDB where to find it if it doesn\u0026rsquo;t automatically. You can provide it with symbol-file and if the path needs to be re-written, you can use set substitute-path to provide rules to re-write file paths in symbol files.\nDebuginfod linux distributions such as Fedora and CentOS now provide debuginfod servers which provide debug symbols for system libraries. This can provide symbols when they aren\u0026rsquo;t installed by default.\nSpack By default, spack compiles cmake projects with RelWithDebInfo which provides basic debugging symbols, however the source is not installed by default. By using spack envionments and spack develop it does the right thing, and you can get the source.\nDebugging Libraries #Listing what shared libraries are loaded sometimes you want to know what libraries where loaded. info sharedlibrary to get a list of them. I used this to detect a version mismatch between uses of two different zfp libraries that were both installed.\nMulti Process/Thread #GDB provides abilities to debug multi-process and multi-thread programs as well as those that call exec. The follow-fork and follow-exec commands do what you would expect. You can use catch fork and catch exec to break on fork and exec.\nNon-stop mode and background commands** allows non-main threads to continue to run when the focused thread is stopped. This is essential in many cases for debugging multi-threaded programs. To debug multiple threads simultaneously you will often need to run commands in the background by appending the \u0026amp; character to them. You can re-forground a thread by usingthe interrupt command\nAdvanced ways to start GDB #In addition to starting a command under gdb, there are two other key ways to use GDB \u0026ndash; core dumps and attaching to running processes.\nCore dumps #What is a core dump Core dumps are a copy of all registers and memory. These can be taken at any time, but are created automatically when a signal such as sigsegv terminates a process.\nEnabling core dumps if they are not enabled, there are a few key steps to enable them: If you are using systemd, you can use coredumpctl gdb if you are a member of the journald group. If you are not a member of journald, and your system uses systemd, you don\u0026rsquo;t really have a way to enable core dumps without root privileges. If you are not using systemd, the core gets written to the location specified by kernel.core_pattern sysctl. Lastly, the process has to have a positive core dump limit which are defaulted by /etc/security/limits.conf\nUsing core dumps once you have a core file, you can debug a core file with gdb -c ./path/to/core ./path/to/program You can\u0026rsquo;t use breakpoints, but you can inspect local variables and final stack information.\nConnecting to a running process #Connecting GDB to a running process require privileges \u0026ndash; namely CAP_SYS_PTRACE or CAP_SYS_ADMIN \u0026ndash; to attach to a process. You also will need permissions (e.g., user or gropus) to access the running process. You can connect to a running process with gdb -p $PID /path/to/command.\nGDB Server #Beyond basic uses of GDB to connect to remote processes, GDB can also attach to remote processes or embeded devices. I\u0026rsquo;ve written a program called mpigdb that uses this functionality to debug MPI programs. To use this functionality start the program under gdbserver with a connection string and other flags you would otherwise pass to gdb. Once the GDB server is started, you can connect to a GDB server with target remote.\nThis functionality is also used for embedded devices. Lastly, GDB server functionality is also used with Valgrind\u0026rsquo;s interactive memory sanitizer to get much more information about memory leaks at runtime.\nOther Advanced Features #GDB has many more advanced features, and only so many of them can be covered here. Here are a few more of the ones that I find very useful\nGDB Scripting #Beyond just running commands at breakpoints, you can put a set of commands in to a file to be executed when GDB starts by passing -x ./path-to-script.gdb.\nSettings you\u0026rsquo;ll want to adjust To write effective scripts there are a few settings you will want to adjust:\nsafe mode \u0026ndash; what directories GDB can load scripts from pagination \u0026ndash; GDB pauses when pagniation happens so using it scripts often requires turning this off. fork/exec/non-stop modes \u0026ndash; set the fork and exec modes you need to be appropriate for your application Setting variables often when writing scripts requires needing variables. GDB allows what are called convenience variables \u0026ndash; variables which are not stored in the program\u0026rsquo;s memory. These variables are prefixed with a $ character and set with the set command.\nPython Extensions #If GDB scripts in it\u0026rsquo;s scripting language are too primitive, you can also write extension commands and pretty printers in Python. These are simple python classes that have access to most of the capabilities of GDB.\nThese functionalities have been used for capabilities like debugging Python modules with C extensions. For example when running a python program (assuming everything has been configured correctly), you can call py-bt to get the python strack trace in addition to the C/C++ backtrack allowing you to vastly improve your ability to debug things.\nWhere to learn more? #I\u0026rsquo;ve just scratched the surface. There are many more powerful features that I haven\u0026rsquo;t discussed such as reverse debugging, trace points, and more.\nFor more information see the GDB documentation\nMany students that I know only use the run, continue, break, step, fin, backtrace and print and next functions. Which if any of these GDB features have you used? How could they improve your debugging process? Write a gdb script that breaks on a function when an argument is a particular value and prints a backtrace before continuing. I hope this helps!\nChangelog # 2023-01-25 Created this document. ","date":"22 January 2023","permalink":"https://robertu94.github.io/learning/gdb/","section":"","summary":"\u003cp\u003eGDB is a powerful tool that is underutilized by most programmers that I\u0026rsquo;ve met.\nIt can tell you the state of one or more running or crashed programs, and even manipulate the memory of a running process.\nIt is an invaluable tool for understanding what is going wrong with your programs.\u003c/p\u003e","title":"Learning to Learn: GDB"},{"content":"MPI is the de-facto standard way to write distributed programs that run on super computers. Many have tried to replace it, but so far none of them have succeeded. Learning to use it successfully, will enable you to write powerful distributed programs.\nOrder of Topics #MPI has a minimal powerful and useful core, but really tries to completely own it\u0026rsquo;s space.\nI strongly reccommend reading \u0026ldquo;Using MPI\u0026rdquo; by Gropp et al. which opened my eyes to the power of MPI I would focus on learning MPI in the following order:\nMPI_Init and MPI_Finalize The 6 basic routines Collectives Communicators to preform collectives on subsets of ranks Asynchronous MPI MPI\u0026rsquo;s Advanced features (inter-communicators, groups, error handling, info structures, dynamic processes, File IO, One-Sided Communication) Research features included in major MPI distributions such as Fault Tolerant MPI (i.e. ULFM) Beginning MPI \u0026ndash; Important Functions #You really can do an amazing amount with just 6 routines:\nMPI_Init \u0026ndash; startup MPI MPI_Finalize \u0026ndash; shutdown MPI MPI_Send \u0026ndash; send a message from a source to a target MPI_Receive \u0026ndash; receive a message from a source at the target MPI_Comm_rank \u0026ndash; ask for the ID of a process MPI_Comm_size \u0026ndash; ask for the total number of processes With these functions you have all of the primitives that you need to write distributed programs. However very quickly, you will want to have groups of processes work together to solve common problems. For this MPI provides collectives:\nMPI_Bcast \u0026ndash; send from one rank to all ranks MPI_Reduce \u0026ndash; compute a reduction from all ranks send the result to one rank MPI_AllReduce \u0026ndash; compute a reduction from all ranks send the result to all ranks MPI_Gather \u0026ndash; send data from all ranks combine them for one rank MPI_Scatter \u0026ndash; split and send data from one rank to all ranks You also will quickly want to compute collectives subsets of processes. For this, MPI_Comm_split and MPI_Comm_split_type.\nHowever to scale, you frequently need non-blocking versions of these calls. Many routines have an I variant which is non-blocking such as MPI_Ibcast for MPI_Bcast. However MPI provides much, much more capabilities allowing collectives and failures to be isolated to a subset o processes. I recommend you learn these as you need them.\nImportant Tools / 3rd party libraries #Debugging Debugging MPI programs can be tricky without appropriate tools. There are specialized debuggers such as TotalView and DDD which are propriety and very expensive. I\u0026rsquo;ve written a simply version built around standard MPI features and new versions of GDB called mpigdb\nBoost.MPI MPI 2 introduced a standard C++ api. Almost no-one used it, and it was removed in MPI 3. If you want a C++ API, either use the C api (which is quite serviceable), or use the Boost.MPI version. I\u0026rsquo;ve also written a lighter weight version called libdistributed provides basic features send, recv, and bcast for C++ types as well as a work-queue.\nTAU (profiler) If you need to profile an MPI application and you don\u0026rsquo;t want to use the provided PMPI features directly, you can use tau. Tau will profile most of the events that you care about, and can be a good place to start.\nMath libraries (PETSc/ScaLAPACK/FFTw/etc) If you want to do things that look like linear algrebra or a fourier transform, don\u0026rsquo;t write them yourself, use one of the standard libraries that allow distributing things using MPI.\nGraph Libraries If you need to do graph algorithms, Boost\u0026rsquo;s graph library is MPI aware and can be distributed across the network.\nStorage Libraries (parallel HDF5) If you need to write data, HDF5 does a number of optimizations to write data from a large number of nodes much more efficiently than a naïve use of POSIX APIs can be. You can also use mpiio instread of hdf5 if you want to avoid a dependency, but hdf5 frequently does what you want, and is much more full featured.\nMajor Concepts Worth Knowing About #A few other major topics that may be deceptively simple, but actually provide substantial depth:\nTags MPI_Send and MPI_Recv allow for a tag object. If multiple sends or receives are in flight simultaneously, they can be matched based on the tag, or you can use MPI_ANY_TAG to recieve any tag. This allows for a work queue style workflow which a lead process checks for any of a collection of worker processes has completed some task.\nMPI Communicators in addition to serving as the basis for collectives, MPI communicators provide a powerful set of abstractions for working with subsets of processes, customizing error handling, and segregating messages sent to different nodes. A messages sent via MPI_Send and MPI_Recv only actually matches if the communicator and tag also match. This allows libraries to segregate their own communication from the communication from the user of the library.\nMPI+Threads by default, MPI does not enable thread safety unless started with MPI_Init_thread and the library was compiled with thread support. If you have a multi-threaded program, you probably want this flag enabled.\nNonblocking MPI Nonblocking MPI uses MPI_Request objects. You can check them with MPI_Test or block waiting for them MPI_Wait. If MPI_Test and MPI_Wait will de-allocate the MPI_Request object if the indicate that the request was completed. If you need this to be idempotent, you need to maintain boolean to track if this call ever returned success for a particular call. There also calls for MPI_Waitany MPI_Waitsome and MPI_waitall to wait for groups of requests at the same time.\nMPI Datatypes allow you to send data structures that have irregular shapes (i.e. arrays, structs, etc\u0026hellip;) in a single MPI_Send or MPI_Recv calls which can dramatically simplify code that uses these.\nOne-sided MPI allows you to preform one-sided RDMA operations including one-sided atomic operations can offer substantial performance increases over two sided MPI operations.\nWhat\u0026rsquo;s Next # Read \u0026ldquo;Using Advanced MPI\u0026rdquo; by Gropp et al. is a great next step after reading \u0026ldquo;Using MPI\u0026rdquo; William \u0026ldquo;Bill\u0026rdquo; Gropp also has an excelent set of lecture slides on MPI that aren\u0026rsquo;t too hard to find online. Read the Standard \u0026ndash; as far as programming system standards go, this document is remarkably easy to read. What features have you used from MPI? What aspects of MPI would simplify your distributed programs further? Changelog #","date":"22 January 2023","permalink":"https://robertu94.github.io/learning/mpi/","section":"","summary":"\u003cp\u003eMPI is the de-facto standard way to write distributed programs that run on super computers.\nMany have tried to replace it, but so far none of them have succeeded.\nLearning to use it successfully, will enable you to write powerful distributed programs.\u003c/p\u003e","title":"Learning to Learn: MPI"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/mpi/","section":"Tags","summary":"","title":"MPI"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/clang/","section":"Tags","summary":"","title":"Clang"},{"content":"Have you ever wanted to identify a list of files that would need modifications to adopt a new API? Clang-Query can make this much easier. I recently wanted to introduce a set of helper functions to simplfy an aspect of compressor configuation in LibPressio. But first, I needed to know what modules were effected.\nClang-query needs a compilation database which can be produced by tools such as bear (if you have a Autotools or Makefile based project) or more sophisticated build systems such as cmake, meson or bazel. After we have these, the files are just JSON, so we can produce a list of files that are effected using jq and reduce the set considered using the select function.\njq \u0026#39;.[].file | select(test(\u0026#34;src/plugins/\u0026#34;))\u0026#39; ./build/compile_commands.json However clang-query needs relative paths (This seems like a bug to me, but 🤷) so we can convert them using | xargs realpath --relative-to .\nclang-query -p ./build/compile_commands.json $( jq \u0026#39;.[].file | select(test(\u0026#34;src/plugins/\u0026#34;))\u0026#39; ./build/compile_commands.json | xargs realpath --relative-to . ) Clang-query will then construct a C++ AST for each of the files which we can then query.\nI knew that the effected modules would provide a definition of a method called set_name_impl and would exist in a certain source directory. The corresponding match expression is\ncxxMethodDecl(hasName(\u0026#34;set_name_impl\u0026#34;), isExpansionInMainFile()) which in turn returns all of the files that should be effected. However, the method that needs to be actually needs to be modified is get_configuration_impl and we want to see if we missed any places that need to be modified, so we can look for the old pattern:\nset traversal IgnoreUnlessSpelledInSource set bind-root false match cxxMethodDecl(hasName(\u0026#34;get_configuration_impl\u0026#34;), hasDescendant(cxxMemberCallExpr(hasArgument(0, unless(cxxMemberCallExpr(on(memberExpr())))), hasDeclaration(cxxMethodDecl(hasName(\u0026#34;copy_from\u0026#34;)))).bind(\u0026#34;r\u0026#34;)), isExpansionInMainFile()) Determing the appropriate matcher can be done iteratively using tab completion, binding, and the dump output mode.\n","date":"28 December 2022","permalink":"https://robertu94.github.io/posts/2022-12-28-refactoring-with-clang-query/","section":"Posts","summary":"\u003cp\u003eHave you ever wanted to identify a list of files that would need modifications to adopt a new API?\nClang-Query can make this much easier.\nI recently wanted to introduce a set of helper functions to simplfy an aspect of compressor configuation in LibPressio.\nBut first, I needed to know what modules were effected.\u003c/p\u003e","title":"Refactoring With Clang Query"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/julia/","section":"Tags","summary":"","title":"Julia"},{"content":"In the past, I\u0026rsquo;ve written pretty glowingly about Julia. It\u0026rsquo;s been a few years since I first used Julia in 2019, and it hasn\u0026rsquo;t completely replaced Python for me. However, I wanted to share a few neat projects that I\u0026rsquo;ve done using it which would have been much more painful without it, and share what I think now about what I wrote in 2019.\nImplementing a Statistical Metric on the GPU #Being able to access heterogenous compute is important to be able to make the most of the availible hardware. Doing so almost always requiring jumping to another (often device specific) language to implmenet a mix of kenrels and high-level reductions. While Julia doesn\u0026rsquo;t allow you to completely get away from device specific code for more than simple cases (for now), it does give you an easy way to mix high level reductions and low-level kernels to run on the GPU.\nThis code computes a metric that I call the quantized entropy which highlights each of Julia\u0026rsquo;s capabilities here:\nWe can easily define a kernel \u0026ndash; even one that uses more advanced kernel features like atomics Writing reductions \u0026ndash; even reductions that aren\u0026rsquo;t provided upstream \u0026ndash; is really easy Many operations (i.e. BLAS/LaPACK operations) just work and are applied on the GPU just by casting to a device array type. This code is pretty concise 40 lines of code. I implemented another metric that uses Multi-GPU capabilies that 36 lines of code and leveraged linear algebra capabilities of the native device libraries which is remarkably consise.\nusing CUDA function quantize_kernel(cu_data, global_bins, N::Int64, dmin::Float32, abs::Float32) global_idx = threadIdx().x + blockDim().x * (blockIdx().x -1) if global_idx \u0026lt;= N bin_idx = trunc(Int32,(cu_data[global_idx] - dmin)/abs) + 1 @inbounds CUDA.@atomic global_bins[bin_idx] += 1 end return end function extreem(x:: Tuple{T,T}, y:: T)::Tuple{T,T} where{T} (min_x, max_x) = x; return (min(min_x, y), max(max_x, y)) end function extreem(x:: T, y:: Tuple{T,T})::Tuple{T,T} where{T} (min_y, max_y) = y; return (min(x, min_y), max(x, max_y)) end function extreem(x:: Tuple{T,T}, y:: Tuple{T,T})::Tuple{T,T} where{T} (min_x, max_x) = x; (min_y, max_y) = y; return (min(min_x, min_y), max(max_x, max_y)) end GPUArrays.neutral_element(::typeof(extreem), T) = (floatmax(T), floatmin(T)) function cuda_qentropy(data; abs=1e-4) cu_data = CuArray(data); N = length(cu_data) (dmin,dmax) = reduce(extreem, cu_data; init=(floatmax(eltype(data)),floatmin(eltype(data)))) bin_counts = round(Int64,(dmax-dmin)/abs) + 1 bins = CUDA.zeros(Int32, bin_counts) n_threads=32 n_blocks=trunc(Int,(N + n_threads+1) / n_threads) @cuda threads=n_threads blocks=n_blocks quantize_kernel(cu_data, bins, N, dmin, abs) -mapreduce(+, bins; init=0.0) do bin if bin != 0 prop = convert(Float64,bin)/N; prop * log2(prop) else 0.0 end end end Implmenting an Interactive Visualization of Slices of 3D Data. #Understanding how data is structured is much easier with interactive tools, but depending on what structure you are looking for, there aren\u0026rsquo;t always pre-built tools that will help you understand that strucuture. I\u0026rsquo;ve consistently been frustrated with the apparent inconsistencies of tools like matplotlib, and I was impressed with what I could build in just a few lines of code.\nThis code highlights\nHow to apply layouts and multiple figure types with ease How to easily make them interactive via reactive programming. How to incorperate keyboard interactivity This is 36 lines of code creates an interactive visualization that shows both a histogram and heatmap of slices of 3d data that can be updated with either a slider or with keyboard commands.\nusing GLMakie function vis_explorer(args::SliceExploreArguments) data = Array{args.type}(undef, args.dims...) read!(args.filename, data) n_slices = args.dims[end] min,max = extrema(data) fig = Figure() s = Slider(fig[2,1:2], range=1:1:n_slices, startvalue=round(Int,args.dims[end]/2)) slice = lift(s.value) do v data[:,:, v] end slice_vec = lift(s.value) do v vec(data[:,:, v]) end hist_title = lift(s.value) do v \u0026#34;Histogram slice $v\u0026#34; end img_title = lift(s.value) do v \u0026#34;SliceView slice $v\u0026#34; end hist_ax = Axis(fig[1,1], title=hist_title, limits=((min,max), nothing)) hist!(hist_ax, slice_vec) img_ax = Axis(fig[1,2], title=img_title) hm = heatmap!(img_ax, slice) Colorbar(fig[:,end+1], hm) on(events(fig).keyboardbutton) do event if event.action == Keyboard.press || event.action == Keyboard.repeat if event.key == Keyboard.k set_close_to!(s, s.value[]+1) elseif event.key == Keyboard.j set_close_to!(s, s.value[]-1) end end end fig end Implementing a Distributed Experiment #Writing experiments that spans several nodes is the bread and butter of writing code in HPC. While it is possible to use libraries such as mpi4py or libdistributed to write experiments that run on any nodes, Julia provides a set of libraries that make this nearly effortless, and many of them are part of the standard library.\nIn this code you can see:\nA distributed for loop where communications between nodes were serialized and abstracted away The ability to distribute libraries and data to all nodes with the @everwhere macro A simple way to store complex metrics to standard formats like CSV even when not all experiments return the same keys And all of this comes in a mere 36 lines of code.\nusing Distributed using Base.Iterators addprocs(6) template = Array{Float32}(undef, 500, 500, 100) read!(expanduser(\u0026#34;~/git/datasets/hurricane/100x500x500/CLOUDf48.bin.f32\u0026#34;), template) # broadcast libraries and data to workers @everywhere using Pressio @everywhere input_data = $template @everywhere library = pressio() configurations = collect((product([\u0026#34;sz\u0026#34;, \u0026#34;zfp\u0026#34;], exp10.(range(-1, -6, length=6))))) results = @sync @distributed vcat for (comp_id, abs) in configurations try comp = compressor(library, comp_id) options = Dict{String,Any}( \u0026#34;pressio:abs\u0026#34; =\u0026gt; abs, \u0026#34;pressio:metric\u0026#34; =\u0026gt; \u0026#34;size\u0026#34;, ) set_options(comp, options) compress(comp, input_data) result = get_metrics_results(comp) result[\u0026#34;error\u0026#34;] = missing catch e result = Dict{String, Any}() result[\u0026#34;error\u0026#34;] = e end result[\u0026#34;comp_id\u0026#34;] = comp_id result[\u0026#34;abs\u0026#34;] = abs [result] end using DataFrames, Tables, CSV CSV.write(\u0026#34;/tmp/results.csv\u0026#34;, Tables.dictcolumntable(results), transform=((col, val) -\u0026gt; something(val, missing))) What Are My Thoughts Since 2019? #I largely stand by my 2019 review. The good things about Julia are still good, and if not have gotten better. While, I still find the matlabisms annoying, I eventually learned enougth of them to be productive. However, the package ecosystem has matured substantially in just 2 years with libraries especially in the areas of language interpop, accllerated computing, and mathmatical computing are now among best in class, but other areas still are lacking. I even think discoverablity has improved with podcasts (although mostly on hiatus) like Talk Julia and the Julia Bloggers and Julia Conn online pressenece improving knowledge of what is being developed in the Julia community. However discoverablity of what function to use could still be improved, but has gotten better with things like LanguageServer.jl.\nHowever, I think 3 things are still largely keeping me from using it more:\nReliance on specific Python or C++ libraries. If a project is 90% relying on a library in some other langauge (i.e. Tensorflow/PyTorch, libfabric) it sometimes makes sense to just use that language. Finding Julia based alternatives takes time, and isn\u0026rsquo;t always worth the effort Lack of knowledge among my colleagues. A big part of my work is sharing my code with others, and many of my coleagues don\u0026rsquo;t use Julia. Although for some packages, it has gotten way better, some packages still have really long pre-compile times (i.e. time to first plot). Hope this helps!\n","date":"15 December 2022","permalink":"https://robertu94.github.io/posts/2022-12-15-three-neat-things-with-julia/","section":"Posts","summary":"\u003cp\u003eIn the past, I\u0026rsquo;ve \u003ca href=\"https://robertu94.github.io/posts/2019-03-21-julia-could-there-be-one/\"\u003ewritten pretty glowingly about Julia\u003c/a\u003e. It\u0026rsquo;s been a few years\nsince I first used Julia in 2019, and it hasn\u0026rsquo;t completely replaced Python for\nme. However, I wanted to share a few neat projects that I\u0026rsquo;ve done using it\nwhich would have been much more painful without it, and share what I think now\nabout what I wrote in 2019.\u003c/p\u003e","title":"Three Neat Things I Did With Julia"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/debugging/","section":"Tags","summary":"","title":"Debugging"},{"content":"High-performance network interconnects, such as Infiniband, are common in high performance computing environments. Recently, my colleagues, and I ran a series of experiments on the Cooley system at the Argonne Leadership Computing Facility. For one set of experiments, the network performance was consistent and fast, but for the others we periodically had poor performance. Up until this point in my work, I had not directly used or configured network libraries below this layer. This post summarizes the steps we took to investigate the performance.\nNot knowing what the possible cause was we considered that the input, configuration, software or hardware could be different. As far as we knew, we were providing the same input to both runs. We ran ldd on the associated binaries for each experiment and confirmed that each experiments were loading the same dynamic libraries giving us confidence that it wasn\u0026rsquo;t the software. Not being super familar with the hardware hardware was on the node. We ran hwloc-ls to see the Node hardware.\n$ hwloc-ls Machine (378GB total) ...\u0026lt;snip\u0026gt;... NUMANode L#1 (P#1 189GB) Package L#1 + L3 L#1 (15MB) L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#6) L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#7) L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#8) L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#9) L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#10) L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#11) HostBridge L#7 PCIBridge PCI 15b3:1011 Net L#10 \u0026#34;ib1\u0026#34; Net L#11 \u0026#34;ib0\u0026#34; OpenFabrics L#12 \u0026#34;mlx5_0\u0026#34; PCIBridge PCI 15b3:1003 Net L#13 \u0026#34;eth2\u0026#34; OpenFabrics L#14 \u0026#34;mlx4_0\u0026#34; The output to our surprise lists two network interfaces: one was an Mellanox version 4 card (mlx4_0) and the other was a Mellanox version 5 (mlx5_0) card. We noted the network device names, and that it one of them was using Ethernet and one was using Infiniband protocol at Layer 2.\nWe then we were curious what devices each of our experiments were configured to use. One set or experiments was using the parallel file system, and the others were using Mochi \u0026ndash; a libfabric based networking library. Running mount told us that the experiments using the file system were transiting over the newer card, but we didn\u0026rsquo;t know what the libfabric library was doing. Libfabric provides some debugging command like fi_info and fi_pingpong. Running fi_info -p verbs we saw both devices were available with the verbs provider, and interestingly the mlx4_0 was returned first. That didn\u0026rsquo;t mean that our code was necessarily using this interface, but it was a hint. We then printed out the address of the clients and servers as determined by our high level library, and sure enough it was using the mlx4_0 card which we confirmed against the output of ip addr.\nNow this wasn\u0026rsquo;t enough to declare victory. We next tried to replicate the issue with fi_pingpong -vvv on both client and server, and sure enough when using the mlx4_0 card we saw the same kinds of hangs we did in our experiments. We then tried the same thing with the mlx5_0 card, and we didn\u0026rsquo;t see any more hangs. We then reconfigured our Mochi based experiments to use ofi+verbs://mlx5_0 to use this card, and sure enough we were able to resolve the issue with hangs.\nHope this helps!\n","date":"14 December 2022","permalink":"https://robertu94.github.io/posts/2022-12-14-debugging-libfabric/","section":"Posts","summary":"\u003cp\u003eHigh-performance network interconnects, such as Infiniband, are common in high performance computing environments. Recently, my colleagues, and I ran a series of experiments on the Cooley system at the Argonne Leadership Computing Facility. For one set of experiments, the network performance was consistent and fast, but for the others we periodically had poor performance. Up until this point in my work, I had not directly used or configured network libraries below this layer. This post summarizes the steps we took to investigate the performance.\u003c/p\u003e","title":"Debugging Inconsistent Network Performance"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/networking/","section":"Tags","summary":"","title":"Networking"},{"content":"CMake is the de-facto C++ build system used by an overwhelming number of C++ projects. Even if you personally favor more modern alternatives such as Meson, Bazel, or Pants, if you ever pull in a 3rd party dependency, there is a good chance that it uses CMake so knowing enough about CMake to understand it is worth knowing.\nHow to get started #I recommend new users to CMake start with the following resources:\nMastering CMake is a high level overview of how to use CMake developed and maintained by the CMake Developers. Modern CMake Is focuses on a useful subset of CMake for a number of common tasks. Using CMake to build something #If a package cmake 3.15 or newer (most people), the following sequence will build and install a cmake project\ncmake -S ./path/to/sourcedir -B ./path/to/builddir cmake --build -j $(nproc) cmake --install Here ./path/to/sourcedir is the directory where you source files (specifically the \u0026ldquo;toplevel\u0026rdquo; CMakeLists.txt is stored). and ./path/to/builddir is a directory (which may not exist) where you want to store build artifacts prior to installation.\nYou can customize the build by passing flags to the first cmake command. I commonly use the following\ncmake -S ./path/to/sourcedir -B ./path/to/builddir \\ -G Ninja \\ -DCMAKE_INSTALL_PREFIX=$(pwd)/.local \\ -DBUILD_SHARED_LIBS=ON \\ -DBUILD_TESTING=ON \\ -DCMAKE_BUILD_TYPE=Debug \\ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \\ -DCMAKE_C_COMPILER_LAUNCHER=ccache \\ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON cmake --build -j $(nproc) cmake --install To prefer the Ninja build generator, and ccache for faster development builds with shared libraries installed to my user\u0026rsquo;s prefix instaed of the the / or /usr prefix which may require admin privledges and to enable LLVM based tooliing (i.e. clangd for completions in my editor). These preferences are encoded into my m build tool.\nCMake also respects GNU style envionment variables (i.e. CXX, CC, CFLAGS, CXXFLAGS, and LDLIBS) to pick up common defaults.\nKey Commands #Required boilerplate that must appear at the top of your top-level cmake\ncmake_minimum_required project Basics\nadd_library and add_executable create build objects target_link_libraries and PUBLIC vs PRIVATE to consume dependancies (including their header file flags) and link libraries target_include_directories to set include paths not provided by link libraries target_compile_features a portable way to set the C++ standard The most essential cmake variables $CMAKE_CURRENT_SOURCE_DIRECTORY and $CMAKE_CURRENT_BINARY_DIRECTORY Basic generator expressions such as $\u0026lt;BUILD_INTERFACE:\u0026gt; vs $\u0026lt;INSTALL_INTERFACE:\u0026gt; for use with target_include_directories add_subdirectory for code organization Adding 3rd party dependencies. Do yourself a favor and put all dependencies imports at the top level so the scope is right for the whole project.\nfind_package to import dependancies built using CMake and certain system dependancies such as threads, MPI, Python, and OpenMP. find_package(PkgConfig) and pkg_search_modules to import dependencies described by pkg-config FetchContent download and provide dependencies as part of the build, but consider instead third party tools such as spack, connan, and vcpackage. I\u0026rsquo;ve written a guide on CMake package dependencies here.\nHow do you currently handle dependencies and builds in your system? Is this portable to other machines where you would like to use this software? If not, what do you need to change and why? Build Options\noption to add build options if for basic control flow configure_file to add a \u0026ldquo;configure file\u0026rdquo; which is typically header which has #defines for certain build system variables set with #cmakedefine or #cmakedefine01 target_add_sources to add additional source files to a library after the fact for example to compile in an optional plugin. Installation\ninclude(GNUInstallDirs) and its associated variables like CMAKE_INSTALL_PREFIX, CMAKE_INSTALL_INCLUDEDIR , and CMAKE_INSTALL_LIBDIR install(TARGETS) to install your libraries and their export files install(DIRECTORY) to install your header files and data files include(CMakePackageConfigHelpers) and configure_package_config_file and write_basic_package_version_file to make your package importable by others See this project for an example on how to properly expose a CMake dependency\nTesting\ninclude(CTest), BUILD_TESTING, and add_test include(GoogleTest) and gtest_discover_tests for high quality integration of googletest based tests CMake Magic Variables to reconfigure builds\nCMAKE_BUILD_TYPE automatically configure optimizations like -O2 and -Og -g with settings like Release or Debug there are also other settings for RelWithDebInfo and MinSizeRel for small build artifacts BUILD_SHARED_LIBS to choose between shared and static linking INTERPROCEDURAL_OPTIMIZATION property enables IPO for faster optimizations Enabling 3rd Party technologies #CMake provides most of what you want on its own, but a few tools are worth knowing about:\nspack a dependencies manager for HPC software connan and vcpackage a dependency managers more common in enterprise ccache and sccache can be provided to CMAKE_\u0026lt;Lang\u0026gt;_COMPILER_LAUNCHER to dramatically speed up re-builds builds ninja is a much faster project builder than Make. You can enable it with -G Ninja passed to cmake ccmake a terminal user interface for setting cmake build options clang-tidy and include-what-you-use for extra static analysis can be used by setting the \u0026lt;LANG\u0026gt;_CLANG_TIDY and \u0026lt;LANG\u0026gt;_INCLUDE_WHAT_YOU_USE Doxygen for automatic documentation form the header-files of your project. Can be used automatically from CMake with find_package(Doxygen) Important Concepts # PUBLIC, INTERFACE vs PRIVATE dependency and configurations SHARED, STATIC, INTERFACE vs IMPORTED libraries. The later you will use directly only seldom but is used for find_package internally. that target_* functions accumulate from everywhere they are used allowing code that adds specific features to be spread out Don\u0026rsquo;t use globbing to add files to a build. List them explicitly for best performance Debugging CMake #CMake can be obtuse at times. A few key commands can help:\n--trace puts cmake into trace mode to print all calls made in a CMake build system. CMAKE_FIND_DEBUG_MODE prints out extra information when finding a packages that can be used to track down how a variable was set. Newer versions have a flag --debug-find-pkg= which can enable this for specific packages. message(WARNING ...) print a warning message to the console with the specified message can be used for tracing and viewing values of a variable at a point in the code Advanced Topics # for, function, and macro for advanced control flow execute_process for when you just need to run a script CUDA and other accelerator language support Where to learn more # The Official CMake Documentation Changelog # 2023-02-09 linked to other cmake resources added section on debugging 2023-01-09 Added basic usage example 2022-11-15 Created ","date":"15 November 2022","permalink":"https://robertu94.github.io/learning/cmake/","section":"","summary":"\u003cp\u003eCMake is the de-facto C++ build system used by an overwhelming number of C++ projects.\nEven if you personally favor more modern alternatives such as Meson, Bazel, or Pants, if you\never pull in a 3rd party dependency, there is a good chance that it uses CMake so\nknowing enough about CMake to understand it is worth knowing.\u003c/p\u003e","title":"Learning to Learn: CMake"},{"content":"tl;dr #Put the following in your .bashrc\nuse_build() { if hostname | grep summit \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading crusher spack\u0026#34; module swap PrgEnv-cray PrgEnv-gnu module load craype-accel-amd-gfx90a module load rocm export clustername=crusher fi #other OLCF Machines export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; source $HOME/git/spack-crusher/share/spack/setup-env.sh source $HOME/git/spack-crusher/share/spack/spack-completion.bash } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-crusher source ~/.bashrc spack compiler find First create packages.yaml at ~/.spack/crusher/packages.yaml with:\ncmake: externals: - spec: cmake@3.17.0 prefix: /usr all: providers: mpi: [cray-mpich] hdf5: externals: - spec: hdf5@1.12.2+mpi+shared+hl prefix: /opt/cray/pe/hdf5-parallel/1.12.1.5/gnu/9.1 buildable: False cray-mpich: externals: - spec: cray-mpich@8.1.16 modules: [cray-mpich/8.1.16, libfabric/1.15.0.0, craype-accel-amd-gfx90a, rocm/5.1.0] buildable: false After that create compilers.yaml at ~/.spack/crusher/cray/compilers.yaml with:\ncompilers: - compiler: spec: gcc@12.1.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: sles15 target: any modules: - PrgEnv-gnu - gcc/12.1.0 - cray-mpich/8.1.16 - craype-accel-amd-gfx90a - rocm/5.1.0 environment: set: PE_MPICH_GTL_DIR_amd_gfx90a: \u0026#34;-L/opt/cray/pe/mpich/8.1.16/gtl/lib\u0026#34; PE_MPICH_GTL_LIBS_amd_gfx90a: \u0026#34;-lmpi_gtl_hsa\u0026#34; And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # Crusher\u0026rsquo;s module system seems to be more picky than most needing a rocm and craype-accel to be present for most uses. We provide these. There are also some magic environment variables that the OLCF docs said to add to the environment. We use Cray\u0026rsquo;s provided MPI and HDF5 to get the most oout of the platform. We load a GNU compiler from a module because it seems to work most reliably Crusher shares a home filesystem with other machines like Summit these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Crusher and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH to keep these cleanly separate. Changelog # 2022-10-26 created this document ","date":"26 October 2022","permalink":"https://robertu94.github.io/guides/spack/crusher/","section":"","summary":"Configure spack for ORNL OLCF Crusher","title":"Spack on Crusher"},{"content":"tl;dr #Put the following in your .bashrc\nuse_build() { if hostname | grep summit \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading summit spack\u0026#34; module load gcc/10.2.0 export clustername=summit fi #other OLCF Machines export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; source $HOME/git/spack-summit/share/spack/setup-env.sh source $HOME/git/spack-summit/share/spack/spack-completion.bash } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-summit source ~/.bashrc spack compiler find First create packages.yaml at ~/.spack/summit/packages.yaml with:\npackages: all: providers: mpi: [spectrum-mpi] openssl: externals: - spec: openssl@1.1.1c prefix: /usr buildable: False openssh: externals: - spec: openssh@8.0p1 prefix: /usr buildable: False cuda: externals: - spec: cuda@11.0.221 prefix: /sw/summit/cuda/11.0.3 buildable: False spectrum-mpi: externals: - spec: spectrum-mpi@10.4.0.03rtm4 prefix: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-10.2.0/spectrum-mpi-10.4.0.3-20210112-ht5bw4jruhjujvkzcvpra5lryg5vfhy4 buildable: False tar: externals: - spec: tar@1.30 prefix: /usr cvs: externals: - spec: cvs@1.11.23 prefix: /usr diffutils: externals: - spec: diffutils@3.6 prefix: /usr groff: externals: - spec: groff@1.22.3 prefix: /usr bison: externals: - spec: bison@3.0.4 prefix: /usr gawk: externals: - spec: gawk@4.2.1 prefix: /usr pkgconf: externals: - spec: pkgconf@1.4.2 prefix: /usr m4: externals: - spec: m4@1.4.18 prefix: /usr git: externals: - spec: git@2.18.4~tcltk prefix: /usr binutils: externals: - spec: binutils@2.30.73 prefix: /usr automake: externals: - spec: automake@1.16.1 prefix: /usr flex: externals: - spec: flex@2.6.1+lex prefix: /usr coreutils: externals: - spec: coreutils@8.30 prefix: /usr autoconf: externals: - spec: autoconf@2.69 prefix: /usr gmake: externals: - spec: gmake@4.2.1 prefix: /usr libtool: externals: - spec: libtool@2.4.6 prefix: /usr findutils: externals: - spec: findutils@4.6.0 prefix: /usr curl: externals: - spec: curl@7.61.1+gssapi+ldap+nghttp2 prefix: /usr texinfo: externals: - spec: texinfo@6.5 prefix: /usr After that create compilers.yaml at ~/.spack/summit/linux/compilers.yaml with:\ncompilers: - compiler: spec: gcc@10.2.0 paths: cc: /sw/summit/gcc/10.2.0-2/bin/gcc cxx: /sw/summit/gcc/10.2.0-2/bin/g++ f77: /sw/summit/gcc/10.2.0-2/bin/gfortran fc: /sw/summit/gcc/10.2.0-2/bin/gfortran flags: {} operating_system: rhel8 target: ppc64le modules: [] environment: {} extra_rpaths: [] And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # Summit is a powerpc machine and uses IBM\u0026rsquo;s Spectrum-MPI for best performance We load a new compiler from a module because the default is IBM XL lacking some features from Modern GCC Summit shares a home filesystem with other machines like Crusher these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Summit and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH to keep these cleanly separate. Changelog # 2022-10-25 created this document ","date":"25 October 2022","permalink":"https://robertu94.github.io/guides/spack/summit/","section":"","summary":"configure spack on ORNL OLCF Summit","title":"Spack on Summit"},{"content":"","date":null,"permalink":"https://robertu94.github.io/guides/cpp/","section":"","summary":"","title":""},{"content":"C is the lingua franca of platform APIs. Most if not all languages provide a mechanism to import APIs from C because they need to do so in order to support the operating system. But given the expressiveness of C++, its often nice to write this code in C++ instead; here\u0026rsquo;s how to do that sanely.\ntl;dr # define a version of your API using only C compatible types in a header Do NOT include a C++ header in the transitive set of headers make the header compatible with C and C++ using #ifdef __cplusplus and extern \u0026quot;C\u0026quot; Why is this needed #C++ is (mostly) a super set of C. If you want C++ to be callable from C, you need to stick to this subset for the declaring code and mark the functions as having a C ABI. C++ by default includes type information in its ABI (to allow things like templates and overloads) Using extern \u0026quot;C\u0026quot; turns this off and defaults back to C-style the user provided name.\nHow to remove C++ from your API #Depends on what you are trying to remove.\nUsing void* #A common pattern is to use a type enum + a void pointer. For example:\ntemplate \u0026lt;class T\u0026gt; T sum(std::vector\u0026lt;T\u0026gt; const\u0026amp; vec) { return std::accumulate(vec.begin(), vec.end(), 0, std::plus\u0026lt;\u0026gt;{}); } Could become\ntypedef enum { int32_dtype, int64_dtype //other types are possible } dtype; double sum_c(void const* data, dtype type, size_t n); in the header file, and in the implementation file:\ndouble sum_c(void const* data, dtype type, size_t n) { switch(type) { case int32_dtype: { const int32_t* begin = static_cast\u0026lt;int32_t const*\u0026gt;data; const int32_t* end = begin + n; return sum(std::vector\u0026lt;int32_t\u0026gt;(begin,end)); } // ... other implementations } } Using #define #Another powerful (but error prone) option is to use C\u0026rsquo;s pre-processor.\n#define sum_definition(type) \\ type sum_##type(type const*, size_t n); #define sum_impl(type) \\ extern \u0026#34;C\u0026#34; type sum_##type(type const* v, size_t n) { \\ type const * begin = static_cast\u0026lt;type const*\u0026gt;(data); \\ type const * end = begin + n; \\ return sum(std::vector\u0026lt;type\u0026gt;(begin,end)); \\ } Then the user can put sum_definition(my_numeric_type) to add new types that they defined, and instantiate their own implementations as needed with sum_impl(my_numeric_type). Note many pre-processors will allow the second macro in a C header since the code isn\u0026rsquo;t type checked until the macro is expanded.\nIt is also pretty common to have code like this in a header\nsum_definition(float) sum_definition(double) sum_definition(int) // continue for more types And this in an implementation\nsum_impl(float) sum_impl(double) sum_impl(int) // continue for more types Using vtable structs #Another option is to define a set of function pointers that implement the same idea:\n//header typedef struct { my_numeric_type* (zero*)(); my_numeric_type* (add*)(struct my_numeric_type*, struct my_numeric_type*); my_numeric_type* (free*)(struct my_numeric_type*); } my_numeric_vtable; typedef struct { my_numeric_vtable const* vtable; void* data; } my_numeric_type; my_numeric_type* new_double(double d); my_numeric_type* sum(my_numeric_type* nums[], size_t n); //impl my_numeric_type* double_zero() { return new_double(0); } my_numeric_type* double_add(my_numeric_type* a, my_numeric_type* b) { double a_v = *((double*)a-\u0026gt;data); double b_v = *((double*)b-\u0026gt;data); return new_double(a_v+b_v); } my_numeric_type* double_free(my_numeric_type* p) { free(p-\u0026gt;data); free(p); } my_numeric_vtable double_vtbl { double_zero, double_add, double_free }; my_numeric_type* new_double(double d) { my_numeric_type* ret = malloc(sizeof(my_numeric_type)); ret-\u0026gt;vtable = double_vtbl; ret-\u0026gt;data = malloc(sizeof(double)); *((double*)ret-\u0026gt;data) = 0; return ret; } my_numeric_type* sum(my_numeric_type* nums[], size_t n) { my_numeric_vtable* vtbl = nums[0]-\u0026gt;vtable; my_numeric_type* total = vtbl-\u0026gt;zero(); for(size_t i; i \u0026lt; n; ++i) { my_numeric_type* next = vtbl-\u0026gt;sum(total, nums[i]); vtbl-\u0026gt;free(total); total = next; } return total; } This has the cost of being verbose and allocation heavy, but allows you to hide the underlying types and allow you or the user to add new ones.\nAdditionally You could replace the definition of my_numeric_type in the header with just struct my_numeric_type;. With this done, it is also possible to make my_numeric_vtable private by removing it from the header since it is only used via pointer. These changes have the trade-off of not allowing users to provide new implementations which may or may not be desirable.\nAvoiding dependencies in headers #These tricks can help you avoid needing to pull in extra headers.\nUse forward declarations generously for non-template C++ types. Except for things in the C++ standard library (because of reserved.names.general), you can forward declare functions and thus not need their implementation as long as all APIs in which you use them consume only pointers. Use this to reduce the number of headers you pull in.\nvoid* pointers can be cast to anything except function pointers and anything can be cast to them. You can use them as \u0026ldquo;data\u0026rdquo; pointers which are casted an interepeted correctly in C++.\nDon\u0026rsquo;t include implementation details in headers, and you can get away with many fewer headers.\nTools like include-what-you-use can help with this.\nUsing a common header file for C and C++ #Once you have a header for your API that uses only C compatible functions, simply mark it as such like so\n//a header guard is a nice thing to do, protects against duplicate definitions #ifndef EXPORTING_C_MARKDOWN_YGKLQ24P #define EXPORTING_C_MARKDOWN_YGKLQ24P #ifdef __cplusplus extern \u0026#34;C\u0026#34; { #endif // all functions here are C accessible #ifdef __cplusplus } #endif #endif /* end of include guard: EXPORTING_C_MARKDOWN_YGKLQ24P */ The extern \u0026quot;C\u0026quot; bit here declares the function as having the C ABI rather than the C++ one.\nHope this helps!\n","date":"24 October 2022","permalink":"https://robertu94.github.io/guides/cpp/exporting_c/","section":"","summary":"\u003cp\u003eC is the \u003cem\u003elingua franca\u003c/em\u003e of platform APIs.  Most if not all languages provide a mechanism to import APIs from\nC because they need to do so in order to support the operating system.  But given the expressiveness of C++,\nits often nice to write this code in C++ instead; here\u0026rsquo;s how to do that sanely.\u003c/p\u003e","title":"How to export a C API from a C++ code"},{"content":"tl;dr #Most of the time simply use find_package, set CMAKE_PREFIX_PATH to include the path specified in CMAKE_INSTALL_PREFIX when the package was installed like so:\nfind_package(std_compat REQUIRED) target_link_libraries(my_library PUBLIC std_compat::std_compat) Tools like spack will set CMAKE_PREFIX_PATH for you. Otherwise specify it using an envionment variable or with a -D flag to cmake.\nHow to add the dependency #Adding a dependency in CMake is pretty straight forward, but depends on where it is coming from. You should do the first one that is applicable.\nIf a library is defined in the same cmake file or one in a parent directory, you can use it directly. If a library is defined in some other cmake project, you often import it with find_package If a library has a pkg-config, you should use pkg_search_module to import it. If a library has a find command such as llvm-config or Tensorflow\u0026rsquo;s you can create an imported target, use execute_process to get the flags, and use the SHELL: prefix for target_compile_options and or target_link_options to set the build options If a library has none of these, locate the library with find_library and header with find_file then create a imported target. Public vs Private #If the thing you are building is a library, generally choose PUBLIC unless it you are writing a header only library in which case choose INTERFACE.\nIf the thing you are building is an executable, choose PRIVATE.\nYou can also choose PRIVATE if none of the headers you link to are in the transitive closure of any of your public header files.\nWhat is the name to pass to find_package and target_link_libraries #If it is one of the built-ins (i.e. MPI/OpenMP), check the CMake docs.\nIf it comes from a user-defined package, look for a directory called cmake installed with the package. In there should be a file called ...Targets.cmake where the ... is some name. The text before Targets.cmake is the entry to pass to find_package. In this file is a list of expectedTargets which will contain the names the package exports.\nImporting PkgConfig libraries #find_package(PkgConfig REQUIRED) pkg_search_module(ZSTD IMPORTED_TARGET GLOBAL libzstd) target_link_libraries(my_library PUBLIC PkgConfig::ZSTD) There isn\u0026rsquo;t anything sacred about ZSTD as the name here. It can be replaced with another name that makes sense. The libzstd bit is what pkg-config looks for. IMPORTED_TARGET GLOBAL makes things \u0026ldquo;modern\u0026rdquo; allowing you to just add a target_link_library command to add the include flags. pkg-config --list-all |grep $name_of_package can help you find package config files.\nImporting libraries with a find tool #Example using Tensorflow:\nfind_package(Python REQUIRED) execute_process(COMMAND \u0026#34;${Python_EXECUTABLE}\u0026#34; \u0026#34;-c\u0026#34; \u0026#34;import tensorflow as tf; print(*tf.sysconfig.get_compile_flags())\u0026#34; OUTPUT_VARIABLE TENSORFLOW_COMPILE_FLAGS OUTPUT_STRIP_TRAILING_WHITESPACE) execute_process(COMMAND \u0026#34;${Python_EXECUTABLE}\u0026#34; \u0026#34;-c\u0026#34; \u0026#34;import tensorflow as tf; print(*tf.sysconfig.get_link_flags())\u0026#34; OUTPUT_VARIABLE TENSORFLOW_LINK_FLAGS OUTPUT_STRIP_TRAILING_WHITESPACE) target_link_options(my_library PUBLIC SHELL:${TENSORFLOW_LINK_FLAGS} ) target_compile_options(my_library PUBLIC SHELL:${TENSORFLOW_COMPILE_FLAGS} ) Transitive Dependencies #tl;dr, add the find_package or similar commands to FooConfig.cmake.in file you wrote for each PUBLIC, INTERFACE or IMPORTED dependency that you add. This isn\u0026rsquo;t required for PRIVATE dependencies. Or LibPressioTools for an example with several optional dependencies.\nSee the docs for more details\nUsing find_library find_program and find_file #Avoid this if possible.\nfind_library(CUFile_LIBRARY cufile PATHS ${CUDAToolkit_LIBRARY_DIR} REQUIRED) find_file(CUFile_HEADER cufile.h PATHS ${CUDAToolkit_INCLUDE_DIR} REQUIRED) # After CMake 3.20, prefer `cmake_path` instead because it doesn\u0026#39;t access the filesystem get_filename_component(CUFile_HEADER_DIR \u0026#34;${CUFile_HEADER}\u0026#34; DIRECTORY) # it\u0026#39;s probably better to create an IMPORTED library here than to just add them all to the library I want to link # cuFile to, but CUDA::cuda_driver and CUDA::cuda_rt are link # dependencies for cufile that I found in the documentation target_link_libraries(libpressio PRIVATE CUDA::cuda_driver CUDA::cudart ${CUFile_LIBRARY}) target_include_directories(libpressio PRIVATE ${CUFile_HEADER_DIR}) These often need to be exported differently. See the docs for best practices to export these\n","date":"19 October 2022","permalink":"https://robertu94.github.io/guides/cpp/cmake_deps/","section":"","summary":"A short verison of how to work with dependencies in CMake","title":"How to add a dependency in CMake"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/experiments/","section":"Tags","summary":"","title":"Experiments"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/hpc/","section":"Tags","summary":"","title":"HPC"},{"content":"So you want to do empirical computer science? Doing good science is difficult. It requires discipline and attention to detail. However there are strategies that can help you focus on answering the questions you can attempt to answer. First you should ask, “is this a scientific question?” Not all questions are scientific questions. Questions about aesthetics, values, ethics are not science questions. “Is A better than B?” is not a scientific question, it’s a question of values. Questions of values require trade offs, and while important can’t be solved with the scientific method of stating assumptions, posing questions, designing experiments, collecting data, and interpreting results. “Can method A achieve more flops than method B in a given specific context?” Is more of a scientific question.\nAnother of the most important questions to ask is, “What are you trying to measure and what could it tell you?”. The latter part about what it could tell you is essential because it helps protect you from spurious conclusions that were not central when designing your system. Answering this pair of questions will often require you to clearly specify a model of your system and understand or at least have an educated guess about how your system works. Once you have answered that question (at least for now; you’ll often have to revise it with new data), my hope is that this document can give you some pointers on how to most efficiently and confidently answer your question.\nSystems Modeling #The process of science begins with a hypothesis — a testable statement about the state or behavior of a system. Constructing a rigorous hypothesis requires a description of your system under test. The description is almost certain underdetermative of the total behavior of a system. It has been said that a model that is just as complex as the real system is just the real system. By design, models omit aspects of the systems they describe because these simplifications allow us to reason about the underlying behaviors we actually care about.\nThe process of creating these models is called systems modeling. While targeted, accurate, and concise systems modeling is challenging often relying on experience and expertise of the modeler or scientist constructing the model, there are a few principles that can be helpful:\nStart with a broad model that is not refined. Done appropriately you may be surprised about how accurate it is. You can always refine a broad model into a more accurate one by decomposing its parts into a more accurate model. Focus on the interfaces, inputs and outputs. These are the trickiest to get right, but also can give you the most leverage to tweak the behavior of your model.\nA rigorous model can often be phrased as a set of logical syllogisms. For example:\nLarger cache sizes have fewer evictions when executing code The more evictions there are, the slower the system will be on application 1 System A has a larger cache than System B Therefore System A will be faster than system B on Application 1 Readers familiar with caching know that is a pretty reductive model. You’ll probably need to account for factors like clock speed, algorithm, and eviction policy to get the accuracy you’re looking for on new applications of your model, but you could certainly start here and account for these differences later as you need them. As you make these changes, you’ll adjust the premises of your model to reflect the system and what you learn about it.\nStatistical Primer #One of the most common mistakes I see in empirical computer science is to ignore the statistical principles when designing the experiments. Statistical principles can help you avoid drawing incorrect conclusions due to measurement or random error in your system.\nThere are a few key principles to keep in mind:\nConsider the 3Rs #randomness, replication, and reduction of noise. When designing experiments as much as is practicable you should repeat the experiments, reduce the effects of non-studied factors, and randomize what you can’t control.\nConsider your assumptions and consequences of violation #Every statistical method has assumptions and consequences of violation. The most common consequence is so called “liberal inference” which means that your experiment could give misleading results. Careful attention to assumptions can protect you from false conclusions.\nKnow the basis of Statistical Inference #Statistical Inference is the process of inferring the probability of the truth of some claim based on the results of some experiment. These methods can be either parametric and non-parametric. Parametric methods make assumptions about the distribution of the values of your observations. Non-parametric make fewer or weaker assumptions. One of the benefits of parametric methods is that they offer stronger evidence of a claim (I.e. require fewer replicas to draw a conclusion) than non parametric methods. The benefit of non parametric methods is that they make fewer assumptions and can be used validly in more cases.\nKnow some basics of Experimental Designs #There are several common designs including factorial, Latin squares/hyper cubes. When the experimental design space becomes large, tools such as orthogonal arrays can reduce the testing space. These methods can help provide structure that can help you avoid violating certain assumptions and keep the cost of your experiments down.\nChoose an appropriate Sample size #Statistical Inference offers a trade off between the rate of false positives and the sample size. Tests that require smaller sample sizes are called more powerful but often require more strict assumptions.\nThere is a lot more to statistics than what I can cover here. It’s worth reading a book or two on statistics methods. One such book is “A first course in design and analysis of experiments” by Gary W. Oehlert.\nSeparating Concerns #There generally are three concerns when conducting a computer science experiment:\nRunning the experiment Parsing the results into a machine readable form Taking the results and running analysis I would advise writing a single script for each of these. That way you can easily run each part of the process independently if needed.\nRunning experiments #When writing scripts and programs to run your experiments there are a number of tricks that can enable more productive code.\nWrite scalable code. Most experimental codes can be formulated as the execution of a “do_experiment” function that takes a configuration as an argument which can then be constructed using the Cartesian product of a set of factors.\nfrom itertools import product def do_experiment(args): # run experiment return ex_results Approaches = [1,2,3] Replicates = 5 Results = [] for _, approach in product(range(replicates), approaches): Results.append(do_experiment(approach)) Code like this can be easily parallelize and distributed using something like a MPI4Py MPICommExecutor (python) or libdistributed work queue (c++). The C++ equivalent of product is std::views::cartesian_product or range-v3’s cartesian_product.\nKnow how to accurately measure things #Measuring fine grained timing can be a nuanced process. First, for sub-millisecond timings, the clock resolution can matter. Second, not all clocks are monotonic. Third, some clocks have high measurement overhead for the first three cases in c++ you generally want the std::chrono::steady_clock in Python the equivalent is time.perf_counter_ns . Fourth, your compiler may optimize away your benchmark unless you force it to keep it (Google Benchmark has a function called benchmark::DoNotOptimize) which generally does the right thing). Fifth, other processes may interfere with your measurement (especially print statements) so avoid doing expensive operations during your benchmark timings. Lastly not all clocks can measure all processes, for example the C++ clock functions cannot measure the timing of Cuda operations accurately because they cannot observe device state.\nKnow your scheduler #A common mistake that I see new students make is to run all jobs interactively, and not make use of the scheduler to run their job. Additionally your scheduler can do many things for you such as email you when your job finishes, start one job after a previous job finishes, handle process pinning and resource allocation and more. Its worth reading what it is capable of.\nKnow a higher level programming language #Systems programming languages like C,C++ and Rust are remarkably powerful and useful tools for writing benchmarks and experiments. However not all parts of the experimental process need this level of performance. For example plotting and data scraping tasks often aren’t as well suited to lower level languages and require large volumes of code for simple tasks. Likewise only recently has C++ gained higher level abstraction for Cartesian iteration. Prefer a higher level language for these tasks when possible.\nUnderstand sources of variability and control them if possible #Computer systems have many sources of variability. This could be clock variability, interference variability, seed variability, or process variability. Many, but not all of these sources of variability can be mitigated with appropriate steps. Attempt to control this variability if it is practical to do so.\nOutput as much context as possible #Especially with long running tasks having appropriate debugging information is key to reduce the number of run, interpret, modify cycles to a minimum. In doing so, remember what the system can record for you such as core dumps when the system crashes. Some context to specifically record include timestamps, task ids (if they are deterministic), complete error messages, error status, and progress indicators.\nKnow a distributed programming framework #For many problems running code on a single node isn’t enough. You want to be able to run experiments over several nodes collaboratively. While it is possible to do this with TCP/IP, it isn’t often the best way. In HPC the tool of choice is often MPI which provides both tools that are easy to start with and provides room to grow. However, other choices include RPC systems such as Mochi and GRPC.\nMake the testing environment reproducible (spack, docker, etc…) #Reproducibility is a crisis in science; do not contribute to this crisis. Tools from industry like containers can make that much easier. Docker/Podman is my favorite, and is reasonably easy to use. Generally these tools boil down to writing a specialized script that reproduces your environment exactly which can then be shipped to users in the form of a compressed archive. Sometimes (especially when specialized hardware is involved) this is easier said than done. You might need a different version of software depending on your hardware platform. Spack is a powerful package manager that can help you solve some of these more complicated dependencies, but warrants its own post. Together these tools give you a powerful platform to run experiments easily on diverse machines. Tools like singularity let you run containers on HPC systems.\nPrefer parseable output, but sometimes you can’t #See the next section on writing parsable outputs. However, because many HPC libraries write to stdout/stderr, you can’t assume that you will have these to yourself. MPI implementations can be especially loud (for good reason) when they don’t believe that they are configured correctly.\nValidate on small cases first if possible #Waiting on the scheduler to run your job can be painful. If possible, have some small subset that you can verify works “correctly” before jumping to a supercomputer where your runtime environment becomes more complex and queuing times slow your ability to iterate on your experiments. Many supercomputers also charge for every execution and the costs can add up quickly.\nWrite the code to do partial writes, and resume-able in case the code crashes #There are few things more frustrating than for code to run for a few hours, crash (or timeout), and not have any results to show for it. It is often preferable to write results incrementally as your experiment is running than waiting until the end. This also often lets you use a pattern known as checkpoint restart where you resume execution at the state after your last failed/incomplete run. This makes it far easier to restart your job later to finish what’s left rather than start over from scratch. This is often as easy as partitioning the output into non overlapping regions or ids and writing the output. If this can’t be done a master process can be delegated to store the output\nWrite the code to handle error conditions as gracefully as possible #Likewise thinking about the error propagation boundaries for your code. In the pseudo code above placing a mandatory error propagation boundary just outside calls do_experiment will do what you want: isolate a fault from one experiment from another experiment allowing as many experiments to continue as possible without a crash in one experiment affecting other experiments. This can be tricky. Things like MPI have the default behavior of terminating the world on a signal. Other things like HDF5 with many blocking function calls might not catch that other threads of execution finished in a error state and will block indefinitely. This long tail of failures to tolerate can be constructed incrementally as you encounter them.\nAs an aside: sigsegv or its cousins sigfpe and sigbus are often implemented as catch-able signal but the mechanisms for recovering from it in standard C/C++ fully and correctly are limited or non-existent. Even if you think you can recover correctly from these signals don’t; you probably shouldn’t and should let the entire C runtime restart instead. What is easier and arguably better is to use another program (ie bash/python scripts) to handle this particular fault boundary.\nHere is an example that tries to implement as much of this as possible:\n#!/usr/bin/env python from mpi4py.futures import MPICommExecutor, as_completed from itertools import product from csv import DictWriter, DictReader from time import perf_counter_ns def do_experiment(args): starttime = perf_counter_ns() replicate, approach = args if replicate == 3: # example error raise RuntimeError(\u0026#34;bad error happened here!\u0026#34;) # do the actual experiment here, then put results in a dict # including a starttime and endtime can really help you debug things ex_results = { \u0026#34;result1\u0026#34;: replicate * 2, \u0026#34;result2\u0026#34;: replicate, \u0026#34;starttime\u0026#34;: starttime, \u0026#34;endtime\u0026#34;: perf_counter_ns() } return ex_results FIELDS = [\u0026#34;approach\u0026#34;, \u0026#34;replicate\u0026#34;, \u0026#34;starttime\u0026#34;, \u0026#34;endtime\u0026#34;, \u0026#34;result1\u0026#34;, \u0026#34;result2\u0026#34;, \u0026#34;err\u0026#34;] approaches = [1, 2, 3] replicates = 5 futures = [] in_checkpoint = set() tasks = list(product(range(replicates), approaches)) fut_to_task = {} with MPICommExecutor() as pool: # only the root mpi rank enters this if-statement # the other processes will be workers if pool is not None: # check our results file to skip all results that are already successful try: with open(\u0026#34;output_file.csv\u0026#34;) as checkpoint_file: reader = DictReader(checkpoint_file, FIELDS) for row in reader: if row[\u0026#39;err\u0026#39;] == \u0026#34;\u0026#34;: in_checkpoint.add((int(row[\u0026#39;replicate\u0026#39;]), int(row[\u0026#39;approach\u0026#39;]))) exists = True except FileNotFoundError: exists = False pass # start all of the tasks we care about for task in tasks: if task not in in_checkpoint: fut = pool.submit(do_experiment, task) futures.append(fut) fut_to_task[fut] = task # write results as we find them with open(\u0026#34;output_file.csv\u0026#34;, \u0026#34;a\u0026#34; if exists else \u0026#34;w\u0026#34;) as outfile: writer = DictWriter(outfile, FIELDS) if not exists: writer.writeheader() for fut in as_completed(futures): try: result = fut.result() task = fut_to_task[fut] print(\u0026#34;task \u0026#34;, task, \u0026#34;completed successfully\u0026#34;) result[\u0026#34;err\u0026#34;] = \u0026#34;\u0026#34; result[\u0026#34;replicate\u0026#34;] = task[0] result[\u0026#34;approach\u0026#34;] = task[1] writer.writerow(result) except Exception as ex: result = {} task = fut_to_task[fut] print(\u0026#34;task \u0026#34;, task, \u0026#34;failed\u0026#34;) result[\u0026#34;err\u0026#34;] = str(ex) result[\u0026#34;replicate\u0026#34;] = task[0] result[\u0026#34;approach\u0026#34;] = task[1] writer.writerow(result) Writing parsing code #When parsing the results of experiments it is often helpful to do so in ways that are machine parse-able #While you almost certainly could write a domain specific language that often isn’t the best use of your time or resources. Sticking to a few well established formats can tremendously simplify the task ahead of you.\nCSV is ubiquitous and easy to generate. For this reason it is usually the first thing I reach for as long as the data isn’t hierarchical or non-scalar in nature. JSON is usually the next step that I reach for when I need simple arrays or basic hierarchies represented in my output. There are decent JSON parsing libraries in nearly every language which makes it easy to work with. Google’s protobuf is also not a bad option when JSON parsing speed is a limitation and you have a stable schema. While not as common in HPC it does a decent job of representing complex scalars and moderate hierarchies in ways that can be checked against a schema for validity. HDF5 as a structured, scalable data container. If you need to store large tensors (higher dimensional matrices) of data with hierarchical relationships, it is one of the few games in town. It also tends to have decent support in a variety of languages. SQLite can be another highly portable output format when your data is highly relational. It is also supported on nearly every platform under the sun. Binary files - writing a binary flat file is a possibility when the data format is relatively simple (ie a single large array), and nearly all languages support this. However it is seldom the best format because it doesn’t communicate some subtleties like endianness and and dimensionality that formats like HDF5 provide in addition to features like compression or attributes. Free-Form Text can be parsed with regular expressions as tool of last resort for getting data out of your experiments. However needing a regular expression is often a suggestion that your output would benefit from greater structure and/or isolation. There are great websites that can help you rapidly prototype and validate your regular expressions for a variety of regular expression dialects. When using tabular outputs like CSV use one row per experiment, one column per field #This makes it much easier to do joins and parsing of your experimental results to other tables of data much easier. If possible avoid literal new line characters or field delimitors in fields (especially error message fields) instead either 1 know how to escape them or replace them with characters without special interpretations.\nWhen using tabular data include a column for errors or execution irregularities #Experimental errors happen. Having a way built in to both filter in/out and summarize these kinds or errors can save you a lot of time trying to interpret your results and issues encountered during your experiments.\nParsing code often doesn’t benefit as much from parallelism so don’t worry about this upfront #Serial execution is a good enough place to start. An exception to this is where your data is a large tensor in which case parallel HDF5 is your friend for scalable IO performance.\nSince the previous example used csv and wrote to a dedicated file, we don\u0026rsquo;t even have to write the parsing code, we can just use pandas.read_csv\nWriting plotting/analysis code #Choose a language which has mature tools for this. #less you are doing sophisticated 3D graphics where libraries like VTK or OpenGL or Vulcan are required, you can accomplish a lot more a lot faster with libraries in Python (Seaborn/Matplotlib) or Julia (Makie/ Plots.jl). C++ is not the best tool for every job.\nSeparate plotting/analysis from parsing parsing the log files #Often one of these tasks will take much longer than you’d expect. By separating these tasks, you can work with the clean data more iteration and quickly drill in on a plot that does what you want.\nPrefer vector graphics for 2D plots #Vector graphics automatically scale to arbitrary resolution because they are described as a series of equations rather than a “map” of colors. In many plotting libraries this is as simple as choosing an eps or svg format output.\nA simple script that plots the runtime for each might look like this:\nimport pandas as pd import seaborn as sns df = pd.read_csv(\u0026#34;output_file.csv\u0026#34;) df[\u0026#39;runtime\u0026#39;] = df[\u0026#39;endtime\u0026#39;] - df[\u0026#39;starttime\u0026#39;] # understand what kinds of errors occured errs = df[df.err.notnull()] print(errs.err.value_counts()) # filter out results with errors success = df[df.err.isna()] # plot the timings sns.set(rc={\u0026#34;figure.figsize\u0026#34;: (10, 7.5)}) sns.set_theme(context=\u0026#34;talk\u0026#34;, style=\u0026#34;whitegrid\u0026#34;) fig = sns.barplot(x=\u0026#34;approach\u0026#34;, y=\u0026#34;runtime\u0026#34;, data=success) fig.get_figure().savefig(\u0026#34;runtime.eps\u0026#34;) Which of the principles in this chapter should you focus on improving in your experimental design? Which do you do well with? Changelog # 2022-10-18 \u0026ndash; initial version ","date":"18 October 2022","permalink":"https://robertu94.github.io/learning/experiments/","section":"","summary":"\u003cp\u003eSo you want to do empirical computer science? Doing good science is difficult. It requires discipline and attention to detail. However there are strategies that can help you focus on answering the questions you can attempt to answer. First you should ask, “is this a scientific question?” Not all questions are scientific questions. Questions about aesthetics, values, ethics are not science questions. “Is A better than B?” is not a scientific question, it’s a question of values. Questions of values require trade offs, and while important can’t be solved with the scientific method of stating assumptions, posing questions, designing experiments, collecting data, and interpreting results.  “Can method A achieve more flops than method B in a given specific context?” Is more of a scientific question.\u003c/p\u003e","title":"Suggestions for the Design of Computational Experiments"},{"content":"#tl;dr\n# set NEW_VERSION to the desired version cd $SPACK_ROOT git pull cd $(spack repo list | grep robertu94 | sed -E \u0026#39;s/robertu94[[:space:]]+//\u0026#39;) \u0026amp;\u0026amp; git pull spack install libpressio-tools ^libpressio${NEW_VERSION} spack load libpressio-tools ^libpressio${NEW_VERSION} libpressio is now Mainline #LibPressio is now in the mainline version of spack (as of commit da6aeaad44e434da5563d32f2fa9900de362b2ed, October 17, 2022). You no longer need to add robertu94_packages unless you need a development version. You can either spack repo rm robertu94, or the next pull of robertu94_packages will include a dummy package to refer back to the builtin version.\nYou can now update like so (without robertu94_packages)\ncd $SPACK_ROOT git pull spack install libpressio-tools ^libpressio${NEW_VERSION} spack load libpressio-tools ^libpressio${NEW_VERSION} the longer version #Updating LibPressio with spack generally has several steps:\nupdate your spack install git pull $SPACK_ROOT if your HPC system has updated (i.e. after system-wide maintenance) you might need to adjust the modules loaded for your compiler, MPI, or other critical external packages. See the guides for the latest guides on the systems that I use. Be sure to check the spack release notes for other configuration changes that might be needed update your robertu94_packages with cd $(spack repo list | grep robertu94 | sed -E 's/robertu94[[:space:]]+//') \u0026amp;\u0026amp; git pull This may not be necessary for many users after LibPressio is merged into upstream Spack. Hopefully in Fall 2022. Users working on development versions of these packages will still want to use the 3rd party repository After LibPressio is merged into upstream spack. Users may want to spack repo remove robertu94 to prefer using the stable versions in upstream spack. Spack install your new version. If you passed variants to your install command, be sure to include them again here. A good place to start is spack install libpressio-tools ^ libpressio+sz Spack load the built libraries. this can be done via the hash of the built package (which is robust to new versions being built) It can also be done via spack load libpressio-tools. The later may error if a new version is added. If you get an error regarding multiple versions, you can uninstall one of the older versions with spack uninstall $HASH where $HASH is the hash of the older version or by requesting the specific version. You can list available versions and their configuration with spack find -lvd --loaded Changelog # 19 October 2022 \u0026ndash; libpressio has been merged in upstream spack and no longer needs robertu94_packages in most cases. ","date":"13 September 2022","permalink":"https://robertu94.github.io/guides/spack/updating_libpressio_with_spack/","section":"","summary":"how can you update your copy of libpressio via spack?","title":"Updating LibPressio with Spack"},{"content":"Modern software does not exist in a vacuum, developers always rely on other developers or engineers to provide aspects of their systems. Even if you write hello world in your programming language of your choice you likely still depend on the standard library from your language. Even if you were to write hello world in the raw binary format of your assembly language, you still depend on the operating system. Even if you wrote your own uni-kernel to produce hello world on the console, you likely rely on vendor provided firmware to initialize the program or memory. And if you are one of the blessed few who can completely fabricate your own hardware and don\u0026rsquo;t need to rely on vendor provided firmware, you likely don\u0026rsquo;t have time or expertise to write efficient higher level systems and need others to provide software to run on your hardware. Therefore the question is not will you use dependencies, but how will you use them responsibly.\nTypes of Dependencies and When to Introduce Them #There are three major kinds of dependencies: build dependencies, link dependencies, and run dependencies. These dependency types effect what portion of the code execution process a dependency is needed for. For example, build dependencies if they are not also link or run dependencies need not be installed on the final system where the code will be executed.\nWhen considering what dependencies to adopt you should consider both its costs and its benefits. Costs of a dependency can include the difficulty for a user to configure or install it, the portability of the dependency to various different architectures and operating systems, the time or size of the installation, the complexity of the dependency or the complexity that the dependency adds to your application, and the rate of change of the dependency. Each of these costs have to be accounted for when adopting a dependency. Dependencies often provide benefits in one of two ways: they either provide some feature or they reduce the maintenance or development burden of some other feature that you may have.\nFor example, introducing MPI as a dependency for an HPC code can either be a lightweight or a heavy dependency depending on the application. Many HPC centers provide an installation of MPI meaning configuring or installing the dependency has little to no cost. MPI is portable to a variety of even heterogeneous distributed systems. And implementations of MPI often have a sufficiently stable interface that they can be adopted at low cost. However, if executing on some average users laptop, MPI can be a much heavier dependency where installation and configuration of MPI is less available. It is therefore essential to consider the context in which a dependency will be used to truly measure its costs.\nWhen considering the cost of a dependency and how it integrates with a larger application here are some factors to consider. First is the interface boundary how the Dependency interacts with the rest of your system. Dependencies are more easily interchangeable if the size of the interface boundary is small and if the number of points of integration is also small. For example if a dependency provides only some simple function such as left patting a string, its interface boundary may be very small, however that operation may be used pervasively a crossed a user facing application. This specific example is notorious after a developer in the JavaScript ecosystem deleted a package that provided this functionality and intern broke a wide variety of JavaScript packages that either directly utilized left pad or utilized left pad via a transitive dependency. This deletion was ultimately an expensive debacle resulting in likely thousands of hours of wasted developer time.\nThe next aspect to consider is whether a dependency should be optional or required. If the functionality provided by a dependency is likely to be unused buy some substantial portion of the user base of the code, it may be valuable to only conditionally require a particular dependency that provides the specific functionality used by that some population. However this choice comes at a cost. On the one hand the conditional dependencies Create a combinatorial explosion of different configurations of your software. So while one conditional dependency may be easy to work around, if you had eight conditional dependencies then you have two to the eighth different possible ways to build your software. This can induce a larger cognitive load on users seeking to adopt your project. A good example of this is the LibPressio compression library. LibPressio allows you to provide optional dependencies on a variety of compressors that can be adopted by users. By making each compressor an optional dependency, users only are required to install the compressors that they actually intend to use. However, users may often unintentionally install a copy of LibPressio lacking the specific compressor that they intend to use.\nAdditionally, when adopting a dependency you may wish to consider how to test the integration of this dependency. Like most pieces of software, software evolves overtime, and this includes your dependencies. It is important to catch changes in the invariants that you expect of your dependencies where it will have a meaningful impact on Gore software. However, this comes at a cost: these tests often must be written and maintained as the project evolves over time.\nLastly, one should consider how the dependency will integrate with your build system. Almost all software projects over a given size utilize some build system that either generates a specific code, or resolves the dependencies that the code adopts. Each build system handles this task in a slightly different way. When should consider how adopting a dependency will impact the complexity required in your build system to handle either variations in your dependency or the simple number of dependencies.\nHow to Integrate a Dependency #There are several methods of integrating a dependency into your application: embedding or vendor in, expecting ambient installation, package management, or requiring external installation.\nRendering or embedding a dependency simply copies the entire source of a dependency into your repository. The most famous example of this is Google who famously uses a single \u0026ldquo;mono repository quote for all of its software. This approach has many benefits if you can simply remain inside of its ecosystem. By adopting a monolithic repository, there is no problem with ensuring that your dependencies change unexpectedly, and that the installation of these dependencies can be automatically handled by your build system. This allows you to easily perform atomic updates across the entire transitive side of dependencies at once without multiple or two stage commits. However, I have found that very few can truly operate in a monolithic repository. It requires an organization of sufficient scale to manage the security and other maintenance that may need to be performed on a dependency that is developed external to your organization. Additionally, in some languages having multiple versions of a dependency installed by different sub components of your application can be problematic resulting in at best a spectacular segmentation fault, and at worst silent undefined behavior.\nAnother option for dependency management is expecting ambient installation. This method assumes that the dependency is already installed. In my experience this works for some specific stable and widely used dependencies. For example, depending on the C standard library In this way is likely a safe choice. Anyone who is using your library almost certainly has a copy of a C standard library of a suitable version to provide the specific functions that you were likely to use. Choosing this method simply punch the entire dependency management process to chance and that requires practically no effort when it works for dependencies such as the C standard library.\nAnother popular option for dependency management is to use a package manager. Package managers are tools that sit along side Your build system to provide the dependency is that the build system will consume. However it seems to me that package management systems are some of the most frequently implemented pieces of software. Your operating system, your programming language, and your users may all use different packaging systems. Maintaining separate packages for each of these systems can be a substantial maintenance burden. However, when the systems work, it can be as simple as managing a short text file describing your list of dependencies to ensure a reproducible build experience a crossed systems.\nLastly, you can require external installation. External installation is unlike expecting ambient installation and that the developer does not expect or cannot reasonably expect that the dependency that they wish to use is available on the user system. This is often an easy choice for developers. They may implement this by describing in a README file to go in separately follow the instructions to install a particular dependency before continuing installing this program. However this is often a terrible experience for users.\nHow To Be A Good Dependency #I will close with the discussion of what it means to be a good dependency. A good dependency provides API stability, discovery ability, portability, consistency, a name space, avoiding unnecessary side effects, and an unnecessarily broad API.\nAPI stability means different things to different people. They are concept of denotational, semantic, and binary stability. Denotational stability is the simplest: from one version of the software to a later version, the same functions still exist with at least the same signatures. However, this does not ensure that the program will continue to function. The authors of the dependency may have left the signatures the same, but changed what the functions did in a way that causes the program to malfunction or crash. To remedy this, semantic stability is required. Semantic stability requires that the meaning of a program from one version to another later version has not changed. However, even this is not enough for some users. Some users further require binary stability. In this form of stability, even the machines representation of a particular function and its calling conventions cannot change from one version to a later version. This particularly strict version of stability is useful for critical security dependencies such as the SSL provider. Requiring binary stability ensures that system administrators can swap to implementations of the SSL provider without re-configuring or re-compiling the applications that depend upon it.Choosing what level of stability or lack there of that you intend to provide and convey to users is a critical aspect of deciding how to maintain your library and be a good dependency. As Linus Torvalds has said \u0026ldquo;don\u0026rsquo;t break user space. \u0026quot;\nDiscover-ability is also a key aspect of being a good dependency. Discover-ability means many things including being accessible from your language or systems primary package management system, but also providing sufficient documentation and high-level overview use of your software for a developer to quickly understand how the dependency is to be used. Improving discover-ability can be a challenging undertaking requiring careful attention to odious tasks such as writing well written documentation.\nPortability is likewise important. Here are some common areas of portability that I think developers often mess: first they use features with restrictive dependencies for example a particularly bleeding edge version of a compiler or an operating system; they utilize hard coded pads extensively in either the build system or in the software itself; or in the worst case they have completely written their own nonportable build system that doesn\u0026rsquo;t operate and is not flexible. In some cases your tooling can warn you when you attempt to use functions that are not available on all platforms. Hard coded pads can often be solved with a central file generated by the build system that provides default paths where needed. Lastly use a well established modern build system to build your software. Those who come to use your software will thank you.\nConsistency requires careful attention to detail to maintain. There are two types of consistency internal and external consistency. Internal consistency requires that similar functions and similar concepts behave in similar ways within your code. External consistency is more difficult to implement and requires using names that are consistent across your domain or field to provide similar denotational in semantic meanings of your functions. A good example of programs that are not externally consistent are APT and DNF I could managers. In apt, the function update does not actually update any software. In DNF the update function actually update software. I would argue that abs missed the boat here. This requires users to learn new paradigms when switching between various Linux distributions.\nProviding a name space that is consistently employed further improve the usability of your software. In languages like sea, all functions are by default included in the global name space. This means that different libraries or dependencies can provide definitions of the same function which unfortunately can behave in different ways. The easiest way to solve this is to prefix all publicly exposed functions with some project specific name code to ensure that these functions from different libraries do not collide. In C++, and other languages that came after, this process is much easier and is often handled using a system often called modules. Modules provide both a mechanism to import definitions as well as a way to name space them after they have been imported. Mini systems additionally provide a way to rename a module if its name could conflict with another function that they use or wishes to provide.\nDependency authors should avoid unnecessary side effects. While some kind of side effects such as printing a relatively obvious while mostly benign, other forms of side effects can be potentially hazardous. For example, your method of error handling can be introducing the side effects into your users code.If your code calls assert on some thing that could otherwise be a runtime error, you deprive your user of agency to determine how they would like to handle the error .Introducing a unintended side effect of program termination. Likewise, functions that refer to global variables can also have unintended side effects. These functions can provide inconsistent behavior if multiple dependency is that a library attempts to use attempt to change these global variables in different ways simultaneously.\nLastly, dependencies should avoid exposing unnecessarily broad API\u0026rsquo;s. The broader an API is, the harder it is to maintain and to implement consistently. In the vein of Occam\u0026rsquo;s razor, an API should be sufficiently complex to master its domain and nothing else. A good example of this and how this can go awry is when a library exposes a function for some primitive operation like Max which can unintentionally be imported into a different context and result in inconsistent semantics.\nI hope the short document has helped you think both about how to consider when to adopt a dependency, how to adopt a consist dependency, and how to write code that itself will be a good dependency. Happy programming.\nChangelog # 2022-07-20 first version ","date":"20 July 2022","permalink":"https://robertu94.github.io/posts/2022-05-05-dependencies/","section":"Posts","summary":"\u003cp\u003eModern software does not exist in a vacuum, developers always rely on other developers or engineers to provide aspects of their systems.\nEven if you write hello world in your programming language of your choice you likely still depend on the standard library from your language.\nEven if you were to write hello world in the raw binary format of your assembly language, you still depend on the operating system.\nEven if you wrote your own uni-kernel to produce hello world on the console, you likely rely on vendor provided firmware to initialize the program or memory.\nAnd if you are one of the blessed few who can completely fabricate your own hardware and don\u0026rsquo;t need to rely on vendor provided firmware, you likely don\u0026rsquo;t have time or expertise to write efficient higher level systems and need others to provide software to run on your hardware.\nTherefore the question is not will you use dependencies, but how will you use them responsibly.\u003c/p\u003e","title":"Thoughts on Dependencies for Scientific Software"},{"content":"Setting up spack #The way that I use spack, there are 6 major steps to configuring spack:\nDetermine what system compiler, python, and MPI you will use Telling spack how to find your compiler Telling spack find major system dependencies such as MPI and OpenSSL as external dependencies Configuring preferred providers for virtual dependencies and target architectures Configuring spack to use a binary cache if it will matter Setting up a shell shortcut for loading spack What system dependencies to use #Spack does a pretty good job \u0026ldquo;living off the land\u0026rdquo; using what ever dependencies you may need. There are a few packages that benefit from just using the versions provided by your system administrator typically your compilers, python (used to invoke spack; not the one you intend to use), MPI, and OpenSSL.\nCompilers and Python are hard dependencies of spack, and it can\u0026rsquo;t be used without them. You probably want to grab a somewhat recent compiler if it is available to you. Spack can install compilers for you, but they often take a long time to compile so reusing a prebuilt compiler if possible is really nice.\nMPI is special both because it often has to be paired with a specific compiler version to work correctly, but also because MPI implementations has so many variants and options, it can be difficult to configure correctly. Using the system MPI hopefully ensures you get the right one.\nOpenSSL is also special because spack itself needs a working OpenSSL to do many thing involving downloading packages, and using a non-standard OpenSSL will break commands like curl and git. You can work around this, but it is often best to just use the system version of this package\nConfiguring the spack compiler #If a module is needed for your compiler load that first.\nAfter that, Spack\u0026rsquo;s built-in spack compiler find command does most of what you will want. However there is one nuance is that won\u0026rsquo;t correctly determine what modules it should load.\nIf you determined you need a compiler provided by a module, you need to tell spack about that now using the module entry in spack.\nConfiguring Major Dependencies #If a module is required for your MPI module, load that first.\nAfter that Spack\u0026rsquo;s built-in spack external find $spack_name_for_mpi_version will do what you want for MPI. Likewise spack external find openssl will handle OpenSSL.\nIf you determined you need a MPI provided by a module, you need to tell spack about that now using the module entry in spack.\nConfiguring Preferred Variants and Targets #Some hardware will have hardware optimized libraries or prefered configurations. Set these now in the all packages variants section of package.yaml.\nOn heterogeneous clusters, setting target= to your common CPU hardware architecture can vastly simplify deployment at modest performance cost. Alternatively, provide a list of architectures to cross compile as needed.\nConfiguring spack to use a binary cache #Some systems (mainly LTS Ubuntu) have binary caches pre-built for spack that can vastly improve install times for common packages builtin in common ways. For Ubuntu 18.04 this works wonders to decrease build times:\nspack mirror add binary_mirror https://binaries.spack.io/releases/v0.18 spack buildcache keys --install --trust The other popular cache is e4s which supports a newer ubuntu LTS release, and some other distros provided you are willing to build your own compiler.\nspack mirror add E4S https://cache.e4s.io spack buildcache keys --install --trust Setting up a shell shortcut #I like to have a single command to load spack and any MPI or compiler modules that I may need which I often call use build.\nSetting LD_LIBRARY_PATH when it is needed #If you have a spack envionment where you need LD_LIBRARY_PATH (e.g. to build or run with a Makfile). Then you can use the following commands to have spack add these variables, run despacktivate and then sspack env activate /path/to/env to reactivate it.\nspack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH] spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH] Updating Spack #I generally recommend rebuilding your spack envirionments whenever you update spack\u0026rsquo;s major version or whenever there is a major change to your HPC system (new modules for core dependencies, new OS version, new hardware). Because with spack environments re-building a specific environment is easy, I often do this by rm -rf spack and re-building from scratch.\nI find that more incremental updates can be more buggy than they are worth.\nChangelog # 2022-07-19 created this document ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/spack/","section":"","summary":"configure spack on a new system","title":"Shortcut Guide for Configuring Spack for a New HPC System"},{"content":"tl;dr #Put the following in your .bashrc\nuse_build() { if hostname | grep bebop \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading bebop spack\u0026#34; module load gcc/11.4.0 module load intel-oneapi-mpi/2021.12.1-tjkrnei module load intel-oneapi-mkl/2024.0.0-jfwrhz5 source $HOME/git/spack-bebop/share/spack/setup-env.sh source $HOME/git/spack-bebop/share/spack/spack-completion.bash export clustername=bebop fi #other LCRC Machines export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-bebop git clone https://github.com/robertu94/spack_packages git/robertu94_packages source ~/.bashrc # for extra packages like libpressio spack repo add ~/git/robertu94_packages Bebop provides default configurations for packages and compilers so you are ready to go! And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # Spack shares a home filesystem with other machines like swing these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Bebop and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH to keep these cleanly separate. The Bebop admins have configured spack with a bunch of sane defaults, so you can leave these alone :) Changelog # 2022-07-19 created this document 2022-08-11 updated to use Intel-MPI 2024-07-11 updated after the bebop update ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/bebop/","section":"","summary":"configure spack for ANL LCRC bebop","title":"Spack on Bebop"},{"content":"tl;dr #Put the following in your .bashrc\nuse_build() { if hostname | grep \u0026#34;^i\u0026#34; \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading improv spack\u0026#34; module load gcc/13.2.0 module load openmpi/5.0.1-gcc-13.2.0 source $HOME/git/spack-improv/share/spack/setup-env.sh source $HOME/git/spack-improv/share/spack/spack-completion.bash export clustername=improv fi #other LCRC Machines export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-improv git clone https://github.com/robertu94/spack_packages git/robertu94_packages source ~/.bashrc use_build spack compiler find # for extra packages like libpressio spack repo add ~/git/robertu94_packages First create packages.yaml at ~/.spack/improv/packages.yaml with:\npackages: all: providers: blas: [openblas] lapack: [openblas] mpi: [openmpi] openmpi: externals: - spec: openmpi@5.0.1+atomics~cuda~java~memchecker~static~wrapper-rpath fabrics=ucx prefix: /gpfs/fs1/soft/improv/software/spack-built/linux-rhel8-zen3/gcc-13.2.0/openmpi-5.0.1-yhyhopk modules: [openmpi/5.0.1-gcc-13.2.0] buildable: False After that create compilers.yaml at ~/.spack/improv/linux/compilers.yaml with:\ncompilers: - compiler: spec: gcc@=13.2.0 paths: cc: /gpfs/fs1/soft/improv/software/spack-built/linux-rhel8-x86_64/gcc-8.5.0/gcc-13.2.0-iyqxotb/bin/gcc cxx: /gpfs/fs1/soft/improv/software/spack-built/linux-rhel8-x86_64/gcc-8.5.0/gcc-13.2.0-iyqxotb/bin/g++ f77: /gpfs/fs1/soft/improv/software/spack-built/linux-rhel8-x86_64/gcc-8.5.0/gcc-13.2.0-iyqxotb/bin/gfortran fc: /gpfs/fs1/soft/improv/software/spack-built/linux-rhel8-x86_64/gcc-8.5.0/gcc-13.2.0-iyqxotb/bin/gfortran flags: {} operating_system: rhel8 target: x86_64 modules: [gcc/13.2.0] environment: {} extra_rpaths: [] And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # We use mpi from the system to avoid configuring them We load a new compiler from a module to have access to newer C++ features Improv shares a home filesystem with other machines like swing these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Improv and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH to keep these cleanly separate. Changelog # 2024-02-22 created document ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/improv/","section":"","summary":"configure spack for ANL LCRC Improv","title":"Spack on Bebop"},{"content":"tl;dr #Put the following in your .bashrc\nfunction use_build { # other ALCF machines if hostname | grep cooley \u0026amp;\u0026gt;/dev/null || hostname | grep cc \u0026amp;\u0026gt; /dev/null; then soft add +gcc-8.2.0 soft add +cmake-3.20.4 soft add +cuda-11.0.2 soft add +mvapich2 source ${HOME}/git/spack-cooley/share/spack/setup-env.sh export clustername=cooley fi export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-cooley git clone https://github.com/robertu94/spack_packages git/robertu94_packages source ~/.bashrc use_build spack compiler find # for extra packages like libpressio spack repo add ~/git/robertu94_packages First create packages.yaml at ~/.spack/cooley/packages.yaml with:\npackages: all: providers: blas: [intel-mkl] lapack: [intel-mkl] fftw-api: [intel-mkl] scalapack: [intel-mkl] mkl: [intel-mkl] git: externals: - spec: git@2.17.1+tcltk prefix: /usr buildable: false openssh: externals: - spec: openssh@7.4p1 prefix: /usr buildable: false curl: externals: - spec: curl@7.29.0+ldap prefix: /usr buildable: false cuda: externals: - spec: cuda@11.0.194 prefix: /soft/visualization/cuda-11.0.2 buildable: false mvapich2: externals: - spec: mvapich2@2.2~cuda~debug~regcache~wrapperrpath prefix: /soft/libraries/mpi/mvapich2/gcc buildable: false rdma-core: externals: - spec: rdma-core@22.4 prefix: /usr buildable: false After that create compilers.yaml at ~/.spack/cooley/linux/compilers.yaml with:\ncompilers: compilers: - compiler: spec: gcc@4.8.5 paths: cc: /bin/gcc cxx: /bin/g++ f77: /bin/gfortran fc: /bin/gfortran flags: {} operating_system: rhel7 target: x86_64 modules: [] environment: {} extra_rpaths: [] - compiler: spec: gcc@8.2.0 paths: cc: /soft/compilers/gcc/8.2.0/bin/gcc cxx: /soft/compilers/gcc/8.2.0/bin/g++ f77: /soft/compilers/gcc/8.2.0/bin/gfortran fc: /soft/compilers/gcc/8.2.0/bin/gfortran flags: {} operating_system: rhel7 target: x86_64 modules: [] environment: {} extra_rpaths: [] And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # We prefer intel-mkl for blas/lapack because of the Intel CPUs We use openssl and mpi (with RDMA Core) from the system to avoid configuring them We load a new compiler from a softenv because the default is ancient. Spack doesn\u0026rsquo;t know about softenv, so paths are passed via the prefix as here Spack shares a home filesystem with other machines like theta these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Cooley and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH variable to keep these cleanly separate. Changelog # 2022-08-11 created this document ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/cooley/","section":"","summary":"configure spack for ANL ALCF cooley","title":"Spack on Cooley"},{"content":"tl;dr #Put the following in your .bashrc\nfunction use_build { # other ALCF machines if hostname -f | grep polaris \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading polaris spack\u0026#34; module swap PrgEnv-nvhpc PrgEnv-gnu module load cudatoolkit-standalone source ${HOME}/git/spack-theta/share/spack/setup-env.sh export CRAYPE_LINK_TYPE=dynamic export clustername=polaris fi export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-polaris git clone https://github.com/robertu94/spack_packages git/robertu94_packages git clone https://github.com/mochi-hpc/mochi-spack-packages git/mochi-spack-packages source ~/.bashrc use_build spack compiler find # for extra packages like libpressio and thallium spack repo add ~/git/robertu94_packages spack repo add ~/git/mochi-spack-packages First create packages.yaml at ~/.spack/polaris/packages.yaml with:\npackages: all: providers: mpi: [cray-mpich] cuda: externals: - spec: cuda@12.0.0 modules: [cudatoolkit-standalone/12.0.0] - spec: cuda@11.8.0 modules: [cudatoolkit-standalone/11.8.0] - spec: cuda@11.7.1 modules: [cudatoolkit-standalone/11.7.1] - spec: cuda@11.6.2 modules: [\u0026#39;cudatoolkit-standalone/11.6.2\u0026#39;] - spec: cuda@11.4.4 modules: [cudatoolkit-standalone/11.4.4] - spec: cuda@11.2.2 modules: [cudatoolkit-standalone/11.2.2] buildable: False libfabric: externals: - spec: libfabric@1.11.0 modules: [\u0026#39;libfabric/1.11.0.4.125\u0026#39;] buildable: False cray-mpich: externals: - spec: cray-mpich@8.1.16+wrappers modules: [\u0026#39;cray-mpich/8.1.16\u0026#39;] buildable: False openssl: externals: - spec: openssl@1.1.1d prefix: /usr buildable: False After that create compilers.yaml at ~/.spack/polaris/cray/compilers.yaml with:\ncompilers: - compiler: spec: gcc@12.3.0 paths: cc: /usr/bin/gcc-12 cxx: /usr/bin/g++-12 f77: /usr/bin/gfortran-12 fc: /usr/bin/gfortran-12 flags: {} operating_system: sles15 target: x86_64 modules: [\u0026#39;libfabric/1.15.2.0\u0026#39;, \u0026#39;cray-mpich/8.1.28\u0026#39;] environment: {} extra_rpaths: [] And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # We use openssl and mpi from the system to avoid configuring them We load a new compiler and programming environment because the default is older. Spack shares a home filesystem with other machines like cooley these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Cooley and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH variable to keep these cleanly separate. Polaris is a cray machine, so needs some cray special sauce like the PrgEnv-gnu to behave more sanely. However, unlike theta it uses CRAYPE_LINK_TYPE=dynamic as the default behavior so we don\u0026rsquo;t need to set it here. As of writing, Polaris uses slingshot 10 and will be updated to Slingshot 11 in Fall 2022 While this should not change the configuration used here, it may change the connection strings for certain applications using mochi/margo based services from using from verbs;ofi_rxm to cxi Changelog # 2022-08-29 created this document ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/polaris/","section":"","summary":"configure spack for ANL ALCF Polaris","title":"Spack on Polaris"},{"content":"tl;dr #Put the following in your .bashrc\nfunction use_build { # other ALCF machines if hostname | grep theta \u0026amp;\u0026gt;/dev/null; then echo \u0026#34;loading theta spack\u0026#34; module swap PrgEnv-intel PrgEnv-gnu module swap gcc/11.2.0 source ${HOME}/git/spack-theta/share/spack/setup-env.sh export CRAYPE_LINK_TYPE=dynamic export clustername=theta export HTTP_PROXY=http://theta-proxy.tmi.alcf.anl.gov:3128 export HTTPS_PROXY=http://theta-proxy.tmi.alcf.anl.gov:3128 export http_proxy=http://theta-proxy.tmi.alcf.anl.gov:3128 export https_proxy=http://theta-proxy.tmi.alcf.anl.gov:3128 fi export SPACK_USER_CONFIG_PATH=\u0026#34;$HOME/.spack/$clustername\u0026#34; export SPACK_USER_CACHE_PATH=\u0026#34;$SPACK_USER_CONFIG_PATH\u0026#34; } Then run\nmkdir -p ~/git git clone https://github.com/spack/spack git/spack-theta git clone https://github.com/robertu94/spack_packages git/robertu94_packages git clone https://github.com/mochi-hpc/mochi-spack-packages git/mochi-spack-packages source ~/.bashrc use_build spack compiler find # for extra packages like libpressio and thallium spack repo add ~/git/robertu94_packages spack repo add ~/git/mochi-spack-packages First create packages.yaml at ~/.spack/theta/packages.yaml with:\npackages: all: providers: mpi: [cray-mpich] blas: [intel-mkl] lapack: [intel-mkl] fftw-api: [intel-mkl] scalapack: [intel-mkl] mkl: [intel-mkl] cray-mpich: externals: - spec: cray-mpich@7.7.14 modules: [cray-mpich/7.7.14] buildable: False subversion: externals: - spec: subversion@1.10.6 prefix: /usr cmake: externals: - spec: cmake@3.10.2 prefix: /usr cvs: externals: - spec: cvs@1.12.12 prefix: /usr texinfo: externals: - spec: texinfo@6.5 prefix: /usr autoconf: externals: - spec: autoconf@2.69 prefix: /usr bison: externals: - spec: bison@3.0.4 prefix: /usr flex: externals: - spec: flex@2.6.4+lex prefix: /usr groff: externals: - spec: groff@1.22.3 prefix: /usr openssl: externals: - spec: openssl@1.1.0i-fips prefix: /usr git: externals: - spec: git@2.26.2~tcltk prefix: /usr diffutils: externals: - spec: diffutils@3.6 prefix: /usr tar: externals: - spec: tar@1.30 prefix: /usr automake: externals: - spec: automake@1.15.1 prefix: /usr gmake: externals: - spec: gmake@4.2.1 prefix: /usr m4: externals: - spec: m4@1.4.18 prefix: /usr openssh: externals: - spec: openssh@7.9p1 prefix: /usr binutils: externals: - spec: binutils@2.35.1 prefix: /usr findutils: externals: - spec: findutils@4.6.0 prefix: /usr pkg-config: externals: - spec: pkg-config@0.29.2 prefix: /usr libtool: externals: - spec: libtool@2.4.6 prefix: /usr libfabric: variants: fabrics=gni mercury: variants: +udreg ~boostsys ~checksum gawk: externals: - spec: gawk@4.2.1 prefix: /usr rdma-credentials: buildable: false version: [] target: [] compiler: [] providers: {} externals: - spec: rdma-credentials@1.2.25 arch=cray-cnl7-mic_knl modules: - rdma-credentials/1.2.25-7.0.2.1_4.3__g67c8aa4.ari After that create compilers.yaml at ~/.spack/theta/cray/compilers.yaml with:\ncompilers: - compiler: spec: gcc@11.2.0 paths: cc: cc cxx: CC f77: ftn fc: ftn flags: {} operating_system: cnl7 target: any modules: - PrgEnv-gnu - gcc/11.2.0 environment: {} extra_rpaths: [] And now spack should work. On your next login, just call use_build to reload spack.\nFor the longer version see the guide on [configuring spack]({% link _guides/spack.markdown %})\nWhat makes this machine special? # We prefer intel-mkl for blas/lapack because of the Intel CPUs We use openssl and mpi from the system to avoid configuring them We load a new compiler and programming environment because the default is ancient. Spack shares a home filesystem with other machines like cooley these machines are completely different hardware wise and use different module systems. The load function loads a copy of spack specifically for Cooley and uses a separate spark instance for other machines. We use spack\u0026rsquo;s SPACK_USER_CONFIG_PATH variable to keep these cleanly separate. Theta is a cray machine, so needs some cray special sauce like the PrgEnv-gnu and CRAYPE_LINK_TYPE=dynamic to behave more sanely. Very often you will want to include --attr enable-ssh=1 in your COBALT flags to enable TCP access to the nodes (i.e. SSH) Changelog # 2022-08-16 created this document ","date":"19 July 2022","permalink":"https://robertu94.github.io/guides/spack/theta/","section":"","summary":"configure spack for ANL ALCF Theta","title":"Spack on Theta"},{"content":"Presentations #","date":null,"permalink":"https://robertu94.github.io/presentations/","section":"","summary":"\u003ch1 id=\"presentations\" class=\"relative group\"\u003ePresentations \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#presentations\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h1\u003e","title":""},{"content":"","date":"15 April 2021","permalink":"https://robertu94.github.io/presentations/ecp2021-lossy/","section":"","summary":"This talk was part of a larger session on Error Bounded Lossy Compression.  It presents updates on LibPressio since the last annual meeting focusing on the automated configuration work completed as part of FRaZ and LibPressio-Opt as well as interface improvements.","title":"LibPressio"},{"content":"","date":"1 November 2020","permalink":"https://robertu94.github.io/presentations/sc2020-doctoral/","section":"","summary":"Compression is commonly used in HPC applications to move and store data. Traditional losslesscompression, however, does not provide adequate compression of floating point data often found inscientific codes. Recently, researchers and scientists have turned to lossy compression techniquesthat approximate the original data rather than reproduce it in order to achieve desired levels ofcompression. Typical lossy compressors do not bound the errors introduced into the data, leading to the development of error bounded lossy compressors (EBLC). These tools provide the desired levelsof compression as mathematical guarantees on the errors introduced. The current state of EBLCleaves much to be desired. The existing EBLC all have different interfaces requiring codes to bechanged to adopt new techniques; EBLC have many more configuration options than theirpredecessors, making them more difficult to use; and EBLC typically bound quantities like pointwise errors rather than higher level metrics such as spectra, p-values, or test statistics thatscientists typically use. My dissertation aims to provide a uniform interface to compression and todevelop tools to allow application scientists to understand and apply EBLC. This canvas presentsthree groups of work: LibPressio, a standard interface for compression and analysis;FRaZ/LibPressio-Opt frameworks for the automated configuration of compressors using LibPressio;and work on tools for analyzing errors in particular domains","title":"Approachable Error Bounded Lossy Compression"},{"content":"","date":"1 September 2020","permalink":"https://robertu94.github.io/presentations/jlesc2020-lossy/","section":"","summary":"As time progresses, the volume of data surrounding machine learning and AI methods continues to grow from training, testing, and validation datasets to the models themselves. As the volume grows, there are increasing challenges in transporting and storing the data. Lossy compression techniques present the opportunity to drastically reduce the volume of data while maintaining or even improving upon the quality of the decisions made by AI. This talk presents a survey of novel research examining the effect of lossy compression on AI decision making in a variety of domains including medical data science and physics. We examine the effects of data ordering, error bounds, compression methodologies to make general recommendations from these and other areas regarding how to effectively leverage lossy compression in AI using the common LibPressio interface.","title":"Lossy Compression for AI"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/software-development/","section":"Tags","summary":"","title":"Software Development"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/spack/","section":"Tags","summary":"","title":"Spack"},{"content":"So you are developing software, and you need the software to build for multiple difference machines, with different version of a library, or with just different compile time options. You can probably manage this by hand, but it very quickly gets out of hand. Spack is a package manager designed for high performance computing, but I would argue is more broadly useful for five reasons:\nIt is explicitly designed to run as an unprivileged user. This makes it much easier to install software for personal use since you won\u0026rsquo;t need sudo. It is almost trivial to create your own even private repositories, mirrors, and build caches. This makes it useful for small teams who have their own specialized software or need to operate behind the firewall. It it allows for combinatorial package management. This means that you can have multiple version of different packages all installed at the same time making it easier to try different combinations of versions. It doesn\u0026rsquo;t make assumptions about compilers or preferred build options. Like Gentoo, it allows for USE flags (called variants) that allow customization of installed packages allowing for more flexible builds. It allows for deterministic package management. Users can simply inquire of Spack, what are all of the versions of all of the packages that this package transitively depends on, and can rebuild this set of packages on demand. It has radically transformed my work-flow and how I work and I think it will for you too. This a quick guide on how to get started with Spack.\nGetting Started Using Spack #Prerequisites #Spack actually doesn\u0026rsquo;t need that much to get started:\nPython A C/C++ tool-chain (doesn\u0026rsquo;t even need to be very new) A handful of very common archive management tools (i.e. tar, gzip, bzip2, xz, and zstd) A few incredibly common development dependencies (i.e. bash, patch, git, curl) If you use a modern Linux distribution, you probably already have all of these installed. Refer to the Installation docs for up to date information\nInstalling Spack #Here is a short script that I use to get started with Spack:\n#clone spack and the main repository git clone https://github.com/spack/spack source ./spack/share/spack/setup-env.sh source ./spack/share/spack/spack-completion.bash #clone and configure my personal repository of extra packages git clone https://github.com/robertu94/spack_packages robertu94_packages spack repo add ./robertu94_packages #tell spack to discover what I have installed spack compiler find spack external find Some other things you might want to do:\nChange the default number of cores used for builds Change the default target architecture (native which is faster) to something more portable (i.e. x86_64) Set some global variants options for packages Set up a mirror, build cache, or upstream instance \u0026ndash; methods for accelerating builds. You can find how to do these things in the docs\nUnderstanding Combinatorial Versioning #With Spack it is important understand the difference between installed and loaded packages. There can be many of the versions of the same package installed; however, only one of them may be loaded at any one time.\nAdditionally, packages can differ not only by version number (i.e. 1.0, 2.1, or develop) but also by variants. Variants are \u0026ldquo;flags\u0026rdquo; that indicate that certain features should be enabled or disabled. Let\u0026rsquo;s consider an example from my repository, libpressio. libpressio has several variants: sz, zfp, mpi, libdistributed, lua, etc\u0026hellip; You can see a list of them with:\nspack info libpressio These variants control which plugins are built for libpressio out of the large array of different available plugins. So let\u0026rsquo;s say you want to install libpressio with sz and zfp: the way to do that is:\nspack install libpressio+sz+zfp The + here refers to the variants that were explicitly enabled for this package not the dependency itself. There is also ~ which explicitly disables a variant. By default, the variants will be set to whatever the package maintainer thought was best. You can always override them in spack\u0026rsquo;s configuration later.\nOk, but what if you wanted to build SZ\u0026rsquo;s random access mode with libpressio version 0.42.1?\nspack install libpressio@0.42.1+sz+zfp ^sz+random_access Here the ^ is the \u0026ldquo;depends upon\u0026rdquo; operator. This says build version 0.42.1 of libpressio with the sz and zfp variants, and when building SZ, build the version that has the random_access variant.\nWhen you are finally ready to use it, you can load it like so:\nspack load libpressio@0.42.1 Or you can refer to it by its specific hash.\nA few other common important commands:\nspack list shows the list of available software spack find shows the list of installed software spack uninstall un-installs a version of software Creating Your Own Spack Packages #Creating and editing your own software couldn\u0026rsquo;t be easier. To create a new package simply call\nspack create -n $package_name $url_to_archive You can even pass a -N $repo_name to put it in your own personal repository\nTo edit an existing package in your $EDITOR:\nspack edit $package_name Spack strongly prefers versions be checksumed, you can do this with:\nspack checksum $package_name For me, most of the time, the defaults were exactly what I needed. If you want to learn more, checkout the official docs\nSpack is a great tool. I hope you found this to be useful!\n","date":"27 August 2020","permalink":"https://robertu94.github.io/posts/2020-08-27-spack/","section":"Posts","summary":"\u003cp\u003eSo you are developing software, and you need the software to build for multiple difference machines, with different version of a library, or with just different compile time options.\nYou can probably manage this by hand, but it very quickly gets out of hand.\nSpack is a package manager designed for high performance computing, but I would argue is more broadly useful for five reasons:\u003c/p\u003e","title":"Spack: a Truly Useful Package Manager"},{"content":" ","date":"1 May 2020","permalink":"https://robertu94.github.io/presentations/ipdps2020-fraz/","section":"","summary":"With ever-increasing volumes of scientific floating-point data being produced by high-performance computingapplications, significantly reducing scientific floating-point datasize is critical, and error-controlled lossy compressors have beendeveloped for years. None of the existing scientific floating-pointlossy data compressors, however, support effective fixed-ratiolossy compression. Yet fixed-ratio lossy compression for scientificfloating-point data not only compresses to the requested ratio butalso respects a user-specified error bound with higher fidelity. Inthis paper, we present FRaZ: a generic fixed-ratio lossy com-pression framework respecting user-specified error constraints.The contribution is twofold. (1) We develop an efficient iterativeapproach to accurately determine the appropriate error settingsfor different lossy compressors based on target compressionratios. (2) We perform a thorough performance and accuracyevaluation for our proposed fixed-ratio compression frameworkwith multiple state-of-the-art error-controlled lossy compressors,using several real-world scientific floating-point datasets fromdifferent domains. Experiments show that FRaZ effectively iden-tifies the optimum error setting in the entire error setting space ofany given lossy compressor. While fixed-ratio lossy compressionis slower than fixed-error compression, it provides an importantnew lossy compression technique for users of very large scientificfloating-point datasets.","title":"FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Floating Point Scientific Data"},{"content":" ","date":"1 January 2020","permalink":"https://robertu94.github.io/presentations/libpressio2020-tutorial/","section":"","summary":"This video provides a basic tutorial of how to install and use LibPressio \u0026ndash; a generic abstraction for lossless and lossy compression for dense tensors.  You can find LibPressio on \u003ca href=\"https://github.com/codarcode/libpressio\" target=\"_blank\" rel=\"noreferrer\"\u003egithub\u003c/a\u003e","title":"LibPressio Tutorial"},{"content":" ","date":"1 December 2019","permalink":"https://robertu94.github.io/presentations/argonne2019-aproachable/","section":"","summary":"Error Bounded Lossy Compression is a powerful technique that combines the power of lossy compression with the strong guarantees for the preservation of the original data. However, using error bounded lossy compression can be challenging. There is a proliferation of interfaces, different semantics, and difficult to understand relationships to user error bounds. This talk presents two tools — LibPressio and FRaZ — that help address these issues. LibPressio provides a usable abstraction across SZ, ZFP, MGARD, as well as traditional image compressors (JPEG, WEBP, PNG, TIFF, etc) and lossless compressors via BLOSC. FRaZ is a tool that uses LibPresssio to compress datasets according to a Fixed Compression Ratio with progress towards supporting arbitrary user error bounds. I present our recent work submitted to IPDPS and current work in progress.","title":"Approachable Error Bounded Lossy Compression"},{"content":"Should a software design be strong or robust? This is a debate that seems to have been developing in recent years with the recent proponents so-called \u0026ldquo;strong-typing\u0026rdquo; advocating new API designs. In this post, I go a little into the debate and discuss its consequences.\nWhat does it mean to be strong vs robust? #Robustness in software engineering is not a new concept, and intuitively a attractive one. Who doesn\u0026rsquo;t want their software to be robust? However, robust to what? And more importantly, at what costs?\nOne of the most lasting definitions of robustness comes from Jon Postel \u0026ndash; one of the early architects of the Internet.\nBe liberal in what you accept, and conservative in what you send\nBy this he argued, that systems should accept a wide array of types and formats as input, while sending out a highly specified return value. Now Postel meant this in terms of the design of inter-networked systems such as the Internet; However, the same principles apply to API design. What is an API except a means by which we accept input from others and return to them a response or do some work on their behalf.\nHowever this comes at a cost. Network programming has bedeviled many a computer science undergraduate and later engineers as they attempt to cope with the proliferation of implementations on the Internet. In some senses, these systems are not easy to use.\nIn a different domain, developers and teachers such as Jonathan Bocara have been advocating for so-called \u0026ldquo;strong types\u0026rdquo;. Bocara describes strong types this way:\nA strong type is a type used in place of another type to carry specific meaning through its name\nAt first this might not make a ton of sense so consider this example which is slightly different than the one Bocara uses:\ntemplate \u0026lt;class Type, class Tag\u0026gt; struct NamedType { Type value; template\u0026lt;class... Args\u0026gt; explicit NamedType(Args\u0026amp;\u0026amp;... value_args): value(std::forward\u0026lt;Args\u0026gt;(value_args)...) {} operator Type() { return value; } }; using NumThreads = NamedType\u0026lt;unsigned int, struct threads\u0026gt;; This allows instances of NumThreads to be used as an unsigned int but instances of unsigned int cannot be used as NumThreads without explicitly converting it to a NumThreads first. Likewise, each class instantiated from, NamedType is incompatible with each other. That means if we made another type called NumProcesses from NamedType it would also be incompatible. You can only use NumThreads for NumThreads and NumProcesses for NumProcesses\nBocara and others argue that this design makes it easier to use APIs that use types like this. Specifically, it helps with the wrong argument order problem and the problems associated with not understanding what a argument means from a call site. To use the example above, if all you have is 4 at the call sight, you have no idea what 4 means, but if you see NumThreads(4) you can be reasonably sure that the function will use 4 threads.\nWhat are the trade-offs? #I see there are two major areas of trade-offs between strong and robust types.\nFirst is the tradeoff between ease of implementation and ease of use and flexibility. Strong Types are self-evidently easier to implement and use. They are self-documenting at the call site, and they are trival to write with helpers such as NamedType above. Additionally, since strong types can only be used in one way, they require less effort to implement.\nStrong types are less flexible. Consider a concurrency library that allows tasks to be executed in parallel. At first, this library could be implemented using operating system threads. This library could have a type called NumThreads which determines the number of threads that are to be used. However, an enterprising developer could add another back-end that uses operating system processes instead. At this point, the developer could use NumThreads for both and the name would be misleading, he could could create a new type called NumProcesses, or he could rename NumThreads to a new name which implies some other other concurrency model that users have to learn and understand while breaking the API for all downstream users who have NumThreads in their code. The second choice is even more insidious because it now makes it harder from some down-stream user to have a generic parallel solution that works regardless of what kind of parallelism the user wants to employ because they now need to introspect the task queue to determine which argument it needs to have passed.\nSecond is modifiability. For either, contracting the API \u0026ndash; i.e. allowing fewer inputs than previously allowed \u0026ndash; is a breaking change to users of the API. This is illustrated by Hyrum\u0026rsquo;s law:\nWith a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody. ~ Hyrum\u0026rsquo;s Law\nFor strong types, expanding the API is typically not a breaking change unless users are doing something unusual (i.e. making a function pointer from your API, need ABI compatibility for an API with default arguments, etc) and requires only an additional overload. For robust APIs, they necessarily allow expansion by definition, but then require a substantially more robust design to accept a wider possible array of input types.\nHow can we do implement these principles? #Every language has different capabilities to implement these design patterns. I take a few case studies from a few languages that I have used: Java, C++, and Python.\nStatically typed languages such as Java and C++ have the ability to implement something like a strong type. Where they differ is their ability to implement robust types.\nJava which prides itself as a language for software engineers and architects has a wide array of options for complementing robust types. It has interfaces, bounded generics and unbounded generics. Using these facilities is straight forward.\nC++ has fewer but arguably more powerful options with template and SFINAE. Since C++ templates are turning complete, it is possible to implement very sophisticated code generation facilities in templates if you don\u0026rsquo;t care about compile times. However unlike Java, these changes are typical not trivial. To implement constrained templates uses will take advantage of SFINAE and built-in classes such as std::enable_if and std::void_t to enable and disable functionality. In C++20, C++ gained concepts which allow an easier way to specify this kind of functionality, but these are as of writing not yet widely available.\nDynamically typed languages such as Python have very little ability to implement strong types without going against the principles of the language. In recent versions, python has gained a facility called Type Hints which can provide some additional information to API users, but it isn\u0026rsquo;t widely used, and isn\u0026rsquo;t as complete a full type system in a language such as C++ or Java.\nWhy does it matter? #So why do these design choices matter? They matter because they change the quality attributes of our designs. As we chose between strong types and robust types, we make decisions that impact the maintainability, usability, and modifiability of our designs. Thinking carefully through these trade-offs will improve our designs.\nHope this helps!\n","date":"17 November 2019","permalink":"https://robertu94.github.io/posts/2019-11-17-strong-or-robust/","section":"Posts","summary":"\u003cp\u003eShould a software design be strong or robust?\nThis is a debate that seems to have been developing in recent years with the recent proponents so-called \u0026ldquo;strong-typing\u0026rdquo; advocating new API designs.\nIn this post, I go a little into the debate and discuss its consequences.\u003c/p\u003e","title":"Strong or Robust?"},{"content":"There\u0026rsquo;s at least for me an inherent coolness to be able to say that I can run code on a super computer. However, there is often a gap between the kind of programming that you learned in your introductory classes, and what kind of code runs well on a computing cluster. In this post, I try to provide an introduction to high performance computing, and some of the differences between it and personal computing.\nWhat is a super computer really? Modern super computers are not really a single physical machine, but instead a collection of commodity machines that are networked together with specialized software that helps them work together to solve problems larger than any individual machine could solve on its own.\nWhy run something on a super computer? Super computers can be used when the problem is either too large to solve or too large to solve in a reasonable amount of time.\nKey First Steps: # Read the user guide for your specific cluster, it will highlight important details that are specific to your cluster such as hardware details and configuration as well as preferred and available software. Every super computer has unique hardware that should be accounted for when choosing what software to run because it can have dramatic performance impacts. Learn a language with good tooling for HPC, supported on your cluster. Writing software for one machine is hard; writing software for a team of machines with diverse and specialized hardware is harder. Using a language that has mature libraries to support different use cases will make things easier. C/C++ Python Fortran, Julia, and Scala (for big data) are also popular Learn how to schedule batch jobs and run interactively on your cluster. On super computers, you often need to request time to run a task rather than just running it. The software that allows you to request time is called the scheduler. Often it provides facilities to coordinate between jobs, monitor their progress, and to notify you when they are finished. Profile first to understand the bottlenecks of your applications. Learn how to user a profiling tool available on your system. Profile an application that you use on a super computing cluster. What parts of it are slow? How efficiently are you using the hardware capabilities of the machine? What libraries or tools exist that could enable you to speed up and better utilize the machine? Common Frameworks and Their Alternatives #It is desire-able to be able to write one set of code and have that code have optimal performance on every machine that you might want to run it on. To this end, library and framework authors attempt to write libraries that will help make this task easier. However truly performance portable code is largely an illusion. Each machine will have different hardware and configuration options that will be best tuned for certain kinds of workloads. In order to take best advantage of these systems, it\u0026rsquo;s important to understand a variety of frameworks and methods to best tune code for a machine.\nInter-Node Parallelism \u0026ndash; MPI (Message Passing Interface) #In HPC, The Message Passing Interface is the de-facto lower level programming framework for coordination between nodes. It provides relatively low level primitives that can be built upon to coordinate work amongst the cluster. Because of its importance in HPC, I plan to write a separate learning to learn on it because of its importance to HPC. However for a deep dive to learn more about MPI consider reading the books \u0026ldquo;Using MPI\u0026rdquo; and \u0026ldquo;Using Advanced MPI\u0026rdquo; to get started followed by the MPI standards specifications for details as needed. You can find more on MPI here\nHowever since MPI is fairly low level, it has encouraged the development of higher level libraries to support applications real world usage. Two of the most notable are PETSc and HDF5. The former provides matrix math facilities and the latter IO.\nHowever there are some notable alternatives to MPI include global memory systems such as upc++ and RPC systems such as Thallium or gRPC.\nIntra-Node Parallelism \u0026ndash; OpenMP (Open Multi Processing) #Historically, MPI has been most used for distributed memory parallelism, and other systems are used for unified memory parallelism because unified memory systems can make simplifying assumptions that can improve performance. One of the most prevalent unified memory systems is OpenMP.\nOpenMP enables multi-threading and heterogeneous programming within a single node. Unlike MPI which is a library, OpenMP is best thought of a series of compiler extensions for C, C++, and Fortran compilers that parallelize code marked with special directives. Recently OpenMP gained concepts for heterogeneous computing as well allowing the programming of GPUs and other specialized accelerators with minimal syntax.\nOther alternatives for OpenMP could be parallel algorithms in your language\u0026rsquo;s standard library, vendor specific offloading tools like Cuda, hip, or OneAPI, the SYCL extensions to C++, as well as user level threading libraries like argobots.\nParallel IO \u0026ndash; HDF5 (Hierarchical Data Format) #For large programs that leverage super computers, IO can become a performance bottleneck. To alleviate this, supercomputers have developed parallel IO libraries to write to the distributed file systems. Often these are built atop MPI\u0026rsquo;s file IO standard.\nFor simple file formats, it is possible to use MPI-IO directly, however in practice it is generally better to just use HDF5. HDF5 provides a set of routines to read and write large self-describing volumes of data. It\u0026rsquo;s documentation is fragmented and dense, but there is tons of straight forward example code that is easy to use. HDF5 also provides high level facilities for operating on tables of data or other specialized structures.\nSome alternatives to HDF5 include Mochi and DAOS which both take a more \u0026ldquo;micro-services\u0026rdquo; based approach to IO.\nMath Libraries #One of the most common operations performed on super computers are matrix math. This is because of both the ubiquity of these operations in scientific codes, but also the relative intensity of these operations. There are a few performance optimized libraries that have are commonly used to support these operations.\nFor dense matrix algebra there are BLAS and LAPACK. The former provides primitives for vector and matrix math. The later tools for solving matrix systems. Later distributed memory parallel implementations of BLAS and LAPACK were developed called PBLAS and ScaLAPACK respectfully. Lastly there are recent developments in libraries like MAGMA which further extend these classic libraries for heterogeneous programming. Each system will have its own set of these libraries to best optimize these routines.\nFFTW is a library for performing fast Fourier transforms. It provides both a distributed and unified memory versions. While many libraries may beat FFTW in raw performance for a little while, it doesn\u0026rsquo;t tend to remain that way for long. One thing to be aware of is that FFTW is GPL licensed, which has implications for how you publish software that uses it. As a result, there have been re-implementations of its interface in libraries like MKL from Intel that have other licenses. Read these licenses carefully before you use these pieces of software.\nSuiteSparse is a collection of primitives and solvers for sparse matrices and vectors. While many HPC problems are dense, a growing number of them are modeled as sparse. SuiteSparse provides similar routines to BLAS and LAPACK but for sparse problems.\nYou probably don\u0026rsquo;t use these directly, instead use a higher level library like Eigen or PETSc if you need distributed computing.\nPatterns for Parallel Computing #As mentioned above, often software needs to be refactored to achieve optimal performance. Here are a list of patterns that are commonly used to adapt software for parallel and distributed environments.\nFundamental Patterns # pattern problem enumeration and grouping identify and group parallel resources send/recv point to point communication collectives distribute or collect factions of work from a collection of nodes functors/map perform many completely independent problems catamorphisms/reduce/scan combine many associative operations anamorphisms/hylomorphisms/divide and conquer problems that can be partitioned into independent sub problems sorting permute elements into a specified order Advanced Patterns # pattern problem linear algebra and BLAS mathmatical primitives operations graph operations scalable grpah operations pipeline a series of steps that need to be preformed in order, but otherwise are independent grid/stencil problems calculations on a grid of values Strategy Patterns # pattern problem auto-tuning at runtime choose an implementation appropriate for the platform and problem hardware specialization utilize specialized hardware features to acceerlate the code optimize common operations bias performance of sets opererations in favor of the more common ones fast-path optimization write code to skip expensive checks/verification in the typical case speculative execution compute a result before it is needed in anticipation of needing it, and discarding it if unused operator fusion and reordering combine and reorder multiple operations to improve performance of the collection Load Balancing Patterns # pattern problem move execution move computation to data to increase locality batching rather than doing many small operations, group them to do more work at once work stealing balance imbalanced workloads by shifting work from overutilized processors to underutilized ones exponential back-off avoid contention by backing off a progressively increasing amount of time on each timeout caching store the results of expensive calculations/load/stores in faster storage for reuse jitter randomize the time for certain operations to reduce contention Resource Management # pattern problem cooperative scheduling avoid prevention to increase throughput on tasks and avoid OS overheads premption and priority prempt execution of long running tasks to share resources backfill utilize a gap in a schedule to schedule smaller tasks pooling allocate a large chunk of resources (memory, disk, nodes, etc\u0026hellip;) to avoid repeatedly preforming an expensive allocation Resilience Patterns # pattern problem check-point restart mitigate the possibility of failure by periodically taking, and allowing restarting from a saved copy of the program state replica preform operations multiple times or store multiple copies in memory to enable restarts in case of failure gossip protocols actively periodically check the state of remote nodes to ensure that they are still functioning or to communicate events heart-beat passively periodically check the state of remote nodes to ensure that they are still functioning bulkhead design the application to anticipate failure of one or more components and recover gracefully reconfiguration upon failure, reconfigure the application to run in a reduced mode reconstruction upon failure, partially rebuild the application using new resources erasure codes a less memory/disk intensive form of replica which can recompute the state provided a certain number of \u0026ldquo;copies\u0026rdquo; survive merkel trees incrementally hash a structure to be able to validate changes in subsets algorithm based fault tolerance introducing slight modifications to algorithms to apply lightweight fault tolerance Approximation Patterns # pattern problem inexact algorithms using approximate algorithms/functions which are faster, but less precise proxy-model use an statistical/ml/ai model which emulates a full computation compression use a compressed representation of the state to save disk/memory/bandwidth lossy compression use a lossy compressed representation of the approximate state to save even more memory/disk/bandwidth sampling use a reduced number of states to approximate the entire state to save compute/disk/memory/bandwidth dimensionality reduction project the state into a lower dimensional representation to save compute/disk/memory/bandwidth Synchronization Patterns # pattern problem atomic instructions use specialized instructions to avoid explicit locking strong consensus and leader election select one or more process dynamically to preform synchronized operations futures represents a handle to a async computation that can be waited upon and possibly canceled queuing/actor model place work to be done on a queue to be evaluated when resources are available two stage commit place work to be done on a queue to be evaluated when resources are available delayed synchronization synchronize less frequently to avoid expensive communication at the possible cost of some accuracy conflict-free replicated data types synchronize less frequently to avoid expensive communication at the possible cost of some accuracy repository store state in a subset of nodes to reduce the time required for syncronization Which if any of these patterns can you use to accelerate your program? Reproduce-ability #Broadly speaking there are two approaches to reproducible HPC programs. Use a specialized package management system that creates as close to a hermetic build as possible like Spack, or use a container runtime environment like singularity with all of your dependencies self contained. These approaches can be complementary and systems like Spack can build containers.\nWhat can you do to make your results more reproducible? What next? #If you are looking for more resources here is where I would get started:\nRead the specific documentation for the libraries and software that you intend to use. Academic conferences like Super Computing, International Super Computing, the International Parallel and Distributed Processing Symposium are major venues where technologies related to HPC are announced and demonstrated. Papers from these conferences are available online from IEEE or the ACM depending on the year. Suggestions to learn HPC #To learn HPC I have a few recommendations:\nBuild your own virtual or Raspberry Pi cluster. Building a cluster will give you a greater appreciation of how the underlying software that drives HPC works. You can either build it in a set of 3-4 VMs or Raspberry Pi or similar single board computer. Many commercial cloud providers also have easy cluster setup systems. Port a code that you have to run on a HPC cluster. This will enable you to get some hands on experience with the software and technologies involved. Change Notes # 2023 Added remarks on profiling 2021 Initial Version ","date":"25 August 2019","permalink":"https://robertu94.github.io/learning/hpc/","section":"","summary":"\u003cp\u003eThere\u0026rsquo;s at least for me an inherent coolness to be able to say that I can run code on a super computer.\nHowever, there is often a gap between the kind of programming that you learned in your introductory classes, and what kind of code runs well on a computing cluster.\nIn this post, I try to provide an introduction to high performance computing, and some of the differences between it and personal computing.\u003c/p\u003e","title":"Learning to Learn: High Performance Computing"},{"content":"So you want to or have to try this thing called \u0026ldquo;Linux.\u0026rdquo; Just like curry powder isn\u0026rsquo;t just one thing, but a distinct mix of spices that come together into a tasty mixture, Linux is much the same. Also like curry, Linux isn\u0026rsquo;t for everyone. In this post I describe the process of choosing the \u0026ldquo;flavor\u0026rdquo; of Linux that will work best for you, introduce a powerful tool that will help you to make the most of Linux, and describe some first steps to take when things go wrong.\nChoosing a Distribution #What even is Linux? Without going too far into the weeds, Linux is an \u0026ldquo;operating system\u0026rdquo;. There is something of a debate about what exactly the words \u0026ldquo;operating systems\u0026rdquo; means amongst operating systems developers. If you think of Windows and macOS you probably envision a pretty substantial collection of software including graphical interfaces. However, the developers of the BSD family of operating systems would argue that it would be slightly smaller and includes only critical software for server use which has practically ubiquitous use. Even more so, developers like Linus Torvalds \u0026ndash; the primary maintainer and original developer of Linux \u0026ndash; argue that an operating system is relatively tiny \u0026ndash; just a set of functions that abstract away the differences between different sets of hardware that can be used by applications that you care about. And there are those who would argue for an even smaller definition\nSince Linux is so small compared to proprietary operating systems, the user has to make a comparatively large number of choices in order to have a practically usable system: What kind of desktop do I want? Do I even want a desktop? What kinds of applications do I want? How will they work together? This is where \u0026ldquo;distributions\u0026rdquo; come in. Distributions make to various degrees decisions about these important questions to make a system like Linux easier to use. But this leaves users with yet another question: What Distribution should I choose and why?\nSo how can I find a list of Linux distributions for my usecase? You can literally just search the Internet for \u0026ldquo;Linux distribution for [type of user]\u0026rdquo; and generally come up with the list of more supported distributions for your use case. You can also use sites like DistroWatch which list maintain an active list of distributions by some measure of popularity. However, by virtue of the site, it tends to highlight more \u0026ldquo;boutique\u0026rdquo; or frequently updated distributions. You can also listen to podcasts about Linux to get a feel for some different distributions: I like the shows by the Jupiter Broadcasting Network. Once you have a list of distributions to consider, I argue that you should consider 3 factors when choosing a Linux distribution:\nThe rate of change and the support cycle. The availability of software The choices of default software Rate of Change and Support Cycle #The first factor to consider when choosing a Linux distribution is the questions of rate of change and support cycle. The rate of change is rate at which the maintainers provide new versions and changes to the software. Since rate of change is hard to measure directly, it is often easer to look at the related to the question of the length of the support cycle. The support cycle is the length of time the developers provide security patches other services to support to older versions the software. Almost always, a short support cycle is coupled with a high rate of change and visa versa.\nThere are Linux distributions at many points along the spectrum of possible options. At one extreme, distributions \u0026ndash; like CentOS \u0026ndash; adopt changes very slowly favoring the test of time to weed out bugs. At the time of writing CentOS has a 10 year support cycle. However there are costs to long support cycles, some you may use may get to be quite old. In the middle are distributions that offer a 1 year, and 2 year, and 5 year support cycle. At the other extreme, other so-called \u0026ldquo;rolling\u0026rdquo; distributions \u0026ndash; like ArchLinux \u0026ndash; adopt changes very quickly to provide users with the latest and \u0026ldquo;greatest\u0026rdquo; of software. Rolling distributions often expect their users to adopt changes every few weeks if not every few days.\nSo when considering what Linux distribution to install, consider where you fall along this spectrum:\nDo you want to be installing updates every few days, or are is a monthly or yearly update with some critical security patches more often more what you are looking for? Do you do work that benefits from the latest and \u0026ldquo;greatest\u0026rdquo; versions of software? Do you want a \u0026ldquo;set it and forget\u0026rdquo; paradigm, or are you looking for something you can \u0026ldquo;tinker\u0026rdquo; with? So how can I tell where a Linux distribution falls on this spectrum? First search for terms such as \u0026ldquo;support life-cycle\u0026rdquo; or \u0026ldquo;rolling release\u0026rdquo; on the distribution web page. Unfortunately, it also important to see if the distribution is actively maintained especially if a distribution is not backed by a company. Just because a distribution says they have a 5 or 10 year support cycle doesn\u0026rsquo;t mean it is still actively supported or maintained. Occasionally leaders retire from maintaining the distribution and for smaller distributions that can cause the distribution to die along with them. Signs a distribution have may have died include:\nNo patches for critical and widely publicized security vulnerabilities A stale website or source code repository that hasn\u0026rsquo;t been updated racily Large numbers of broken links A social media or blog post that describes that distribution is ending Considering how stable or rolling you want your system to be is an important first step to choosing a Linux distribution.\nAvailability of Software: Package Management and Software Selection #Another factor to consider is will the particular distribution support the software packages that I want to run? If you ultimately cannot find software for your distribution, then Linux may not be for you. However, in recent years, there have been great strides in availability for many kinds of software. Here are the steps that I recommend users take to find software for their distribution:\nIs the software available for your distribution? Either in: Packages in your distribution\u0026rsquo;s software repository Packages in cross-distribution Container Package Managers From a custom build script or self-contained archive Is the there an alternative for the software? Look in the previous sources (1.1-1.3) for software that has the same function Consider \u0026ldquo;web-based\u0026rdquo; alternatives If all else fails consider virtualization Distribution Package Managers #You may have used an \u0026ldquo;app store\u0026rdquo; on various proprietary operating systems. Linux had these long before the proprietary alternatives and they are called \u0026ldquo;package managers\u0026rdquo;. They are the primary way of installing software on Linux. And, just like the app stores you are familiar with each have different applications, each Linux distribution has its own software available for it.\nPackage managers have many benefits. They manage updates for the software on your computer eliminating the need for separate updater processes for each program. They help eliminate the redundancy of having multiple copies of the same libraries and data on your computer. They also install required dependencies automatically. Generally packages from your distributions package manager will be the most tested software for your particular distribution meaning you are less likely to run into bugs. For these reasons, you should always check your package manager first before installing from other sources.\nLinux package managers are not universal between distributions \u0026ndash; in fact many distributions have their own unique package manager. While these package managers offer similar feature sets, each offers its own features that distinguish it for the particular use case the developers had when they designed it. This means that if you decide to change Linux distributions you will likely need to learn a new package manager.\nEven between Linux distributions that use the same package manager there can be differences. For example, almost every distribution that I have ever used had a different name for the package containing the graphical version of the vim text editor. These different names make it difficult to use packages even for the same package manager from a different distribution because the dependency listings have the wrong name for the corresponding dependencies on a different distribution. And that doesn\u0026rsquo;t even consider the possibility of differently choose default options for packages of software that can conflict.\nSome package managers support the ability to add third-party application sources. The quality of these third-party packages can vary greatly. Some such as EPEL (Extra Packages for Enterprise Linux) for RedHat Enterprise Linux or CentOS are very reliable, but others can fall out of date quickly, miss important security updates, be untested, or even worse contain malicious software. If you choose to use them, use them with care.\nContainer Package Managers #Recently there have been efforts to create package repositories that work across Linux distributions. These work by packaging a \u0026ldquo;runtime\u0026rdquo; with the applications so that it works regardless of what distribution they are installed on. These run-times can be large, but through some functionality in the Linux kernel, they can be shared between applications. They also offer features to lock down applications so that they can only access part of the system by default.\nAt time of writing there are 3 leading contenders for desktop applications: flatpak, snap, and appimage. Each has their own advantages and disadvantages which are changing daily. Additionally not all distributions support all of these formats \u0026ndash; they may not have a new enough kernel or have the right kernel features enabled. However, most distributions support them.\nAnother alternative is to use software designed for packaging server software such as Docker or Podman. Because docker was designed for server software, the default security options are so strict that it tends disallow things that desktop software needs to properly function. However it can be an option for software with fewer permission requirements.\nCustom built software #Unlike other proprietary operating systems, Linux has a long tradition of users building their own software from source code. This source software can be found on company websites in .tgz or .zip archives or in git repositories like github. Once you have the source you will need to build and install it. There are a number of tools for doing this, including cmake, meson, and autotools but each package adopts a different approach. By convention the instructions for how to compile and install software can be found in a file called README or INSTALL.\nThere are reasons to use custom build software:\nYou have software that you developed yourself and are only using on a small number of computers. You are using a Linux environment where you don\u0026rsquo;t have access to the package manager such as work, school, or a super computer. You want an software option selected that isn\u0026rsquo;t widely enabled. That said, there are a number of reasons to avoid installing a number of packages this way. First, it won\u0026rsquo;t update with the other packages. Second, it doesn\u0026rsquo;t manage dependencies like the previous methods. Finally, it is substantially more complicated than the previous methods.\nWhere to find alternatives #While software availability for Linux is much better than it was a decade ago, there will be software you use that doesn\u0026rsquo;t exist on Linux. However, often times there are good alternatives that preform the same or a similar enough function. Good places to look for alternatives include the extensive ArchLinux list of applications, specialized search engines such as alternativeto, and general web search. Once you\u0026rsquo;ve found an alternative you like, you can install it using any of the previously described methods.\nIn addition to native applications, its also important to consider web applications as possible alternatives. For example Microsoft has a version of Office called Office 365 Online which implements many of features of Desktop versions of Office except it runs in a web browser. While it doesn\u0026rsquo;t do everything that the Desktop version does, it gets me most of the way to doing everything I need to do for the few cases that Libreoffice doesn\u0026rsquo;t work.\nOne final area to consider is Google Chrome\u0026rsquo;s Android app runtime environment. It allows you to run many android apps as in the Chrome browser. It doesn\u0026rsquo;t work for all applications on Android \u0026ndash; for example my preferred Bible reader software Olive Tree \u0026ndash; but can provide some needed applications not available otherwise.\nConsider Virtualization #Lastly if all else fails, Linux has excellent virtualization support. Virtualization allows users to run other operating systems on top of Linux. While this is a heavy weight solution for the situation where there is absolutely no substitutes for the software that you need. A complete treatment of virtualization is out of scope for this article, but Red Hat\u0026rsquo;s Documentation is best in class albeit a tad technical.\nYou may see recommendations to use Virtual Box for for the virtualization host. While this is true for Windows \u0026ndash; and perhaps MacOS \u0026ndash; this is largely not true for Linux. Virtual Box is notorious for being harder to maintain and less efficient with resources than other alternatives for Linux such as QEMU/KVM. Previously, Virtual Box offered trade offs like better client tools, but this is no longer the case with the advent virtio drivers for most major operating systems (including recent versions of MacOS and Windows).\nSo What Distributions do I Recommend? #The distributions that I recommend most often are:\nDistribution Purpose Fedora Workstation Daily use for users who are early adopters Ubuntu LTS Daily use for users who prefer stability Fedora Server For a server with more modern software CentOS For a server that needs to run with minimal changes for 10 years ArchLinux If you want bleeding edge sometimes unstable software Gentoo If you have a highly customized workload that nothing else works for What Linux distribution is well suited for your needs? Where are you going to get key software that you need? Desktop Environments #One piece of software that new Linux users will need to select that they probably haven\u0026rsquo;t before is the desktop environment. The desktop environment is the software that provides the desktop that you see when you log into a graphical environment. Unlike Windows and MacOS, Linux offers multiple desktop environments that serve particular needs. I think there are roughly three types of desktop environments to consider:\nModern Environments #Modern Environments tend to be heavier on resources, but offer more ease of use features and smooth animations in exchange. The two leading examples of these are KDE\u0026rsquo;s Plasma desktop and Gnome.\nKDE is the more customizable of the two. It allows customization of nearly every aspect of the desktop experience, from where windows for recently started applications are drawn to how and when notifications are sent. It offers a relatively sane set of defaults, but often takes some tinkering to get it exactly how you want it. If you favor lots of customization options, KDE may be for you.\nGnome offers a simple and reliable out of the box experience. It doesn\u0026rsquo;t have nearly the customization options that KDE had; instead, it offers a opinionated, easy to use, modern desktop. Gnome does have some weird opinions that may be foreign to new users. For example, it at time of writing doesn\u0026rsquo;t allow icons on the desktop. Instead, users should search for the application and documents they want using the built-in search feature. However, if you embrace the Gnome work flow, I have found it to be quite productive.\nClassic Environments #Classic desktop environments will be more conservative with animations and features, but instead try to replicate a computing paradigm from an earlier time.\nXFCE is probably the most conservative of the bunch. The offer an extremely light weight computing experience that performs well even on older hardware that doesn\u0026rsquo;t feature a built-in GPU. That doesn\u0026rsquo;t mean it is completely lacking in features. It offers significant capability to power users who are willing to peer in to some of the hidden functionality using scripting. Even though, I don\u0026rsquo;t use XFCE as my main desktop, I actually choose to use their file manager Thunar for this reason.\nMate is based on an earlier version of the Gnome desktop before they became quite as modern. It offers several default appearances that resemble MacOS, and different versions of Microsoft Windows. It requires a bit more resources than XFCE, but in exchange offers a few more options without having to adopt scripting.\nNon-Traditional Environments #Linux also offers a number of non-traditional desktop environments that may not be for new users, but offer substantial productivity improvements in the long run.\nOne such family is tiling window managers such as i3. Tiling window managers don\u0026rsquo;t encourage you to place windows manually, but rather they are automatically tiled to make the most use of the screen. They are often the most customizable desktop environments, but also require the user to find the functionality they want and either adopt or implement it. However in exchange the user gets a custom fit experience that can be quiet efficient.\nIt is also possible to use Linux with a graphical environment and instead manage it using the command line. While this probably isn\u0026rsquo;t great for desktop use, it is very common for server environments. A graphical desktop and its dependancies are just one more thing to take up resources on the machine and simply aren\u0026rsquo;t needed of machines that are primarily web-servers or network attached storage for example.\nUsing the Command Line #You won\u0026rsquo;t be able to use Linux to its fullest without using its command line. However, learning it use it can be daunting. In this section, I want to motivate why you should take the effort and suggest a path for doing so.\nWhy use the command line? #Using the command line grants the power to automate, access to additional functionality, and the ability to quickly integrate several smaller tools to solve larger problems.\nUnix \u0026ndash; the predecessor to Linux \u0026ndash; was designed as time sharing system where companies would automate large batch processes. It is unsurprising then that Linux retains much of this heritage to automate tasks. The command line gives the user the ability to trigger tasks at a particular time, when a file or directory is modified, when status of a service changes and many others. The power to run a task when events occur allows the user to respond to them in a way that may require a large army of humans to do individually. Because the command line only produces and accepts streams of text, it becomes easy to have an interface that adapts to all of these kinds of events as well as new events as they are developed and introduced.\nAdditionally, many commands expose additional functionality via their command line interfaces. It is often easier to develop command line interfaces for a particular task than graphical ones. In fact, most programming languages do not provide graphical window interface toolkits in their standard libraries. This means that developers will often put provide interfaces to the command line before they develop a graphical interface so they can prototype functionality. Rather than remove this functionality, the command line interface is retained to allow for batch processing and automation.\nFinally, chaining several small commands yields a combinatorial explosion of functionality. The developers of Unix developed a design philosophy for how tools in their system should be used. While the whole of the philosophy is worth consideration, let\u0026rsquo;s focus on guideline 1 and 2.\nMake each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new \u0026ldquo;features\u0026rdquo; Expect the output of every program to become the input on another, as yet unknown, program. Don\u0026rsquo;t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don\u0026rsquo;t insist on interactive input. Quoted from: UNIX Time-Sharing System: Foreword\nTools that adhere to guideline one are likely to have only one specific side effect if they have any at all. While tools that adhere to guideline two are endomorphisms which means they are inherently compose-able. Together tools that follow both guidelines one and two can be combined and pipelined without fear of unwanted side-effects. This creates a combinatorial exposition of functionality that provides great power to the user.\nSo for example, imagine you have a tool that monitors how full your hard disk is, another tool that makes graphs out of numeric data, and another tool that sends email. You could use these tools to send an email if the drive gets too full, or you could use them to make a plot of how your disk has filled over time. Then if you got a new application that allows sending text messages, you could easily swap out the last tool in the change and send the alert the user via text message with minimal changes to the rest of the pipeline.\nCommon Objections #Some will argue that the command line is harder to use than graphical applications.\nThey might argue that the textual interface of the command line is harder to use than a graphical interface. For some tasks this is almost certainly true: video editing, document formatting, or web browsing. However, more often than not, it is an expression of personal lack of experience than actual measurable increases in difficulty. When you face this frustration, consider if you are working with or against the strengths of the command line. Are you working on an aesthetic or interactive task or are you working on a data-oriented, batch/non-interactive task where you are integrating many data sources where a command line pipeline will shine? Using the tool for its intended purpose will improve its ergonomics.\nThey may also argue that its daunting to learn due to the sheer number commands. They might compare it to the relative elegance of a language like C or python. Yes, the shell does have a large number of possible commands, but that doesn\u0026rsquo;t mean you need to use them all. My computer has at least 2661 such commands installed; I have used no more than 384 of them. To get useful things done you need even less you need no more than about 30. And because each tool is designed to do one thing, each tool is not particularly complicated.\nLastly they may argue that command line tools don\u0026rsquo;t share a set of design guidelines for how commands should respond to arguments. This is obviously true, the oft maligned tar is one of the most egregious examples. However, are graphical interfaces any better? Especially graphical tools that are supposed to be designed so \u0026ldquo;Expect the output of every program to become the input on another, as yet unknown, program\u0026rdquo;. I would argue graphical tools are more complicated and just as inconsistent. Yes, many graphical programs implement \u0026lt;ctrl-s\u0026gt; to save the current state, but not all. And despite exceptions, the command line has similar conventions: -h or --help for help, -v or --verbose for increased verbosity, -q or --quiet for less, etc.\nWhat to learn #This is a minimal set of tools that I would recommend learning in the order that I would learn them:\nPurpose Tool Why Files and Directories ls List the working directory Files and Directories cd Change the working directory Files and Directories mv Move a file Files and Directories cp Copy a file Files and Directories rm Permanately Delete a file Files Path Manipulation dirname Get the name of the directory a file is in Files Path Manipulation basename Get the name of the file without its directory Files and Directories cat Concatonate one or more files to the terminal Pagination less Create a scrollable view on the output Documenation man View documenation Control Flow bash Most common shell on Linux platforms; the glue used to put things together Files and Directories find Identify files with a set of properties Archive management tar Work with many compressed archives; often used with downloads Text Processing head Print the first lines or bytes from a file Text Processing tail Print the last lines or bytes from a file Text Processing grep Search for lines in a set of files that match a pattern Text Processing sort Sort lines on some criterion Text Processing uniq Remove duplicate lines from sorted lines Text Processing wc count the number of words, lines or characters in a file Text Processing echo Convert arguements to a string and print it to the command line Privledge Escalation sudo Request administrative priledges Package Mangement apt/dnf/pacman/emerge/etc\u0026hellip; Install, remove, or update software sudo warrants a more detailed explanation. Linux and other Unix like operating systems protect key functions such as installing software system-wide by often restricting them to a user called by convention \u0026ldquo;root\u0026rdquo; which is nearly omnipotent on the system. Sudo stands for Superuser Do. sudo allows requests the root user (super user on the system) to take an action on your behalf. You generally should avoid using this power. However one place you will need to use it is with package management.\nPackage managers also warrant additional explanation. Many Linux distributions sadly seem to suffer from not-invented here syndrome and has reinvented their own package manager with its own esoteric options and usage. Don\u0026rsquo;t try to learn them all, just learn how to use the one for your distribution. Refer to the Pacman Rosetta to see how to accomplish common tasks for your distribution.\nIf you have some more time or a special need, I would additionally pick up these commands:\nPurpose Tool Why Advanced text processing awk Work with field delimited data; i.e. CSV, log files Advanced text processing sed Make sophsticated changes to lines of a file Scheduling and events systemd-run Schedule commands to run in the future or after an event Network transport curl Download files using a wide range of protocols Network transport ssh Run commands securely on a remote machine Format Converters convert Convert between image formats Format Converters pandoc Convert between text/document formats Format Converters jq Extract information from JSON formatted data Format Converters xpath Extract information from XML formatted data For small problems, you can often write everything you need in the command line prompt. However eventually, you will probably want to put combinations of commands into a file. When you do, you\u0026rsquo;ll need a text editor Text editors are a very personal tool and no one tool will satisfy all users. I\u0026rsquo;ll elaborate on the choice between these in a later post, but I encourage new users to try one of these:\nvi/vim/neovim \u0026ndash; the vi-family of text editors feature a focused feature set and a famously powerful keyboard shortcuts, and they are nearly ubiquitous on Linux systems. emacs \u0026ndash; is a powerful \u0026ldquo;batteries included\u0026rdquo; text editor and extensible text editor. vscode \u0026ndash; a more modern graphical text editor An Integrated Development Environment (IDE) \u0026ndash; generally focused on a specific programming language or programming language. Examples include Eclipse or the JetBrains IDEs It is also possible to use other lighter-weight tools that are often packaged by default with Linux distributions, but I don\u0026rsquo;t encourage this. Other tools don\u0026rsquo;t always offer the room to grow and require the user to learn a different tool when they no longer meets their needs. These editors scale to even large projects such as the Linux kernel with million lines of code.\nDon Knuth and Donald McIlroy both famously wrote a program to count the nth most frequently used words and print out the words and their frequencies. Knuth used a variant of Pascal taking over 10 pages. McIlroy used composition of shell scripts taking only 6 lines. When is shell scripting vs more traditional programming languages the better solution? Finding help and Debugging first steps #So, where do you find examples of how to use these command line commands and find help when I need it?\nIs the problem replicate-able? If the problem cannot be reproduced, it is much harder to fix so it is worth taking some time to figure out how to replicate the problem. -h or --help options on specific commands man pages. Linux has a set of built-in manuals. Most of them can be accessed by calling man followed by the name of the command. Online documentation Often each project will have a home page with documentation and examples of usage. Look here first if it exists. The Grymoire \u0026ndash; Extensive documentation on a number of older Linux/Unix commands. They have some of the best free awk and sed documentation. The Archwiki \u0026ndash; ArchLinux \u0026ndash; a Linux distribution for experienced power users who want to live at the bleeding left \u0026ndash; has very detailed documentation on a number of Linux commands. search engines \u0026ndash; you can often search \u0026ldquo;name of command\u0026rdquo; with the word \u0026ldquo;example\u0026rdquo; to get examples on sites like Stack Overflow or Geeks4Geeks. However, sometimes these resources can be inefficient uses of computer resources, wrong, or out of date. Prefer official documentation whenever possible. When looking at forums like Stack Overflow, consider the following: (1) how old is the post? (2) is the recommendation consistent with the documentation? From time to time, a post will live on way past the project being updated and will reccomend a deprecated or inefficent way of doing things. Checking the documentation can help avoid that. (3) Is it recommending a hack-ish work arround that will leave you insecure? For example, some posts on Stack Overflow will recommend disabling role based access control or the firewall which will greatly compromise the security of your system. It\u0026rsquo;s worth reading and understanding what the suggested fix does before using it. There are also a number of paid resources that I feel are worth considering reading for specific tools:\n\u0026ldquo;Unix Powertools\u0026rdquo; a pretty comprehensive book on command line utilities by Shelly Powers, Jerry Peek, Tim O\u0026rsquo;Reilly, and Mike Loukides \u0026ldquo;sed \u0026amp; awk\u0026rdquo; - by Dale Dougherty and Arnold Robbins ","date":"25 August 2019","permalink":"https://robertu94.github.io/learning/linux/","section":"","summary":"\u003cp\u003eSo you want to or have to try this thing called \u0026ldquo;Linux.\u0026rdquo;\nJust like curry powder isn\u0026rsquo;t just one thing, but a distinct mix of spices that come together into a tasty mixture, Linux is much the same.\nAlso like curry, Linux isn\u0026rsquo;t for everyone.\nIn this post I describe the process of choosing the \u0026ldquo;flavor\u0026rdquo; of Linux that will work best for you, introduce a powerful tool that will help you to make the most of Linux, and describe some first steps to take when things go wrong.\u003c/p\u003e","title":"Learning to Learn: Linux"},{"content":"Communication is essential to the enterprise of knowledge. Without communication, we would never be able to build upon the works of others or have them build upon our own. One of the most important \u0026ndash; and challenging \u0026ndash; arts within communication is writing. Writing is unique among commutative forms in its durability and portability. Once written, the words of the author can transcend even their death and travel places they never dreamed to tread. However writings portability and durability are the same feature that present its central challenge: With the author long gone, how can the he still communicate his or her intent? In this post, I ponder how I face these challenges and encourage others to do the same.\nIn short, the best papers: write with a purpose, recognize every technical document is persuasive, do not neglect the importance of structure, and are written for the public.\nWrite with a purpose #Before writing it is essential to consider your purpose. Your purpose will determine many aspects about how you write.\nFor example, the purpose of these blog posts is to save me time.\nHow can writing save time? From time to time, my classmates and others ask me questions. Often time the first and second time, I give a response, but by the time that I get the same question three times, I add the question to the list of topics for these posts. Then when the post is written, I can then give these posts to anyone else who inquires and use it as a starting place for discussion.\nGiven this purpose, it informs how I write. I need to be comprehensive if I want the post to actually save me time; however if the posts are too long, they never will be read. The way that I attempt to make this compromise is by providing extensive references rather than the inlining them into the text itself. That way the interested party can investigate only the topics of greatest interest, and the lackadaisical one can simply skim on.\nWhen writing I often consider, \u0026ldquo;how can I increase the signal to noise ratio for my audience?\u0026rdquo; The higher this ratio, the better investment reading it is for them. The better investment it is for them, the more likely they are to use it.\nIf they don\u0026rsquo;t care about something, it shouldn\u0026rsquo;t be included. This manifests itself in several ways: Some audiences don\u0026rsquo;t care about details and an endless slog of details makes a text unapproachable and a waste of their time and attention. Even engaged audiences don\u0026rsquo;t care about everything. Is what I have written going to help them do their job, help them teach this to others, or help them enjoy the text? Therefore, I write with a presumption of apathy and find something useful only when I can reasonably are interesting to the audience. \u0026ldquo;tl;dr\u0026rdquo; \u0026ndash; too long didn\u0026rsquo;t read \u0026ndash; is a reality and I fear a common one.\nRecognize every technical document is persuasive #A corollary to writing with a purpose is that every non-fiction document is persuasive. Informative writing attempts to persuade the reader of the truth of certain facts; proposals are attempt to persuade the reader of value of an idea. Something as simple as a receipt is attempting to persuade its reader that the holder has obtained a good or service. Even \u0026ldquo;objective\u0026rdquo; writing that desires to show various sides of an argument argues at a minimum that they are presenting each viewpoint accurately. The author\u0026rsquo;s purpose at some fundamental level is to persuade of some truth.\nSo what does this mean for writing? If every document is persuasive, then every document has an argument. Making this argument as clear as possible improves the writing in several ways:\nFirst, it makes it easier to spot flaws in the reasoning of the argument. These flaws, called fallacies, devastate the integrity of your arguments and perhaps your own as well. Identifying them in your writing and those of your sources preempts possible objections.\nWhat about the counter-claim that not every truth claim can be accessed by logic? While it is true that you may not be able to present a definitive logical argument for every situation, you can almost always make a case on the plurality of the evidence \u0026ndash; an inference to the best explanation.\nSecond, well reasoned papers are easier to follow. A well reasoned paper, has a inherent structure \u0026ndash; namely its argument \u0026ndash; that makes it easier for the reader to follow. Immanuel Kant called this structure a schema in his work \u0026ldquo;Critique of Pure Reason\u0026rdquo;. Since then others such as Jean Piaget in his work \u0026ldquo;Origins of Intelligence in Children\u0026rdquo; have argued that schema are critical to the formation of idea and cognitive development .\nFinally, it makes it easier to identify irrelevant sections of your writing. If a statement or section does not contribute to the overall argument of the piece, it should be discarded.\nHowever, even the clearest writing lacking support is unlikely to garner assent. This means that conclusions, motives, and methods are equally important. An unsupported argument is often as useful as an unmade one. Here are some common mistakes that I have seen:\nNot describing the methods in sufficient detail to replicate them. Describing methods is the \u0026ldquo;showing your work\u0026rdquo; from your high-school math class. Yes, eventually you can show progressively less and less work as you get a more sophisticated audience, but if the audience could and wanted to completely do these work of proving your point themselves, they wouldn\u0026rsquo;t be reading your paper. This also helps you identify methodological flaws in your work before they are written as criticism because it forces you to reckon with your methods and their methods. Not justifying the reasons for your methods. Nearly all experiments have some number of arbitrary choices. However, there can be great significance in which choices were arbitrary and which were informed. Explaining to your audience which is which is one of the few ways they know the difference. Conclusions must follow logically from the arguments. Often in my primary education, I heard it described that conclusions ought to summarize the essay followed. I submit this is insufficient. Conclusions need to explicitly and logically follow from that which came before. Otherwise, what is it concluding? What is the purpose of your document? What is the argument presented by your document? Does the argument support your purpose or is their missing or extraneous information? Writing Scientific Papers #So what is the argument of scientific papers? Scientific papers argue for their own acceptance by the community. Papers ought to be accepted if and only if they are interesting, novel, important, self-contained, and correct. Interesting speaks to the technical challenge/intellectual merit of the work and is most frequently addressed in the \u0026ldquo;problem formulation\u0026rdquo;, \u0026ldquo;design\u0026rdquo;, and \u0026ldquo;evaluation\u0026rdquo; sections of the paper. Novel speaks how the work innovates compared to its peers and is most frequently addressed in the related work section of the paper. Importance speaks to the meaningfulness of the work and to its broader impacts and is most frequently addressed in the \u0026ldquo;introduction\u0026rdquo;, \u0026ldquo;evaluation\u0026rdquo;, and \u0026ldquo;conclusion\u0026rdquo; sections of the paper. Self-contained speaks to the interpret-ability and reproduce-ability of the work and is most frequently addressed in the \u0026ldquo;background\u0026rdquo;, \u0026ldquo;methodology\u0026rdquo; and \u0026ldquo;evaluation\u0026rdquo; sections of the paper. Correct speaks to the methodology of the work and is most frequently addressed in a \u0026ldquo;methods\u0026rdquo; section where you include the boring but necessary details to reproduce your work (e.g. OS version, Compiler version etc\u0026hellip;) but also the \u0026ldquo;evaluation\u0026rdquo; section as you explain each experiment.\nIt is important to note that each of these topics is discussed at multiple points in the paper. The order that you make the case for your paper may not necessarily be a strict ordering of topics from interesting to novel, to important, etc\u0026hellip; The topics may be intermingled in order to optimize the ordering of the paper to reduce forward references or to combine treatments of similar topics. However, do not be afraid to repeat yourself some. Some repetition encourages that a topic is available where different readers may look for it.\nBeyond these criterion, the best papers are also enjoyable, concise, and skim-able. No one enjoys reading a overly long or dry paper Appropriate use of varying style and format, and in some cases a vibrant illustrative metaphor or playful title can help reduce the imposing nature of a wall of text'' to make the paper more enjoyable. Lastly, one of my first managers told me early on that your manager thinks in pictures\u0026rsquo;\u0026rsquo; which I also think describes many scientists and speaks to the skim-ablity of paper. The idea being that when we study how experts in their fields read documents we find that they seldom read top to bottom; they jump straight to the sections that will impact them. Therefore, skim-able papers can be read in two un-intuitive ways:\nBy reading only the figures and captions (or formula in math heavy venues) in the paper without missing the key ideas of the paper. By reading only the first sentences of every paragraph. Aside: Writing Journal Extensions #In computer science specifically, there is the notion of a journal extension. These are longer versions of papers that are published in journals after a conference paper is published. Generally these need to have at least 30% new content relative to the original.\nI have two pet-pieves about journal extensions:\nAlways provide a diff from the original conference paper to the journal extension \u0026ndash; this makes it far easier for the reviewer to judge if you have provided sufficient new material. Always re-write the sections of the paper where the arguments has changed on account of your new extensions. This should almost certanly happen in the introduction, conclusion, and evaluation, but other sections may need to change as well. Using Evidence #Arguments are evaluated by the evidences for them. As the writer, you choose what evidence you present and how you present it. While evidence fundamentally boils down to a series of a series of premises, how you present these premises can effect the credence that the reader assigns to them and their ability to quickly digest the information. When writing academic papers, there are several common ways that you can present evidence:\nmethod description benefit space citation include a reference to someone else who has demonstrated the point low-mid small inline evidence a brief statement of experimental findings low-mid low logical arugment structured formal logic to support your point mid mid cartoon illustrate a concept without real data mid-high large table structured numbers which represent your point mid-high large mathmatical model a detailed, creible model of the behavior mid-high large figure graphical representation of your point with real data high large hand wave do not say anything none free As the writer, you have a fixed (and unknown) budget of attention from your reader. Your goal is to provide the most benefit to the arguments that you believe the reader will want the most support for within that budget. Things like tables, figures, and mathematical models can go a long way to convince your reader that your argument is accurate, but take substantial space on the page and for the reader to consume them. Things like citations and inline evidence provide little benefit on their own \u0026ndash; most reviewers do not follow references in my experience and may not take evidence presented at face value, but can be expressed quickly. While these values of benefit and space are subjective and differ from discipline to discipline, they can be indicative of how much people may pay attention to the evidence they present.\nOne method though bears special attention, \u0026ldquo;hand waving\u0026rdquo;. Hand waving leaves an aspect of your argument for the reader to supply the argument on their own. This is free, you don\u0026rsquo;t have to write anything thus saving on your attention budget, but comes with substantial risk the reader will doubt it having no evidence. Another place where hand waving is used is when you want to ignore a orthogonal concern or something you expect the reader to know. How effective this is depends greatly on whether your audience agrees with you that the concern is truly orthogonal or is aware of it. For this reason, I have a love-hate relation with hand waving, you can\u0026rsquo;t restate everything on a topic every time, but also don\u0026rsquo;t want to inadequately support your argument either\nFor your argument, what pierces of evidence do you have? What format(s) are appropriate for each aspect of your argument? Do not neglect the importance of structure #Just as important as logical structure, physical structure is also key. The physical structure of writing should make easy for the reader to find what they care about. Making information easy to find takes many forms:\nIt means putting the information the reader cares about earlier in the document. Every additional sentence, reduces the chances your reader is paying careful attention when they get to the end. How is this different from identifying the logical structure and eliminating irrelevant details? Consider the classical argument:\nA. Socrates is a man B. All men are mortal C. Therefore, Socrates is mortal There are 6 possible arrangements of this argument that are all equally logical valid. Not all 6 of these arrangements are equally comprehensible. It generally makes sense that conclusions (part C) ought to follow after the premises (parts A and B). However, there are still two possible arrangements of parts A and B if C is to go last. Putting the argument that the reader cares about first can help them find the information the care about more quickly.\nIt can also mean the formatting and typesetting of a document. Headings, bold, and italics help reduce the homogeneity of your text giving the reader something to easily find their way back to if their eyes or mind drift. However, too much formatting can distract the reader. Remember the example of a book in which everything is highlighted: when everything is highlighted, nothing is effectively highlighted.\nStructure includes grammar. While perfect grammar is seldom necessary, careless grammar can be distracting \u0026ndash; or worse confusing. Everyone should have and use a grammatical reference; I recommend Strunk and White\u0026rsquo;s Elements of Style.\nFinally for longer texts, a good index, table of contents, and glossary can also make a difference. They allow a narrowly interested reader to jump directly to a topic of interest rather than having read everything.\nDo indexes still make sense in the age of electronic documents and search functionality? I think so, but the threshold for a good index has risen. No longer is it sufficient to identify the locations of key terms, indexes now should help the reader determine the passages where a topic is most directly addressed and where related topics are. For example:\ntasks 1 User 1 Kernel 2,3 OpenMP 27-30 MPI 31-35 Hadoop 70-72 This index shows that tasks exist for users, the kernel, OpenMP, MPI, and Hadoop. Yes, the user could search for each of these terms independently, words like \u0026ldquo;user\u0026rdquo; are likely often used all over the document. Providing an index can help the reader find the instances that are actually useful to them.\nWhat structuring of your argument is most consistent and logical? How does the ordering of points impact the strength of your argument? Consider Writing for the public #Far too often, I read a paper in my own field that has so much mathematical notation or technical jargon and find myself questioning what it even meant. You never know who will read your paper or why. You will never know what context they are coming from. For this reason, one should always try to make their paper more approachable to the general public. It opens your work to people who may be interested in your topic, but currently lack expertise. It also makes it easier for experienced readers to quickly understand and appreciate your work.\nThis manifests itself in several concrete suggestions:\nTry to appeal to universal examples. Taking all of your examples from a signal domain reduces the portability of your writing across domains. Don\u0026rsquo;t assume the reader knows the specific name for a concept or example you use. Describe each briefly just in case an someone outside your intended audience reads your paper. Avoid unnecessary notation and terminology. Notion and terminology are powerful in that they allow us to allow us to concisely represent a specific concept. However, the average human brain can only manage 7±2 things at a time. Each term that the author introduces increases the required mental bandwidth to understand the paper. If you use notation and terminology, use it consistently. Inconsistent notation not only can pollute the brains name space for notation wasting one of the 7 scarce places for temporary storage, but it also can lead to ambiguities in your argument which lead to misunderstanding. Consider the importance of abstractions. Remember the brain has limited short term storage. Abstractions allow the reader to fit multiple related items the same unit of working memory. Good abstractions are concise, coherent, and extensible. In my opinion, there is substantial overlap between creating good object-oriented designs and creating good abstractions. I would read the short article \u0026ldquo;Principles of OOD\u0026rdquo; by Robert Martin on object oriented design for some of the principles to consider. Make your work self-contained. A key tool to make a paper self contained is to include background and related works sections. A good background section allows the general public to have a sufficient understanding of your topic to understand your writing. In contrast, related work exists exists to demonstrate the distinctiveness of a work relative to its competitors. How can you adapt your writing to improve how you write for the public? Writing a Learning to Learn #Let\u0026rsquo;s look how I try to apply advice to write a \u0026ldquo;Learning to Learn\u0026rdquo; post. Here is the outline that I start with when writing a learning to learn post for a technology:\nHow do you get started? This is where I highlight the resources I recommend for true beginners to the topic, and perhaps the field in general. What are the important tools and utilities to know about? What are the most important Standard/Included features? Here I try to make recommendations about what parts of the technology are worth prioritizing learning What are the important 3rd party libraries? Most tools/technologies leave some aspects of their systems to 3rd parties to implement. Here I try to summarize which 3rd part libraries/plugins one should encourage someone to adopt this technology or accomplish basic tasks left out of the standard functionality. Tools that make using the technology better? Here is where I summarize things that operate on the language or tool rather than extend it. The classic example would be a debugger or a profiler for a programming language. You likely won\u0026rsquo;t use a debugger from the language itself, but it\u0026rsquo;s an invaluable tool to learn to use it. Tools made better by using this technology. Some tools can be extended via use of a given technology. A clear example of this would be python extensions for GDB. Here I try to highlight some of these kinds of synergies. What are the important concepts? Here I try to summarize the concepts that are so important, that I feel they are worth a paragraph or two of dedicated discussion. What topics should I read more about, but aren\u0026rsquo;t really an introductory topic? Here I try to list important topics that a beginner or intermediate user doesn\u0026rsquo;t need to know to get started, but should learn to become more advanced. Where can I go for more information? Lastly I try to highlight some resources that I use for more advanced or complete information on a topic. That doesn\u0026rsquo;t mean that I stick exclusively to this outline. For example, C++ doesn\u0026rsquo;t have section 2.4 because in my opinion writing dynamically loaded library extensions isn\u0026rsquo;t a feature most programmers are going to use, and is tricky to get write correctly. I also feel free to add additional sections as I feel they are important. For example the Linux post spends a lot of time discussing how to get software and choose a distribution rather than provide a general overview of how to use it. That\u0026rsquo;s because I\u0026rsquo;m not convinced the best way to get started learning Linux is going to remain stable long enough for this kind of post to remain useful. Instead I try to point users to more important meta questions about how to adapt to changes and find new work flows.\nChangelog # February 2023 - Added section on using evidence August 2020 - Added section on writing a learning to learn Hope this helps!\n","date":"1 July 2019","permalink":"https://robertu94.github.io/learning/writing/","section":"","summary":"\u003cp\u003eCommunication is essential to the enterprise of knowledge.\nWithout communication, we would never be able to build upon the works of others or have them build upon our own.\nOne of the most important \u0026ndash; and challenging \u0026ndash; arts within communication is writing.\nWriting is unique among commutative forms in its durability and portability.\nOnce written, the words of the author can transcend even their death and travel places they never dreamed to tread.\nHowever writings portability and durability are the same feature that present its central challenge: With the author long gone, how can the he still communicate his or her intent?\nIn this post, I ponder how I face these challenges and encourage others to do the same.\u003c/p\u003e","title":"Learning to Learn: Writing"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/writing/","section":"Tags","summary":"","title":"Writing"},{"content":"Efficient study of a topic is a powerful skill for a wide variety of domains. However \u0026ndash; despite having several classes on writing and research in High School and Undergraduate Studies \u0026ndash; it wasn\u0026rsquo;t something that I truly appreciated and learned how to do well until Graduate School. I think this really boils down to lacking the kind of personal application that makes these efforts meaningful. However now that I have come to appreciate it, I want to highlight how I think how I study new topics, and how this applies to many topics outside of academic research.\nI think its important to clarify that this post is not designed to present tactics but strategy about how to approach new topics. First, effective tactics vary widely between person to person. Some will prefer the use of electronic tools for certain tasks, while others will favor pen and paper. It is important to utilize the tools and techniques that work best for you. Second, effective tactics can vary widely by topic. Some topics have extensive free resources that cover a topic. Other require access to journals and conference papers to really approach a topic in depth.\nBroadly speaking, I think effectively studying a new topic involves several steps:\nBegin with a Research Question(s) \u0026ndash; What are you going to research? Build a Candidate Reading List \u0026ndash; What places could have the information? Read the Papers Intentionally \u0026ndash; What do those places say? Record, Organize, and Categorize \u0026ndash; How can I remember what I\u0026rsquo;ve learned? Focus and Refine Your Reading List, Update the Research Question(s) \u0026ndash; What should I look at next? If you are thinking this looks similar to the PQRST approach that I mentioned in the Learning to Learn Reading post, it does, but I think there are some different aspects to studying multiple works rather than just one that warrant studying this further. For example, in doing a comprehensive study, doing comparisons and contrasts are more important. Most works by a single author will present a single point of view on a topic \u0026ndash; or at the very least favor a particular perspective. However by reading works by multiple authors, you can expose subtleties that wouldn\u0026rsquo;t appear if you just reviewed one topic. Secondly, with a comprehensive study, archival and retrieval becomes more important. This is mostly because of the shear increase in the volume of information which can make it difficult to keep it all in memory. With comprehensive studies it becomes essential to be able to find the knowledge that you already had quickly which is less important with reading a single work.\nHow Do I Craft a Research Question? #Your initial research questions should be design to answer the question, \u0026ldquo;Why am I looking into the topic?\u0026rdquo; The key tool in doing efficient research is maximizing the signal to noise ratio. You don\u0026rsquo;t gain anything from reading irrelevant documents. Starting with question, \u0026ldquo;Why am I looking at this?\u0026rdquo;, gives you a clear guide as to what is or isn\u0026rsquo;t relevant allowing you to ignore irrelevant sources faster to increase the signal to noise ratio.\nIn that vein, it is helpful to be as specific as possible! This too will help cut down on reading irrelevant documents. For example rather than \u0026ldquo;performance\u0026rdquo;, search instead for \u0026ldquo;network bandwidth\u0026rdquo; if you are really focused on the bandwidth of computer networks. You can always increase the scope of your research question later if you want to look at performance more broadly.\nSo what if you still don\u0026rsquo;t know where to start? I often start with a variant of one of these questions:\nWhat are the views on a topic? Understating how others view a topic has many benefits: it improves understanding and empathy, it can provide an concept of the prominence of an idea, and it can tell you what others view as fruitful areas of research. How does this topic apply to some other topic? One problem with academia is the apparent lack of application. Focusing on an application from the beginning can improve this. Many aspects of research involve applying some underlying skill or knowledge to a new topic. How can I solve X problem? This the fundamental engineering question. It may look like subset of the previous question, but has an important distinction: the previous question begins two or more candidate topics and looks for a candidate problem. The later begins with a problem and examines possible solutions. Both approaches can yield valuable results. What are the trade-offs for X? This is really the key scientific question. Understanding trade-offs can allow engineers and others to build upon your work to consider new topics and expose new tools. Try asking the questions above of the topic that you want to research. What aspects of your topic did you discover when thinking about it systematically? How Do I Build a Reading List? #The next important question to ask is what to read and what to read first.\nIts key to remember while building your reading list, you don\u0026rsquo;t have to read the entire document. I generally do a first pass on titles, then I filter more aggressively on abstracts, and finally I filter even more aggressively on skimming the relevant sections of an document. This is where tools like electronic search can be very helpful to look for key words and phrases and topics. I would suggest taking the time to read only the most information dense resources in full. When you go to cite or use a resource, then you should go back and get a more full context.\nIf you are completely new to a topic, it can be helpful to begin with some preliminary reading. This can take several forms. Here are some of the ones that I use most often:\nBegin with a survey or a book on the topic \u0026ndash; I really don\u0026rsquo;t know anything about a topic the first thing that I try to find is a book on the subject. While most people have heard of books, some many not know about survey articles. Survey articles are documents that provide a comprehensive overview of a topic and tell you a wide variety of research going on or views on an area. So then why a book or survey article in the age of the Internet? First, Books and surveys can offer a holistic view of a topic and have fewer constrains to length or subject that a conference, journal, or article can have which allows them to be more introductory and provide more context. Second, books often are edited and reviewed which means that the document can be easier to read than that of some journals/articles. Search for components of the research question \u0026ndash; Once you have a fairly precise research question, you can begin to search the Internet for resources on a topic. In additional to general search engines like Google or Google\u0026rsquo;s Scholar search engine, sometimes specialist search engines such as the ACM digital library search or the IEEE Explore in the field of computer science can provide more specific results for their topic area. A use of both can vastly improve the variety of results that you may find. Look for authors, keywords, and further/active questions. As you are reading, make note of authors, key words/phrases and active research questions that these papers identify. This can aid in the development of further and more precise research questions to review later. Once you have an initial corpus of documents, you can expand your reading list to contain other possibly relevant materials. There are three key ways that I try to do this:\nRead books that cite/are cited by other works that were relevant \u0026ndash; Academic work seldom occurs in a vacuum. In additional to understanding the topical context, it can be valuable to understand the temporal context as well. Why are the authors writing about this topic now? How did there work change over time and why? Understanding the answers to these questions can give you an idea of the barrier of entry to a useful contribution. Read more than just websites. As I said the subsection on preliminary reading, not all writings are written in the same style. Even within a given genre of non-fiction, there are substantial differences between journals, conferences, books, debates, tech reports, and websites. Understanding how each of these are used in your specific discipline can improve your ability find the kinds of information that you are interested in. Consider alternative perspectives. This is especially important in topics that are contentious or unclear. For these topics it can be enlightening to get past the caricatures often used by opposite sides of a contentious issues to understand their fundamental concerns. While you don\u0026rsquo;t have to agree with them, understanding their perspective is helpful. If you\u0026rsquo;ve followed all this advice in generating a reading list, you likely have more reading material than you have time to read it. Thus it becomes important to cut the list of things you have to read. Remember, the goal is to maximize the signal to noise ratio. I often will mentally sort the resources by expected value versus the cost of time to read. Then I mentally budget how much time that I have to read and will cut off the list when I run out of budget. As I suggested earlier in this section, you can do this in multiple progressively time intensive phases.\nHow Can I Record, Organize and Categorize My Reading #I think the biggest distinction between reading for pleasure, reading a single document, or doing a systematic study is the importance of recording, organizing, and summarizing information. Again remember the goal is improve the ability to find and retain information so that you don\u0026rsquo;t have to read the entire document again in full. Every time that you need to review an entire document in full for the same information or have to spend a long time searching for a document that you already found, you often are not using time as efficiently as possible. This is not to say that you shouldn\u0026rsquo;t read documents twice, but you should strive to find new information each time you do.\nThis takes two key forms forms: minimizing time to find a document and minimizing time to find information within a document. These are in some senses the same problem: finding the source (or part of a source) that has the information you are looking for. Finding information within a document requires or uses similar kinds of techniques but done at a more granular level. I would describe these techniques as indexing, summarizing, and archiving.\nIndexing is the problem of making it easy to find a large number of documents often times this looks like organizing information by topic or keywords, tagging information with keywords, or putting it into a system/database that records this informant. Picking the indexes to use can be challenging. You often what a combination of specific and general tags to help you find information for when you want to find a specific thing as well as a variety of things about a topic.\nSummarizing is a similar problem, but slightly more verbose, it provides a good way to zero in on a specific resource on a topic of interest, but provides greater context. Doing this well is challenging, if a summary is written two distinctly, it can be hard to find similar documents, but a summary that is too generic can not serve as a meaningful index. Again, the goal is to minimize the time to find information you\u0026rsquo;ve previously ingested. Often times it is important to record not only what you found, but how and when you found it. The more tags that you can associate with a piece of informant, the easier it will be to find later. Additionally, this will often can enable you to find new information because often times information is found in clusters.\nLastly is archiving. Websites go down all the time, books go out of print or are damaged, and sometimes your ability to access certain resources changes over time. Having a good record of information is useless if you cannot retrieve the information later. It is often important to make a copy of the information that you found so that you can retrieve it later even if the source is hard to find. However there can be costs to this, everything you store requires time to search through when you want to approach a topic. It can often be important to have a tiered system of archival for documents that are more recently reverent versus those which are more less currently relevant.\nThink about how you index and achieve your findings right now. Can you quickly find resources that you identified before? Why or why not? Closing Suggestions #In closing, I would encourage the reader to apply these techniques to a wide variety of research problems. I use this both for work, but also when I attempt to answer questions posed to me by friends, or for personal study of topics that are important to me such as my faith.\nHappy learning!\n","date":"12 April 2019","permalink":"https://robertu94.github.io/learning/litrature-review/","section":"","summary":"\u003cp\u003eEfficient study of a topic is a powerful skill for a wide variety of domains.\nHowever \u0026ndash; despite having several classes on writing and research in High School and Undergraduate Studies \u0026ndash; it wasn\u0026rsquo;t something that I truly appreciated and learned how to do well until Graduate School.\nI think this really boils down to lacking the kind of personal application that makes these efforts meaningful.\nHowever now that I have come to appreciate it, I want to highlight how I think how I study new topics, and how this applies to many topics outside of academic research.\u003c/p\u003e","title":"Learning to Learn: Studying New Topics"},{"content":"There is a constant problem with programming language design: fast, generic, easy to write; pick two. The principle is that programming languages cannot be all three at once. Code that is Fast and Generic like C++ isn\u0026rsquo;t exactly easy to write. Code that is Generic and easy to write like Python isn\u0026rsquo;t always fast in the sense that C/C++ programmers mean it. Code that is Fast and Easy to Write isn\u0026rsquo;t always Generic in the sense that Python is. There is a new language on the block \u0026ndash; Julia \u0026ndash; which strives to challenge these assumptions. In the remainder of this post, I highlight what I like about it and describe my experience using it over the last semester.\nThe Good #Complexity when you want it, but fast by default #A friend of mine recently wanted to write this python, but was surprised when it was slow:\nfrom collections import defaultdict import numpy as np import timeit data = (np.random.rand(1000000, 40) * 100).astype(np.int) rows,cols = data.shape def locs(data): locs = defaultdict(list) for row in range(rows): for col in range(cols): locs[data[row,col]].append((row,col)) return locs print(timeit.timeit(\u0026#39;locs(data)\u0026#39;, globals=globals(), number=1), \u0026#34;seconds\u0026#34;) I thought to myself, I could make that run faster in c++:\n#include \u0026lt;random\u0026gt; #include \u0026lt;unordered_map\u0026gt; #include \u0026lt;chrono\u0026gt; #include \u0026lt;vector\u0026gt; #include \u0026lt;iostream\u0026gt; class Matrix{ public: Matrix(int x, int y): values(x*y), shape(std::make_pair(x,y)) {} int get(int x, int y) const {return values.at(x + shape.first*y);} void set(int x, int y, int value) { values.at(x + shape.first*y) = value;} std::pair\u0026lt;int,int\u0026gt; size() const {return shape;} private: std::vector\u0026lt;int\u0026gt; values; const std::pair\u0026lt;int,int\u0026gt; shape; }; std::unordered_map\u0026lt;int, std::vector\u0026lt;std::pair\u0026lt;int,int\u0026gt;\u0026gt;\u0026gt; locs(Matrix const\u0026amp; data) { std::unordered_map\u0026lt;int, std::vector\u0026lt;std::pair\u0026lt;int,int\u0026gt;\u0026gt;\u0026gt; locs; auto shape = data.size(); for (int i = 0; i \u0026lt; shape.first; ++i) { for (int j = 0; j \u0026lt; shape.second; ++j) { auto value = data.get(i,j); locs[value].emplace_back(i,j); } } return locs; } int main(int argc, char *argv[]) { Matrix data(1000000, 40); std::default_random_engine eng; std::uniform_int_distribution\u0026lt;int\u0026gt; dist{1,100}; auto rand = [\u0026amp;]{ return dist(eng); }; auto shape = data.size(); for (int i = 0; i \u0026lt; shape.first; ++i) { for (int j = 0; j \u0026lt; shape.second; ++j) { data.set(i,j, rand()); } } auto begin = std::chrono::high_resolution_clock::now(); locs(data); auto end = std::chrono::high_resolution_clock::now(); std::cout \u0026lt;\u0026lt; std::chrono::duration_cast\u0026lt;std::chrono::milliseconds\u0026gt;(end-begin).count()/1000.0 \u0026lt;\u0026lt; std::endl; return 0; } But as you can see this isn\u0026rsquo;t the most readable code, and I\u0026rsquo;ve had to re-implement a 2D Matrix class to make the code as readable as this is. My friend who knows python probably wouldn\u0026rsquo;t have been able to write this.\nSo I decided to explore Julia as an option:\nmodule Locs import DataStructures #slowest function find_locs_generic(A) locs = DataStructures.DefaultDict(() -\u0026gt; Vector()) rows, cols = size(A) for i = 1:rows for j = 1:cols push!(locs[A[i,j]], (i,j)) end end locs end #faster function find_locs_partial(A::AbstractArray{T,2}) where {T} locs = DataStructures.DefaultDict(() -\u0026gt; Vector{Tuple{Int,Int}}()) rows, cols = size(A) for i = 1:rows for j = 1:cols push!(locs[A[i,j]], (i,j)) end end locs end #fastest function find_locs_full(A::AbstractArray{T,2}) where {T} locs = DataStructures.DefaultDict{T, Vector{Tuple{Int, Int}}}( () -\u0026gt; Vector{Tuple{Int,Int}}() ) rows, cols = size(A) for i = 1:rows for j = 1:cols push!(locs[A[i,j]], (i,j)) end end locs end data = rand(1:100, 1000000, 40) macro benchmark(exp) quote println($exp) local val = @time $(esc(exp))(data) val end end @benchmark find_locs_generic @benchmark find_locs_partial @benchmark find_locs_full end So how does Julia stack up? I\u0026rsquo;m pretty sure my friend could have written the slowest version fairly quickly. I could have helped them add the type annotations in a few minutes to get it to the last version.\nI ran some quick benchmark numbers on my machine:\n#python (9 lines of code) 14.683772362000127 seconds #julia (10 lines of code) Main.Locs.find_locs_generic 6.856236 seconds (80.11 M allocations: 2.282 GiB, 48.27% gc time) Main.Locs.find_locs_partial 2.661613 seconds (80.10 M allocations: 1.989 GiB, 12.73% gc time) Main.Locs.find_locs_full 1.364066 seconds (40.26 M allocations: 823.874 MiB, 11.68% gc time) #C++ (26 lines of code, algorithm 13) 0.548 seconds As you can tell, even the slowest version of julia is twice as fast as Python. Yes, we probably could have gotten the python faster using something like numbra, but that\u0026rsquo;s not what they are familiar with and wasn\u0026rsquo;t packaged for their distribution. There looks like there is still some room for C++ for the absolutely performance critical applications, but Julia strikes a remarkable balance between being fast and easy to write.\nPowerful features #Julia has a few very powerful features that make it a true developer\u0026rsquo;s language.\nThe first is static multiple dispatch. Multiple dispatch allows a function call to dispatch based on all types in the function signature. This might seem like a small feature, but it allows for a incredibly robust programming model. It completely obviates the need for object oriented programming while also eliminating the need for some of the more complicated object oriented patterns such as visitor pattern.\nadd(x::Integer, y::Integer) = println(\u0026#34;int int \u0026#34;, x+y) add(x::Integer, y::AbstractFloat) = println(\u0026#34;int float \u0026#34;, x+y) add(x::AbstractFloat, y::AbstractFloat) = println(\u0026#34;float float \u0026#34;, x+y) add(x::AbstractFloat, y::Integer) = println(\u0026#34;float int \u0026#34;, x+y) The second is functions as a first class citizens in the type system. This allows a robust functional style programing style when it makes sense such as for data oriented processing code. Combined with multiple dispatch and by-default generic typing, this almost completely eliminates the need for many of the traditional gang of four patterns making it a surprisingly productive language.\nfunction mymap(func, list) results = [] for i in list push!(results, func(i)) end results end The third is Julia\u0026rsquo;s sophisticated user-defined type promotion system. This allows the user to quickly make adaptations when a type does not conform to the interfaces provided by some library author. It also means that any type can utilize the rich promotion facilities allowed by the language and standard library so even a average developer can modify core classes if needed. Look at the promotion.jl file to see how this works.\nMacros are the final and perhaps most power feature that I\u0026rsquo;ll highlight. These are not textual macros in the style of C, but rather object oriented macros in the style of Lisp. This allows the average developer to make extensions to the compiler or add feature to the language. The developers often brag that these features allow them to implement most of Julia in Julia itself with a concise syntax. There is (admittedly simple) example of this above that does the benchmarking of the julia code printing out the method name of the method passed in.\nPackaging System #Julia\u0026rsquo;s packaging system takes the best from modern package managers like golang and rust. It accomplishes this with a few neat tricks:\nIt uses git as a package manager like golang. This makes distributing julia packages really easy. This particular features is a blessing and a curse; there is a lot of garbage out there. But it also means that its pretty simple to get started which encourages future developments as people scratch their own itches. It internally stores a commit hash for every library that is installed in the current environment so it is trivial to reproduce a package set installed on someone else\u0026rsquo;s computer making reproducing bugs easy. Unlike python and ruby it only stores exactly one copy of each version of each library that you have installed. This makes it easy to have different versions where it is needed, but save space where it is possible. Best in Class Packages #Julia has a number of excellent packages that I would say challenge the packages that I use on a regular basis. For example, Plots.jl is what I wanted Matplotlib to be. More builtin types just work with plot and the names of arguments are in my opinion easier to remember and use. Other great packages include OnlineStats and JuliaDB which allow for distributed massive statistics applications. It also has great packages for machine learning such as Flux and differential equations DifferentialEquations.\nReally Good Interoperability with Other Languages #Julia has more interop packages than any other language that I\u0026rsquo;ve ever used. Several of them are quite good including PyCall, RCall, JavaCall. This allows you to use packages that you know and are familiar with without having to learn something new right away or if it doesn\u0026rsquo;t exist.\nThe Bad #Matlabisms #Julia does have a number of matlabisms the most annoying of which is that container indexes begin at 1. This also shows itself in the name of several function names, the use of ^ for exponentiation rather than **, and the whitespace is used to separate array literals. None of these Matlabisms disqualify it in my opinion, but explain some of the more annoying syntactical quirks of the language.\nRelative Maturity of Packages #Julia is a new language, and that means that the libraries that support it are also immature. For example the Cxx package for c++ foreign function calls still doesn\u0026rsquo;t compile on the stable release of the language. Other examples include the Clustering library which uses rows for features rather than columns unlike almost every other machine learning library I\u0026rsquo;ve ever used. I\u0026rsquo;m confident that both of these will be fixed soon. There are GitHub issues open about both of these problems which are actively worked on.\nThis is largely ameliorated by the vast array of high quality interop packages that allow you to pull libraries from other languages while they are being ported to Julia natively.\nThe Ugly #The absolute worst aspect of Julia is the ability to discover packages.\nRight now \u0026ndash; just after the 1.0 release \u0026ndash; there are a number of packages that are broadly recommended (including at time of writing Cxx an amazing best in class C++ wrapper) but no longer compile on the latest release. Allegedly, this should get better now that Julia has made a stable release. However, only will time will tell on this issue.\nIts also not clear to determine what package to use at any given time. The Julia Observatory is a graveyard to packages that were highly used on previous releases, but have been abandoned since the 1.0 or even in some cases 0.6 release in favor of newer less discoverable packages. A clear example of that of this would be the Debuggger. Now there is a great package called Debugger that works super well, but if you search for debuggers you will find Gallium or ASTInterpreter2 which don\u0026rsquo;t really work for the 1.0 release.\nThere is a similar problem with determining which function to use. Let\u0026rsquo;s say you want to generate interpreter-like printing output of some datatype. So you search for print in the help menu and you find the builtin function, but it\u0026rsquo;s not clear which print you are looking for. You can find it if you follow the hint that print calls show. show has two overloads, one that takes a mime-type and one that doesn\u0026rsquo;t, but even it doesn\u0026rsquo;t print exactly like the repl. What does that is repr, but only if the MIME type passed in is text/plain which is documented only in the repr help page. This is further confused by the difference between show(stdout, \u0026quot;text/plain\u0026quot;, [1 2; 3 4]) and show(stdout, [1 2; 3 4]) which is supposed to use a default argument of text/plain according to the docs. These kinds of bumps appear in a surprising number of places in the language. Again, this is not to say that these bumps won\u0026rsquo;t be ironed out, but they do exist.\nConclusion #So could there only be one? Maybe. Julia is the first language that I\u0026rsquo;ve encountered in a while that I\u0026rsquo;ve decided to learn in my own time. With recent developments like a full featured Debugger, it has me interested enough to keep using it. Hope this helps!\nHappy Programming!\n","date":"4 April 2019","permalink":"https://robertu94.github.io/posts/2019-03-21-julia-could-there-be-one/","section":"Posts","summary":"\u003cp\u003eThere is a constant problem with programming language design: fast, generic, easy to write; pick two.\nThe principle is that programming languages cannot be all three at once.\nCode that is Fast and Generic like C++ isn\u0026rsquo;t exactly easy to write.\nCode that is Generic and easy to write like Python isn\u0026rsquo;t always fast in the sense that C/C++ programmers mean it.\nCode that is Fast and Easy to Write isn\u0026rsquo;t always Generic in the sense that Python is.\nThere is a new language on the block \u0026ndash; Julia \u0026ndash; which strives to challenge these assumptions.\nIn the remainder of this post, I highlight what I like about it and describe my experience using it over the last semester.\u003c/p\u003e","title":"Julia: Could There be One Language?"},{"content":"Python is a relatively simple language compared to others such as C++. Despite its simplicity, Python really shines because of its robust standard library, extensive 3rd party library ecosystem (especially for statistics and data analysis), and intuitiveness. This post tracks my process of learning how to program well in Python.\nOrder of Topics #The listing of topics here is arbitrary and represents the path that I took to learn Python roughly organized by topic. My advice is that with Python you choose topics that are useful to you and focus on them.\nBeginning Python #Everyone needs to start somewhere, but there are limited, current, comprehensive, and free documents that describe an overview for the language.\nI would recommend in starting in one of two ways:\nRead \u0026ldquo;Dive into Python 3\u0026rdquo; by Mark Pilgrim. It used to be that this book was all you needed to read to get started. However the author of this book has since retired and Python has grown since version 3.0. To supplement this reading, I would recommend reading up on the following newer topics from the Python Release Notes: Pathlib \u0026ndash; the newer and more user friendly file interface introduced in Python 3.4. f-Strings \u0026ndash; a newer way for writing formatted strings which cuts down on the verbosity. Async-IO \u0026ndash; as of Python 3.5, Python developed a robust set of faculties for asynchronous programming. While you probably won\u0026rsquo;t start out writing async programs, know what they are and how they work will make reading them less surprising. Type Hints \u0026ndash; as of Python 3.5 Python now supports the ability to provide type information for arguments. While the information is only a hint and is automatically ignored at runtime, it can be checked statically by tools. Read the \u0026ldquo;Python Tutorial\u0026rdquo; which is part of the standard documentation. This resource is a bit terse for non-programmers, but provides a well written and more modern introduction to the language. I also would read \u0026ldquo;The Zen of Python\u0026rdquo; by Tim Peters. You can find it by typing import this in a Python interpreter. It is poem that describes some of the design philosophy about Python. Keep these lines in mind as you watch and read the following items in this post.\nI would also watch these videos by Raymond Hettinger who is one of the Python Core Developers that overview the design philosophy around Python:\nTransforming Code into Beautiful Idiomatic Beyond PEP8 \u0026ndash; Best Practices for Beautiful Intelligible Code Tooling and Libraries #Python is famous for the set of libraries and tools that it supports and that are built using it. In this section I give a limited overview of the libraries and tools that I use most often.\nStandard Library #Python\u0026rsquo;s claim to fame is that it has one of the best \u0026ldquo;batteries included\u0026rdquo; standard libraries. Learning to use it well is essential to any budding Python programmer. For these tools, I reccomend the web documentation on docs.python.org. These docs are written in such a way so that they can be read as introduction rather than using the help() method which is intended to be a reference. Here are some of the parts of the standard library that I use most.\nName Use datetime Parsing, formatting, and calculating dates collections A set of useful data structures random A robust but easy to use random numbers pathlib Modern, easy to use filesystems library os OS generic facilitates sqlite3 Built-in SQLite support csv CSV parsing json JSON parsing logging A robust easy to use logging library argparse All but the most complex command line parsing threading Spawning and using threads multiprocessing Spawning and using multiple processes concurrent.futures Modern futures based multithreaded/process subprocess Run other programs and get their results asyncio Asynchronous IO library typing Type hint support itertools Tools for working with iterators functools Tools for higher level functions unittest Built-in unit testing framework. doctest Another unit testing framework that uses doc strings 3rd Party Libraries #In addition to the fantastic standard library, Python has a number of famous 3rd party libraries that are considered the best in class across all programming languages. While I list these specifically later, I would be remiss in failing to mention the NumFOCUS libraries which includes MatplotLib, NumPy, SciPy, Pandas, and others up front as some of the best open source (arguably overall) libraries for statistical analysis and numerical calculation that exist. Unfortunately the quality of the documentation for these tools varies highly. One alternative to strictly reading the documentation is to look at the unit-tests for the library. Because of the builtin unit testing frameworks being quite good, Python libraries often have extensive test suites.\nName Use Description pandas data anlaysis Data analysis framework for tabular data scipy data anlaysis Fast scientific functions numpy data anlaysis Fast numeric framework famous for arrays matplotlib plotting Defacto plotting library for Python seabourn plotting Ease of use layer for matplotlib bokeh plotting Web-first plotting library plotly plotting Interactive plotting library beautiful soup scraping Sane HTML parsing scrapy scraping Web Scraping Framework requests scraping/web requests Vastly superior HTTP library django web app opinionated easy to use CMS framework flask web app Flexible web app framework selenium testing Automated web browser controller psycog2 database PostgreSQL bindings simpy simulation Simulation library mpi4py HPC Distributed memory programming executor tensorflow machine learning Faster but more complex machine learning sklearn machine learning Easy to use but slower machine learning networkx graphs Easy to use but slow graph library networkit graphs Slightly harder to use but faster graphs Tools for developing Python #Due to the popularity of Python, there are a number of tools that exist to make it easier to work with. A number of these tools have great documentation. There is one key exception which is setuputils. For setuputils I would read this guide on packaging Python libraries.\nName Use pdb Python debugger pip Tool for installing packaged libraries setuputils Python library/tool for packaging libraries flake8 Linter for Python that respects PEP 8 jedi Code completion for Python mypy Python Static Type iPython Friendly interactive shell JuPyter Notebook software commonly used for science python-language-server Implementation of the language server protocol \u0026ndash; I pair this with vim for my development environment pycharm/spyder More fully featured alternative IDEs Other tools that work well with Python #Another reason to learn Python is be able to use it as part of other tools. Most of the time, the libraries are fairly straight forward to use and to learn to use with good project level documentation. Generally speaking these tools come in one of two flavors, applications that are written in Python and are easy to extend and those that are written in C/C++ and need a wrapper library.\nname purpose gdb well supported native executable debugger lldb easy to use native executable debugger ansible easy to use system orchestration tool saltstack more \u0026ldquo;Pythonic\u0026rdquo; system orchestration tool GDB and LLDB are C/C++ applications that use a wrapper library. There are two good tools for doing generating Python bindings for lower level C/C++ applications:\nswig \u0026ndash; easy to use, but can have significant overhead for some applications. Supports other languages. boost::Python \u0026ndash; harder to use, but lower overhead Both have extensive examples of how to get started with these tools online and pretty good project documentation. The biggest hiccup that people find when they go to use these tools is that they require the use of the development libraries for Python which are often installed separately from the Python interpreter.\nGenerally, the strategy that is recommended for using these tools is to create a class or module that exposes the functionality you would like to configure from Python and applying these tools to just that module. Then as an added benefit, you have a façade for your library to make it easier to use in the lower level library.\nWatch this video from CppCon that describes why you may want to do this and how to do it well.\nFunctional Programming in Python #One common misconception about Python is that it is primarily object oriented programming language. Rather, Python is primarily a functional language that supports object orientation where it makes sense. In functional programming, programmers avoid the use of state and have so called \u0026ldquo;higher level programming\u0026rdquo; faculties that accept functions and return functions. You should watch this video on why object oriented Python is overused.\nThe functional capabilities of Python are best seen in the following features of the language:\nFunctions as first class types. List, Generator, and Dictionary Comprehensions. Easy to write Iterators. Great library support for iterators, and higher-level programming. I would read the appropriate sections of the Python docs especially the introduction to functional programming in python as well as the itertools and functools docs.\nOne common question that comes up when considering generators versus list comprehensions is which to use. Generally, you should prefer generators because they use less memory and can be converted to lists later if multiple passes or random access is required. This video is on Python 2, but shows why you should prefer generators over lists\nObject Oriented Programming in Python #That said, Objects in Python are not considered second class citizens. If you have not read the key text on Object Oriented design [Design Patterns Elements of Reusable Software](Python\u0026rsquo;s Class Development Toolkit https://youtu.be/HTLu2DFOdTg) you should read that first. It explains why you would want to use object oriented design in the first place. However much of what is in this book is a language feature of Python. I would read this site which shows that many of the classical patterns of Object Oriented design are just features of Python. I would also watch/read the following resources to better understand object oriented Python:\nPython\u0026rsquo;s Class Development Toolkit which provides a high level overview of how to write good classes in Python. Super Considered Super! this video overviews Python\u0026rsquo;s multiple inheritance model and how to use it effectively. Other helpful topics #There are a number of other useful topics that I would commend to your consideration when you have finished reading about the topics above.\nCoroutines/Asynchronous Programming are a really easy way to write iterators and are an easy way to abstract asyncronous programming in Python in general. Read the standard library docs on AsyncIO to get an overview of why and how to use these. Decorators are a great way to do aspect oriented programming in Python. I would read this paper on decorators and I would read about how Flask uses them to get the high points. Metaclasses are a powerful way to change how classes are constructed. If you are in doubt if you need meta-classes you probably don\u0026rsquo;t need them. However they are how Django built their Model class and how much of the standard library is built. I would read this Stack Overflow answer about how they work if you need to learn how to use them. Performance Python generally is slower than compiled languages such as C/C++. You can look into tools like embedded C/C++ modules, cython, pypy, or numbra if things are too slow. Understanding profiling tools is also helpful consider watching this video as a starting place. Parallel and Distributed Programming Python has some of the best parallel and distributed programming features out there. I would read/watch the following resources: Read the docs for multiprocess and theading in the standard library. They describe the executor abstraction which is core to parallel and distributed programming in Python. Watch this video from Raymond Hettinger on parallel programming which describes some common pitfalls and tools for parallel programming. mpi4py documentation \u0026ndash; mpi4py is a Python binding for openmpi, but providing substantial ease of use benefits. If you want to run on a HPC cluster, mpi4py is the way to do it. What\u0026rsquo;s Next #Once you\u0026rsquo;ve done the above, I would consider the following resources for father reading:\nPyCon \u0026ndash; PyCon is a set of related conferences about Python programming. Some of the keynotes and presentations are quite well done. Many are posted online. Read the standard library code for some advanced examples of good Python code. As a reminder, Python code is almost always distributed as source code which means you can read it. Read the Python Enhancement Proposals (PEPs) to get an idea about where the language is going. Suggestions to Learn the Language #I have one key suggestion to learning Python: Refuse the temptation to write C in Python. Python has great functional and object oriented facilities use them.\nPick a project that is useful you: Python has libraries for almost everything. Some much better than others. I would suggest picking a project then do some quick research about what libraries people use to do that kind of work. After you have written the code, re-write it after watching videos like the ones from Raymond Hettinger or considering the Zen of Python. This will help you understand Pythonic style more and write more natural code. Hope this helps!\n","date":"23 January 2019","permalink":"https://robertu94.github.io/learning/python/","section":"","summary":"\u003cp\u003ePython is a relatively simple language compared to others such as C++.  Despite its simplicity, Python really shines because of its robust standard library, extensive 3rd party library ecosystem (especially for statistics and data analysis), and intuitiveness.  This post tracks my process of learning how to program well in Python.\u003c/p\u003e","title":"Learning to Learn: Python"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/python/","section":"Tags","summary":"","title":"Python"},{"content":" ","date":"1 November 2018","permalink":"https://robertu94.github.io/presentations/sc2018-predicting/","section":"","summary":"Techniques for predicting error bounded lossy compression ratios from data distribution and compressor settings.","title":"Predicting Optimal Error-Bounded-Lossy-Compression Configuration"},{"content":"From time to time, I get questions about how I read and retain as much information as I do. While it is never easy \u0026ndash; especially with dry technical documents \u0026ndash; there are a few strategies that I have learned along the what that I find helpful. In this post I provide some general suggestions along with some that are more useful in computer systems research.\nStrategies for Reading in General #My chemistry and physics teacher \u0026ndash; Dr. Brown \u0026ndash; in high school always began his classes by teaching the class a technique called P.Q.R.S.T. which I have found very helpful regardless of what genre I find myself reading. P.Q.R.S.T. stands for Pre-Read, Question, Read, Summarize, Test.\nPre-Read \u0026ndash; Scan the document quickly. Look for key aspects such as abstracts, characters (for historical or fictional documents), topics, concepts, diagrams, formulas, figures, headings, comparisons, key sources and citations, quotations, and dates. These often give you a mental schema on which you will associate the rest of the knowledge you are about to assimilate. For me, I try to take about 15 minutes for every 10 printed pages in a technical document. Question \u0026ndash; Develop a list of questions that you would like answers to as you read the document. The list of questions should be affected by the complexity of the document (more questions for denser documents), the key aspects you identified during Pre-Reading, and an appropriate set of questions the based on genre. You should prioritize this list of questions based on what you hope to retain from the paper. Read \u0026ndash; Read the document slowly. Take time as you read to look for and answer the questions you developed during the last stage. The purpose of this reading is to be able answers the questions throughly and prepare your self to summarize the material. Summarize \u0026ndash; Summarize the document. I find about 1 paragraph is for every 10 pages of a standard technical document to present a high level overview. The summary should answer the most important questions you developed. If you can\u0026rsquo;t get 1 paragraph, there are three possible problems 1) the document is simply too dense and you need more text, or 2) you are not actually summarizing but paraphrasing, or 3) which is most likely you don\u0026rsquo;t understand the paper well enough to summarize it, and you should read it again with some more questions based on what you learned last time. I\u0026rsquo;ll reiterate the key piece of advise I learned from his class: \u0026ldquo;be concise\u0026rdquo;. Test \u0026ndash; Test your knowledge of the document using the questions you developed. If you can\u0026rsquo;t retain the information without looking at your notes you should try again by starting at the question phase using the questions you did not retain well. It will probably help you to go through the summarize and reading phases again so as to rehearse the information in your mind. try applying PQRST to a paper or book that you need to read. How did the questions you asked help in your understanding? Strategies for Reading Source Code #Not all systems are well documented. For example the Linux kernel has over 400,000 lines of Documentation which seems like a lot until you consider that it has 22,802,098 of C source files and headers as of 4.18-rc6. This means that if you want to understand them you will need to do some digging. Reading the source code can also give you insight at a level of detail that the documentation does not. But how do you get started with a project that large? Here are a few suggestions:\nStart with main() \u0026ndash; most user level applications have some concept of an entry point which start execution. This is a bit more tricky in the kernel (which doesn\u0026rsquo;t have a main()) or systems like systemd which have a lot of them. In the case of programs without a main, you can look for methods that are called some variant of \u0026ldquo;start\u0026rdquo;, \u0026ldquo;init\u0026rdquo;, or are decorated/marked as a constructor. Additionally for kernels, there is often a custom linker script which specifies the entry section. You can simply look for the method that decorated with that attribute. In the case of systemd you can use documentation to help determine which ones are salient to your use case. Read the tests \u0026ndash; most large computer systems have automated tests. By design these are often easy to digest, currently working sections of code. Therefore you can use them to better understand the functional aspects of the code. Look for usages of a type or function in the code itself. Aspects of how to use a type or a family of functions can often be found in other places throughout the code base. You can use tools like cscope, ctags, or an IDE to provide some of this functionality. Look for each access to a given variable. This information should give you insight into the invariants that code expects. Build an interaction diagram. Look for which functions call other functions and in what contexts. Having a clear picture of what functions call which other functions can give insight into the layers in the architecture or suggest what other aspects of the system to study next if necessary. Use tools like ltrace, strace, xray, or perf to gather information about the runtime call usage. This can provide insight into what the system does under live conditions. Use a debugger. When you are curious about a specific use case of a system, you can use a debugger to see what values are set or where they are changed through the use of breakpoints and watchpoints; However this comes with a performance cost, and is notoriously difficult to do on distributed systems. use the techniques in this section to summarize some source code for a project that you use, but are not familiar with. What did you learn? Strategies for Reading Journals and Conference Papers #Reading Conference or Journals papers isn\u0026rsquo;t much different than reading generally technical papers. You can and should apply P.Q.R.S.T. to help cut them down to digestible portions. However, there are a few things about the format of a well-written journals article or academic paper that makes them easier to process if you use them.\nAbstracts \u0026ndash; abstracts are designed to be short summaries of the key findings of the paper. Read these very carefully as you Pre-Read because they can often provide useful hints for what questions you should be thinking about in the context of the paper. Citations \u0026ndash; academic work seldom appears in a vacuum. A good paper will cite a mixture of old and new academic papers as well as other technical resources, tools, and datasets that compare, contrast, and support their research. Once you have found a related paper, you can follow the citations in the paper to find other papers that are relevant or look at other papers by the sames authors. This can help you better understand the context of the work before you. Also pay attention to what authors cite as novel versus call out as their own contributions or general knowledge. Related Work \u0026ndash; Often papers will have a related work section that provides compares and contrasts a paper to others in the same field. These comparisons and contrasts can help you better ascertain the exact contributions of the paper. Methodology \u0026ndash; Often papers will have a short section at the beginning of the methodology section that is much more down to earth than the often flowery and exaggerated language in the introduction and abstract. Look in this section for what the authors actually did versus what they claimed they did. High quality papers do exactly what they claim they did or more. Limitations \u0026ndash; a high-quality academic paper will engage seriously with the problems and limitations of their approach. Read these carefully to help contextualize the work relative to other work in the same field. Academic papers are often peer-reviewed. It is also often helpful to think like a reviewer as reading papers:\nHow interesting are these results to others in my field and why? How technically sound are these results? Are their logical structure and statistical evidence of their results valid? How well presented is this paper? Is it clear, concise, make effective use of figures, tables, sectioning, titles, abstracts, and keywords? Are the reference sufficient and appropriate? Is there some topic that they are missing? Do they have new and old papers? How many of the citations are peer reviewed? How well organized is the paper? Why did the authors organize the paper the way that they did, and does that reduce forward references and duplication while presenting the information in a logically ordered manner that flows well? How confident am I about my ratings? If you can\u0026rsquo;t answer these questions confidently, you should probably read some more of the cited work in the paper so that you have a better understanding of norms and conventions in the field they are studying. It may also be helpful to do a Google Scholar search for key concepts in the paper. Identify 3 key citations and read papers that cite or are cited by the paper you read. How does this work compare to similar papers? Hope this helps!\n","date":"8 August 2018","permalink":"https://robertu94.github.io/learning/reading/","section":"","summary":"\u003cp\u003eFrom time to time, I get questions about how I read and retain as much information as I do.  While it is never easy \u0026ndash; especially with dry technical documents \u0026ndash; there are a few strategies that I have learned along the what that I find helpful.  In this post I provide some general suggestions along with some that are more useful in computer systems research.\u003c/p\u003e","title":"Learning to Learn: Reading"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/reading/","section":"Tags","summary":"","title":"Reading"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/ansible/","section":"Tags","summary":"","title":"Ansible"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/chef/","section":"Tags","summary":"","title":"Chef"},{"content":"So you know you need a configuration management system and you have an idea of which one will work for you. So what should I think about about before deploying one of this systems? In this third and final post in this series, I present some suggestions about using these systems in a way that is flexible and scalable to larger numbers of systems.\nEven within an operating system like Linux, there is a lot of variation between Linux distributions. You would be surprised how often different Linux distributions will name software, which options they will build with, what paths they expect software to be installed at, which users are expected to run the software. So if you want to support different distributions here are a few things to keep in mind:\nTry installing and getting the provisioning working on one distribution rather than multiple at first. It will help you learn the Provisioner\u0026rsquo;s tools, and it introduces substantial complexity to get this done right. So don\u0026rsquo;t add it if you don\u0026rsquo;t need it. When installing a software package, prefer to install a list of software packages. It is not uncommon for distributions to split or combine packages relative to other distributions. For example clang-query and clang are both part of the sys-devel/clang package on Gentoo but on Fedora, they are split between clang and clang-tools-extra. By installing a list of software, the list can simply include 2 packages on Fedora while only one on Gentoo. When installing a new/uncommon software package consider creating a package and serve it from a local package repository rather than building from source using the provisioner. Some distributions (looking at you CentOS) have ancient upstream package repositories (even with epel) so some newer things like neovim aren\u0026rsquo;t in the upstream repositories. Doing this isn\u0026rsquo;t too hard, and it is way easier than using a provisioner as a package build system. When installing a configuration file, consider using a variable for the path. Almost every Distribution installs configuration files into different directories. Systemd has done wonderful things standardizing this, but consider networking configuration is installed. Is it /etc/network/interfaces, /etc/sysconfig/network-scripts/* or /etc/systemd/network/*.network? It depends which Linux distribution you are using. Different Linux distributions require different users as system user accounts. For example, some distributions use the user www to serve web files, some use apache, some use httpd. In short keep in mind that you may need to support different user identities. Always test from both existing and new systems. Configuration Management that rebuilds your entire infrastructure is of no use if it does not work. Additionally it does you no good if it doesn\u0026rsquo;t respond to the updates that you intend to deliver to your systems. I would recommend using an approach similar to Chef\u0026rsquo;s kitchen with a set of VMs to manage testing. An alternative is to use containers, just be aware that certain key features of your system that you might like to manage using configuration management may not be amenable to containers. Use group vars, hiera, pillars or attributes to separate production and testing machines. One of the worst feelings that exists is when you apply a half-baked configuration module to production machines that requires you to reinstall them from scratch because you\u0026rsquo;ve bricked your login. Beware of how variable merging occurs. Each system has some concept of merging information from multiple sources, and they are almost all different and may not work the way that you would hope they would. Pay very close attention to how multiple variables get applied to the same host if it is part of conflicting groups. Manage each resources in at most one module if at all possible. The more that you split out these tasks the harder it is to debug your modules and ensure that they operate correctly. Make use of dependency management systems in the tools to ensure dependent modules execute before the modules that depend on them. Most importantly, always make changes only via the configuration management tool. Often user set changes will get overwritten later by the tool, and debugging why a hot-fix disappeared is not a problem that you want to have. This takes discipline, but its worth it \u0026ndash; trust me. Well that wraps up my suggestions for configuration management tools. Good luck and happy computing!\n","date":"31 July 2018","permalink":"https://robertu94.github.io/posts/2018-07-31-configuration-management-pitfalls/","section":"Posts","summary":"\u003cp\u003eSo you know you need a configuration management system and you have an idea of which one will work for you.\nSo what should I think about about before deploying one of this systems?\nIn this third and final post in this series, I present some suggestions about using these systems in a way that is flexible and scalable to larger numbers of systems.\u003c/p\u003e","title":"Configuration Management: Common Pitfalls"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/puppet/","section":"Tags","summary":"","title":"Puppet"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/saltstack/","section":"Tags","summary":"","title":"SaltStack"},{"content":"So, you need a Configuration Management System, so which one do you choose? This post is the second in a three part series on configuration management. In this post, I\u0026rsquo;ll highlight the strengths of these systems and their respective weaknesses.\nEvery evaluation needs to have criteria to be useful. Here are some of the criteria that I have had when I thought about this question.\nEase of Use Time to Setup Difficulty of Adding More Machines Difficulty of Creating New Modules Difficulty of Supporting More Configurations Quality of Documentation Puppet #Puppet is the oldest open source configuration management system, and perhaps the mostly widely deployed in enterprise environments. In my opinion, it is the most different out of the four open source configuration management systems. It uses a completely declarative language that requires some getting used to. It managements ordering by the use of explicit ordering statements. Additionally each resource, may only be declared once. The problem this proposes is with inter-module dependency management. Puppet solves this problem with the use of virtual resources. Virtual resources then can be \u0026ldquo;realized\u0026rdquo; multiple times, but only one instance of the resource will be created. For those unfamiliar with this concept, it takes a while to get used to. Using the completely graph oriented execution policy allows Puppet to create highly parallel and fast execution graphs to apply the specified policy.\nPuppet uses a client-server architecture with multiple components. Each managed box runs a copy of the agent which is intended to run in concert with a master. The agent registers with the Puppet master which then applies the policy with according to a specified regularity. The other key components of Puppet are , facter, hiera, and puppetdb. facter is responsible for gather \u0026ldquo;facts\u0026rdquo; about the nodes such as the hostname, number of cpus, or the ip-address. hiera is responsible for providing hierarchical information to the nodes such as what group(s) a node belongs to and group based variables. puppetdb acts as a \u0026ldquo;fact cache\u0026rdquo; and repository of events that happen on the nodes.\nAdding a new box generally requires the agent requesting a master to sign a certificate request. Without the signing their request, the agents will not be delivered a configuration to deploy. This can provides a high level of security, but is admit auntly harder to configure than some of the other solutions.\nSupporting more configurations is somewhat easy to do using Puppet. There are two common ways this is accomplished. The first is using so-called \u0026ldquo;parameter modules\u0026rdquo; which are small Puppet files that use conditional expressions to specify an appropriate values based on the facts on a system. The other method is using hiera to specify different values. The hiera should be thought of providing different values based on role, where as \u0026ldquo;parameter modules\u0026rdquo; should be thought of providing for variations on the same role. However, the exact point to draw the line between the two is not always clear which in my opinion cases some confusion.\nCreating new resources types can be done with ruby. Puppet types are simply ruby classes that provide certain members and methods. For those who know ruby, extending Puppet is a breeze.\nPuppets documentation is perhaps its worst facet. It is at times impenetrable at others hopeless vague and unhelpful. Its gotten better since when I first used it over four years ago, but it is still the worst of the four by far.\nAnsible #Ansible is another early open source configuration management system. Unlike Puppet, Ansible uses \u0026ldquo;playbooks\u0026rdquo; which are linear lists of instructions that generally execute from top to bottom. While this makes it easier to use and reason about, it can make it slower to execute. There is additionally no restraint on having resources managed in only one location. This creates a greater possibility of error if there are conflicting requirements, but makes it easier to work with common resources. Additionally recently, it was acquired by Red Hat making it likely to stick around and be well supported for years to come.\nInstead of requiring its own agent infrastructure, Ansible reuses ssh and python making it fairly easy to adopt since most systems already have these installed. The ansible agent is temporarily loaded on to the system when the master connects. This obviates the need to maintain or update the agent since it is loaded fresh each time. This temporary agent does all required fact finding and receives the appropriate set of host and group facts from the master.\nAdding a new box is a breeze with Ansible. Simply use ssh-copy-id to copy the master\u0026rsquo;s ssh key on the node, and ensure that a somewhat recent venison of python is installed. It may be one of the easiest system to get started.\nSupporting new configurations with Ansible is easy. Ansible has a clearly defined variable precedence hierarchy. Simply define variables with the appropriate level of precedence. There is one exception to this: Windows. Windows systems often require different modules that make it more difficulty to integrate them using the same role files.\nCreating new ansible modules (what Puppet calls a resource) can be done with python. They are simply scripts that accept from standard input and return JSON to standard output as text. Additionally, Ansible provides a large number of helper methods in python to make it easy to develop new modules. For windows systems, they provide a set of power shell modules that provide similar features.\nAnsible\u0026rsquo;s documentation is the best of the four by far. It has both extensive high level documentation making it easy to get started, but also extensive and up-to-date low level documentation to make it easy to reference for experienced users. The documentation has ample examples of usage that demonstrate a variety of use cases.\nSaltStack #SaltStack is one of the slightly newer configuration management systems. SaltStack brings refinement to the modules design presented by ansible. Many of the cross distribution abstractions in SaltStack are more refined than their Ansible counter parts. One annoying weaknesses of Salt is there is no concept of a comment in the files. This makes it harder to documentation more obscure configuration choices.\nLike puppet, it recommends a agent to be running on the client machines. However like Ansible it does not strictly require one via the SaltSSH feature; technically puppet has a currently experimental feature called \u0026ldquo;bolt\u0026rdquo; that provides this functionality, but it very much is experimental. The agent still must register with the master for the agent based mode, but this is relatively easy.\nSupporting new configurations in SaltStack is about as easy as Puppet. Simply define a pillar (SaltStack\u0026rsquo;s admittedly simpler, but less flexible version of hiera) with the appropriate variables defined or create a new grain (SaltStack\u0026rsquo;s version of a fact) that determines the appropriate information.\nCreating new SaltStack modules is easily done with python. Modules are simply functions in a python file that accept and return a dictionary. This is substantially more polished than the clunky text oriented approach used by Ansible. This makes them fairly easy to implement, but perhaps not as refined as Puppet\u0026rsquo;s object oriented approach.\nSaltStack\u0026rsquo;s documentation is not as well refined as Ansible\u0026rsquo;s. While it still has excellent low level documentation, its high level documentation is not quite as refined. I believe this combined with the less support for various modules than Ansible has hindered SaltStack\u0026rsquo;s adoption.\nChef #Chef is the other prominently used configuration management tool. Unlike SaltStack, Puppet, or Ansible, Chef uses nearly pure ruby to describe its configuration which it calls cookbooks. This has the advantage of being flexible, but the disadvantage of being harder to learn to use well than the descriptive domain specific languages used by the other three. Unfortunately of the four commonly used configuration management tools it is the one that I have the least experience with. I have used each of other systems to manage several systems that I consider production, and I have only managed test systems with Chef.\nLike Puppet and SaltStack, Chef uses a client-server architecture. However, unlike Puppet and Chef getting started on new nodes is very easy because of the chef knife feature. The knife tool allows you to easily bootstrap the chef agent on other machines making it almost as easy to get started as Ansible. Additionally the kitchen tool allows you to create and manage virtual machines for the purposes of testing applied configurations. This gives it a very compelling story form a getting started perspective.\nSince chef supports nearly arbitrary ruby code in \u0026ldquo;cookbooks\u0026rdquo;, making supporting new configurations is easy, but perhaps not quite as clean as some of the other systems such as Ansible. Chef like SaltStack has respectable and reasonably mature abstractions to make cross distribution development reasonable.\nCreating new chef types is as simple as writing new ruby classes. Since it uses nearly pure ruby, you can reach into the vast array of existing ruby libraries to build your modules.\nChef has ok documentation, but I find that it misses some of the high level documentation that really makes Ansible shine, however I would rate it above SaltStack.\nWhich is Better? #Before deciding on which system you should use, you should think about what requirements you have and why you have them. I\u0026rsquo;d also consider writing a small project in each when evaluating if it meets your needs. Some things like ease of use are highly subjective, so don\u0026rsquo;t take my word for them try them yourself. I would recommend something simple like installing your dotfiles.\nThat brings an end to part 2 of this series. In the final part of this series, I\u0026rsquo;ll offer some advise built on years of experience using configuration management systems.\n","date":"31 July 2018","permalink":"https://robertu94.github.io/posts/2018-07-31-configuration-management-comparison/","section":"Posts","summary":"\u003cp\u003eSo, you need a Configuration Management System, so which one do you choose?\nThis post is the second in a three part series on configuration management.\nIn this post, I\u0026rsquo;ll highlight the strengths of these systems and their respective weaknesses.\u003c/p\u003e","title":"Configuration Management: the Battle Royal"},{"content":"Configuration Management Systems like Ansible, Chef, Puppet, and SaltStack are in my opinion are nearly essential if you find yourself managing more than 5 machines. But what exactly are they, which is better for my circumstances, do I still need them if I use a container based infrastructure, how do I get started? This post is the first in a series of posts that will attempt to answer these questions.\nConfigurations Management systems enforce a software-defined policy against the running state of computer systems. I think this is best thought of in terms of some other important but different systems.\nShell Scripts #Like shell scripts, Configuration management systems allow you to write a series of configuration steps into code that are executed on machines. However, unlike shell scripts, these systems strive for idempotency by default \u0026ndash; actions are run if and only if they are needed. Additionally configuration management systems often manage the parallelism of running the configuration on a variety of different machines. Together idempotency potency and built-in parallelism make implementing scalable, repeatable systems substantially more pleasant.\nPackage Management Systems (dnf, apt, pacman, portage, etc\u0026hellip;) #Package managers can be used to install software files, configure users, network configurations, and more through the use of specially designed custom packages. However, using systems to manage things like users or network configurations can be tricky, because it is required to track all potential starting points of the systems and code to get them to the desired current state. Some package managers explicitly prohibit any changes to files owned by other packages. While this is good for ensuing a consistent file systems, it makes it hard if not impossible to manage files like /etc/hostname that are often owned by some base file system layout package using these mechanisms.\nFirstboot Management Systems (packer, cloudinit, kickstart, sytstemd, etc\u0026hellip;) #Packer is a software package that allows you to construct a distributable image your base system that can define things like your preferred dhcp configuration or ensure that your configuration management system agent is installed if required to bootstrap later management. Likewise the couldinit, kickstart, or systemd\u0026rsquo;s ConditionFirstBoot= option allow you to specify many of these initial configuration requirements required to bootstrap everything that comes after. However, since these operate on first boot only, they can\u0026rsquo;t effectively be used to manage on-going configuration management without making the base OS immutable which isn\u0026rsquo;t done that often outside of container environments that are designed to work in this manner.\nContainer Management Systems (docker, kubernetes) #Container Management systems such as Docker and Kubernetes in many ways can stand in the place of Configuration Management systems. Often in container-oriented architectures, you have some resilient, distributed key store such as etcd that stores configuration for services running in the containers. Additionally, for state-ful services you have something like a PersistantStorageClaim that reserves some amount of storage on a possibly remote storage device or service (iSCSI, RDB, FiberChannel, etc\u0026hellip;) or a separate database service. Then the container manager ensures that you have a sufficient number of replicas running at any given time. Some approaches like the one taken by Fedora-CoreOS (formally Fedora Project Atomic and Container Linux) operate by distributing an immutable image that is then first-booted every time and managed using the tools described above. In many cases, this can obviate the need for a configuration management system.\nHowever, there are some cases where configuration management systems are still useful. Managing the underlying services that support the container environment is often difficult because fundamentally these services are hard to contain: they often need direct, physical access to hardware; need to manage the container engine itself; or need privileges that you don\u0026rsquo;t want to provide to a container. A configuration management system can help fill this gap in managing the base infrastructure required: for example allocating new VMs or machines to join your kubernetes cluster, configuring the routers or switches so that they can be later managed by Kubernetes or Docker.\nThat concludes the first post in this series. Next time, I\u0026rsquo;ll cover some of the major systems that operate in this space, and the means by which I compare them.\n","date":"31 July 2018","permalink":"https://robertu94.github.io/posts/2018-07-31-configuration-management-related-systems/","section":"Posts","summary":"\u003cp\u003eConfiguration Management Systems like Ansible, Chef, Puppet, and SaltStack are in my opinion are nearly essential if you find yourself managing more than 5 machines.\nBut what exactly are they, which is better for my circumstances, do I still need them if I use a container based infrastructure, how do I get started?\nThis post is the first in a series of posts that will attempt to answer these questions.\u003c/p\u003e","title":"Configuration Management: the Related Systems"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/cuda/","section":"Tags","summary":"","title":"Cuda"},{"content":"GPU programming has the potential to make embarrassingly parallel tasks very quick. But what if you want to perform the same task on a variety of different types? In this post, I walk through a generic testing code that preforms a vector add on GPU and CPU to verify the correctness.\nThe Test Harness #Our main function is pretty simple:\nint main(int argc, char* argv[]) { check_type\u0026lt;int\u0026gt;(); check_type\u0026lt;long\u0026gt;(); check_type\u0026lt;double\u0026gt;(); check_type\u0026lt;float\u0026gt;(); return 0; } So how do we write check_type? Since it does most of our work, let\u0026rsquo;s break it down step by step. First we create a function that will perform our check and allcoate some data pointers we will use later.\ntemplate \u0026lt;class T\u0026gt; void check_type() { //create an array for 1024 items of type T constexpr int elms = 1024; std::array\u0026lt;T, elms\u0026gt; a, b, c, d; T *d_a = nullptr, *d_b = nullptr, *d_c = nullptr; // device vectors Generating random data #To have the best guarentee that the code is working, it is best to test with random data. To do that we need to generate random data according to a distribution. In C++11, you do this by creating a random number distribution and seeding it with a random number engine.\n//create a random number generator std::random_device rd; std::mt19937 eng(rd()); typedef typename std::conditional\u0026lt; std::is_integral\u0026lt;T\u0026gt;::value, std::uniform_int_distribution\u0026lt;T\u0026gt;, std::uniform_real_distribution\u0026lt;T\u0026gt;\u0026gt;::type generator; generator dist(0, 100); auto trand = [\u0026amp;dist, \u0026amp;eng]() { return dist(eng); }; In the code above we first create a std::random_device which is responsible to seed our random number generator. However, since this uses hardware randomness (if available), it can be slow. So it is not uncommon to switch to a pseduo-random number generator which we do with the marsenne twister generator mt19937 on the next line. For the generator, we need to decide if we need integral data (i.e. integers) or floating point data. We accomplish this using std::conditional and std::is_integral to select at template instantiation time between a std::uniform_int_distribution and a std::uniform_real_distribution. Finally we create a lambda expression to create a version that behaves like classic C style rand.\nComputed expected on the CPU #If you\u0026rsquo;ve never worked with C++\u0026rsquo;s \u0026lt;algorithm\u0026gt; header it makes boiler plate code like this a dream.\n//check_type cont... //generate random inputs std::generate(std::begin(a), std::end(a), trand); std::generate(std::begin(b), std::end(b), trand); //compute the expected output on the CPU std::plus\u0026lt;T\u0026gt; pl; std::transform(std::begin(a), std::end(a), std::begin(b), std::begin(d), pl); std::fill(std::begin(c), std::end(c), 0); First we generate two arrays with random numbers. After that we compute the results by transforming the two arrays we generated into the result array by adding them. Finally we fill the result array with zeros to ensure the comparison fails latter if some thing goes poorly.\nMoving data to the device #Unlike CPU programming, GPU programming requires explicit data movement to be efficient. Before we run the kernel, we need to explicitly request space on the GPU and move the data to it.\n//check_type cont... //allocate the device buffers for inputs and outputs size_t size = (sizeof(T) * elms); check_error(cudaMalloc((void**)\u0026amp;d_a, size)); check_error(cudaMalloc((void**)\u0026amp;d_b, size)); check_error(cudaMalloc((void**)\u0026amp;d_c, size)); //load the inputs onto the device check_error(cudaMemcpy(d_a, a.data(), size, cudaMemcpyHostToDevice)); check_error(cudaMemcpy(d_b, b.data(), size, cudaMemcpyHostToDevice)); Notice we compute the size based off the number of elements and the size of the type, but for C programmers who use malloc this isn\u0026rsquo;t a huge surprise. We take advantage of the .data() routine to work with std::array instead of raw data pointers.\nYou\u0026rsquo;ll notice calls to a method called check_error, for now just remember that it performs error handling. We\u0026rsquo;ll talk more about it in a later section.\nCreating and Launching the Kernel #Not much changes in launching the kernel and retrieving the results. All we have to do is add \u0026lt;T\u0026gt; to dispatch to the correct typed version of vadd.\n//check_type cont... //launch the kernel int threadsPerBlock = 256; int blocksPerGrid = (elms + threadsPerBlock - 1) / threadsPerBlock; vadd\u0026lt;T\u0026gt;\u0026lt;\u0026lt;\u0026lt;threadsPerBlock, blocksPerGrid\u0026gt;\u0026gt;\u0026gt;(d_a, d_b, d_c, elms); //copy the results back to the host check_error(cudaMemcpy(c.data(), d_c, size, cudaMemcpyDeviceToHost)); So, how is vadd defined?\ntemplate \u0026lt;class T\u0026gt; __global__ void vadd(const T* a, const T* b, T* c, int elms) { int idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx \u0026lt; elms) { c[idx] = a[idx] + b[idx]; } } If you\u0026rsquo;ve ever used cuda, this code isn\u0026rsquo;t that surprising. They only key difference is the introduction of the template \u0026lt;class T\u0026gt; bit to make the code generic.\nWe could have accomplished the same impact with some code like this:\n#define vadd_impl(T) \\ __global__ void \\ vadd_##T(const T * a, const T * b, T * c, int elms) \\ { \\ int idx = blockIdx.x * blockDim.x + threadIdx.x; \\ if (idx \u0026lt; elms) { \\ c[idx] = a[idx] + b[idx]; \\ } \\ } vadd_impl(int) vadd_impl(long) vadd_impl(float) vadd_impl(double) # define vadd(T) vadd_##T which would then be called in the driver like so:\nvadd(T)\u0026lt;\u0026lt;\u0026lt;threadsPerBlock,blocksPerGrid\u0026gt;\u0026gt;\u0026gt;(d_a, d_b, d_c, elems) But what do we lose by doing this? We loose some readability and compiler support. Notice the vadd_impl(int), vadd_impl(long), and so on. This has to be explicitly defined for each type. Not only is this tedious, if cuda decided to add a new type, we have to update our library to support it. We also generate the code for each method whether or not we use it. But what do we gain by doing it this way? As much as I hate to say it, opencl support for most hardware available right now. Opencl 2.2 introduced the c++ kernel language which includes template support, but very few vendors support this syntax. Without it, you are left to use macros to get the same generic code.\nPrinting results # //check_type cont... //check if the results are the same and report to the user std::cout \u0026lt;\u0026lt; name\u0026lt;T\u0026gt;::n \u0026lt;\u0026lt; \u0026#34; \u0026#34; \u0026lt;\u0026lt; (std::equal(std::begin(c), std::end(c), std::begin(d)) ? \u0026#34;success\u0026#34; : \u0026#34;failed\u0026#34;) \u0026lt;\u0026lt; std::endl; cleanup: cudaFree(d_a); cudaFree(d_b); cudaFree(d_c); What is name\u0026lt;T\u0026gt;::n? You may recall that C++ currently doesn\u0026rsquo;t really have fully featured reflection (that may change if Herb Sutter gets his way\u0026hellip;). So if you want the name of a class, you have two choices, typeid or some clever template programming. The problem with typeid.name() is that it returns an implementation defined name. For clang and gcc, this is the mangled name that is generated by the compiler for purposes getting unique names for linking. This can be cleaned up into a human readable name using an implementation specific demangle function. However, this is far from a perfect procedure, so I opted for some clever template programming. name is a struct that defines the name of the type is passed in as its template parameter. It is defined like this:\ntemplate \u0026lt;class T\u0026gt; struct name { static const char* n; }; #define decl_name(T) \\ template \u0026lt;\u0026gt; \\ struct name\u0026lt;T\u0026gt; \\ { \\ static const char* const n; \\ }; \\ const char* const name\u0026lt;T\u0026gt;::n = #T; decl_name(int); decl_name(double); decl_name(float); decl_name(long); It defines a macro and uses the # operator to convert the template type parameter to a string. The macro creates a template specialization of the name base template provided above. This provides a compile-time way to get the appropriate type name with requiring demangling. There is a weakness to this method. It can\u0026rsquo;t have separate entries for type aliases (i.e. size_t). But since I didn\u0026rsquo;t need that flexibility for my code, it was fine.\nSo what is with the label cleanup? Remember the method called check_error? It prints a failure message and cleans up memory if a cuda function fails. It is defined like this:\n#define check_error(x) \\ do { \\ cudaError_t err = (x); \\ if (err != cudaSuccess) { \\ std::cerr \u0026lt;\u0026lt; \u0026#34;Failed: \u0026#34; \u0026lt;\u0026lt; __LINE__ \u0026lt;\u0026lt; \u0026#34; \u0026#34; \u0026lt;\u0026lt; cudaGetErrorString(err) \\ \u0026lt;\u0026lt; std::endl; \\ goto cleanup; \\ } \\ } while (0) Macros that are intended to act like functions can have bizarre interactions if used in non-standard places. Wrapping a macro in a do { } while(0)} allows the macro to be used almost everywhere a function call can. It also has the effect of creating a scope that can prevent naming conflicts. The parens about (x) are also to prevent weird order of operations bugs that can results from macros. Finally we goto cleanup if the error happens which simplifies the control flow from being a huge if pyramid.\nI hope you find this helpful. Until next time, happy programming!\n","date":"12 May 2018","permalink":"https://robertu94.github.io/posts/2018-05-12-generic-cuda/","section":"Posts","summary":"\u003cp\u003eGPU programming has the potential to make embarrassingly parallel tasks very quick.\nBut what if you want to perform the same task on a variety of different types?\nIn this post, I walk through a generic testing code that preforms a vector add on GPU and CPU to verify the correctness.\u003c/p\u003e","title":"Generic Cuda"},{"content":"C++ is a huge language. It has tools form imperative, functional, object-oriented, and generic paradigms. And that leaves out the extremely fine control over things like memory allocation strategies in the standard library not generally available elsewhere. In this post, I present my learning path through C++ and offer some suggestions for learning this multi-faceted language.\nOrder of Topics #This is not intended to be an exhaustive (for that would be far too long), or optimal (for that would be context dependent) listing of the topics, but rather the path that I took through the language.\nBeginning C++ #Everyone needs to start somewhere, for C++ I would start here.\nFor this section, I would read at least Effective C++ and Effective Modern C++, and then I would skim the C++ Core Guidelines. Together these will give you a broad basis to learn more about C++. The other references are still useful, but maybe not as pressing.\nEffective C++ by Scott Meyers - still the best beginners book for C++ and overviews common design concerns in C++. While it focuses on C++98, much of this book still applies. Effective STL C++ by Scott Meyers - Overview the standard template libary (set of containers and algorithms included in the standard libary) and how to use the common parts effectively. More Effective C++ by Scott Meyers - Extensions to \u0026ldquo;Effective C++\u0026rdquo; but more special purpose than the base book. While it focuses on C++98, much of this book still applies. Modern Effective C++ by Scott Meyers - How to effectively use new features in C++11 which radically modernized the language. C++ Core Guidelines - How to effectively use newer parts of the language. Here are some challenges to improve your C++ knowledge\nImplement std::unique_ptr and then std::shared_ptr from scratch. Implement std::tuple, std::variant and std::any from scratch. Implement a function fmap that takes a arbitrary Container of with elements of type T, a function which may convert each T to a possibly different type V and then stores the result into an instance of that container. Ensure your function works for std::vector, std::map, std::optional, and std::tuple. Implement a function curry that take a function of arity k and an argument of the type of the first argument and returns a function of arity k-1. Implement a function flatten that takes possibly arbitrarily nested sequence containers and returns all of the items fully un-nested. Standard Library #It almost goes without saying that the C++ Standard library is incredibly useful, and you should almost always start here. Its not as complete as say Python\u0026rsquo;s standard library, but its far more flexible. I would use cppreference.com or devdoc.io to read documentation on the standard library.\nI would at least know about the following core objects in the standard library ordered by how roughly important I find them.\nName Use \u0026lt;algoritm\u0026gt; generally useful functions std::unique_ptr pointers that are the only reference to an object in memory std::shared_ptr pointers that automatically count references std::format a typesafe printf-like alterative to iostreams std::array a statically allocated array with handy bindings std::vector a dynamically resizing array std::map key value store, often implemented as a red-black tree std::unordered_map key value store, often implemented as a hash table std::set key store, often implemented as a red-black tree std::unordered_set key store, often implemented as a hash table Iterator Concepts/Ranges simplifies accessing members of a collection std::string dynamically resizing character data std::span a nonowning view into an existing container std::string_view a non-owning reference to character data std::tuple a generic version of a struct useful for generic programming std::list a doublely linked list std::ostream output to file, string, or stdout std::istream input from file, string, stdin std::exception indicates extra-ordinary circumstances Build Systems and the C++ Ecosystem #Tools are important to getting actual work done while programming. Because of the nature of C++ I find that it has and need more types of tools than you may have used in other languages. I target this section to Linux/Unix platforms because it is what I use most often. When I list multiple tools, you generally only need one, and I list them in order of personal preference.\nFor this section, read what each class of tool does; then use it as a reference when you need an instance of that kind of tool. By no means is this a comprehensive list.\nPurpose Tool Why Backend Builder Ninja Much faster than Make Backend Builder Make Incredibly common on Unix platforms Backend Builder Bear Generates compile-commands.json files for other projects Build System Meson Easy to use, Very fast Build System CMake More easy to use Build System Autotools Works on esoteric platforms where other choke Compile Cache ccache Dramatically speeds up incremental builds Distributed Builds icecc/icecream Spread builds out to a server of faster machines Code Formatting clang-format Easy, Highly customizeable, sane defaults Compilers clang Better Error messages, clang/llvm Ecosystem Compilers GCC g++ Currently still faster, more common IDE Integratin clangd Clang based IDE integration shows warnings and advice Debugger lldb Highly programmable, handles templates well, easy to use Debugger gdb The standard debugger, esoteric interface, more on gdb here Debugger templight Specialized debugger for compile time C++ code Debugger metashell An older, ease of use tool built atop templight Indexing Exhuberant Ctags The de facto tool for this job Linting clang-tidy Most extensive and \u0026ldquo;correct\u0026rdquo; linter Profiling perf Linux Specific, Extremely robust, easy to use Searching ag/ripgrep insanely fast, sane defaults, not syntactic sensitive Searching clang-query Slow, semantic sensitive Searching grep Common, pretty fast, not semantic sensitive Tracer ltrace Trace shared library calls Tracer strace Trace system calls Tracer dtrace Trace/Profile kernel and elsewhere, not widely available Tracer llvm-xray Tracer with similar design principles to dtrace, but requires recompilation and is more available Leak Checking Leak Sanitizer locate various memory leaks in programs Undefined Behavior Checking Address Sanitizer locate various memory misuses in programs Undefined Behavior Checking Memory Sanitizer locate uninitialized reads in programs Undefined Behavior Checking UndefinedBehavior Sanitizer locate other undefined behavior in programs Undefined Behavior Checking Thread Sanitizer locate data races in programs Package Manager Spack Package manager focused on HPC; Similar to PIP Package Manager VcPkg/Connan/Wraptool C++ centric package managers; you miliage may very depending your usecase I\u0026rsquo;ve recently added a learning to learn document for CMake\nYou\u0026rsquo;ll also eventually decide that you need/want some libraries to get useful work done. This post would be remiss if I didn\u0026rsquo;t mention a few cross cutting libraries.\nBoost - a family of libraries that stretch the limits of what C++ can do. Libraries from Boost often become standardized. The best reference I have found is The Boost C++ libraries book. Qt - While primarily a UI toolkit, it features a bunch of useful features. The online documentation is great. Abseil - Google\u0026rsquo;s take on a general purpose library. Provides backward compatibility for new standard concepts. The comprehensive docs are sparse, but the code is well documented. Here is a list of other libraries that I have used and would recommend.\nPurpose Library Why Commandline parsing getopt Very common command line parser, very portable, not pretty Unit Testing Google Test Very common testing tool, easy to use Benchmarking Google Benchmark Very common benchmarking tool, easy to use Networking Protobuf/gRPC Networking Server/Communication framework Networking Boost.ASIO De facto C++ native networking library Networking sys/socket.h Still works, arguably simpler than ASIO Graphs Boost.Graph Tons of standard graph algorithms and structures Date Math date.h Incredibly fast, easy to use; now in the standard library JSON/XML Parsing Boost.PropertyTree Easy to use JSON Parsing nholman-json Extreemly easy to use Distributed Programming HPX A different take on HPC; I would argue easier to use Distributed Programming Boost.MPI More native than standard MPI; fewer interfaces Distributed Programming OpenMPI/MPICH Very flexible Profile some C++ that you wrote. What is are the bottlenecks in your code and why? Port the build system of a package that you use to another build system. What was easier or harder? Configure your build system to run your tests, run clang-format, clang-tidy and clangd Create a program trace with ltrace, strace, llvm-xray, and perf compare and contrast the outputs Write a clang-query script to find all references to a function or of an enum value. Try accelerating your build with ccache/sccache or icecream/distcc. Profiling and Speeding up C++ builds #C++ can take a long time to build especially when your code is really template heavy. First just adopt ccache, it will dramatically speed things up for you for incremental builds.\nHere is how I profile C++ builds. First if you are using ninja, you can clean your build, delete your .ninja_log file, and run your build. This will populate this file with a log of the build process. Next, You can extract timings for the build using this ninja stats script written in AWK. Once you\u0026rsquo;ve narrowed down your search to a few files, you can use clang\u0026rsquo;s -ftime-trace to get a detailed view of what is taking so long to compile/instantiate. See The Blog post \u0026ldquo;time-trace: timeline / flame chart profiler for Clang\u0026rdquo; for more information about how to use these traces.\nOnce you\u0026rsquo;ve made as many changes to your problematic structures and functions as you can, you may further reduce build times by using \u0026ldquo;precompiled headers\u0026rdquo; and \u0026ldquo;unity builds\u0026rdquo;. Precompiled headers work by serializing the compilers internal state so that the parsing phase of C++ can be skipped. Unity builds reduce the time required to instantiate identical headers by grouping source files for compilation. These both are not without their drawbacks for tooling which relies on source file names such as clangd, but I am hopeful that both of these techniques will be obviated with C++20 modules while improving the tooling situation. However, at time of writing, C++20 modules are not yet completely implemented by most major compilers.\nuse clang\u0026rsquo;s -ftime-trace or a similar feature in another compiler to profile your build. What could you do to accelerate the build? Object-Oriented C++ #C++ has a uniquely complicated object oriented system. Most of this is due to the use of templates for generic programming and Turing-completeness of templates.\nFor this section, the ordering is less important. Several of these books apply to more than just C++, but are especially important in a language as verbose as C++.\n\u0026ldquo;Design Patterns Elements of Reusable Software\u0026rdquo; by the \u0026ldquo;Gang of Four\u0026rdquo; - before the pitch-forks come out: 1) the examples of patterns in this book are often in C++, 2) object-oriented programming is object-oriented programming, and it doesn\u0026rsquo;t change much from language to language. I also find that most people struggle in OO to reinvent the wheel labeled by these four in 1994. It changed the way that I structure programs that I write. \u0026ldquo;Refactoring: Improving the Design of Existing Software\u0026rdquo; by Martin Fowler - over the course of learning C++ you\u0026rsquo;ll discover that you\u0026rsquo;ve done horrible, terrible things. This book while targeted at a general audience will help you fix them. It also puts the proper focus on usable, understandable interfaces and code over the spaghetti I often see in new C++ developers. The Pimpl Idiom - Pointer to Implementation is a powerful technique to reduce compile time dependancies. Read the set of two articles entitled \u0026ldquo;Compilation Firewalls\u0026rdquo; by Herb Sutter. The RAII Pattern - Resource Acquisition is Initialization is a fundamental to memory and exception safe programming in C++. Read the article about RAII on CPP Reference. Modern C++ Design: Generic Programming and Design Patterns Applied by Andrei Alexandrescu - One of the few uniquely C++ Object-Oriented books I have read. It assumes a fair bit of knowledge on generics and templates so read about it first. Metaclasses: Thoughts on Generative C++ by Herb Sutter - This video highlights were object oriented C++ maybe going at time of writing in early 2018. Wrap a c library that you use to use RAII. How much more concise is the user code? Try implementing generic versions of the Gang of Four patterns from \u0026ldquo;Object Oriented Design Patterns\u0026rdquo;. The flyweight pattern and factory pattern are especially rewarding to implement. Generic C++ and Templates #Generics and Templates are among C++ most powerful features. Every C++ programmer should know the basics.\nFor this section, the ordering is especially important, the later items are quite advanced and build on the previous items.\n\u0026ldquo;C++ Templates the Complete Guide\u0026rdquo; by David Vandevoorde, Nicolai M. Josuttis, and Douglas Gregor - An easy as possible introduction to the sometimes black magic of C++ templates. Curiously Recurring Template Pattern - while discussed in the Vandevoorde book, I found the article, \u0026ldquo;Polymorphic Clones in Modern C++\u0026rdquo; by Jonathan Boccara a much more practical example of where this may be used. \u0026ldquo;Modern C++ Design: Generic Programming and Design Patterns Applied\u0026rdquo; by Andrei Alexandrescu - I listed this in the object-oriented section, but this book opened my eyes the possibilities of C++ templates and generic programming in general. If you can\u0026rsquo;t figure out why a particular template won\u0026rsquo;t compile, consider using templight or metashell.\nFunctional C++ #This is an evolving category of C++. As such, I expect these to be much more in the coming years.\nFor this section, I would read all of the articles, they are not too long.\nlambda expressions - a key building block for functional programming in C++. Read the CPP reference documentation on lambdas. std::function - a generic means of storing type-specified references to a function. Read CPP references documentation on std::function \u0026lt;algorithm\u0026gt; - a example of a library that makes extensive use of functional concepts. Read CPP references documentation on \u0026lt;algorithm\u0026gt; What next? #When you\u0026rsquo;ve read most of the above, and are still looking to improve, I use the following to stay sharp.\nCPPCon - the yearly C++ developers conference. Filled with thought provoking talks and most are on YouTube. Read llvm\u0026rsquo;s libtooling, clang\u0026rsquo;s libC++ or Google\u0026rsquo;s abseil source code to see clear examples of well-written C++ code. Read the C++ Standard - it\u0026rsquo;s not for the feint of heart. The final version is behind a pay-wall, but the drafts are not. Read this if you want to understand some of the deeper behaviors of C++. C++Weekly - A video podcast that overviews different aspects of C++ in bite size chunks. CPP Cast - An audio podcast that covers upcoming C++ news Suggestions to Learn the language #I have two key suggestions to learn the language:\nStart small: choose a subset of the language that you want to learn well. The language is too large for most if not all people to be an expert on all parts of the language. Somethings like parameter passing should be in everyone\u0026rsquo;s subset, but the oddities of std::atomic\u0026lt;\u0026gt;, vector\u0026lt;bool\u0026gt; or the CRTP probably don\u0026rsquo;t need to be. Expand your subset as needed: choose a series of small projects that motivate why you want to learn various aspects of the language. This will help you practice and remember what you\u0026rsquo;ve learned. I hope you find this useful. Until next time!\nChange Notes # 2023 - improved formatting, updated tools 2020 - Added section on build profiling, updated tools, library components to learn, and further resources 2018 - Initial version ","date":"12 January 2018","permalink":"https://robertu94.github.io/learning/cpp/","section":"","summary":"\u003cp\u003eC++ is a huge language.  It has tools form imperative, functional, object-oriented, and generic paradigms.  And that leaves out the extremely fine control over things like memory allocation strategies in the standard library not generally available elsewhere. In this post, I present my learning path through C++ and offer some suggestions for learning this multi-faceted language.\u003c/p\u003e","title":"Learning to Learn: C++"},{"content":"Foundation Parallel Operations #Fundamental to developing high-performance software is having an understanding of the basics of parallel architecture.\nEnumeration and Grouping #Intent #Label each processing element in a group with a distinct label and describe new groups in a way that is consistent across nodes.\nMotivation #Enumeration and grouping is foundational to other parallel primitives such as Send/Recv, which need these labels to identify sources and targets. It is often also used as a primitive in larger algorithms to statically partition work between a collection of processing elements. Lastly, programs might create multiple distinct enumerations for various subtasks to facilitate collectives on subsets of processes better, or create new groups dynamically to via collaboration with the scheduler to expand the quantity of resources for a task. This also applies to GPU device programming (e.g. threadId.x in CUDA).\nApplicability # As a fundamental primitive, it\u0026rsquo;s applicable to almost all use cases For systems with rapidly changing group members, other work partitioning schemes and addressing schemes that hide this aspect from the user may be more important Structure # classDiagram class User:::user class ResourceManager:::optional { +group(): Group } class Scheduler:::optional { +allocate(r: Request): Lease +deallocate(l: Lease) } class Element:::required class Group:::required { +id(Element: e): int +size(): int } Group \"1..n\" \u003c-- \"1\" Element: Belongs To ResourceManager --\u003e Group: Creates ResourceManager \u003c-- Scheduler: Provisions Scheduler \"1\" --\u003e \"*\" Element: Allocates/Deallocates User \"1\" --\u003e \"1\" Scheduler: Requests classDef required fill:#f9f,stroke:#333,stroke-width:4px; classDef optional fill:#bbf,stroke:#333,stroke-width:4px; classDef user fill:#eea,stroke:#333,stroke-width:4px; Participants/Elements # Elements - an individual processing element Groups – collections of processing elements (optional) resource manager – a leader (see leader election) that decides identity within a group (optional) scheduler – used to allocate/deallocate additional elements Collaboration with other patterns # A fundamental concept used in most other operations Dynamic group management often requires interaction with the scheduler and thus resource management patterns Dynamic group management allows fault tolerance on collectives and thus resilience patterns Code Examples #MPI Comm examples using Sessions, which interact with the resource manager to create processes aligned with hardware resources\n#include \u0026lt;mpi.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;stdlib.h\u0026gt; #include \u0026lt;string\u0026gt; int main(int argc, char *argv[]) { // Initialize an MPI session MPI_Session session; int err = MPI_Session_init(MPI_INFO_NULL, MPI_ERRORS_RETURN, \u0026amp;session); if (err != MPI_SUCCESS) { fprintf(stderr, \u0026#34;MPI_Session_init failedn\u0026#34;); MPI_Abort(MPI_COMM_WORLD, err); } //query the runtime what processes sets exist (implementation defined) int num_p_sets; MPI_Session_get_num_psets(session, MPI_INFO_NULL, \u0026amp;num_p_sets); std::string pset_name; for(int i = 0; i \u0026lt; num_p_sets; ++i) { int psetlen = 0; MPI_Session_get_nth_pset(session, MPI_INFO_NULL, i, \u0026amp;psetlen, NULL); pset_name.resize(psetlen - 1, \u0026#39;0\u0026#39;); MPI_Session_get_nth_pset(session, MPI_INFO_NULL, i, \u0026amp;psetlen, pset_name.data()); //match the specified process set if (pset_name == argv[1]) break; } //groups on their own do not facilitate communication, they just enumerate processes MPI_Group world_group; if (MPI_Group_from_session_pset(session, pset_name.c_str(), \u0026amp;world_group) != MPI_SUCCESS) { return 1; } int grank, gsize; MPI_Group_rank(world_group, \u0026amp;grank); MPI_Group_size(world_group, \u0026amp;gsize); printf(\u0026#34;Hello from grank %d out of %dn\u0026#34;, grank, gsize); //create a communicator to actually communicate between the processes MPI_Comm comm; if (MPI_Comm_create_from_group(world_group, \u0026#34;world_set\u0026#34;, MPI_INFO_NULL, MPI_ERRORS_RETURN, \u0026amp;comm) != MPI_SUCCESS) { return 1; } int rank, size; MPI_Comm_rank(comm, \u0026amp;rank); MPI_Comm_size(comm, \u0026amp;size); printf(\u0026#34;Hello from rank %d out of %dn\u0026#34;, rank, size); // Cleanup MPI_Comm_free(\u0026amp;comm); MPI_Group_free(\u0026amp;world_group); MPI_Session_finalize(\u0026amp;session); return 0; } MPI Comm example using COMM_WORLD\n#include \u0026lt;mpi.h\u0026gt; int main(int argc, char* argv[]) { MPI_Init(\u0026amp;argc, \u0026amp;argv); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, \u0026amp;rank); MPI_Comm_size(MPI_COMM_WORLD, \u0026amp;size); printf(\u0026#34;Hello from rank %d out of %dn\u0026#34;, rank, size); MPI_Finalize(); } CUDA kernel launch examples\n#include \u0026lt;random\u0026gt; #include \u0026lt;vector\u0026gt; __global__ void vadd(float* a, float* b, float* c, size_t n) { //enumerates a block i = threadIdx.x + blockIdx.x * blockDim.x; //because the number of tasks may not be evenly divisible by what is //efficent on the hardware you almost always get a more threads than tasks if (i \u0026lt; n) { c[i] = b[i] + a[i]; } } int main(){ constexpr size_t dims = 1024; //performs std::vector\u0026lt;float\u0026gt; h_a(dims); std::vector\u0026lt;float\u0026gt; h_b(dims); std::uniform_real_distribution\u0026lt;float\u0026gt; dist; std::mt19937 gen; auto rng = []{ return dist(gen); }; std::generate(std::begin(h_a), std::end(h_a), rng); std::generate(std::begin(h_b), std::end(h_b), rng); float *d_a,*d_b,*d_c; cudaMalloc(\u0026amp;d_a, sizeof(float)*dims); cudaMalloc(\u0026amp;d_b, sizeof(float)*dims); cudaMalloc(\u0026amp;d_c, sizeof(float)*dims); cudaMemcpy(d_a, h_a.data(), dims*sizeof(float), cudaMemcpyHostToDevice); cudaMemcpy(d_b, h_b.data(), dims*sizeof(float), cudaMemcpyHostToDevice); size_t threads_per_block = 256; size_t blocks_per_grid = (dims+threads_per_block-1)/threads_per_block; //requests creation of block_per_grid*threads_per_block threads //threads_per_block has a max limit based on hardware width \u0026lt;\u0026lt;\u0026lt;blocks_per_grid, threads_per_block\u0026gt;\u0026gt;\u0026gt;vadd(d_a, d_b, d_c, dims); cudaMemcpy(h_c.data(), d_c, dims*sizeof(float), cudaMemcpyDeviceToHost); cudaFree(d_a); cudaFree(d_b); cudaFree(d_c); } // etcd\nConcequences (pros and cons of use) # Use of groups may simplify programming by allowing collective operations to be used as opposed to independent operations The creation of strongly consistent groups is a syncronization point which inroduces overhead especially for extreemly large number of processing elements. Implementation considerations # Static vs Dynamic number of processing elements – this addresses whether or not groups can be created or destroyed at runtime by introducing or removing processing elements. The creating groups proses an operational challenge if the number of nodes exceeds the current resource request because the overall system may not have sufficent resources to satisfy the request. In which case the scheduler accepts immediately, either refueses, or accepts with a delay. Refusing could result in the task having insufficent resources, delaying often causes waiting while resources are allocated. The case where the new request is fits within an existing allocation is much more operatoinally simple, but requires additional resources be allocated by scheduler but for some of the job might be unused. Static vs Dynamic membership of groups – this addresses whether or not an existing groups membership can change. Allowing changes of group membership allows for more better handling of failures and more dynamic right-sizing of resource needs, but increases complexity to handle cases where a node is deallocated from or added to a group. Strict consistency vs Weak consistency – can group members have an inconsistent view of group membership? Weak consistency can enable lower overhead (from less syncronization), more scalablity (from less syncronization) and greater fault tolerance (because groups can remain if one of its members die). Strong consistency of group membership is dramatically easier to reason about. Known uses # MPI_Communicators are strongly consistent MPI_Comm_size, MPI_Comm_rank – get the number of processing elements in a group and the id of a specific processing element MPI_Comm_split – allows creating subgroups from existing resources MPI_Comm_spawn – allows creating new groups on additional resources MPI_Comm_connect – allows connecting two groups etcd, zookeeper, mochi-ssg – implements weakly consistent dynamic group management CUDA – strongly consistent, uses dynamic group creation, but not dynamic membership changes Send/Recv #Intent #Fundmental primiative describing point to point communication between two processing elements.\nMotivation #Without some mechanism to communicate between physically distinct hardware, parallel and distributed software does not exist. It important to note this even applies withing a single node on systems featuring multiple Non Uniform Memory Access (NUMA) domains which violate traditional Von Neuman assumptions around system architecture by having some memory that is “lower latency” to computation on certain processing elements than others. It can seldonly be completely avoided.\nApplicability # As a fundemental primitive, it is applicable to almost all use cases Some communication patterns can be more consisely expressed using higher level primatives which can then be then be specialized for the underlying hardware capabilties. In such cases, higher level collective operations can be used. Some uses cases (e.g. consistent and partition tolerant systems like traditional relational database managment systems like PostgresQL, some aspects of filesystems) require frequent syncronization to maintain consistency. In such systems, avoiding distributing the workload requires fewer expensive syncronizations that occur over the network and may improve performance. Structure #Participants/Elements # Message – what is being sent Sender – who is communicating information Reciever – who is obtaining information Switch(es)/Interconnect – intermediate network devices/nodes that transpartently conveys a message Collaboration with other patterns # Scatter/Gather/Broadcast – depending on hardware, these higher level collectives are implemented in terms of a point-to-point communication method. Hardware specialization – these routines are so primiative that often one or more aspect of them are implemented in dedicated hardware Pooling – often underlying resources use for sending and recieving messages are pooled Syncronization and Resilance patterns – use these as a primitive Code Example(s) #// example from an RPC based system\n// example from MPI\n// example from MPI One-sided\n// example from MPI partitioned send\n// example from a GASNet based approach\nConcequences # Communication incurs overhead which is often orders of magnitude slower than simple computations. Communication enables greater access to resources via horizontal scaling (more nodes) which is often cheaper than vertical scaling (more powerful nodes) past a certain scale due to the difficulty of implicity maintaining coherence on progressively larger systems. Communication introduces complexity of managing distributed state to the application Communication enables fault recovery and resilance by reducing the probability of a single point of failiure from a single node failure. Implementation considerations # Implicit (Global Address Space) vs Explicit (e.g. Message Passing). There are two fundemental models of communication – explicit models where the programmer specifically describes which processes send and which recieve – this describes UNIX sockets as well as HPC oriented solutions such as MPI. There is however an alternative with global addressing systems which instead make communication implict much like the communication between threads. At scale, there tends to be a tradeoff between performance (explicit) and productivity (implicit), but depending on the usage pattern, this performance overhead can be either minimal or catestrophic. Message Sizes – The performance of small messages are dominated by the latency to send any data at all. Large messages are dominated by the bandwidth of the system. Often there is a tradeoff between latency and throughput. Frequenly a small amount of throughput can be sacrificed for much lower latency. Frequenly, these choices are made by your networking library, but may be tunable for a specific code. Addressing – How do you identify a specific sender or reciever pair? Systems such as MPI use an integer value `rank` to identify a specific processing element within a communicator (group of processing elements). For recieving operations, MPI allows recieving from any process in a group which enables implementing constructs like work queues. MPI further specifies this with TAGs which allow multiple distinct messages to be sent concurrently – e.g. to impement a cancelation channel or to indicate an exception channel. Routing – how is the route between nodes determined. Often this is the responsability of the networking library or hardware. Some network topologies (e.g. fat-trees, toruses) have specialized routing algorithms that minimize contention or ensure low latency. Other topologies feature slow links (e.g. wide area networks) that may need to be deprioritized. Lastly, some networking libraries and hardware allow for “multi-path” communication which enables higher throughput at the cost of additional adminstrative and implementation complexity. Asyncronous vs Syncronous – are the routines to initate sends blocking with respect to the caller, or not? Non-blocking routines enable the overlapping of computation with communication, but at the cost of increased complexity. Modern hardware is natively asyncronous with respect to the CPU. Zero Copy – are copies require to send messages? In UNIX sockets, the user provided buffer supplied to send() is copied to user space to kernel space before the routine returns. Then the kernel copies from the kernel based buffer to the network interface where the message is transmitted to the recieving node which then copies the data from its network interface to a kernel buffer, before it is finally copied to the user memory. In zero copy networking, the user writes directly to the sends network interface’s buffer and the recieving node’s network interface writes directly to user memory bypassing copies to the kernel. This potentially has security trade-offs if implemented poorly, but can dramtically improve send/recv latency. OS Bypass – In UNIX sockets, a system call write()/read() is required to initiate a read or write on the network which incurs overhad from context switching as well as potential overhead from context switching to other tasks which are prefered along system call boundries. “Queueing” and SENDRECV – even when using zero copy transfers, it is possible that the network interface may not have suffcent capacity to accomidate all read/write requests and a request will need to queued. To prevent deadlocks from queuing in large pair-wise exchanges, “buffered sends” or “asyncronous sends and recieves” which pre-allocate the network interface memory or alternatively combined “sendrecv” can be used which atomically swap buffers can be used to avoid deadlock. Queuing can be mangaed by the network interface or or coordinated by the switchs/routers on the network (in the case of infiniband). Partitioned Operations and Device Initiated Sends – currently some devices (e.g. GPUs) have limited ability to initate network operations on their own, and require another device (e.g. the CPU or the network interface) to initate the request. Some fabrics and devices implment so-called partiioned sends which divide the declaration that a send should occur from the declaration that memory is ready to send, and from the actual send of the data. In such cases, the network interface enques a task to initate the sends/recieves on the behalf of the device using low level routines that allow the device to wait for a condition (e.g cuStreamWaitValue32 which issues a callback when a particular memory address is set to a particular value). This is frequently combined with atomic additions or bitwise operations to indicate that an operation is ready. At time of writing this requires close collaboration between the network and device (e.g. HPE Slingshot 11 and Nvidia CUDA). A special case of this exists for sends (writes) and recvs (reads) from the filesystem under branding like GPUDirect Storage. Reliability – does the protocol (e.g. tcp/udp, infiniband RC/UC) ensure that messages are actually delievered. Reliable messages are easier to reason about, but are less robust to heavy contention or packet loss scenarios where latency can spike as messages are retransmitted to ensure delivery. Unreliable messages in contrast have no delivery garuntees, but if the application tolerates some information loss (e.g. video transmission, gossip messages), unreliable messages can be used to greatly reduce latency. Error Handling – how are errors identified and reported to the user? Some errors can be definitivley return (e.g. ENOPERM for disallowed operations), but others can be harder to detect especially in the context of node failures which can be difficult to distinguish from heavy contention or insufficent progress. See heart beat and gossip protocols for more information. One Sided vs Two Sided – are both sides of the communication involved with the communication or is just one? Two sided operations are classic in explicit message passing systems. One sided operations utilize a hardware feature known as remote direct memory access (RDMA) to allow remote hosts to issues commands to read or write to user space memory on a remote host and are present in both global address based systems as well as modern explicit message passing libraries. In the case of two sided operations, the sender and reciever are syncroized when the message is communicated, but one-sided operations, some other mechanism is used to explicity syncronize calls to lower overhead (e.g. communication epochs). Progress – how does the system decide when to actually perform operations. Some system require explicit progress to give control about when operations are performed to provide lower latency. Others implicity handle progress either specific calls that advance progress as collateral to other operations, or with a dedicated progress thread which can be simplier to use. Atomisity – what operations can be performed on remote memory? At its simplist, read/writes are allowed, but more modern systems allow certain atomic operations such as arithmatic, bitwise, and certian other operations to be preformed in such a way that either the entire operation is performed or none of the operation is performed allowing lower overhead and not requring return messages. Hardware frequently limits this to a single 64 bit instruction having atomic operations. If an array of such values are “atomically” operated on, each value is independently treated as atomic but the ordering of operations to each value may be arbitarily interleaved. Security – in HPC systems, security of communication is often handled at the network level and communication initation stage rather than at the node level while non-HPC system often implement security at the node level assuming the network is untrusted. This allows less overhead enforcing access controls, but requires a trusted adminstrative domain. Known uses # Lower Level TCP/UDP/ROCE over IP IO_URing Infiniband Machine Specific HPC Fabrics Slingshot/ToFuD Higher Level MPI Implementations of Send/Recv, MPI_Win one sided functions RPC systems (e.g. Mochi, GRPC) GASNet based Languages (e.g. Chapel, upc++) Device \u0026lt;-\u0026gt; Host in GPU Programing (e.g. OpenMP Target, CUDA, SYCL, etc…) Collectives #Intent #A collective operations that send data from one node to collection of other nodes either in whole (broadcast) or in part (scatter) of others or visa versa to send from many nodes to one node (gather) or all nodes (allgather), or from all nodes to all other nodes (all to all).\nMotivation #These are foundational communication patterns for group of nodes. This might be used to spread work out to a group of nodes, inform them of a request, wait for them (or some subset of them) to complete a request.\nApplicability # As a fundemental primitive, it is applicable to almost all use cases Collectives are often “implicit” in global address space schemes so may not be explicity invoked by the user making them less relevent in this context. The larger the group of processes collectively performing some task, the greater the overhead can be from a straggler who takes a disproportionately long time to complete the collective operation. Structure #Participants/Elements # (optional) a root process [for broadcast, gather, scatter] A group of processing elements Collaboration with other patterns # As a fundemental primative, this patterns is used to build many large patterns Code Example #Concequences # Collective simplify code and allow performance portability accross diverse architectures. Collectives can become a point of failure in large jobs when one or more nodes fail during a collective Implementation considerations # Syncronous vs Asyncronous – asyncronous collectives allow computation or other communication to occur during a collective operation, but increase the complexity of the underlying implementation Non-contigious collectives – collectives may support sending different quantties of information during a collective (e.g. gatherv/scatterv) Cancelation – can an operation be canceled after it has been started, if so is the cancelation premptive or collaborative? Allowing cancelation makes it easier to implement patterns such as “racing” queries where you want the first and then not wasting resources on the remaining operation Toleration of stagglers/failures – does the collective require all nodes to complete the operation, or mearly some predetermined fraction of them? Topology awareness – especially in the context of wide area network such as on the Eagle supercomputer which spans multiple datacenters and some links between nodes are dramatically slower than others. In such as system, these algorithms should be specialized to avoid sending more communication over the slow link than is strictly nessisary either communication batching strategies or ordering of pairwise steps in the collective. Choice of algorithms Ring – process i sends to process i+1%n Torrus – a special case of ring algorithms where the ring is aligned to the torus network topology Butterfly– has a consistent more consistent runtime per process in the collective Bionomial Tree – minimizes volumes of communication and maximizes throughput by communicating in k-nary tree Linear – offers lower latency in some cases (e.g. small numbers of processes sending small messages) Known uses # MPI Collectives NCCL (network)/CUB(device) collectives for CUDA Map #Intent #Applies the “same” function to a collection of tasks in parallel\nMotivation #This is a foundational programming pattern. It can be used in parallel or serial cases whenever \u0026ldquo;the same\u0026rdquo; independent operation needs to be performed on a collection of elements.\nApplicability # As a fundamental primitive, it is applicable to many use cases This pattern is still applicable for use cases where different operations (including no-ops) need to be performed on non-overlapping subsets of elements provided that the decision of which operation should be performed can be determined only by considering each element individually. Not appropriate for cases where tasks need to synchronize/communicate with each other, instead consider reductions or scans Structure #Participants/Elements # Workers – who performs the tasks Tasks – the actual work to be performed (optional) Scheduler – decides which tasks process each work element Collaboration with other patterns # Load balancing – load balancing patterns can be used to deal with imbalanced workloads which require inconsistent quantities of resources per task Resource Management – how tasks are assigned to resources can dramatically effect performance Hardware specialization – common and foundational operations are often implemented in hardware (e.g. vector addition, matrix matrix multiply for small matrices) in a paradigm referred to as Single Instruction Multiple Data (SIMD). GPUs implement Single Instruction Multiple Threads (SIMT) which allows for an additional level of hierarchy. Fault Tolerance - how errors and cancellation are handled has implications for fault tolerance of parallel maps especially checkpointing, bulkheads, replicas and algorithm based fault tolerance. Code Example #Consequences # Tasks can be distributed over processing elements in parallel allowing for less wallclock execution time. Starting/tearing down parallel resources has some overhead, or may introduce less regular data access patterns resulting in less efficent data access than the serial case. Implementation considerations # Serial vs parallel \u0026ndash; are the tasks performed in serial or in parallel. Performing tasks in serial may be optimal when the overhead for creating the workers or scheduling the tasks exceeds the benefits from parallel execution. Task heterogenitity vs homogenity \u0026ndash; heterogeneous tasks often feature greater load imbalance than homogeneous tasks and may be harder to schedule on certain kinds of hardware platforms. For example on GPUs, there is both a limit on the number of heterogenous tasks that can execute concurrently (which can be lower than on CPUs), but even for homogenous workflows also when there is divergence in the workflow (e.g. some tasks take different branches of an \u0026ldquo;if\u0026rdquo; statement), GPUs may pause execution on for threads while other threads in the same team take a different branch resulting in dramatically increase execution time. Static vs dynamic task assignment \u0026ndash; once a task is assigned to a worker, can the task be re-assigned to another worker? Cluster wide HPC schedulers (e.g. Slurm, PBS) tend to perform static task assignment which is much easier to implement. Node local, tasking runtime, and cloud oriented schedulers (e.g. Linux, OpenMP, Kubernetes) may perform dynamic task assignment to better balance load, to facilitate system maintenance, or to perform efficient packing of jobs. Dynamic scheduling often requires careful attention to task migration, but is easier in the case of \u0026ldquo;Map\u0026rdquo; tasks which are independent. See work stealing for additional discussion. Collective vs Non-Collective \u0026ndash; do all workers obtain the results of the other workers after the map is completed (i.e. Collective) or not (i.e. non-Collective)? A non-collective implementation can be made to be collective by adding a collective operation (e.g. gather or all-gather) after the map; collective implementations can be easier to reason about. See collectives. Returning void \u0026ndash; sometimes the user does not care about the return value of the function only that the side effects of the function occur eventually (e.g. Providing a progress notification to the user, or reporting some metrics to metrics collection endpoint). These are sometimes called \u0026ldquo;fire-and-forget\u0026rdquo; functions or \u0026ldquo;detached\u0026rdquo; functions. In practice, I find the true use cases for this to be rare; you almost always want to know if something failed unexpectedly and be able to handle that condition or to ensure that something eventually completes (e.g. asynchronous checkpointing). Instead prefer queuing style systems where error conditions and progress can be handled in a structured way. Preserving the ordering to tasks \u0026ndash; if tasks are provided in the order from 1..N, are they returned in the same order or some other order. Returning tasks in a different order (e.g. In the order of completion) may reduce synchronization between tasks, but introduces synchronization where the elements are returned as tasks compete to be the next value to return. lazy vs eager \u0026ndash; are the tasks launched as soon as the function and returned synchronously (e.g. Vector\u0026lt;T\u0026gt;) is called or are they called later either all at once (e.g. Future\u0026lt;Vector\u0026lt;T\u0026gt;\u0026gt;) or batch by batch (e.g. Generator\u0026lt;T\u0026gt; or Generator\u0026lt;Vector\u0026lt;T\u0026gt;\u0026gt;). Lazy execution can be more resource efficient if not all tasks are required or can reduce the operating memory overhead by not materializing the entire structure in memory. Static or dynamic queue contents \u0026ndash; are all tasks known at the beginning of the execution of the first task, or are tasks added to the queue as the workflow progresses? Static queue contents may allow for more detailed and optimal scheduling decisions than in the dynamic case. If tasks are added to the queue, how are they scheduled relative to existing tasks? See resource management for additional discussion. Cancellation \u0026ndash; At which points can tasks be canceled? Popular choices include never unless the program is killed (e.g. CUDA), prior to task execution, at dedicated points during task execution (e.g. Green threads, pthread_cancel with deferred cancellation, see cooperative scheduling), or at any point (e.g. Linux threads, most HPC schedulers see task preemption). Allowing cancellation may requires additional synchronization or overhead compared to not allowing cancelation, but allows tasks that are no longer needed to be discarded. Error Handling \u0026ndash; what happens if an error occurs during the execution of a task? Does the entire program attempt to terminate (e.g. MPI_ERRORS_FATAL), does the task complete with an exception, does the task complete and the user is required to implement error handling? Like cancellation, this allows tasks to end earlier than expected, but unlike cancellation may effect fault tolerance scheduling \u0026ndash; discussed in more detail for scans and reduces which introduces task dependencies. Even in the case of maps, scheduling decisions can be made based on expected/allocated execution time, resource utilization and availability (e.g. requires a GPU but none is currently available), status (e.g. waiting for a file read), fairness (e.g. how to ensure the long tasks continue to make forward progress), priority, and in the case of real-time systems deadlines. Known uses # the Map in MapReduce in Hadoop #pragma omp parallel for in OpenMP Kokkos::parallel_for in Kokkos Job Arrays in OpenPBS/PBSPro/Slurm Kubernetes Jobs with multiple completions Scan/Reduce #Intent #This pattern combines multiple elements of a data structure into a single cumulative result. The result can represent a single element (a reduction), or a collection of elements (a scan).\nMotivation #Reductions and scans are foundational parallel programming patterns. They can be used in parallel and serial use cases where multiple element of a data structure need to be combined to produce a result.\nApplicability # As a fundamental primitive, it is applicable to almost all use cases where you need to combine the values of multiple elements of a data structure such as filter operations and aggregations (e.g. count, mean, etc\u0026hellip;) While it is possible to implement map using a scan, this will likely introduce undue synchronization overhead. Structure #Participants/Elements # tasks \u0026ndash; what work needs to be performed \u0026ldquo;the zero/identity element\u0026rdquo; \u0026ndash; what is the initial state of the reduction, or alternatively how is the inital state created? the binary operation \u0026ndash; how are elements of the data structure combined? Is the operation associative, or effectful? workers \u0026ndash; where the work of the reduction preformed scheduler \u0026ndash; how tasks are mapped to workers Collaboration with other patterns #Many of the same collaborations as Map. This list focuses on distinctive of Reduces/Scans.\nOperator Fusion and Reordering \u0026ndash; the ordering of reductions compared to other operations can have substantial effects on the amount of work to be performed Load balancing – reduces and scans are inherently less load balanced than map (i.e. the root of the reduction has more work to do than the leaves) so is a more important consideration. Hardware specialization – common and foundational operations are often implemented in hardware (e.g. Vector dot product) in a paradigm referred to as Single Instruction Multiple Data (SIMD). GPUs implement hierarchical primitives (e.g. Shuffles, warp/group primitives) that can also accelerate these functions. Synchronization patterns \u0026ndash; unlike map, synchronization is inherent to reductions/scans so careful attention to these patterns is critical for performance and correctness. Fault Tolerance \u0026ndash; Unlike map, there are more meaningful intermediate states that can potentially be checkpointed, and more complexity in recovering from failures because the loss of a node necessitates additional communication with the other nodes that would otherwise need to cooperate in a reduction/scan. Code Examples #Consequences of using pattern # parallel reductions can require less wallclock time by distributing the work to be performed over multiple processing elements. more-so than map, reductions/scans require synchronizations that introduce overhead when conducted in parallel. Implementation considerations # What kind of iterator model exists for the collection? For random access and continuious iterators parallelism is possible because multiple processors can \u0026ldquo;jump\u0026rdquo; to a portion of the collection for processing. For input, output, forward, and bidirectional iterators limited parallelism is possible without first converting to a random or continuous iterator because the inherently serial nature of advancing iterators one by one prohibits parallelism. Are you producing a single result or a collection of results? scans (scanl operation init collection :: Foldable f =\u0026gt; (b -\u0026gt; a -\u0026gt; b) -\u0026gt; b -\u0026gt; f a -\u0026gt; f b ) produce a collection of elements instead of a single element for example a prefix sum, a maximum value in each subtree, or a shortest path from one node to any node in a graph. reductions (foldl operation init collection :: Foldable f =\u0026gt; (b -\u0026gt; a -\u0026gt; b) -\u0026gt; b -\u0026gt; f a -\u0026gt; b) produce a single element such as a maximum, the final state of a state machine over a series of events, the depth of a tree, or the size of the largest connected component. There are also variants of these operations such as \u0026ldquo;Reduce by Key\u0026rdquo; which uses a fused scan with a reduce to process a group of elements with the same key together. There are other variants of reductions that take additional arguments to the reduction function (e.g. paramorphisms that pass the original data structure in addition to the reduction state, histomorphisms which can look at multiple preceding states) For more information refer to the paper \u0026ldquo;Fantastic Morphisms and Where to Find Them A Guide to Recursion Schemes\u0026rdquo;. What are the properties of the binary operation? magmas vs semi-groups and associativity. A magma is just a binary operation that peserves type. A semi-group godes further and requires that the operation be associative. Associativity is a key property in that it allows for operations to be grouped in an arbitary order which enables parallelism by allowing operations to be scattered to multiple parallel processing elements and performed in an arbitary order. Sometimes, it is possible to conduct these operations as if they are a true semi-group, but are not. For example floating point operations are not truely associative, but are approximately associative. By relaxing the constraint and allowing associtaivity, the computation can be performed faster, but at the cost of some accuracy. Further more a semi-group can be an Abelian semi-group further allowing communativity \u0026ndash; or reordering of operations. This enables a further set of performance improvements by allowing more arbitary work stealing which is useful in the case of load imbalanced problems. semi-groups vs monoids and the \u0026ldquo;zero\u0026rdquo;/\u0026ldquo;identity\u0026rdquo; element. A monoid is just a semi-group that has a \u0026ldquo;zero element\u0026rdquo; also known as an \u0026ldquo;identity element\u0026rdquo;. The identity element combined using the binary operation with any other element produces the other element unchanged. For addtion, this is 0; for multication this is 1. Monoids offer an advantage over semigroups for parallelism because multiple elements can be \u0026ldquo;lifted\u0026rdquo; into the monoid using the identity element allowing for multiple starting positions instead of a single element. In contrast, for semi-groups, the user needs to provide an explicit base case of the reduction or scan. monoids vs monads and effectful reductions. Monads are \u0026ldquo;programmable semi-colons\u0026rdquo; \u0026ndash; they describe how to take the value in a \u0026ldquo;wrapped\u0026rdquo; value and return a new \u0026ldquo;wrapped\u0026rdquo; value \u0026ndash; for example optionals and futures are monads. There are more abstact notions of monads such as IO in haskell that model side effects of calling functions (e.g. printing a value). If the order of the effects \u0026ldquo;matters\u0026rdquo; for the correctness of your program, this will limit parallelism. Just as you can sometimes \u0026ldquo;cheat\u0026rdquo; and ignore the ordering imposed by magmas you can do the same for monads (e.g. if two prints are out of order, that can be confusing for the debugging/monitoring of a program, but likely not impactful for correctness). The optimal implementation of a scan/reduce is effected by the programming model tree-based methods, 2 pass reduction. This pattern maps nicely onto many hardware types and is the principle way to implement distributed reductions. In this pattern, each iteration elements are paired up and passed to the binary function, and then the process is repeated taking the outputs of the pairs are passed to the as the inputs to the next iteration until only a single element remains. Some optimizations may implement this as a multi-level reduction the width of the nodes changes depending on their depth in the tree. To implement a scan using this design, a \u0026ldquo;upward\u0026rdquo; pass that computes the reduction of each subtree is computed, and then a \u0026ldquo;downward\u0026rdquo; pass computes the actual prefix values. decoupled lookback \u0026ndash; On CUDA and GPUs, scans can be efficiently implemented by leveraging the aspect of the hardware that threads are scheduled in a montonistically increasing manner. This means that the $i^{th}$ thread knows that at least threads $i-1$ have begun to execute (even if they haven\u0026rsquo;t finished). Decoupled lookback combines this with hardware atomic operations to mark reductions as pending, prefix available , or aggregate available. In the case that fewer than $2^62$ elements are processed, the atomic store of the status can be stored with the aggregate into a single 64 bit integer register atomically. For NPU and TPUs which feature hardware instructions for matrix-vector multiplication scans and reductions for certain datatypes can be implemented by multiplication by a specially designed matrix. For a scan, this is the upper triagular matrix of ones. For a reduction, it is the matrix where the matrix with the top row contains only ones. error handling and cancellation \u0026ndash; unlike maps, reductions and scans have more complex error handling because the operations that fail may be intermediate results from the scan or reduction instead of from the raw elements. In this case, it may be desirable for a reduction to accumulate the errors either in an aabitary order (see magmas vs semigroups) or in the order of the original elements. This also has implications for recovering from failures as resuming from a failure in the middle of a reduction requires additional effort and careful syncroinzation during recovery. See the discussion on checkpointing of directed acyclic graphs for additional discussion. Known Uses # MPI features Scan and Reduce collectives OpenMP features Reduce and now Scan intents for loops. Cuda\u0026rsquo;s CUB (now part of NCCL) impement device (and collective) reductions respectively In MapReduce/Hadoop, the Reducer implements a reduce by key. Functional programming languages such as Haskell have the most robust notions of reductions (called fold) and scans. ","date":"12 January 2018","permalink":"https://robertu94.github.io/learning/patterns/","section":"","summary":"\u003ch1 id=\"foundation-parallel-operations\" class=\"relative group\"\u003eFoundation Parallel Operations \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#foundation-parallel-operations\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h1\u003e\u003cp\u003eFundamental to developing high-performance software is having an understanding of the basics of parallel architecture.\u003c/p\u003e","title":"Learning to Learn: Software Patterns"},{"content":"From time to time I get questions about how I stay apprised on all of the topics that I know something about. The short answer is a lot of reading and listening.\nI make extensive use of RSS Feeds and Podcasts as part of my information intake process. In this post, I provide the list of principles that I use to manage the list of information sources that I use.\nChoosing new Podcasts and Feeds #The \u0026ldquo;Jones Theory\u0026rdquo; - named after an ex-board-game-podcaster, Cody Jones, who used it for managing large board game collections. The Jones theory is roughly: \u0026ldquo;you should have only one [podcast, source, or] game that fills a particular need.\u0026rdquo; For example, you probably only need one automotive Podcast if you are not a mechanic; therefore, you should pick one and listen to just that one. Otherwise, you are prone to have way more Podcasts than you can ever possibly listen to.\nHowever, some things like political news are more inherently biased. For these topics, I choose to have different \u0026ldquo;buckets\u0026rdquo; based around the prospective of the speaker (Conservative, Liberal, Libertarian, Democrat, Republican, etc\u0026hellip;) and their focus area (US News, Tech News, International News). I use this principle to decide both when to add a Podcast (what needs do I have that are not being filled), and to remove a Podcast (what needs are filled twice).\nWhen adding a new Podcast, listen to at least 3 non-holiday episodes before deciding to add a Podcast. Listening to 3 Podcasts ensures that you get a aggregate perspective on the topics they cover and the quality of their show. Many Podcasts have a year-end best of clip-show. I personally don\u0026rsquo;t enjoy these and just delete them. Listening to non-holiday episodes helps weed out duds like these. The one possible exception would be extremely long Podcasts, but that should be evaluated on a case by case basis.\nApply the \u0026ldquo;Jones Theory\u0026rdquo; to your intake areas, what is missing and what is over-represented? Keep a High Signal to Noise Ratio #I have found for me the higher the amount of noise in my Podcast/RSS feed, the less likely I am to use it. For that reason, I try to keep the signal to noise ratio as high as possible.\nIf you find yourself regularly skipping or avoiding a podcast, consider dropping it from your Podcast list. I don\u0026rsquo;t always like to do this, some Podcasts that I really enjoyed have had large sections of things I wasn\u0026rsquo;t interested in or have stooped to sub-par work. If you are really concerned about it, you can always try re-adding it in the future when your interests have changed.\nConsider how long a Podcast is and how often it releases to the value your derive from it. For example, there is a Podcast that I enjoy the content, but it releases 3 hours of content per weekday. That is simply too much for me on a topic that is only tangential to my interests. For that reason, I don\u0026rsquo;t subscribe to that Podcast. Instead, I occasionally \u0026ldquo;pull\u0026rdquo; specific Podcasts from its feed when I have the time. Choosing to \u0026ldquo;pull\u0026rdquo; rather than have the information \u0026ldquo;pushed\u0026rdquo; is one way to control these good, but overwhelming Podcasts.\nAnother factor to consider with Podcasts with revolving guests is what is the quality of the guests. Another Podcast I originally liked when listening to has revolving guests. However, I found that I only like three of their ten or so revolving guests. I found myself skipping their Podcast regularly so I dropped it.\nWhat is the signal to noise ratio on what you consume now? Are there better sources? Tools of the Trade #Choose a good Podcast Player and a good RSS Feed Reader. I use Overcast for podcasts and Feedly for RSS. In the past, I have found the built-in players to be more buggy than I like. Overcast and Feedly have both been bug-free and are easy to use. For Overcast, I really enjoy its playback control features. Feedly has really nice keyboard controls.\nI would also consider listening to Podcasts at 2x or 1.5x speed. Several Podcasters speak much slower than than average when recording so they are easier to understand. Speeding up the playback returns them to normal speed and enables you to listen to more podcasts per unit time.\nI hope you find this useful. Until next time!\n","date":"1 January 2018","permalink":"https://robertu94.github.io/learning/intake/","section":"","summary":"\u003cp\u003eFrom time to time I get questions about how I stay apprised on all of the topics that I know something about.  The short answer is a lot of reading and listening.\u003c/p\u003e","title":"Learning to Learn: Intake"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/podcasts/","section":"Tags","summary":"","title":"Podcasts"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/rss/","section":"Tags","summary":"","title":"RSS"},{"content":"Over the last two months, I spent a significant amount of time using Clang\u0026rsquo;s libtooling. Libtooling is a great way to quickly develop tools to analyze and modify large quantizes of C++. In this article, I share some lessons learned working with libtooling.\nBeware the Stability Guarantees. #The biggest problem with libtooling is that it has very few if any Stability guarantees. When I was learning libtooling, I watched Peter Goldsborough\u0026rsquo;s video excellent \u0026ldquo;clang-useful: Building useful tools with LLVM and clang for fun and profit\u0026rdquo;. The video was filmed in May of 2017, and by September 2017, the code samples no longer compiled on the latest Clang. The developers made one of the methods he overloaded non-virtual. It wasn\u0026rsquo;t super difficult to patch his code to make it compile, and Clang/LLVM explicitly warns that you should use libclang instead if you desire a stable interface.\nHowever, this makes it really difficult to learn the interfaces when they are constantly changing. Hopefully, after a few years, libtooling will become slightly more stable and adopt at least a 6 month stability guarantee, or perhaps some refactoring tools to update to the new interfaces such as what has suggested by Google\u0026rsquo;s project Abseil.\nLearning Libtooling is an Exercise in Code Reading #For the most part Clang, LLVM, and Libtooling codebases are paragons of exceptional code. They are clear, use meaningful names consistently, public interfaces are documented. However, there are few unclear passages of code that I stumbled across (Clang 6.0):\nclang::SourceManager has several methods for getting locations: such as getSpellingLoc, getImmediateSpellingLoc, getExpansionLoc, getFileLoc, PresumedLoc. However it is not immediately clear what the difference between them. RecursiveASTVisitor contains some macro-processing black magic to generate the stub methods for each AST class. While you would be hard pressed to write a simpler implementation, it makes to very hard to know that you have written correct code. Additionally they choose not to make these methods virtual (presumably for performance reasons), so you can\u0026rsquo;t even use override to check correctness. There is also some very high level documentation about the roughs aspects of libtooling usage. However, this documentation too leaves much to be desired:\nSome important features for writing tools aren\u0026rsquo;t discussed in the high level docs, such as SourceManager aren\u0026rsquo;t even mentioned. There isn\u0026rsquo;t even an attempt to describe the full grammar of C++. This makes it hard to determine if you have caught all the possible edges cases in the grammar which have a nasty habit of showing up large low-level codebases. The closest is the very useful, ASTMatchers Reference, but it still leaves a lot to figure out with the deep inheritance hierarchies in Clang. The best reference for learning libtooling is the clang-tools-extra repository. Unlike the myriad of other examples on the Internet, these get updated whenever the libtooling API changes because they are managed by the upstream release. Some of these tools are fairly complex, and perhaps aren\u0026rsquo;t great starting places. Another essential tool is clang-query which allows you to dump the AST of programs that you may encounter.\nThe LLVM Support Library is Great #For various reasons, I needed to compile and run my libtooling application on a CentOS 6 box. CentOS 6 uses GCC 4.4 which at the time of writing is ancient in terms of compilers (2009) before gcc (or libstdc++) had many C++11 features. Not only did it compile without user-visible ugly kludges, LLVM\u0026rsquo;s support library provides interfaces to provide useful C++ features like std::make_unique as llvm::make_unique even when std::make_unique isn\u0026rsquo;t available in your standard library which was useful when I went to use the code on an up-to-date Gentoo box with gcc 7.2.0.\nEven if you don\u0026rsquo;t have to support bizarre compiler versions, LLVM support library provides a large variety of useful data structures that I found useful such as llvm::BitVector\nFollow the Build Instructions #LLVM and Clang recommend building your clang/llvm tools inside of an LLVM build tree. To me, this feels kind of gross, requiring users to download clang/llvm sources to build their applications even if they are already installed on your box for using clang. However, from this project I found this is truly for the best. It is surprising how many different locations distributions install Clang and LLVM libraries. Sometimes they don\u0026rsquo;t even install clang and LLVM libraries in the same directories. While it is doable, I found it to be much more trouble than its worth. For example, libtooling needs to lookup paths to clang/llvm headers during execution, it references them with a relative path installed at compile time. If the build is done out of source, you have to add a symlink so that your application can find the libraries. While you could just patch libtooling to respect the system location, I found this to be surprising difficult to locate where this setting was made, and gave up after two hours of searching. Just build your applications in tree, you\u0026rsquo;ll be much happier.\nConclusion #All and all, I found libtooling to be very useful and despite the faults listed above reasonably easy to learn and use. I reccomend that you take a look at it if you need to write tools for C++. I hope you found this helpful. Until next time happy programming!\n","date":"10 December 2017","permalink":"https://robertu94.github.io/posts/2017-12-10-life-with-libtooling/","section":"Posts","summary":"\u003cp\u003eOver the last two months, I spent a significant amount of time using Clang\u0026rsquo;s libtooling.\nLibtooling is a great way to quickly develop tools to analyze and modify large quantizes of C++.\nIn this article, I share some lessons learned working with libtooling.\u003c/p\u003e","title":"Life with Libtooling"},{"content":"Ever notice that every matrix and graph library seems to have a different interface for constructing matrices? Also notice that each only only supports some subset of common matrix formats? With a little help from the Adapter and Builder design patterns we can actually solve this problem.\nDesign Overview # In this design, we have 2 main actors: Parser andBuilder as well as their implementations ParserImpl and BuilderImpl. It allows us to write code like this in c++:\n#include \u0026lt;fstream\u0026gt; #include \u0026lt;matrixloader/parser/matrixmarket.hpp\u0026gt; #include \u0026lt;matrixloader/builder/boostgraph.hpp\u0026gt; int main(int argc, char *argv[]) { using namespace matrixloader; typedef BoostGraphBuilder\u0026lt;double\u0026gt; bgb; std::ifstream matrix_file{argv[1]}; auto builder = std::make_unique\u0026lt;bgb\u0026gt;(); auto parser = std::make_unique\u0026lt;MatrixMarketParser\u0026lt;bgb\u0026gt;\u0026gt;(std::move(builder)); auto matrix = parser.load(matrix_file); return 0; } I have chosen to mock up the interface in C++ because of its expressiveness with generic types. I have also created an implementation of the library in python and as a header only library in C++.\nBuilder Interface #The Builder class is an implementation of the Builder pattern so that Parser classes have a common interface to use to construct matrices and graphs. The Builder class is an abstract generic class that provides three methods: add_edge, reserve and build, and require 2 types: Value and Matrix.\nThe Value type represents the type of a particular edge weight. The Matrix type represents the graph or matrix of Values. Often these types vary jointly, but are, in fact, different. For example double is not viable where a vector\u0026lt;dobule\u0026gt; is. Additionally it is not always possible to extract the subtype from the collection like it is with vector\u0026lt;int\u0026gt;::value_type. Sure, you could probably do some SFINAE or reflection nonsense to make a reasonable guess for most cases, but this is not always possible.\nreserve #reserve is used to provide hints about the size of the resulting matrix and takes 3 parameters rows, columns, and nonzeros.\nAn earlier version of this interface did not provide the nonzeros argument. It was added because the earlier interface was not suitable for providing accurate hints for sparse matrices which often expect the number of nonzero entries for efficient construction.\nNotice, the use of the word hints. The implementation is not required to honor the arguments to this function if not practical. Take for example the Graph class form python\u0026rsquo;s networkx package, it provides no means to indicate how many nodes or edges \u0026ndash; let alone nonzeros \u0026ndash; to expect. Therefore its implementation of Builder has a no-op for this method.\nadd_edge #add_edge is used to indicate a value of an edge. This function unconditionally requires an edge value because in the event that weights are not meaningful, the parser implantation can simply pass 1 to create an adjacency matrix.\nbuild #build constructs or returns the final matrix. Some libraries like networkx do not provide batch more efficient batch construction methods, whereas libraries like scipy.sparse are not efficient without them. Therefore the implementation could choose to construct the graph/matrix representation when the constructor is called, or wait until build is called to finally construct the matrix. For this reason, calling build more than one time may cause undefined behavior.\nBuilderImpl #The BuilderImpl is simply an Adapter to the specific matrix interface of the matrix construction methods. It exists so that we have a common interface for matrix construction.\nParser Design #Parser is an abstract generic class that provides one method Builder::Matrix parse(std::istream\u0026amp;). It is generic on only one type Builder which is expected to conform the Interface Builder. For languages that do not allow for type alias to be members of a class (IE Java), It should be generic on two types Builder and Matrix where Matrix is the Matrix type passed to Builder.\nThe reason for taking an std::istream over something more common like a std::string or a const char * is threefold:\nThere is a class called std::istringstream that provides an adapter from these types to std::istream The most common use case for this use case is loading from a file. For several reasons we may not read the whole file and thus do not need to load it all into memory. A previous version of the this interface accepted std::string by line and return void (requiring a call to build on Builder), but this was changed for three reasons:\nIt avoids excess code on the common case by wrapping it. Some matrix file formats are not newline delimited making an awkward interface to implement for binary files. It reduces the number of objects that need to be passed around. ParserImpl #ParserImpl simply implements the Parser interface for the format desired such as MatrixMarket or EdgeList in terms of contained the Builder interface by reading the file.\nConclusion #While by no means is this a definitive interface for matrix construction, I think it serves a good start. Well until next time happy coding!\n","date":"23 September 2017","permalink":"https://robertu94.github.io/posts/2017-09-23-matrixloading-library/","section":"Posts","summary":"\u003cp\u003eEver notice that every matrix and graph library seems to have a different interface for constructing matrices?\nAlso notice that each only only supports some subset of common matrix formats?\nWith a little help from the Adapter and Builder design patterns we can actually solve this problem.\u003c/p\u003e","title":"Design of A Matrix loading Library"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/graph/","section":"Tags","summary":"","title":"Graph"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/matrix/","section":"Tags","summary":"","title":"Matrix"},{"content":" C++ Templates Staring into the abyss\nRobert Underwood\nOverview Why templates? Basics Template Methods Templates with Polymorphism Classes vs Functions Applications Trait Specialization and Policies Template Meta-programming Tuples Functors Other Uses Warning Here be dragons!\nWhy templates? Generics – code that accepts any type Performance – type specific implementation Code Reuse – let the compile do the writing Basics What are they? Allows you to: delay binding of type to variables write generic code of some type T trades code size for binary size (and readability?) Template Methods Will deduce but not coerce types They can be overloaded Classes vs Functions Functions have type deduction Classes/Structs have partial specialization At least for now… Templates with Polymorphism If possible, Put only template methods in a template class Each expansion takes space in the binary Each expansion increases compile time Variadic Templates Templates of many arguments Use overloading to handle recursion SFINAE Substitution Failure is not an Error\nA blessing and a curse Almost incomprehensible error messages Allows overloading and “compile-time reflection” Reflection – change behavior on class attribute/methods Applications Trait Specialization and Policies Select an algorithm/behavior through: reflection on types template arguments Implementations vary SFINAE-based – more flexible, harder to implement Specialization-base – less flexible, almost trivial Template Meta-programming Solve a recursive problem at compile time often inline, beware template recursion depth Tuples collections of types very similar to a struct can be expanded and passed as function arguments can be used to extract variables via std::tie Functors Accept a function as argument Write functional paradigm code Easier to parallelize Easier to reason about Other Uses Smart Pointers Iterator Generics Curiously Recurring Template Pattern Parameterized Type Attributes Generic implementations of Design Patterns Further Reading Many of concepts are found C++ Templates: The Complete Guide Modern C++ Design Questions Send us feedback at rr.underwood94@gmail.com!\nThis material available under CC By-SA 4.0\n","date":"1 August 2017","permalink":"https://robertu94.github.io/presentations/templates/","section":"","summary":"C++ templates are one of the most powerful, and probably least understood aspects of C++ programming. Trying to understand error messages from templates code can feel like staring into the abyss. In this talk, I introduce many of the various features and dive into applications of C++ templates that are used in the wild. I will cover several of the common pitfalls and design patterns that result from this feature of C++. I will also stare right into the heart of the abyss and explain why the error messages can be cryptic, why its a good thing, and how to begin to decipher those error messages.","title":"C++ Templates: Starring into the Abyss"},{"content":"When most people think of Qt, I imagine that they think about the Graphical User Interface components. But Qt has a variety of other components beyond just being a GUI framework. In this post, I highlight some of what I find to be the more interesting features.\nObject Communication via Signals and Slots #One of the coolest features of Qt is its very clean implementation of signals and slots. Signals and slots are a means of communicating information (called signals) between objects via special callbacks (called slots). For example, a class may define a signal for when its state has changed. Other classes may then connect() to that signal to indicate their interest in the event. In c++11 or later, you can also connect() a lambda function that will be called when the event occurs.\nSo what can you use this for? The clearest example is the implementing pub-sub pattern (also known as the Observer pattern). However unlike traditional Object Oriented observer which requires coupling between the Subject and the Observer (generally in the Subject having a list of Observers or calling a method on the Observer) The signals and slots implementation need not have this coupling. All that is required is that the signature of the slot accepts at least all of the connected signal. This allows for a very robust framework of communication that makes it easy to implement a variety of different class frameworks.\nClass Meta-Data via Properties #Classes that inherit from QObject and use the Q_OBJECT macro may use also the Q_PROPERTY macro. Q_PROPERTY provides a set of neat features over standard c++ member variables variables.\nFirst, they are observable. The user can define a NOTIFY function name that will be called whenever the value is changed. Other classes can then connect to this \u0026ldquo;signal\u0026rdquo; to respond to changes in the value.\nSecond, they support reflection. Qt registers each of the property values exposed in a class with the meta-object system. This allows the user to ask the meta-object what properties are present on an object and make decisions appropriately.\nThirdly, properties can be dynamic, and additional properties can be added to a class instance at run-time. This allows the application to add meta data about an object as needed.\nFinally, they expose the value to Qt\u0026rsquo;s other framework components such as the scripting system or gui framework.\nEven more features #Its hard to cover so many features in such a short post, but here are a few more features that peaked my interest. QtConcurrent provides a non-c++11 implementation of parallel std::transform, std::reduce, and futures called mapped(), mappedReduced(), and QFuture respectively which as of the time of writing is not yet fully implemented in gnu\u0026rsquo;s libstdc++ or clang\u0026rsquo;s libc++ standard libraries. Qt provides versions of the standard data types that are inherently serialize able allowing them to be easily marshalled out to disk or across the network. Qt provides a robust state machine framework which can be used to implement the state pattern, and it can transition from signals to make transitions easy. Qt also provides an event system to allow applications to listen to a variety of application inputs.\nWhat black magic is this? #Most these features are made possible by Qt\u0026rsquo;s meta-object compiler (moc). The Moc uses a combination of special templates, macros, and other special compiler directives that Qt uses to generate a c++ file to implement these features. This makes it such that 1) you don\u0026rsquo;t have to write quite as many trivial methods, 2) it works even on really old/bizarre compilers, 3) it takes very little source code.\nQt has a bunch of useful features beyond standard C++. Check them out, you are bound to learn something.\n","date":"23 May 2017","permalink":"https://robertu94.github.io/posts/2017-05-23-qt-as-library/","section":"Posts","summary":"\u003cp\u003eWhen most people think of Qt, I imagine that they think about the Graphical User Interface components.\nBut Qt has a variety of other components beyond just being a GUI framework.\nIn this post, I highlight some of what I find to be the more interesting features.\u003c/p\u003e","title":"Qt is for more than just GUIs"},{"content":"Authentication and authorization is one of foundational aspects of any security system. However writing an authentication and authorization system can be anything but: There are complex, ever-changing requirements, not to mention differences for differing interfaces it can quickly become daunting. However, there already exists a system on Linux and Unix that allows for dynamic and complex authentication: PAM.\nPAM Modules and Authentication Types #Fundamentally, PAM is a collection of modules that provide several methods. These methods provide: authentication (auth: pam_authenticate), account management (account pam_acct_mgmt), session management (session pam_open_session/ pam_close_session), and password/token management (password pam_chauthtok). For each service the administrator provides a \u0026ldquo;chain\u0026rdquo; of modules that will be used for each method. The admin can specify if a module is required (must succeed for the chain to succeed), sufficient (alone is enough to succeed), or optional (must be run but need not succeed).\nThere are already PAM authentication modules for a variety of services including:\nTraditional UNIX passwd Biometrics such as fingerprint readers Time based passwords such as Google Authenticator Enterprise authentication systems such as LDAP or Kerberos SQL/NoSQL databases The session modules allow developers to preform actions when a users opens a session on their service. Some examples include:\nmount the users home directory check if the user has mail print a message of the day Password modules allow the user to update their login credentials and are generally paired with an authentication module. Finally, account modules determine if a user is authorized to view specific information. Some existing modules include:\nTime of day restrictions location restrictions Additionally developers can easily write their own modules if the needed modules do not yet exist.\nPAM with Different Services #But the user must be thinking, what if my application involves multiple kinds of information with differing requirements. The command to start a PAM connection takes a service identifier as an argument. PAM uses this information to query the file system for a configuration file that specifies the chain to be used for this service. Additionally PAM provides a set of inheritance like facilities to allow applications to inherit login permissions from existing applications/configurations.\nWhat about non-terminal applications? #You may be thinking that this sounds great, but doesn\u0026rsquo;t account for graphical or web applications. Well the PAM authors thought of that too. All communication with the user is done via a communication callback function. If you are using a graphical application, use a call back that presents message boxes (or similar) to the user. If you are using a web application, use a callback that returns a web form to the user. It is a surprisingly robust tool.\nHope this excites you to read more. Happy Programming!\n","date":"14 May 2017","permalink":"https://robertu94.github.io/posts/2017-05-14-pluggable-authentication/","section":"Posts","summary":"\u003cp\u003eAuthentication and authorization is one of foundational aspects of any security system.\nHowever writing an authentication and authorization system can be anything but:\nThere are complex, ever-changing requirements, not to mention differences for differing interfaces it can quickly become daunting.\nHowever, there already exists a system on Linux and Unix that allows for dynamic and complex authentication: PAM.\u003c/p\u003e","title":"Pluggable Authentication With PAM"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/security/","section":"Tags","summary":"","title":"Security"},{"content":"Recently, I was working on a project for 2D Game Development where I had to use SDL 2.0. SDL 2.0 is a family of media libraries designed for writing cross platform games in C. However it can be difficult to remember where various resources are allocated and deallocated. Resource Acquisition is Initialization (RAII) is a common pattern in C++ programming that solves this problem. So I wrote a series of wrappers for SDL 2.0 that use RAII and various other improvements.\nRAII and std::shared_ptr #One of the key reasons for the wrapper was to take advantage of RAII. It is fairly common in SDL to have code like so (constants provided for clarity):\nSDL_Init(SDL_INIT_VIDEO); SDL_Window* window = SDL_CreateWindow(\u0026#34;animation\u0026#34;, posX, posY, WIDTH, HEIGHT, 0) SDL_Renderer* renderer = SDL_CreateRenderer(window, FIRST_WINDOW, SDL_RENDERER_PRESENTVSYNC); /* use pointers */ SDL_DestroyRenderer(renderer); //order matters to prevent memory leaks SDL_DestroyWindow(window); SDL_Quit(); In my library it looks more like this:\nauto lib = std::make_shared\u0026lt;SDLInit\u0026gt;(SDL_INIT_VIDEO); auto window = std::make_shared\u0026lt;SDLWindow\u0026gt;(\u0026#34;animation\u0026#34;, posX, posY, WIDTH, HEIGHT, 0); auto renderer = std::make_shared\u0026lt;SDLRenderer\u0026gt;(window, FIRST_WINDOW, SDL_RENDERER_PRESENTVSYNC); /* use pointers */ Here RAII and shared pointers in each of the classes ensure that each relevant class initialized correctly prior to use. And also the reference counting of std::shared_ptr ensures that they are freed when no longer used in an order that prevents leaks.\nSDLRect #One of the core data classes used in SDL is the SDL_Rect. SDL provides several methods that all operate on SDL_Rects. Providing them as methods on the class greatly reduced duplication and simplified user code. Additionally, std::make_{shared,unique} don\u0026rsquo;t understand struct aggregate initialization. So I grouped them the constructors and methods into a class and provided constructors\nstruct SDLRect: public SDL_Rect { SDLRect(); //aggregate initialization and make_{shared,unique} don\u0026#39;t mix; provide a constructor SDLRect(int x, int y, int w, int h);\tSDLRect(SDL_Rect const\u0026amp;);\tSDLRect intersectionRect(SDL_Rect const\u0026amp; rhs) const; SDLRect unionRect(SDL_Rect const\u0026amp; rhs) const; double dist(SDL_Rect const\u0026amp; rhs) const; bool encloses(SDL_Rect const\u0026amp; rhs) const; double angle(SDL_Rect const\u0026amp; pos) const; bool empty() const; bool hasIntersection(SDL_Rect const\u0026amp; rhs) const; bool operator==(SDL_Rect const\u0026amp;) const; }; Also for those unfamiliar with c++, structs are just classes with a default public access permission. I choose to make SDLRect a struct to preserve API compatibility with the SDL_Rect used extensively in SDL.\nNow before you run away screaming, \u0026ldquo;But SDL_Rect has a non-virtual destructor!\u0026rdquo;. Yes it does, but in this case, the subclass allocates no additional memory from that of the base class. So this code does not contain a memory leak.\nExceptions for Error Handling #On of the other major design decisions that I made in my wrapper was to replace error code checking with exceptions. The basis of this effort was creating a class called SDLException.\nclass SDLException: public std::exception { public: SDLException(); virtual ~SDLException() noexcept; virtual const char * what() const noexcept=0; }; std::ostream \u0026amp; operator\u0026lt;\u0026lt; (std::ostream \u0026amp; out, const SDLException \u0026amp; e) { return out \u0026lt;\u0026lt; e.what() \u0026lt;\u0026lt; \u0026#34; \u0026#34; \u0026lt;\u0026lt; SDL_GetError(); } Then for each type of failure in the SDLWrappers library, I created a subclass that printed a better error in context of what failed. Some would argue that using exceptions introduce additional overhead because the exception must be checked. However, if you create the equivalent code C style code that checks the error messages, I\u0026rsquo;ve found there isn\u0026rsquo;t much difference. In addition, you written 100s of lines less code by using exceptions.\nWell that wraps up this post. Happy programming!\n","date":"8 May 2017","permalink":"https://robertu94.github.io/posts/2017-05-08-sdl-raii/","section":"Posts","summary":"\u003cp\u003eRecently, I was working on a project for \u003ca href=\"brianmalloy.com\"\u003e2D Game Development\u003c/a\u003e where I had to use SDL 2.0.\nSDL 2.0 is a family of media libraries designed for writing cross platform games in C.\nHowever it can be difficult to remember where various resources are allocated and deallocated.\nResource Acquisition is Initialization (RAII) is a common pattern in C++ programming that solves this problem.\nSo I wrote a series of wrappers for SDL 2.0 that use RAII and various other improvements.\u003c/p\u003e","title":"Resource Acquisition is Initialization and SDL"},{"content":"Object Pools are a commonly used pattern used in operating systems, game, and high performance computing development. However just as it can be easy to forget to return a pointer to memory, it can be easy to forget to return the memory to the pool. In this article, I layout a class that I recently used to automatically manage memory from a pool.\nThe template pool class has 4 parts: an onEmpty policy, an allocation policy, a reset policy, and an object proxy. Let\u0026rsquo;s take these one at a time:\nAllocation Policy #The allocation policy is responsible for allocating (and possibly deallocating) instances of the class. It also provides some meta-data about the state of the allocated objects.\nHere is an implementation of an allocation policy that use perfect forwarding to a constructor: Other policies might call a factory method.\n#include \u0026lt;memory\u0026gt; template \u0026lt;class Value\u0026gt; class ForwardingAllocation { public: ForwardingAllocation(): allocated(0U) {} template \u0026lt;class... Args\u0026gt; std::make_unique\u0026lt;Value\u0026gt; Allocate(Args\u0026amp;\u0026amp;... args) { allocated++; return std::make_unique\u0026lt;Value\u0026gt;(std::forward\u0026lt;Args\u0026gt;(args)...); } void Deallocate(std::unique_ptr\u0026lt;Value\u0026gt;\u0026amp;\u0026amp;) { //intensional no-op unique_ptrs //free their own memory when they fall of of scope allocated--; } unsigned int getAllocated() const { return allocated; } protected: ~ForwardingAllocation()=default; private: unsigned int allocated; }; Reset Policy #The reset policy is responsible for resetting an object after it is allocated before it is returned This implementation uses perfect forwarding and in-place construction. Other policies might call a specific reset method.\ntemplate \u0026lt;class Value\u0026gt; class inPlaceReset { public: void Reset(Value* ptr, Args\u0026amp;\u0026amp;... args) { new (ptr) Value (std::forward\u0026lt;Args\u0026gt;(args)...); } protected: ~ForwardingAllocation()=default; } onEmpty Policy #Now to put these policies together. The onEmpty policy is responsible for what the pool should do when it exhausts all avail bile pointers. Here is an implementation of the onEmpty policy that allocates a single new object when the pool is empty.\ntemplate \u0026lt;class Value, class ListType, class AllocationPolicy = ForwardingAllocation\u0026lt;Value\u0026gt;, class ResetPolicy = inPlaceReset\u0026lt;Value\u0026gt; \u0026gt; class onEmptyLoad: public AllocationPolicy, public ResetPolicy { public: template\u0026lt;class... Args\u0026gt; void onEmpty(ListType\u0026amp; free_list, Args\u0026amp;\u0026amp;... args) { free_list.emplace_back( this-\u0026gt;Allocate(std::forward\u0026lt;Args\u0026gt;(args)...) ); } }; Object Proxy #The object proxy is where the magic happens. It is responsible for calling release on an object when it falls out of scope. I implemented it as a enclosed class in the pool.\nclass ProxyType { public: typedef std::shared_ptr\u0026lt;ObjectPool\u0026gt; Pool; ProxyType(Ptr\u0026amp;\u0026amp; ptr, Pool pool): ptr(std::move(ptr)), pool(pool) {} ProxyType(ProxyType\u0026amp;\u0026amp; rhs)=default; ProxyType\u0026amp; operator=(ProxyType\u0026amp;\u0026amp; rhs)=default; ProxyType(ProxyType\u0026amp;)=delete; ~ProxyType() { //don\u0026#39;t forget that moved from pointers also also destructed!! if(ptr.get() != nullptr) { pool-\u0026gt;release(std::move(ptr)); } } //expose pointer a la std::{shared,unique}_ptr Value* operator-\u0026gt;() const { return ptr.get(); } Value\u0026amp; operator*() const { return *ptr.get(); } Value* get() { return ptr.get(); } private: Ptr ptr; std::shared_ptr\u0026lt;ObjectPool\u0026gt; pool; }; Putting it all together #Now we simply need a class that puts all of this together. This class uses a free list but other implementations are possible.\ntemplate \u0026lt;class Value, template \u0026lt;class...\u0026gt; class EmptyPolicy = onEmptyLoad, template \u0026lt;class...\u0026gt; class ListType = std::vector \u0026gt; class ObjectPool: public std::enable_shared_from_this\u0026lt;ObjectPool\u0026lt;Value,EmptyPolicy,ListType\u0026gt;\u0026gt;, public EmptyPolicy\u0026lt;Value, ListType\u0026lt;std::unique_ptr\u0026lt;Value\u0026gt;\u0026gt;\u0026gt; { public: typedef std::unique_ptr\u0026lt;Value\u0026gt; Ptr; typedef ListType\u0026lt;Ptr\u0026gt; List; ObjectPool()=default; ObjectPool(List\u0026amp;\u0026amp; free): EmptyPolicy\u0026lt;Value, List\u0026gt;(free.size()), free(std::move(free)) {} class ProxyType {...}; template \u0026lt;class... Args\u0026gt; ProxyType request(Args\u0026amp;\u0026amp;... args) { Ptr back; if(free.empty()) { this-\u0026gt;onEmpty(free, std::forward\u0026lt;Args\u0026gt;(args)...); back = std::move(free.back()); free.pop_back(); } else { back = std::move(free.back()); this-\u0026gt;Reset(back.get(), std::forward\u0026lt;Args\u0026gt;(args)...); free.pop_back(); } return ProxyType(std::move(back), this-\u0026gt;shared_from_this()); } void release(Ptr\u0026amp;\u0026amp; ptr) { if(ptr.get()==nullptr) { throw std::logic_error(\u0026#34;ptr cannot be null\u0026#34;); } free.emplace_back(std::move(ptr)); } size_t getFree() const { return free.size(); } size_t getInUse() const { return this-\u0026gt;getAllocated() - free.size(); } private: ListType\u0026lt;std::unique_ptr\u0026lt;Value\u0026gt;\u0026gt; free; }; And that\u0026rsquo;s it; a pool that auto-magically manages memory using c++14. Happy programming!\n","date":"30 April 2017","permalink":"https://robertu94.github.io/posts/2017-04-29-smart-pool/","section":"Posts","summary":"\u003cp\u003eObject Pools are a commonly used pattern used in operating systems, game, and high performance computing development.\nHowever just as it can be easy to forget to return a pointer to memory, it can be easy to forget to return the memory to the pool.\nIn this article, I layout a class that I recently used to automatically manage memory from a pool.\u003c/p\u003e","title":"Smart Pool"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/templates/","section":"Tags","summary":"","title":"Templates"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/c++11/","section":"Tags","summary":"","title":"C++11"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/functional-programming/","section":"Tags","summary":"","title":"Functional Programming"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/shell/","section":"Tags","summary":"","title":"Shell"},{"content":"Functional programming is a surprisingly useful programming paradigm. The best aspects of functional programming have this odd habit of showing in places you would never expect.\nThe Shell and Endomorphisms #Arguably one of the most powerful features of the Unix shell is the pipe. It is one of the core building blocks of the Unix philosophy of many small tools working together each doing one thing well. However, long before Unix, the idea of the endomorphisms was developed. The endomorphisms are a cornerstone of functional programming and I would argue why the pipe is so powerful.\nA few rough definitions:\nA morphism is \u0026ldquo;A structure preserving map from one type to another\u0026rdquo; For example, functions are morphism. A endomorphism is a morphism for which the co-domain (or range) is the domain of the morphism. In functional programming, endomorphisms are useful because they are can be composed in arbitrary order. That is not to say that they are associative, only that the output of the function is well formed regardless of how they are composed.\nNow consider what a standard Unix process uses as input and output: They accept text as input. They and emit text as output. In other words, they are endomorphisms.\nWhich brings us to the pipe operator. Pipes take take the standard output of one processes and use it as the standard input to another process. So in essence, pipe is the composition operator of shell programming. With compositions and endomorphisms you will go far.\nof Optional types and Pointers #Pointers are the source of many a computer science student\u0026rsquo;s frustration. However, used wisely, they can be used to implement an moadic Optional type.\nTo be a monad, a type must:\nbe parameterized by a specific type have a unit function that converts an instance of the type to an instance of the monad have a bind operation that takes a instance of monad type and return another monad type A couple observations from the bowels of standards for C and C++:\nfree(nullptr); and delete nullptr; both perform no operation. NULL and its C++11 cousin nullptr evaluates to false, all other pointers evaluate to true. With these two facts we can write a type generic Optional monad (see this repository) for a full implementation).\n//in Optional.h class OptionalBase { // for type safety public: OptionalBase() = default; virtual ~OptionalBase() = default; }; template \u0026lt;class T\u0026gt; class Optional: public OptionalBase { public: Optional\u0026lt;T\u0026gt;() : OptionalBase(), managed(nullptr) {} Optional\u0026lt;T\u0026gt;(T v) : OptionalBase(), managed(new T(v)) {} Optional\u0026lt;T\u0026gt;(T* v) : OptionalBase(), managed((v)? new T(*v) : nullptr) {} Optional\u0026lt;T\u0026gt;(const Optional\u0026lt;T\u0026gt; \u0026amp;v) : OptionalBase(), managed((v.managed) ? new T(*v.managed) : nullptr) {} ~Optional() {delete managed;} Optional\u0026amp; operator= (const Optional\u0026amp; rhs) { if(\u0026amp;rhs == this) return *this; delete managed; managed = ((rhs.managed) ? (new T(rhs.managed)): (nullptr)); return *this; } bool operator==(const Optional\u0026lt;T\u0026gt; \u0026amp;rhs) { if (managed == nullptr \u0026amp;\u0026amp; rhs.managed == nullptr) return true; // empty == empty else if (managed == nullptr || rhs.managed == nullptr) return false; // empty != full else return *managed == *rhs.managed; } template \u0026lt;class Ret, class Args\u0026gt; Ret Bind(std::function\u0026lt;Ret(Args)\u0026gt; \u0026amp;f, typename std::enable_if\u0026lt; std::is_object\u0026lt;Ret\u0026gt;{} \u0026amp;\u0026amp; std::is_pointer\u0026lt;Args\u0026gt;{} \u0026amp;\u0026amp; std::is_convertible\u0026lt;T, typename std::remove_pointer\u0026lt;Args\u0026gt;::type\u0026gt;{} \u0026amp;\u0026amp; std::is_base_of\u0026lt;OptionalBase, Ret\u0026gt;{}\u0026gt;::type * = 0) const { return f(managed); } private: T* managed; }; And to show that it verifies the monad laws (which is more than you can say about Java\u0026rsquo;s implementation)\n#include \u0026lt;iomanip\u0026gt; #include \u0026lt;funtional\u0026gt; #include \u0026#34;Optional.h\u0026#34; int main() { std::function\u0026lt;Optional\u0026lt;int\u0026gt;(int*)\u0026gt; f = [](int* v){ if(v == nullptr) { return Optional\u0026lt;int\u0026gt;(-1); } else if (*v == 2) { return Optional\u0026lt;int\u0026gt;(); } else { return Optional\u0026lt;int\u0026gt;(*v + 1); } }; int * one = new int(1); int * two = new int(2); cout \u0026lt;\u0026lt; (Optional\u0026lt;int\u0026gt;{2}.Bind\u0026lt;int\u0026gt;(f) == f(two)) \u0026lt;\u0026lt; std::endl; cout \u0026lt;\u0026lt; (Optional\u0026lt;int\u0026gt;{1}.Bind\u0026lt;int\u0026gt;(f) == f(one)) \u0026lt;\u0026lt; std::endl; cout \u0026lt;\u0026lt; (Optional\u0026lt;int\u0026gt;{nullptr}.Bind\u0026lt;int\u0026gt;(f) == f(nullptr)) \u0026lt;\u0026lt; std::endl; //doesn\u0026#39;t work in java Option type } Happy programming!\n","date":"23 February 2017","permalink":"https://robertu94.github.io/posts/2017-02-17-suprisingly-funtional/","section":"Posts","summary":"\u003cp\u003eFunctional programming is a surprisingly useful programming paradigm.\nThe best aspects of functional programming have this odd habit of showing in places you would never expect.\u003c/p\u003e","title":"Surprisingly Functional"},{"content":" ","date":"1 February 2017","permalink":"https://robertu94.github.io/presentations/docker/","section":"","summary":"This talk introduces Docker and why you should care about it.  It highlights differences between Docker and VMs while clarifying common misconceptions.  I talk about how to create docker containers and containerize applications.","title":"Dockerize All the Things!"},{"content":"Ansible is probably my favorite provisioning and configuration management tool. Its syntax is concise, expressive, and elegant. Unlike other tools in its category, it has excellent documentation with working examples and intuitive naming. Learning it use it effectively can help you be a more productive developer.\nSpeeding Up Ansible #Anyone that has used ansible for more than a few hosts with more than a few tasks knows that by default it can be really slow. However with two simple options, you can dramatically improve the responsiveness of ansible playbooks without any other changes. For one set of playbooks that I wrote, these changes cut the run-time from an hour to 15 minutes.\nFirst is the number of forks. Forks controls how many hosts ansible will connect to at a time. So as not to overwhelm a small ansible master, the default is set to a measly 5 forks. Most of the time, the master is only responsible for doling out tasks to the slave nodes. And since most modern computers have at least 8 cores, you can easily increase this to a larger value. I use 25 on my 8 core machine.\nSecond, is ssh pipelining. Pipelining allows ansible, to reuse the same ssh connection for multiple actions which dramatically improves performance. However, it is not the default because there are a few old systems that do not support this configuration by default.\nTo enable both of these features, put the following in a file called ansible.cfg or in /etc/ansible/ansible.cfg\n[defaults] forks = 25 pipelining = True Don\u0026rsquo;t Repeat Yourself #DRY or Don\u0026rsquo;t Repeat Yourself is a cornerstone of software design. For most of Ansible 1.X, there were several examples of often repeated stanzas that would appear in Ansible playbooks: The two most common were tags and become that allow you to run some subset of a playbook and to execute with elevated privileges respectfully. Ansible 2.0 introduced the concept of blocks. Blocks allow you to apply common options to a set of commands. For example, in Ansible 1.X:\n- name: install vim dnf: name=\u0026#34;vim\u0026#34; state=\u0026#34;installed\u0026#34; become: True tags: - install - name: update all packages dnf: name=\u0026#39;*\u0026#39; state=latest become: True tags: - install Becomes:\n- block: - name: install vim dnf: name=\u0026#34;vim\u0026#34; state=\u0026#34;installed\u0026#34; - name: update all packages dnf: name=\u0026#39;*\u0026#39; state=latest become: True tags: - install Don\u0026rsquo;t Re-run The Whole Playbook #Suppose that almost all of your playbook runs correctly, but you hit the end, and there is one command that fails. Previously, you would either have to create a playbook with just that task/role, and re-run it or use tags for each action. Now you can simply use the --start-at-task=\u0026quot;\u0026lt;name\u0026gt;\u0026quot; option to re-run just the commands you are interested in. There is one slight issue: If the name that you want to start at is based on a section conditional using a custom fact module, you need to ensure that the custom fact is run first.\nDon\u0026rsquo;t Wait On a Slow Host #Another possible pain point for ansible is if one host in a set needs to do more work than the others. Ansible 2.X added the concept of strategies which control how a role is executed. One of these strategies is free which allows each host in the list to advances as through the play as quickly as possible. For plays that depend on other hosts in the managed group, be sure to use a wait_for command to ensure the servies you are expecting exists.\nHappy Programming!\n","date":"29 January 2017","permalink":"https://robertu94.github.io/posts/2017-01-29-faster-than-light/","section":"Posts","summary":"\u003cp\u003e\u003ca href=\"https://docs.ansible.com\" target=\"_blank\" rel=\"noreferrer\"\u003eAnsible\u003c/a\u003e is probably my favorite provisioning and configuration management tool.\nIts syntax is concise, expressive, and elegant.\nUnlike other tools in its category, it has \u003ca href=\"https://docs.ansible.com/ansible/playbooks_best_practices.html\" target=\"_blank\" rel=\"noreferrer\"\u003eexcellent documentation\u003c/a\u003e with working examples and intuitive naming.\nLearning it use it effectively can help you be a more productive developer.\u003c/p\u003e","title":"Faster than light"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/parallel/","section":"Tags","summary":"","title":"Parallel"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/c/c++/","section":"Tags","summary":"","title":"C/C++"},{"content":"C++ is a both a fantastic language and a mess. It supports at least 4 programming paradigms (procedural, functional, object-oriented, template meta-programming). In some senses, many languages give you one great way to do things: C++ gives you every way and trusts you to use them well. With this flexibility comes problems that C++ seems to have beyond what other languages experience. Therefore, having effective tooling to develop and use C++ is essential.\nThe LLVM project is was designed to be a very modular compiler infrastructure. While it was designed to write compilers, the LLVM project has facilitated a variety of code analysis and source-to-source transition tools.\nGetting started #Many LLVM based tools require a \u0026ldquo;compilation database\u0026rdquo; file in order to know what flags to pass to the compiler when extracting the abstract syntax tree. Generating this file by hand is a bit of a pain, but if you use CMake, it will generate the compilation database. Just add the following line to your CMakeLists.txt file\nset(CMAKE_EXPORT_COMPILE_COMMANDS ON) Code Quality Tools #Of the various tools included, those that warn about code quality problems are probably the most useful. The LLVM code quality warnings are divided into 3 types:\nErrors and Warnings emitted by the compiler: these are almost always not desired behavior. Warnings emitted by the static analyzer: these are more often not desired behavior or are less reliable checks that have a higher false positive rate than those in the compiler. Warnings emitted by the clang-tidy: these are more subjective tests such as those regarding to a style guide or those that discourage c-based constructs in favor of c++ equivalents. Since most are familiar with compiler warnings, I will focus on the latter two. The easiest way to run the static analyzer is via the scan-build utility: The scan-build utility instruments a build by modifying the CXX, CXXFLAGS, and other similar environment variables. One common stumbling block is that scan build will only instrument active builds. Therefore, if the file in question is \u0026ldquo;up to date\u0026rdquo; according to your build system, then scan build will not check that file. Just run a \u0026ldquo;clean\u0026rdquo; on the object files first:\nmake clean scan-build make Clang-tidy provides some additional advise about improving the quality of your code. Of particular use is the moderize-* checkers which advise on how to use c++11 constructs in existing code. Also useful are the cert-* checkers which look for common security vulnerabilities\n#list all avialible checks clang-tidy -checks=\u0026#34;*\u0026#34; -list-checks #run all checkers on all files clang-tidy -p /path/to/build/dir --checks=\u0026#34;*\u0026#34; *.cc #run just the moderize checks on all files clang-tidy -p /path/to/build/dir --checks=\u0026#34;moderize-*\u0026#34; *.cc It is also possible to write your own checks using AstMatchers\nOther Tools #At the time of writing, general refactoring tools in LLVM are still in their infancy. There are two that are considerable farther along:\nclang-rename \u0026ndash; rename a symbol from a fully qualified name or offset. Currently this tool only renames a symbols within a single translation unit which is useful enough, but leaves you wanting with large object oriented projects. clang-include-fixer \u0026ndash; add missing includes from within the projects. This project requires a yaml database in addition the compile.json file. This file can be created using the run-find-all-symbols.py (which on Gentoo is installed into /usr/share/clang/run-find-all-symbols.py). There are several other useful tools worth mentioning:\nclang-format a C++ code formatting tool. clang-query a tool to inspect the clang Ast that you can use in building a custom tool. lldb the llvm debugger which is considerably more script-able than GDB. There are too may tools to cover in a single post. Check out the clang docs for a more complete listing. For further reading, I recommend the work of Eli Bendersky who has written several posts on LLVM and clang.\nHappy programming!\n","date":"22 January 2017","permalink":"https://robertu94.github.io/posts/2017-01-22-llvm-tooling/","section":"Posts","summary":"\u003cp\u003eC++ is a both a fantastic language and a mess.\nIt supports at least 4 programming paradigms (procedural, functional, object-oriented, template meta-programming).\nIn some senses, many languages give you one great way to do things: C++ gives you every way and trusts you to use them well.\nWith this flexibility comes problems that C++ seems to have beyond what other languages experience.\nTherefore, having effective tooling to develop and use C++ is essential.\u003c/p\u003e","title":"LLVM Tooling for C++"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/testing/","section":"Tags","summary":"","title":"Testing"},{"content":"","date":null,"permalink":"https://robertu94.github.io/tags/flex/bison/","section":"Tags","summary":"","title":"Flex/Bison"},{"content":"The Interpreter pattern from the \u0026ldquo;Design Patterns: Elements of Reusable Object Oriented Software\u0026rdquo; can potentially be a very powerful pattern. It allows you to use a domain specific language to represent a complex computational situation. However, writing interpreters in practice can be tricky and time consuming. It really helps to know something about some fundamental parsing algorithms and techniques.\nThe most naive approach to writing an interpret involves manually matching each possible next phrase and creating an if else soup to match each possible outcome. Although, doing string parsing in C or even C++ can be a real pain point in implementing interpreters. Often you result to character by character comparisons using methods like std::string::compare or strcmp\nHowever, it is often not necessary to write these scanning and parsing routines by hand. One way to generate this boilerplate code is to use tools such as flex and bison. Flex is a scanner generator; it handles the job of finding tokens in a corpus. Bison is a parser generator; it handles the job of assembling tokens into a semantic tree. Unlike most unix tools, the full manuals for Flex and Bison are not contained in the man pages. Instead, you will need to use the info utility to read how to use these tools.\nBy default, these flex and bison generate code that is suitable for a standalone executable. However, it is possible to embed flex and bison as part of a larger program. Using this facility you can write an interpreter with ease. Here are a few things to note when embedding parsers:\nGetting Output #Normally, the entry point to bison parsers is the function int yyparse(void). This doesn\u0026rsquo;t provide a way to get access to data from the scanner or parser. But, if you use the %parse-param option in bison, you can change the arguments to yyparse to whatever you desire. Here, I get an integer via pointer to make the signature int yyparse(int * output):\n%parse-param { int * output } If you need to pass multiple arguments, you can either use a pointer to a struct, or you can pass multiple parameters like so: Note the second pair of braces, the effected functions are being generated like macros so you need both sets of braces.\n%parse-param { int * output } { void * more_output } Multiple Scanners and Parsers #If you have multiple scanners and parsers in a program, it becomes important to name space them. Especially, if you use you put a scanner in a library that others might use. You can namespace all external facing bison functions using the %define api.prefix option. Here I rename yyparse() and friends with names like ab_parse():\n%define api.prefix {ab_} Embedding a Scanner and Parser #Finally, here is an example of embedding a flex and bison scanner-parser into an application. Here I pass a socket as input to flex and bison:\nint client_fd = accept(...) /* arguments omitted for conciseness */ FILE * client_file = fdopen(client_fd, \u0026#34;r+\u0026#34;); if(!client_file){ perror(\u0026#34;failed to create file for socket\u0026#34;); exit(errno); } yyset_in(client_file); if(yyparse(\u0026amp;value)==0){ /* the parse succeded and value is valid */ } else { /* the parse failed */ } It is even possible to use flex and bison in threaded applications. Getting this setup can be tricky and nuanced, but once setup it is pretty easy. Once we have this, we have everything that we need to write a threaded library that uses an embedded flex scanner and bison parser.\nHappy coding!\n","date":"15 January 2017","permalink":"https://robertu94.github.io/posts/2017-01-15-interpretors-made-easy/","section":"Posts","summary":"\u003cp\u003eThe Interpreter pattern from the \u003ca href=\"https://en.wikipedia.org/wiki/Design_Patterns\" target=\"_blank\" rel=\"noreferrer\"\u003e\u0026ldquo;Design Patterns: Elements of Reusable Object Oriented Software\u0026rdquo;\u003c/a\u003e can potentially be a very powerful pattern.\nIt allows you to use a domain specific language to represent a complex computational situation.\nHowever, writing interpreters in practice can be tricky and time consuming.\nIt really helps to know something about some fundamental parsing algorithms and techniques.\u003c/p\u003e","title":"Interpreters Made Easy"},{"content":"I really like orchestration tools such as Ansible or SaltStack. They can make running tasks on a group of machines a breeze. But sometimes you can\u0026rsquo;t or don\u0026rsquo;t want to install these tools on a machine. In cases like these, it is helpful to know how to parallelize some tasks in the shell.\nYou can do this via Unix/shell job control:\ncmd=\u0026#34;systemctl enable --now docker.service\u0026#34; hosts=(host{1..4}) for host in ${hosts[@]} do ssh \u0026amp; $host $cmd done However from experience, this can be very error prone. For example, The placement of the \u0026amp; is important so as to background the ssh command and not the command on the remote machine. Additionally, what if you had a lot of hosts and you didn\u0026rsquo;t want to run all of them at once. Instead, you want to utilize a bounded pool of processes.\nThere are a few ways of doing this: most ways are messy or or fairly non-portable. On systems with the util-linux installed you might use flock or lockfile, but then you have essentially to implement semaphores using mutex locks and shell arithmetic. If you don\u0026rsquo;t have util-linux you can accomplish the same thing taking advantage of the atomicity of mkdir on most (but not all) file systems:\nHowever rather than doing this, take a look at the fairly pervasive xargs command. According the manpage, xargs \u0026ldquo;builds and executes command lines from standard input.\u0026rdquo; It has an option -P \u0026lt;NUM_PROCS\u0026gt; that takes determines how many commands to run in parallel. With this, it is just a matter of formatting commands in a way that xargs understands.\ncmd=\u0026#34;systemctl enable --now docker.service\u0026#34; hosts=(host{1..4}) numprocs=8 echo ${hosts[@]} | xargs -P $numprocs -d\u0026#34; \u0026#34; -I{} -n 1 ssh {} $cmd Admittedly this looks a bit cryptic. It helps to know that -d is setting the delimiter from the default newline to space, -n \u0026lt;NUM_ARGS\u0026gt; sets the number of arguments to pass to each command, and -I \u0026lt;REPLACE_STR\u0026gt; is setting the replacement string for xargs so that ssh {} $cmd becomes ssh host1 $cmd for the first command and so on. The xargs command also accepts an input file option (-a \u0026lt;file\u0026gt;) where we could put each host on a newline to simplify the call.\nNow we can easily create process pools in a mostly portable fashion in shell scripts. There are lots of useful things you could do with this, but here are two recipes that I came up with:\n#copy a file to many nodes function pcopy(){ filename=$1 dest=$2 shift 2 echo $* | xargs -d\u0026#34; \u0026#34; -P 8 -I{} -n 1 scp $filename {}:$dest } pcopy somefile.txt host{1..8} #retrieve files from many nodes function pget(){ filename=$1 shift echo $* | xargs -d\u0026#34; \u0026#34; -P 8 -I{} -n 1 scp {}:$filename $(basename $filename).{} } Happy shell scripting!\n","date":"8 January 2017","permalink":"https://robertu94.github.io/posts/2017-01-08-poor-mans-parallelism/","section":"Posts","summary":"\u003cp\u003eI really like orchestration tools such as \u003ca href=\"https://docs.ansible.com\" target=\"_blank\" rel=\"noreferrer\"\u003eAnsible\u003c/a\u003e or \u003ca href=\"https://docs.saltstack.com\" target=\"_blank\" rel=\"noreferrer\"\u003eSaltStack\u003c/a\u003e.\nThey can make running tasks on a group of machines a breeze.\nBut sometimes you can\u0026rsquo;t or don\u0026rsquo;t want to install these tools on a machine.\nIn cases like these, it is helpful to know how to parallelize some tasks in the shell.\u003c/p\u003e","title":"Poor Man's Parallelism"},{"content":" ","date":"1 October 2016","permalink":"https://robertu94.github.io/presentations/provisioning/","section":"","summary":"Ever sit down to setup a set of computers and wonder, “why on earth do I have to do this manually on each and every node?” Writing scripts can solve the problem, but handling error is annoying and time consuming. It does not have to be that way. Configuration management tools such as Ansible, SaltStack, and Puppet take the pain out of managing small to large groups of computers. By the end of the presentation, You’ll learn some of the strengths and weaknesses of each so that you can decide which tool will fit your needs.","title":"Provisioning at the Speed of Thought"},{"content":" ","date":"1 September 2016","permalink":"https://robertu94.github.io/presentations/exploitable_3/","section":"","summary":"This seminar provides and overview of reverse engineering and how you might use it as a developer.  I overview a variety of tools ranging from binary analysis, ptrace, kernel tracing tools, and kernel hacking and when to use each.","title":"Exploitable 3: Reverse Engineering"},{"content":" ","date":"1 August 2016","permalink":"https://robertu94.github.io/presentations/semantic/","section":"","summary":"Spaghetti code is a nightmare.  It is hard to read, harder to understand, and harder still to debug and change.  In this talk, I introduce the discipline of refactoring which transforms code in a way that maintains its external behavior while improving it\u0026rsquo;s internal structure.  I will present several examples of poor design often seen in imperative code, and describe some techniques to refactor this code  to easier to understand semantic code from Martin Fowler\u0026rsquo;s book \u0026ldquo;Refactoring: Improving the Design of Existing Software\u0026rdquo;.","title":"Writing Semantic Code: Using Refactoring Patterns for Better Code"},{"content":" ","date":"1 April 2016","permalink":"https://robertu94.github.io/presentations/autograding/","section":"","summary":"The purpose of this talk is to introduce how instructors could approach the topic of using automation in the classroom.  We will make a case for automation both from a student\u0026rsquo;s and instructor\u0026rsquo;s perspective by showing that automation can save time, encourage test driven development, and improve testing for knowledge.  We will also address concerns regarding grading automation including accuracy, confidentiality, and security.  Then we will get into detail about techniques and tools that can be used for classroom automation.  Finally we will examine case studies of how automation has and can be implemented.  We will examine how this has been done technically as well as part of the classroom environment by faculty members at the School of Computing.","title":"Automation in the Classroom"},{"content":" ","date":"1 April 2016","permalink":"https://robertu94.github.io/presentations/exploitable_2/","section":"","summary":"This talk introduces Docker and why you should care about it.  It highlights differences between Docker and VMs while clarifying common misconceptions.  I talk about how to create docker containers and containerize applications.","title":"Exploitable 2"},{"content":" ","date":"1 April 2016","permalink":"https://robertu94.github.io/presentations/python/","section":"","summary":"An introduction to idiomatic python programming for new computer science students.","title":"Python: A Parser Tongue Primer"},{"content":" ","date":"1 February 2016","permalink":"https://robertu94.github.io/presentations/linux/","section":"","summary":"Introduction to Linux for new computer science students.  Presented multiples times,","title":"Linux is Scary"},{"content":" ","date":"1 February 2016","permalink":"https://robertu94.github.io/presentations/projects/","section":"","summary":"A talk that provides an overview of how to approach computer science projects with less stress and effort.  This talk was given multiple times at various Clemson ACM venues.  Also titled, \u0026ldquo;Perfecting Your Projects\u0026rdquo;.","title":"Think Different"},{"content":" #Block Chinese address to ssh and web table \u0026lt;chuugoku\u0026gt; persist file \u0026quot;/etc/cn.zone\u0026quot; block in quick proto tcp from \u0026lt;chuugoku\u0026gt; to any port { 80 22 }\n","date":"1 February 2016","permalink":"https://robertu94.github.io/presentations/firewall/","section":"","summary":"This talks talks about what firewalls are, why you need one, and how to setup a basic firewall using iptables, firewalld, and pf.","title":"Thou Shall Not Pass! Introduction to Open Source Firewalls"},{"content":" ","date":"1 October 2015","permalink":"https://robertu94.github.io/presentations/exploitable/","section":"","summary":"This seminar talks about what is ethical hacking, how to learn how to do it, and how to set up a ethical hacking lab using Docker.","title":"Exploitable: An Introduction to Ethical Hacking"},{"content":" ","date":"1 October 2015","permalink":"https://robertu94.github.io/presentations/nmap/","section":"","summary":"This talk overviews NMAP a network mapping tool that is useful for understanding and monitoring your network.","title":"NMAP"},{"content":" ","date":"1 September 2015","permalink":"https://robertu94.github.io/presentations/git/","section":"","summary":"This talks overviews the git distributed version control system, what problems it solves, and how to use it.  This talk was presented multiple times also under the title \u0026ldquo;Git Thee to a Version Control System\u0026rdquo;","title":"Git Well Soon"},{"content":" \u0026lt;/code\u0026gt;\u0026lt;/pre\u0026gt; ","date":"1 March 2015","permalink":"https://robertu94.github.io/presentations/tools/","section":"","summary":"Overview of command line tools for POSIX platforms.  This talk covers some of the most useful POSIX scripting tools including awk, sed, gdb, find, and more.","title":"N Unix Tools in O(N ) Minutes"},{"content":" ","date":"1 February 2015","permalink":"https://robertu94.github.io/presentations/vim/","section":"","summary":"An intermediate level talk that aims to showcase neat features of vim above and beyond what is covered in vimtutor.","title":"Intermediate Vim"},{"content":"","date":null,"permalink":"https://robertu94.github.io/categories/","section":"Categories","summary":"","title":"Categories"}]