Try my estimation tool.

Quick History

At the time I created it I was struggling with the accuracy of the estimates I was doing. I was either underestimating because of optimism bias, or over estimating because I offset my bias too much.

The solution as startlingly effective. I used someone else’s ideas and built a basic tool and changed my behavior. That first version was enough to vastly improve my estimating ability, and give me a tool to help other team members think about their own. When I’ve been able to check estimate vs actual all but once I’ve been within on standard deviation of the mean (the “likely range” in that tool’s results).

A few years later I needed to estimate costs in addition to hours. So I created a second version which added cost projections to the process. That helped in a few scenarios, but honestly not as much as I’d hoped. Still, I learned stuff, therefore making it a useful exercise. Mostly it got to me fix some old bugs, and add scatter plot graphs to deal with complex outputs.

Those versions were driven purely by hour range estimates. They were extremely useful in helping me improve my estimates, if limited in estimating approach.

Time To Add New Approaches

Now I work on a team that estimates with Fibonacci story points and t-shirt sizing. I need my tool to support those approaches.

So over the last few weeks I’ve done more updates and upgrades. I now have a new version that will let you estimate using either of those techniques. It automatically converts those estimates to day ranges and runs the same simulations as it did before.

New Options

My experience is that teams using Fibonacci estimates, story points, or t-shirt sizing are usually estimating in days. Calendar time, or business days, are often what matters to product teams or in house teams.

Consultants track billable hours. In house teams hit a deadline.

To give users control over the conversion between the raw inputs and the day ranges the tool offers two modes. Which mode is best depends on what you’re doing, how you team works, and how well you track productivity now.

The first is a simple points to days conversion that takes maps the Fibonacci number to a number of days. It is labeled “Calendar Days” and lets users control the mapping. This approach is better for long-term estimation covering months or more.

The second mode is for teams that are good at measuring their velocity for agile methodologies. This mode let’s users specify a number of points per sprint, and the length of a sprint. It uses those factors to determine the number of points per day, and converts from there. This approach is better for shorter term estimate cover a few sprints.

Other Updates

While I was doing those updates I addressed a few other long standing issues. Older versions lacked feedback that a simulation was running. I also fixed several accessibility issues, improved test coverage, and added line item results summaries. The ReadMe file is now far more complete with better descriptions of process and more instructions.

Importantly, I changed the behavior for under-run estimates. Previously 50% of the time when the simulated value was outside the range provided it would pick a number below the specified range. Now that happens only 25% of the time, the other 75% generate numbers above the range. It also uses a calculation similar to the existing overestimate approach – the lower your confidence the wider a range it uses. This greatly reduces the number of 0 hour/day tasks in the simulations. It also changes the typical graph shape a bit to long-tail more on the high side. I think that is likely more realistic for most projects.

Welcome Our New AI Overlords

To accelerate the updates I used Copilot to help draft and review the code. AI coding still feels like cheating, but it is far faster. The code new is a bit sloppier in spots than I’d like and required several rounds of revisions at times. However, the AI added a lot of speed and reduced research level for new features. That saved time meant I added polishes I’ve been thinking about for years, but never took the time to do.

For example the tool now outlines scatter plots using a kernel density estimate curve. I added save buttons for the images, and there are print styles that make it easy to print into a PDF. Hopefully all those changes will make it easier to share the results. I only bothered to make those changes because AI made it easy to add them.

That process was aided by having had a reasonably good structure to the code I wrote in earlier editions. And I did check all the new work, but keeping tabs on 200 lines of new code from one prompt is a challenging exercise at times. The code base needs some additional refactors still to deal with some of the bloat it picked up. I don’t care that Claude and Copilot assure me all cleanup already been handled, the 2000 line index.js tells me otherwise.

You can find the full code my estimation tool on Github. If you have suggestions, please open an issue there and we can discuss it.

So, as a public service to the community, I provide here a list of common lies and a sentence or two about why you should stop telling them.

My code is self-documenting

Not once in my career have I heard a developer say this and found their code to be “self-documenting”. Never. Nope. (If we’re friends: yeah, not even yours.)

Developers who claim to write self-documenting code seem to solve problems in the least-obvious way available. Or at least it feels that way to me. Maybe if they left comments explaining why they used their approach it would seem brilliant. I don’t know; it’s never come up.

Self-documenting code doesn’t mean following a variable naming convention and having no comments at all. You still need doc comments. You still need good functional decomposition. You still need comments to say why your code does things. Sure, I can usually figure out what it does, but not why.

If you still want to argue with me make sure you read Chapter 15 of Mythical Man Month before calling.

See Also: Nobody else needs to understand this code; I’m the only one working on it

Just because you are the only one working on it right now, doesn’t mean that will be true 4 years from now. I recently had to update code that had been running happily without attention for 5 years. By the time I was working on it, everyone had forgotten it existed let alone how it worked.

And don’t forget you might be that person who needs to remember these details 5 years from now. Guess who I called first when we realized that code needed updating? Yup, the guy who wrote it. No, he didn’t remember why he had made important choices either.

The only time I am not disappointed in the person who committed code in my name 6 months ago is when he wrote me really good comments to explain what he was thinking. When he didn’t do that: that guy was an idiot.

Future you, your colleagues, friends, clients, strangers, they all will thank you for explaining yourself. Doesn’t that feel better than having them all hate you?

I don’t need to use Git

Yes. You. Do.

It’s 2025. Bell Labs invented source control in 1972. “I don’t know how to use it” isn’t a valid excuse. If you’ve been putting off learning to use these tools, it’s time you admit they aren’t going away.

Just because the code exists in a Salesforce sandbox org does not mean you don’t benefit from tracking its history, knowing who (including you) changed what when (and if they wrote good comments why!). The first time it saves you because another developer overwrote your code, it’ll have saved you all the time setup required and more.

I can setup a new repository, with Salesforce configuration, in 90 seconds. It takes 10 seconds to do a commit. Sure, you can have merge problems sometimes – that usually happens when two people make conflicting changes. When do you need source tracking the most? When two people make conflicting changes.

When you see two sandboxes with two different versions of code, how do you know which is right? On my projects it’s the one that matches Git, cause Git is always canonical.

There really is no good argument against putting every line of code into a repo. Just because you have an argument does not mean you have a good argument. It’ll take you less time to start using Git than it will take to argue with me.

See also: I’ll just comment out this section of code for now, and delete it later

No you won’t. Nope, stop kidding yourself. I’ve seen your 200 line blocks of commented out “stuff” (without notes about why it was there in the first place, or why it’s commented out). You want to keep it for 3 or 4 minutes while you test a change – okay sure. But this is part of what source control does for you. Deleted something you want back? Okay, go get it out of your source control system. Delete your block of commented out code before your next commit.

I will fix this hard-coded ID later

If it’s easy to fix, just do it now.

If it’s going to be a bunch of work later, you know as well as I do that you’ll find excuses not to bother at that point either.

Either write the query to get the Id you need now, or move it to custom metadata/setting/environment variable/whatever now. It’ll take probably take 5 minutes to do it right now, and 20 minutes to find the stupid thing later (and 10 minutes to fix it cause you’ll be tired from the 20 minute hunt).

My invocable action doesn’t need to be bulkified

Sure, damn near every example invocable action only processes the first record in the list. Yes, that encourages us to think it’s okay to write code that way. And yes, bulkifying code is hard sometimes.

You might think that your record triggered flow will only need to deal with one user-entered update at a time. And then someone loads a 100,000 records and your action breaks the whole process.

And sure, you always feel like your action is only getting used in that one Screen Flow. But 9 months from now when someone realizes they can get the inputs other ways and call your action with the screens and puts it in a record triggered flow your code will still work.

There exist settings where the screen flow thing is true. They are rare, but I’ve seen them. Use this measure to determine if it applies to your project: if you cannot write a rigorous proof which is good enough for your high school geometry/trig teacher (college CS or Math Professors will also count here) to accept showing this code will only ever be run in a one-record context – you better plan for 200 records.

I will convert this test setup to use a data factory when I have more tests

Why? Why would you think you will take the time to do this conversion later? When you arrive at “later” you will just need one more object added to your setup. Or maybe 2. And you can copy-paste it into the next test setup method. Then may add 2 more objects. And then you’ll just…

If you don’t lay down good patterns up front, you won’t convert to them later. Starting a test factory class doesn’t take a lot of time. Just do it now, and thank yourself later when you realize you’ve slowly built a factory that’s supporting 5 different test suites and hundreds of tests.

I’ll move my business logic out of the trigger once it’s working

Once your code is working you have almost no incentive to clean it up. Chances are you’ll just deploy your code and wash your hands of it. It wont be until you’re neck deep in a conflicting sets of requirements that you’ll decide it’s time to refactor this mess – and then you’ll wonder what each conditional was for in the first place (see self-documenting code above).

Even during a recent technical interview, I took the 15 seconds needed to create the trigger and have it call a service class. I didn’t plan to keep that code for more than 15 minutes and I still bothered. I really think it took 15 seconds. Bing, bang, done. Sorting it out later is so much harder then just doing it on the first pass.

As longs as I hit the test coverage requirement, my tests are good enough.

Automated test coverage is not about hitting a certain number of lines run – that’s the minimum threshold. Good automated tests ensure you code works correctly. Good test coverage now means that when you revise the code later you know if your revisions broke something.

• Do not play games with getting the coverage percentage up.
• Do not have tests with no assert statements.
• Do not decide that 75% is just fine because to get to 90% you’d have to write three more tests – write the three tests!
• Do not let test coverage meaningfully changes when someone auto-formats your code.

Write tests that ensure your code meets specifications, not tests that make the deployment process happy.

I don’t need automated code standards enforcement

Salesforce Code Analyzer is a free tool you can install right now and will help ensure you don’t do stupid stuff in your code. It will also make sure you team follows the same habits. These tools have been around for Salesforce developers for years, but for some reason we don’t all consistently turn them on.

This is standard fare for professional developers. What makes you special that you don’t need to follow standards? Why would you think you can enforce standards on a team without automation?

Every time I have to do a commit to reformat code to be consistent with all the other code on a project, a [insert sad thing here] happens. This pretty much only comes up with people who don’t have auto-formatting of their own. Their code had blends of tabs and spaces, different line endings within the file, off-by-one spacing error (dear God I hope no one uses 3 on purpose), and all kinds of things that cause other editors to sputter and complain.

If you don’t like Salesforce’s default analyzer rules, you can setup your own. It’s just enforcing ESLint and PMD patterns. I have extremely little interest in debating code standards – just have some!

Special Thanks For Shared Salesforce Developer Lies

The following people sent me lies to include in this list, and I appreciate their shared pain. My rants about each are mine alone.

• Peter Churchill
• Michelle Hansen
• Raul Roque

If you would like to join us please check out our up coming events on MeetUp for meeting times, locations, and remote connection information.

We frequently use these presentations to practice new presentations, try out heavily revised versions, and test out new ideas with a friendly audience. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback. If you want to see a polished version checkout our group members’ talks at camps and cons.

If you are interested in giving a practice talk, leave me a comment here, contact me through Drupal.org, or find me on Drupal Slack. We’re excited to hear new voices and ideas. We want to support the community, and that means you.

For data loading their official answer is typically to copy the database to another server, run your job there, and copy the database back onto their server. That’s fine if you can afford to freeze updates to your production site. And have the time to setup a process to mirror changes into your temporary copy. And can afford some additional project overhead. But sometimes those things are not an option. Or the data load takes too long, or happens too often, for that to be practical on a regular basis.

I recently needed to do a very large import of records into Drupal on a Pantheon hosted site. The minimize user impact I started to play around with solutions that would allow me to ignore those time limits. We were looking at dong about 50 million data writes for the project. When I first estimated the process the running time was over a week.

The Outline

Since Drupal’s batch system was created to solve this exact problem it seemed like a good place to start. For this solution you need a file you can load and parse in segments, like a CSV file, which you can read one line at a time. It does not have to represent the final state of your data. While you can actually load the data raw, you can also load each record into a table or a queue to process later.

One quick note about the code samples, I wrote these based on the service-based approach outlined in my post about batch services and the batch service module I discussed there. It could be adapted to a traditional batch job, but I like the clarity the wrapper provides for this discussion.

The general concept here is that we upload the file and then progressively process it from within a batch job. My code samples below provide two classes to achieve this. The first is a form that provides a managed file field which create a file entity that can be reliably passed to the batch processor. From there the batch service uses a bit of basic PHP file handling to copy data into the database. If you need to do more than load the data into the database directly (say create complex entities or other tasks) you can set up a second phase to run through the values to do that heavier lifting.

Load your file

To get us started the form includes this managed file:

   $form['file'] = [
     '#type' => 'managed_file',
     '#name' => 'data_file',
     '#title' => $this->t('Data file'),
     '#description' => $this->t('CSV format for this example.'),
     '#upload_location' => 'private://example_pantheon_loader_data/',
     '#upload_validators' => [
       'file_validate_extensions' => ['csv'],
     ],
   ];

The managed file form element automagically gives you a file entity. The value in the form state is the id of that entity. This file will be temporary and have no references once the process is complete and so depending on your site setup the file will eventually be purged. Which adds up to mean we can pass all the values straight through to our batch processor:

$batch = $this->dataLoaderBatchService->generateBatchJob($form_state->getValues());

If the data file is small, a few thousand rows at most, you can load it right away. But that runs into both time and memory concerns and the whole point of this is to avoid those. With my approach we can ignore those and we’re only limited by Pantheon’s upload file size. If the file size is too large for that you can upload the file via sftp and read from the file system, so you have options.

As we setup the file for processing in the batch job, we really need the file path not the ID. The main reason to use the managed file is they can reliably get the file path on a Pantheon server without us needing to know where they have things stashed. Since we’re about to use generic PHP functions for file processing we need to know that path reliably:

$fid = array_pop($data['file']);
$fileEntity = File::load($fid);
$ops = [];

if (empty($fileEntity)) {
  $this->logger->error('Unable to load file data for processing.');
  return [];
}
$filePath = $this->fileSystem->realpath($fileEntity->getFileUri());
$ops = ['processData' => [$filePath]];

Create your batch

Now we have a file know where it is. Since it’s a csv we can load a few rows at time, process them, and then loop back.

Our batch processing function needs to track two things in addition to the file: the header values and the current file position. So in the first pass we initialize the position to zero and then load the first row as the header. For every pass after that we need to find point we left off. For this we use generic PHP files for loading and seeking the current location:

// Old-school file handling.
$path = array_pop($data);
$file = fopen($path, "r");
...
fseek($file, $filePos);

// Each pass we process 100 lines, if you have to do something complex
// you might want to reduce the run.
for ($i = 0; $i < 100; $i++) {
  $row = fgetcsv($file);
  if (!empty($row)) {
    $data = array_combine($header, $row);
    $member['timestamp'] = time();
    $rowData = [
             'col_one' => $data['field_name'],
             'data' => serialize($data),
             'timestamp' => time(),
    ];
    $row_id = $this->database->insert('example_pantheon_loader_tracker')
             ->fields($rowData)
             ->execute();

    // If you're setting up for a queue you include something like this.
    // $queue = $this->queueFactory->get(‘example_pantheon_loader_remap’);
    // $queue->createItem($row_id);
 }
 else {
   break;
 }
}
$filePos = (float) ftell($file);
$context['finished'] = $filePos / filesize($path);

The example code just dumps this all into a database table. This can be useful as a raw data loader. If you need to add a large data set to an existing site that’s used for reference data or something similar.  It can also be used as the base to create more complex objects. The example code includes comments about generating a queue worker to run on cron or as another batch job. the Queue UI module provides a simple interface to run those on a batch job.

Final Considerations

I’ve run this process for several hours at a stretch.  Pantheon does have issues with systems errors if left to run a batch job for extreme runs. I ran into problems on some runs after 6-8 hours of run time. So a prep into the database followed by running on queue restart has been more reliable.

After several hours on hold, 5 or 6 visits (I’ve lost count), and them testing everything in sight, I finally started writing software to see if I can help find the problem.

At strange intervals there is a run of noise on the line that kills my service.  The noise eventually goes away and my service resumes. But figuring out those intervals has been a challenge since it turns out I don’t actually sit around 24 hours a day watching for service interruptions.

So after the last tech was unable to find a problem, and implied that it was my cable modem, I wrote a simple python script that pings Google every 5 seconds and records the response time (or failure). I setup an old Linux laptop to sit around and do nothing but ping Google 24 hours a day and record the results of each request.

I learned that my service interruptions tend to start and stop on the hour or half hour.  For example, the first night it started to be bad at noon and improved at 2:30 the next morning.  That pattern of human friendly times and data was enough to convince them to try again (although they aren’t willing to consider it’s something scheduled since they aren’t aware of any scheduled processes on their network).

So at the moment I have a loaner modem – which has ruled out my device as the problem – and they are closing on having replaced every piece of hardware between me and the nearest node.  To their credit, it has gotten much less bad.  Most outages now last five minutes (more or less exactly), and I have only have a few a day (instead of a few an hour).

Anyone have guesses about what processes run on cable networks that last five minutes and reattempt for a few hours at a stretch before giving up for roughly 13 or 14 hours?

My takeaway so far has been that if your internet provider is unable to figure out what’s wrong, offer them more data. Even if they don’t know how to use it, it them to take the problem seriously.

Drupal 8 provides many great improvements, but one I’ve been finding to be the most exciting is the easy ability to create my own plugins. They are actually a bit addictive, and it’s easy to start seeing projects as an excuse to create a new plugin manager. At first it can take a little bit of effort to get your head around the process, but I got great details from Unraveling the Drupal 8 Plugin System from Drupalize.me. But the problem with those, and all the examples I could find, was they were either too deep (the drupalize.me article should be considered required reading, but the example is too abstract for most people) or too shallow ( the drupal.org Plugin API examples currently lack context about why I would want to do this in practice).

So my goal is to straddle that gap.

What’s a plugin?

Plugins are a design pattern that let you extend a module or service. Drupal 8 has at least four plugin discovery systems:

• Annotated ← I’m mostly talking about this one, its used for blocks, views, rules, Queue API, and more.
• YAML ← These are used for menus, routes, and services.
• Hook ← D7 carry over hook system, still used in many places.
• Discovery Decorators ← Wrap around other plugin systems to replace/improve on the classic info alter hooks from previous Drupal versions.
• Static ← These are used for test code, and I’m ignoring here.

Basic Commonly Used Plugins

As a Drupal 8 developer you probably use several plugins when building a site, particularly if there is any custom functionality required. When creating controllers you’ll need a route, and likely a menu item, which means you’ll use YAML files to define those aspects.  For blocks, the queue api, or Rules you’ll create an annotated plugin.  Theme functions are still commonly defined with hook_theme().

Why do I create new plugin?

Between core, and major contrib modules (like Rules and Search API), plugins options abound, but it’s not always obvious why you might want to create one for a more focused project. For large projects that may have a large number of related functions, poorly defined use cases, or use cases that are expected to expand over time Plugins provide three major advantages over other approaches:

• Plugins make it easy to easily keep classes small and focused.
• Plugins make it easy to extend a service.
• Plugins allow one module to easily extend the function of another.

Actual Examples

Cyberwoven has a client who needed an internal training portal. They want to provide some basic gamification, particularly badges, and they wanted a more detailed tracking of user behavior than the core statistics module provides. The badges present a function that has many related, but different, use cases and is likely to grow over time. So I created a pluggable service that allows us to easily define a set of conditions for creating a new badge. Each plugin’s annotation lists the events it cares about, and when one of those events is fired the service runs the plugin’s test function. If the badge has been earned the plugin returns the information needed to generate that entity. All the event listening, entity generation, and error checking happens in the service, so each plugin is only focused on the details of the one badge. The event tracker is similar, with each plugin defining a filter for the major system events to determining what details should be recorded, and the service handling the logging of those details and reporting. As the client needs have evolved we’ve been able to add and modify the plugins to meet their needs.

The second place I’ve created a custom plugin system for was an internal project at Cyberwoven (it was actually the project I used to learn how to create custom plugins). Our testing server has always had a service to perform some basic operations on the server, like pulling repos via webhooks and similar tools to help developers. Setting up our development environments for D8 meant it was time for a new test server, and therefore new tools to manage it. From the start I wanted to create a tool that helped both developers and account managers. So I created a D8 site to manage the sites that provides various tools to perform task operations we all need (not only code handling, but also checking if security releases apply to any sites). All the various operations we want exposed through that site are created as plugins to our custom manager. Each task plugin is focused on just that one task. All the other various needed to track the sites, route requests to run a task, and render their responses are all handled in other places.  It’s also now making it easy to add features that wrap around the plugins since the plugins all have a consistent interface (like adding new displays and running some tasks through cron).

Introducing Site Manager

The main module defines an entity for each site, a couple displays (like the listing page displayed above), and the annotated plugin base so we can add new features easily. Plugins can be enabled and disabled on a site-by-site basis, and can be run from the plugins dropbutton.

Once we had the basics in place (like getting the git and drush status’s via simple plugins) we started to add more powerful features to turn the tool into a dashboard.  We added the module search block you see on the left as a separate module that searches all sites for a specific Drupal module (nice when security updates are released). The search module also provides a plugin for each site that lists the modules for just that site.

The last site in the screen shot has a reference to scanning the HTTPS certificate for a specific site.  This is another feature we added recently as this tool increasingly starts to serve as a dashboard. The HTTPS Scanner module again provides both a batch job to scan all sites, and a task plugin that scans just one.

As of this writing, all told there are nine working task plugins across four modules (there are a couple more underway as well). Tasks can also have their results displayed in a block when you look at the details for a site.

Building your own manager

Hopefully by now your convinced that it’s worth having your own plugin manager at least sometimes. So the next question is how to build them.

Elements of a custom plugin

Plugins have several parts, most of which are small and simple:

• Annotation Plugin Definition
• Plugin Manager Service
• Plugin Interface
• Plugin Base
• Example Implementation
• Controller (optional)

Annotation Plugin

The first step is to define the annotation for your plugin. This is a simple class that extends Drupal\Component\Annotation\Plugin to define the variables you want listed in the annotation comments of the plugin. This file goes in custom_module/src/Annotation, and as always the file name and class name match:

Plugin Manager Service

The plugin manager itself is a service, but the vast majority of its function comes from the parent classes.  The convention is to place this file in custom_module/src/Plugin (as will the rest of the base definitions we’re about to provide).  In this case we call it: TaskPluginManager.php.

The full version of this provides a few more improvements to this class (I overrode the getDefinitions() method to provide some filtering options to support disabling plugins), but those aren’t actually required for it to work well.

In short, this class just provides some general definitions to separate it from all the other annotated plugins.

And since the plugin manager is a service, we add it to our module’s service.yml file:

services:
   plugin.manager.task_plugin.processor:
     class: Drupal\sitemanager\Plugin\TaskPluginManager
     parent: default_plugin_manager

Plugin Interface

Next we define the plugin’s interface.

For this plugin I defined 4 critical public functions, three just provide basic information about a plugin, but fourth (run) will be the interesting one in a minute.

Plugin Base

Finally we’re getting to something interesting (or at least something that requires code).  The plugin base is an abstract class which all the actual plugins will extend. This goes in custom_module/src/Plugin/TaskPluginBase.php.

The things worth taking notice of here are the fact that I inject several services into the base class, so I have them for each plugin: translation, main sitemanager module configuration, and a wrapper on Symfony’s process service to make running drush and git commands easier.  The three informational functions are fully defined here, but the run function stays abstract since it’s the function that actually justifies creating a plugin at all.

In theory I could stop right there. I’ve created a plugin manager and defined all the things someone else needs to be able to use it. But there are two technically optional parts that are useful if you like to work and play well with others:  an example implementation and a controller for use or testing purposes.

Example Implementation

We went through all the trouble to create the plugin manager, before you stop you should create at least one plugin to prove all the work we just did actually works.

This is a simple task to use drush to run cron for a site.  The site entities know the location on the file system for Drupal root, and the simple task service handles the extra settings to control where the commands are run (and time outs and error trapping), so we can safely feed that to command and the location to get the response.

The method returns a render array, that can be used as part of a response.

Controller

The controller is truly 100% optional.  There are two reasons you might want to create one: to actually run the plugin if that makes sense (it does for the site manager), and to provide for easy testing of a plugin.

For site manager I created the controller because I actually needed it to run the tasks from the dropbutton (its the link used by all the links on those buttons).  But when I created the others for clients, I realized they can be very hard to test. They are buried inside several layers of complexity, making a test suite is a challenge to create correctly. And since are of the reason to use a plugin for a project with growing definitions, the test suite is often of limited use for rush additions. But by having a carefully built (and secured) controller and route, you can create an endpoint to use to test the plugin.

Because it’s optional, and well covered elsewhere (or heck just use Drupal console to generate it), I’m going to skip over most of the details of actually building the route, and linking that to the controller, and focus on the single function of running the task itself. To keep this short I’ve removed lots of extraneous details like security and error trapping – creating a tool like this isn’t for beginners so I’m assuming you know how to do those things.

The plugin manager service was injected into the controller (code not shown) so I just have it create an instance of the task requested. The controller calls the run function.  If the task is successful the output (a render array) is used for the response, otherwise a simple message is send. This sends an AjaxResponse to work with Drupal’s standard JS handlers, but it could generate a full page just as easily.

Final thoughts

Being able to create your own plugins easily is really nice.  In Drupal 7 custom plugins where generally considered an advanced developer task. The improvements in the abstraction (and the fact that true plugin support is now in core not pulled from ctools or other critical modules), means that while its still not a beginner task it is something all Drupal developers should be learning.

But remember you don’t always need or want the complexity of your own plugins. I’ve found the idea a bit addictive, and I get unreasonably excited when I find a place plugins are used or that it makes sense for me to create a new plugin type. For custom work you can almost always do the same thing using extra functions on a service, controller, or event listener, so this is a judgement call in the end.

Early in my career I spent a lot of time as the only technical person on project, and therefore believed that I didn’t need to document my work carefully since I was the only person who had to understand it later. It turned out that if a project was back burnered for a few months the details were pushed out of my mind by the details of eight other projects.

Any project that takes more than a couple hours to complete involves too many details for most people to remember for more than a few days. We often think about project documentation as something for other people – and it is – but that other person may be you in six months.

I learned to start keeping notes that I could go back to, those notes would turn into documentation that I could share with other people as the need developed. My solutions were typically ad-hoc: freeform word documents or wiki pages. For a while I had a boss who wanted every piece of documentation created by IT to fit a very predictable format and to be in a very specific system. It took two years for him to settle on the system, process, and format to use. By then I had a mountain of information in wiki pages that documented the organization’s online tools in detail, and no one else is IT had anything substantial. It was two more years before the documentation of other team members got to be as good as my ad-hoc wiki.

That’s not to say a rogue solution is best, but the solution that I used was better than his proposed setup for at least three years. That experience got me to think about what makes documentation useful.

Rules of thumb for good project documentation:

• Write up the notes you’d want from others when coming into a project: think of this as the Golden Rule of documentation. Think about what you’d want to have if you were coming into the project six months from now. You’d want an outline of the purpose of the project and the solution used, and places they deviated from any standards your team normally uses. You’ve probably read documents that are explaining something technical to an expert that are hard for anyone else to understand – if I’m reading the documentation I want to become an expert, but I’m probably not one already.
• Keep it easy to create and edit while working: if you have to stop what you’re doing and write your notes in a totally different environment that your day-to-day work you will not do it. Wikis, markdown files, and other similar informal solutions are more likely to actually get written and updated than any formal setup that you can’t update while doing your main work.
• Document as you go: we all plan to go back and write documentation later and almost none of us do. When we do get back to it, we’ve forgotten half the details we need to make the notes useful to others. So admit you’re not going to get back to it and don’t plan to: write as you go and edit as you need.
• Make sure you can come in in the middle: People skim project documentation, technical specifications, and any other large block of text. Make sure if someone has skipped the previous three sections they can either pick up where they left off, or give them directions to the parts the need to understand before continuing.
• Track all contributions: Use a system that automatically tracks changes so you you can see contributions from others and fix mistakes. Tools like MediaWiki, Wordpress, and Drupal do this internally. Markdown or text files in a code repository also have this trait. Avoid solutions like MS Word’s track changes that are meant for editing a final document not tracking revisions over time.

• 

  Don’t fear editing: follow the Wikipedia community’s encouragement to Be Bold. You should not fear making changes to the team’s documentation. You will be wrong in some of what you write, and you should fix any mistake you find – yours or someone else’s. Don’t get mad if someone makes a change that’s not quite right, revert the change or make a new edit and more forward.
• There is always an audience: even if you are the only person on the project you have an audience of at least your future-self. Even if it feels like a waste in the moment having documentation will help down the road.

Remember even if you are working alone you’re on a team that includes at least yourself today and yourself in the future. That future version of you probably won’t remember everything you know right now, and will get very annoyed at you if you don’t record what they need to know. And if the rest of your team members aren’t just versions of yourself they may expressed their frustration more directly.

Ceramics and pottery taught me to be a craftsman, accept critical feedback, and admit failure. For all the courses I took in college, the time I spent in Ceramics and Pottery studio taught me more about how to be good at what I do than any other course work (including CS). Basic pottery is a craft that doesn’t allow you to save a piece once you’ve made mistakes. If you are making cylinders the walls aren’t straight throw it away and start again. My professor in those courses set very high standards and pushed us to meet them through brutally honest, but helpful, critiques. He taught me to appreciate honest feedback, and to be skeptical of my work before showing it others. In those courses we had to do good work, recognize our weaknesses, and hit deadlines.

History taught me to care about communicating. Between the courses I took, and dating (and eventually marrying) a history major (now professor) I came to understand how important it is to communicate well in speech and in writing. I started this blog in part because I wanted to make sure I was spending more time writing to help maintain those skills. I frequently find myself in meetings having to explain highly technical issues to non-technical clients and colleagues. Knowing how to adjust to my audience, without insulting their intelligence, is a critical part of my day to day work.

Religious Studies and Philosophy taught me to make a well reasoned argument and express the ethics of my position. Religious Studies and Philosophy both require you to make arguments based primarily on the strengths of your ideas – you can’t research your way out of a bad concept. If you cannot assemble a well reasoned argument you leave yourself open to easy counter attacks. When you work at a nonprofit you often are building your ideas and your explanations from a mix of the facts of the issue, your worldview, and the assumptions you are forced to make because no one has all the information they really need.

Economics taught me how to view the world through the lens of money and trade. While I don’t think it’s the only, or even best, way to view the world it is a very important worldview that dominates the news and political spaces. Understanding the strengths and weaknesses of economics as a field helps me think about plans put forward by various politicians. It also helps me think about how the companies I work for function, and helps make sure internal systems and plans I propose have a strong business case to support them (and because I learned to throw things away in ceramics I also know that when the case is bad to stop and try something different).

The most important skill I have is the ability to learn new skills. Since I graduated from college the technology I work with every day has changed several times. I’ve had to learn tools, techniques, and strategies that didn’t exist when I was in college. And my career has evolved through multiple employers, disciplines, and areas of expertise.

I consider this unacceptable from old companies, but Atlantic BB was founded in 2004 – there are probably articles about not making assumptions about people’s names that are older than their company.

Folks, it is 2016, when companies insult people and then blame their databases it is because they do not consider all their customers worthy of equal respect.

So let’s get a few basics out of the way:

• Software reflects the biases of the people who write it and buy it.
• If your database tells someone their name is invalid your database is not neutral. Just because you don’t get the push-back that Facebook sees when they mess this up does not mean what you’re doing is okay.
• If your database assumes my household follows 1950s social norms, the company that uses it considers 1950s social norms acceptable in 2016 – and there are probably a few of those they don’t want to defend (I hope).
• When an email, phone rep, or letter calls me by my wife’s last name or her by mine, in both cases they are assuming she has my last name not that I have hers. This is a sexist assumption that the company has chosen to allow.

Of course Atlantic isn’t the only company that does this: Verizon calls me Elizabeth in email a couple times a week because she must be primary on that account (one person must lead the family plan), and Nationwide Insurance had to hack their data fields for years so my wife could appear on our car insurance card (as required by law) every time we moved because their web interface no longer allowed the needed changes. The same bad design assumptions can be insulting for other reasons such as ethnic discrimination. My grandmother was mis-addressed by just about everyone until she died because in the 1960s the Social Security Administration could not handle having an ’ in her name, and no one was willing to fix it in the 50 years that followed SSA’s uninvited edit to her (and many other people’s) name.

In all these cases representatives all say something to the effect of “our computers cannot handle it.” And that of course is simply not true. Your systems may not be setup to handle real people, but that’s because you don’t believe they should be.

Let’s check Atlantic Broadband’s beliefs about their customers based on how they address us (I’m sure there are some additional assumptions not reflected here but these are the ones they managed to encode in one line in this letter):

• They assume they are addressing one primary account holder: I happen to know from my interactions with them that they list my first name as: “Aaron & Eliza”, and my last name as “Crosman Geor”. Plenty of households have more than one, or even two, adults who expect equal treatment in their home. Our bank and mortgage company know we are both responsible adults why is this so hard for an ISP (or insurance company, or cell provider, credit card, etc)?
• They assume my first name isn’t very long: They allowed 13 characters, but 4 more is too many. I went to high school with a kid who broke their database by exceeding the 26 character limit it had (they didn’t ask the kid to change his name, the school database admin fixed the database), but Atlantic can barely handle half that.
• They assume my last name isn’t very long: Only 12 characters were used and they stopped in a strange place. I know many people with last name longer than that: frequently people who have hyphenated last names blow past 12. Also the kid with a 26 character first name – his surname was longer.
• They assume my middle name isn’t an important part of my name: If they had a middle name field, they could squeeze a few more letters in and make this read more sensibly.  But they only consider first and last names important. Plenty of people have three names – or more – they like to have included on letters.
• They assume it is okay to mis-address me and my wife: The name listed is just plain wrong, but they believe it’s okay to keep using this greeting. They assume this even after they have been told it’s not, and even after we’ve reduced service with them (if another ISP provided service to my house I’d probably cut it entirely although mostly for other reasons). They believe misaddressed advertisements will convince me I need a landline or cable package again.

Now I’ll be fair for just a minute and note something they got right: they allow & and spaces in a name so Little Bobby Tables might be able to be a customer without causing a crisis (partially because his name is too long for them to fit a valid SQL command into the field).

Frequently you’ll hear customers blame themselves because their names are too long or they have done something outside the “norm”. Let’s be clear: this is the fault of the people who write and buy the software. Software development is entirely too dominated by men, as is the leadership of large companies. When a company lacks diversity in key roles you see that reflected in the systems built to support the work. Atlantic’s leadership’s priorities and views are reflected in how their customers are addressed because they did not demand the developers correct their sexist assumptions.

These problems are too common for us to be able to refuse to do business when it comes up. I will say that when we switched our insurance to State Farm they did not have any trouble understanding that we had different last names and their systems accommodated that by default.

If you do business with a company that makes these (or other similar mistakes) I think it’s totally reasonable to remind them every time you reasonable can that it’s offensive. Explain that they company is denying you, your loved ones, and/or your friends a major marker of their identity. Remind them they are not neutral.

If you write data systems for a living: check the assumptions you’re building into your code. Don’t blame the technology because you used the wrong character set or trimmed the field too short: disk is cheap, UTF-8 has been standard for 15+ years, and processors are fast. If the database or report layout doesn’t work because someone’s name is too long the flaw is not the name.

We all make mistakes and bad assumptions sometimes, but that does not make it okay to deny people basic respect. When we make a bad assumption, that’s a bug, and good developers are obligated to fix it. Good companies are obligated to prevent it from happening in the first place.

By driving on the left I discovered how many basic driving habits I have that are built around driving on the right. The clearest being that the whole time I was in New Zealand I never knew if anyone was behind me, and the whole time I couldn’t figure out why. The mirrors on the car worked just fine, but it turned out I wasn’t looking at them.

Driving home from the airport after we returned to the US I realized that every few seconds my eyes jump to the upper right of my vision to check the mirror. In New Zealand I spent the whole time glancing at the post between the windshield and the driver’s side window (which had seemed massive to me while I was there) instead of the mirror.

It made me conscious of my driving habits in a way I haven’t been in years, and as a consequence, I think it’s made me a better driver. I’m thinking about little details again; I’ve been more aware of where I am on the road and what I’m doing to keep track of the other cars around me.

A few years ago I was watching videos from the MIT Algorithms course to refresh some of my basics, and because I wanted to know what had been added in the decade since I’d taken that class at Hamilton. During the review of QuickSort the professor mentioned that it wasn’t originally a divide-and-conqueror process, but a loop based approach meant to work on a fixed length array (so you could use a fixed block of memory). And as I recall he suggests that the students should work out the loop based version.

So riding on the train home from work I pieced it together, and found that it’s an elegant process. It’s not something I ever expect to have cause to implement, but it did help me improve my thinking about when to use recursive functions vs when to use a loop, and helped me think about when to use recursion, loops, and other tools for processing everything in a list.

There was a session by John Kary at DrupalCon this year on rethinking loops that pushed me again to revise some of how I made those decisions. Again his talk took the reverse view of much of my previous thinking and was therefore very much worth my time.

If you’re feeling like you are in a good groove on something, try doing it backwards and see what you discover.