How much memory is needed to run 1M Erlang processes?

2023-06-10T00:00:00+00:00
Recently benchmark for concurrency implementation in different languages</a>. In this article Piotr Kołaczkowski</a> used Chat GPT to generate the examples in the different languages and benchmarked them. This was poor choice as I have found this article and read the Elixir example:</p>
tasks </span>=</span></span>
    for</span> _</span> <-</span> 1</span>..</span>num_tasks </span>do</span></span>
        Task</span>.</span>async</span>(</span>fn -></span></span>
            :timer</span>.</span>sleep</span>(</span>10000</span>)</span></span>
        end</span>)</span></span>
    end</span></span>
</span>
Task</span>.</span>await_many</span>(</span>tasks</span>,</span> :</span>infinity</span>)</span></span></code></pre>
And, well, it's pretty poor example of BEAM's process memory usage, and I am
not talking about the fact that it uses 4 spaces for indentation.</p>
For 1 million processes this code reported 3.94 GiB of memory used by the process
in Piotr's benchmark, but with little work I managed to reduce it about 4 times
to around 0.93 GiB of RAM usage. In this article I will describe:</p>

how I did that</li>
why the original code was consuming so much memory</li>
why in the real world you probably should not optimise like I did here</li>
why using ChatGPT to write benchmarking code sucks (TL;DR because that will
nerd snipe people like me)</li>
</ul>
What are Erlang processes?#</a>
</h2>
Erlang is well</del> known of being language which support for concurrency is
superb, and Erlang processes are the main reason for that. But what are these?</p>
In Erlang process</em> is the common name for what other languages call virtual
threads</em> or green threads</em>, but in Erlang these have small neat twist - each of
the process is isolated from the rest and these processes can communicate only
via message passing. That gives Erlang processes 2 features that are rarely
spotted in other implementations:</p>

Failure isolation - bug, unhandled case, or other issue in single process will
not directly affect any other process in the system. VM can send some messages
due to process shutdown, and other processes may be killed because of that,
but by itself shutting down single process will not cause problems in any
process not related to that.</li>
Location transparency - process can be spawned locally or on different
machine, but from the viewpoint of the programmer, there is no difference.</li>
</ul>
The above features and requirements results in some design choices, but for our
purpose only one is truly needed today - each process have separate and (almost)
independent memory stack from any other process.</p>
Process dictionary#</a>
</h3>
Each process in Erlang VM has dedicated mutable</em> memory space for their
internal uses. Most people do not use it for anything because in general it
should not be used unless you know exactly what you are doing (in my case, a bad
carpenter could count cases when I needed it, on single hand). In general it's
here be dragons</em> area.</p>
How it's relevant to us?</p>
Well, OTP internally uses process dictionary (pdict</code> for short) to store
metadata about given process that can be later used for debugging purposes. Some
data that it store are:</p>

Initial function that was run by the given process</li>
PIDs to all ancestors of the given process</li>
</ul>
Different processes abstractions (like gen_server</code>/GenServer</code>, Elixir's
Task</code>, etc.) can store even more metadata there, logger</code> store process
metadata in process dictionary, rand</code> store state of the PRNGs in the process
dictionary. it's used quite extensively by some OTP features.</p>
"Well behaved" OTP process#</a>
</h3>
In addition to the above metadata if the process is meant to be "well behaved"
process in OTP system, i.e. process that can be observed and debugged using OTP
facilities, it must respond to some additional messages defined by sys</code></a>
module. Without that the features like observer</code></a> would not be able to "see"
the content of the process state.</p>
Process memory usage#</a>
</h2>
As we have seen above, the Task.async/1</code> function form Elixir must</strong> do
much more than just simple "start process and live with it". That was one of the
most important problems with the original process, it was using system, that was
allocating quite substantial memory alongside of the process itself, just to
operate this process. In general, that would be desirable approach (as you
really, really, want the debugging facilities</strong>), but in synthetic benchmarks,
it reduce the feasibility of such benchmark.</p>
If we want to avoid that additional memory overhead in our spawned processes we
need to go back to more primitive functions in Erlang, namely erlang:spawn/1</code>
(Kernel.spawn/1</code> in Elixir). But that mean that we cannot use
Task.await_many/2</code> anymore, so we need to workaround it by using custom
function:</p>
defmodule</span> Bench</span> do</span></span>
  def</span> await</span>(</span>pid</span>)</span> when</span> is_pid</span>(</span>pid</span>)</span> do</span></span>
    # Monitor is internal feature of Erlang that will inform you (by sending</span></span>
    # message) when process you monitor die. The returned value is type called</span></span>
    # "reference" which is just simply unique value returned by the VM.</span></span>
    # If the process is already dead, then message will be delivered</span></span>
    # immediately.</span></span>
    ref </span>=</span> Process</span>.</span>monitor</span>(</span>pid</span>)</span></span>
</span>
    receive do</span></span>
      {</span>:</span>DOWN</span>,</span> ^ref</span>,</span> :</span>process</span>,</span> _</span>,</span> _</span>}</span> -></span> :</span>ok</span></span>
    end</span></span>
  end</span></span>
</span>
  def</span> await_many</span>(</span>pids</span>)</span> do</span></span>
    Enum</span>.</span>each</span>(</span>pids</span>,</span> &await</span>/</span>1</span>)</span></span>
  end</span></span>
end</span></span>
</span>
tasks </span>=</span></span>
  for</span> _</span> <-</span> 1</span>..</span>num_tasks </span>do</span></span>
    # `Kernel` module is imported by default, so no need for `Kernel.` prefix</span></span>
    spawn</span>(</span>fn -></span></span>
      :timer</span>.</span>sleep</span>(</span>10000</span>)</span></span>
    end</span>)</span></span>
  end</span></span>
</span>
Bench</span>.</span>await_many</span>(</span>tasks</span>)</span></span></code></pre>
We already removed one problem (well, two in fact, but we will go into
details in next section).</p>
All your lists are belongs to us now#</a>
</h2>
Erlang, like most of the functional programming languages, have 2 built-in
sequence types:</p>

Tuples - which are non-growable product type of the values, so you can access
any field quite fast, but adding more values is performance no-no</li>
(Singly) linked lists - growable type (in most case it will have single type
values in it, but in Erlang that is not always the case), which is fast to
prepend or pop data from the beginning, but do not try to do anything else if
you care about performance.</li>
</ul>
In this case we will focus on the 2nd one, as there tuples aren't important at
all.</p>
Singly linked list is simple data structure. It's either special value []</code>
(an empty list) or it's something called "cons-cell". Cons-cells are also
simple structures - it's 2ary tuple (tuple with 2 elements) where first value
is head - the value in the list cell, and another one is the "tail" of the list (aka
rest of the list). In Elixir the cons-cell is denoted like that [head | tail]</code>.
Super simple structure as you can see, and perfect for the functional
programming as you can add new values to the list without modifying existing
values, so you can be immutable and fast. However if you need to construct the
sequence of a lot of values (like our list of all tasks) then we have problem.
Because Elixir promises that list returned from the for</code> will be in-order</strong>
of the values passed to it. That mean that we either need to process our data
like that:</p>
def</span> map</span>([],</span> _</span>),</span> do:</span> []</span></span>
</span>
def</span> map</span>([</span>head </span>|</span> tail</span>],</span> func</span>)</span> do</span></span>
  [</span>func</span>.(</span>head</span>)</span> |</span> map</span>(</span>tail</span>,</span> func</span>)]</span></span>
end</span></span></code></pre>
Where we build call stack (as we cannot have tail call optimisation there, of
course sans compiler optimisations). Or we need to build our list in reverse
order, and then reverse it before returning (so we can have TCO):</p>
def</span> map</span>(</span>list</span>,</span> func</span>),</span> do:</span> do_map</span>(</span>list</span>,</span> func</span>, [])</span></span>
</span>
def</span> map</span>([],</span> _func</span>,</span> agg</span>),</span> do:</span> :lists</span>.</span>reverse</span>(</span>agg</span>)</span></span>
</span>
def</span> map</span>([</span>head </span>|</span> tail</span>],</span> func</span>,</span> agg</span>)</span> do</span></span>
  map</span>(</span>tail</span>,</span> func</span>, [</span>func</span>.(</span>head</span>)</span> |</span> agg</span>])</span></span>
end</span></span></code></pre>
Which one of these approaches is more performant is irrelevant1</a></sup>,
what is relevant is that we need either build call stack or construct our list
twice</em> to be able to conform to the Elixir promises (even if in this case we do
not care about order of the list returned by the for</code>).</p>
^1</sup>Sometimes body recursion will be faster, sometimes TCO will be
faster. it's impossible to tell without more benchmarking. For more info check
out superb article by Ferd Herbert</a>.</p>
</div>
Of course we could mitigate our problem by using Enum.reduce/3</code> function (or
writing it on our own) and end with code like:</p>
defmodule</span> Bench</span> do</span></span>
  def</span> await</span>(</span>pid</span>)</span> when</span> is_pid</span>(</span>pid</span>)</span> do</span></span>
    ref </span>=</span> Process</span>.</span>monitor</span>(</span>pid</span>)</span></span>
</span>
    receive do</span></span>
      {</span>:</span>DOWN</span>,</span> ^ref</span>,</span> :</span>process</span>,</span> _</span>,</span> _</span>}</span> -></span> :</span>ok</span></span>
    end</span></span>
  end</span></span>
</span>
  def</span> await_many</span>(</span>pids</span>)</span> do</span></span>
    Enum</span>.</span>each</span>(</span>pids</span>,</span> &await</span>/</span>1</span>)</span></span>
  end</span></span>
end</span></span>
</span>
tasks </span>=</span></span>
  Enum</span>.</span>reduce</span>(</span>1</span>..</span>num_tasks</span>, [],</span> fn</span> _</span>,</span> agg </span>-></span></span>
    # `Kernel` module is imported by default, so no need for `Kernel.` prefix</span></span>
    pid </span>=</span></span>
      spawn</span>(</span>fn -></span> :timer</span>.</span>sleep</span>(</span>10000</span>)</span> end</span>)</span></span>
</span>
    [</span>pid </span>|</span> agg</span>]</span></span>
  end</span>)</span></span>
</span>
Bench</span>.</span>await_many</span>(</span>tasks</span>)</span></span></code></pre>
Even then we build list of all PIDs.</p>
Here I can also go back to the "second problem* I have mentioned above.
Task.await_many/1</code> also construct a list</em>. it's list of return value from all
the processes in the list, so not only we constructed list for the tasks' PIDs,
we also constructed list of return values (which will be :ok</code> for all processes
as it's what :timer.sleep/1</code> returns), and immediately discarded all of that.</p>
How we can better? See that all</strong> we care is that all num_task</code> processes
have gone down. We do not care about any of the return values, all what we want
is to know that all processes that we started went down. For that we can just
send messages from the spawned processes and count the received messages count:</p>
defmodule</span> Bench</span> do</span></span>
  def</span> worker</span>(</span>parent</span>)</span> do</span></span>
    :timer</span>.</span>sleep</span>(</span>10000</span>)</span></span>
    send</span>(</span>parent</span>,</span> :</span>done</span>)</span></span>
  end</span></span>
</span>
  def</span> start</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> start</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    this </span>=</span> self</span>()</span></span>
    spawn</span>(</span>fn -></span> worker</span>(</span>this</span>)</span> end</span>)</span></span>
</span>
    start</span>(</span>n </span>-</span> 1</span>)</span></span>
  end</span></span>
</span>
  def</span> await</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> await</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    receive do</span></span>
      :</span>done</span> -></span> await</span>(</span>n </span>-</span> 1</span>)</span></span>
    end</span></span>
  end</span></span>
end</span></span>
</span>
Bench</span>.</span>start</span>(</span>num_tasks</span>)</span></span>
Bench</span>.</span>await</span>(</span>num_tasks</span>)</span></span></code></pre>
Now we do not have any lists involved and we still do what the original task
meant to do - spawn num_tasks</code> processes and wait till all go down.</p>
Arguments copying#</a>
</h2>
One another thing that we can account there - lambda context and data passing
between processes.</p>
You see, we need to pass this</code> (which is PID of the parent) to our newly
spawned process. That is suboptimal, as we are looking for the way to reduce
amount of the memory (and ignore all other metrics at the same time). As Erlang
processes are meant to be "share nothing" type of processes there is problem -
we need to copy that PID to all processes. it's just 1 word (which mean 8 bytes
on 64-bit architectures, 4 bytes on 32-bit), but hey, we are microbenchmarking,
so we cut whatever we can (with 1M processes, this adds up to 8 MiBs).</p>
Hey, we can avoid that by using yet another feature of Erlang, called
registry</em>. This is yet another simple feature that allows us to assign PID of
the process to the atom, which allows us then to send messages to that process
using just name, we have given. While atoms are also 1 word that wouldn't make
sense to send it as well, but instead we can do what any reasonable
microbenchmarker would do - hardcode stuff</em>:</p>
defmodule</span> Bench</span> do</span></span>
  def</span> worker</span> do</span></span>
    :timer</span>.</span>sleep</span>(</span>10000</span>)</span></span>
    send</span>(</span>:</span>parent</span>,</span> :</span>done</span>)</span></span>
  end</span></span>
</span>
  def</span> start</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> start</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    spawn</span>(</span>fn -></span> worker</span>()</span> end</span>)</span></span>
</span>
    start</span>(</span>n </span>-</span> 1</span>)</span></span>
  end</span></span>
</span>
  def</span> await</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> await</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    receive do</span></span>
      :</span>done</span> -></span> await</span>(</span>n </span>-</span> 1</span>)</span></span>
    end</span></span>
  end</span></span>
end</span></span>
</span>
Process</span>.</span>register</span>(</span>self</span>(),</span> :</span>parent</span>)</span></span>
</span>
Bench</span>.</span>start</span>(</span>num_tasks</span>)</span></span>
Bench</span>.</span>await</span>(</span>num_tasks</span>)</span></span></code></pre>
Now we do not pass any arguments, and instead rely on the registry to dispatch
our messages to respective processes.</p>
One more thing#</a>
</h2>
As you may have already noticed we are passing lambda to the spawn/1</code>. That is
also quite suboptimal, because of difference between remote and local call</a>.
This mean that we are paying slight memory cost for these processes to keep the
old module in memory. Instead we can use either fully qualified function capture
or spawn/3</code> function that accepts MFA (module, function name, arguments list)
argument. We end with:</p>
defmodule</span> Bench</span> do</span></span>
  def</span> worker</span> do</span></span>
    :timer</span>.</span>sleep</span>(</span>10000</span>)</span></span>
    send</span>(</span>:</span>parent</span>,</span> :</span>done</span>)</span></span>
  end</span></span>
</span>
  def</span> start</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> start</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    spawn</span>(</span>&</span>__MODULE__</span>.</span>worker</span>/</span>0</span>)</span></span>
</span>
    start</span>(</span>n </span>-</span> 1</span>)</span></span>
  end</span></span>
</span>
  def</span> await</span>(</span>0</span>),</span> do:</span> :</span>ok</span></span>
  def</span> await</span>(</span>n</span>)</span> when</span> n </span>></span> 0</span> do</span></span>
    receive do</span></span>
      :</span>done</span> -></span> await</span>(</span>n </span>-</span> 1</span>)</span></span>
    end</span></span>
  end</span></span>
end</span></span>
</span>
Process</span>.</span>register</span>(</span>self</span>(),</span> :</span>parent</span>)</span></span>
</span>
Bench</span>.</span>start</span>(</span>num_tasks</span>)</span></span>
Bench</span>.</span>await</span>(</span>num_tasks</span>)</span></span></code></pre>Results#</a>
</h2>
With given Erlang compilation:</p>
Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]</span></span>
</span>
Elixir 1.14.5 (compiled with Erlang/OTP 25)</span></span></code></pre>

Note no JIT as Nix on macOS currently2</a></sup> disable it and I didn't
bother to enable it in the derivation (it was disabled because there were some
issues, but IIRC these are resolved now).</p>
</blockquote>
^{2</sup>
Nixpkgs rev bc3ec5ea</code></p>
</div>
The results are as follow (in bytes of peak memory footprint returned by
/usr/bin/time</code> on macOS):</p>
Implementation</th> 1k</th> 100k</th> 1M</th></tr></thead>

Original</td> 45047808</td> 452837376</td> 4227715072</td></tr>
Spawn</td> 43728896</td> 318230528</td> 2869723136</td></tr>
Reduce</td> 43552768</td> 314798080</td> 2849304576</td></tr>
Count</td> 43732992</td> 313507840</td> 2780540928</td></tr>
Registry</td> 44453888</td> 311988224</td> 2787237888</td></tr>
RemoteCall</td> 43597824</td> 310595584</td> 2771525632</td></tr>
</tbody></table>
As we can see we have reduced the memory use by about 30% by just changing
from Task.async/1</code> to spawn/1</code>. Further optimisations reduced memory usage
slightly, but with no such drastic changes.</p>
Can we do better?</p>
Well, with some VM flags tinkering - of course.</p>
You see, by default Erlang VM will not only create some data required for
handling process itself^{3</a></sup>:</p>}
^{3</sup>
Again, word here mean 8 bytes on 64-bit and 4 bytes on 32-bit architectures.</p>
</div>

Data Type</th> Memory Size</th></tr></thead>

…</td> …</td></tr>
Erlang process</td> 338 words when spawned, including a heap of 233 words.</td></tr>
</tbody></table>
-- Erlang Efficiency Guide: 11. Advanced</a></p>
</blockquote>
As we can see, there are 105 words that are required and 233 words which are
used for preallocated heap. But this is microbenchmarking, so as we do not need
that much of memory (because our processes basically does nothing), we can
reduce it. We do not care about time performance anyway. For that we can use
+hms</code> flag and set it to some small value, for example 1</code>.</p>
In addition to heap size Erlang by default load some additional data from the
BEAM files. That data is used for debugging and error reporting, but again, we
are microbenchmarking, and who need debugging support anyway (answer: everyone,
so do not</strong> do it in production). Luckily for us, the VM has yet another flag
for that purpose +L</code>.</p>
Erlang also uses some ETS</a> (Erlang Term Storage) tables by default (for
example to support process registry we have mentioned above). ETS tables can be
compressed, but by default it's not done, as it can slow down some kinds of
operations on such tables. Fortunately there is, another, flag +ec</code> that has
description:</p>

Forces option compressed on all ETS tables. Only intended for test and
evaluation.</p>
</blockquote>
Sounds good enough for me.</p>
With all these flags enabled we get peak memory footprint at 996257792 bytes.</p>
Compare it in more human readable units.</p>
</th> Peak Memory Footprint for 1M processes</th></tr></thead>

Original code</td> 3.94 GiB</td></tr>
Improved code</td> 2.58 GiB</td></tr>
Improved code with flags</td> 0.93 GiB</td></tr>
</tbody></table>
Result - about 76% of the peak memory usage reduction. Not bad.</p>
Summary#</a>
</h2>
First of all:</p>

Please, do not use ChatGPT for writing code for microbenchmarks.</p>
</blockquote>
The thing about micro</em>benchmarking is that we write code that does as little as
possible to show (mostly) meaningless features of the given technology in
abstract environment. ChatGPT cannot do that, not out of malice or incompetence,
but because it used (mostly) good</em> and idiomatic code to teach itself,
microbenchmarks rarely are something that people will consider to have these
qualities. It also cannot consider other features that wetware</a> can take into
account (like our "we do not need lists there" thing).</p>


Who Watches Watchmen? - Part 1
2022-01-17T21:22:18+01:00
I gave talk about this topic on CODE Beam V Americas, but I wasn't really
satisfied with it. In this post I will try to describe what my presentation was
meant to be about.</p>
If you are wondering about the presentation, the slides are on SpeakerDeck</a>.</p>
Abstract#</a>
</h2>
Most of the operating systems are multi-process and multi-user operating
systems. This has a lot of positive aspects, like to be able to do more than one
thing at the time at our devices, but it introduces a lot of complexities that
in most cases are hidden from the users and developers. These things still need
to be handled in one or another way. The most basic problems are:</p>

some processes need to be started before user can interact with the OS
in meaningful (for them) way (for example mounting filesystems, logging,
etc.)</li>
some processes require strict startup ordering, for example you may need
logging to be started before starting HTTP server</li>
system operator somehow need to know when the process is ready to do their
work, which is often some time after process start</li>
system operator should be able to check process state in case when debugging
is needed, most commonly via logs</li>
shutdown of the processes should be handled in a way, that will allow other
processes to be shut down cleanly (for example application that uses DB should
be down before DB itself)</li>
</ul>
Why we need system supervisor?#</a>
</h2>
System supervisor is a process started early in the OS boot, that should handle
starting and managing all other processes that will be run on our system. It is
often the init process (first process started by the OS that is running with PID
1) or it is first (and sometimes only) process started by the init process.
Popular examples of such supervisors (often integrated with init systems):</p>

SysV which is "traditional" implementation that originates at UNIX System
V (hence the name)</li>
BSD init that with some variations is used in BSD-based OSes (NetBSD,
FreeBSD), it shares some similarities to SysV init and services description is
provided by shell scripts</li>
OpenRC that also uses shell-based scripts for service description, used by
Linux distributions like Gentoo or Alpine</li>
launchd</code> that is used on Darwin (macOS, iPadOS, iOS, watchOS) systems that uses
XML-based plists</code> for services description</li>
runit</code> which is small init and supervisor, but quite capable, for example
used by Void Linux</li>
Upstart created by Canonical Ltd. as a replacement for SysV-like init system
in Ubuntu (no longer in use in Ubuntu), still used in some distributions like
ChromeOS or Synology NAS</li>
systemd</code> (this is the name, not "SystemD") that was created by Red Hat
employee, (in)famous Lennart Poettering, and later was adopted by almost all
major Linux distributions which spawned some heated discussion about it</li>
</ul>
In this article I will focus on systemd, and its approach to "new-style system
daemons".</p>

DISCLAIMER</strong></p>
Each of the solutions mentioned above has its strong and weak points. I do not
want to start another flame war whether it is good or not. It has some good in
it, and it has some bad in it, but we can say that it "won" over the most used
distributions, and despite our love or hate towards it, we need to learn how to
live with that.</p>

Why systemd</code>?#</a>
</h2>
systemd</code> became a thing because SysV approach to ordering services' startup was
mildly irritating and non-parallelizable. In short, SysV is starting processes
exactly in lexicographical order of files in given directory. This meant, that
even if your service didn't need the DB at all, but it somehow ended further in
the directory listing, you ended in waiting for the DB startup. Additionally,
SysV wasn't really monitoring services, it just assumed that when process forked
itself to the background, then it is "done" with the startup, and we can
continue. This is obviously not true in many cases, for example, if your
previous shutdown wasn't clean because of power shortage or other issue, then
your DB probably need a bit of time to rebuild state from journal. This causes
even more slowdown for the processes further in the list. This is highly
undesired in modern, cloud-based, environment, where you can often start the
machines on-demand during autoscaling actions. When there is a spike in the
traffic that need autoscaling, then the sooner new machine is in usable state
the sooner it can take load from other machines.</p>
Different tools take different approach to solve that issue there. systemd</code>
take approach that is derived from launchd</code> - do not do stuff, that is not
needed. We can trigger on action of other services (obviously), but also on
stuff like socket activity, path creation/modification, mounts, connection or
disconnection of device, time events, etc.</p>

DIGRESSION</strong></p>
This is exactly the reason why systemd</code> has its infamous "feature creep", it
doesn't "digest" all services like Cron or udev</code>. It is not that these are
"tightly" intertwined into systemd</code>. You can still replace them with their
older counterparts, you will just lose all the features these bring with them.</p>

Such lazy approach sometimes require changes into the service itself. For
example to let supervisor know, that you are ready (not just started), you need
some way to communicate with supervisor. In systemd</code> you can do so via UNIX
socket pointed by NOTIFY_SOCKET</code> environment variable passed to your
application. With the same socket you can implement another useful feature
- watchdog/heartbeat process. This mean that if for any reason your process
became non-responsive (but it will refuse to die), then supervisor will
forcefully bring process down and restart it, assuming that the error was
accidental.</p>
About restarting, we can define behaviour of service after main process die. It
can be restarted regardless of the exit code, it can be restarted on abnormal
exit, it can remain shut, etc. Does this ring a bell? This works similarly to
OTP supervisors, but "one level above". If your service utilize system
supervisor right, you can make your application almost ultimately self-healing
(by restarts).</p>
Basic setup#</a>
</h2>
Now, when we know a little about how and why systemd</code> works as it works, we
now can go to details on how to utilize that with services in Elixir.</p>
As a base we will implement super simple Plug application:</p>
# hello/application.ex</span></span>
defmodule</span> Hello</span>.</span>Application</span> do</span></span>
  use</span> Application</span></span>
</span>
  def</span> start</span>(</span>_type</span>,</span> _opts</span>)</span> do</span></span>
    children </span>=</span> [</span></span>
      {</span>Plug</span>.</span>Cowboy</span>, [</span>scheme</span>:</span> :</span>http</span>,</span> plug</span>:</span> Hello</span>.</span>Router</span>]</span> ++</span> cowboy_opts</span>()},</span></span>
      {</span>Plug</span>.</span>Cowboy</span>.</span>Drainer</span>,</span> refs</span>:</span> :</span>all</span>}</span></span>
    ]</span></span>
</span>
    Supervisor</span>.</span>start_link</span>(</span>children</span>,</span> strategy</span>:</span> :</span>one_for_one</span>)</span></span>
  end</span></span>
</span>
  defp</span> cowboy_opts</span> do</span></span>
    [</span></span>
      port</span>:</span> String</span>.</span>to_integer</span>(</span>System</span>.</span>get_env</span>("</span>PORT</span>", "</span>4000</span>"))</span></span>
    ]</span></span>
  end</span></span>
end</span></span></code></pre># hello/router.ex</span></span>
defmodule</span> Hello</span>.</span>Router</span> do</span></span>
  use</span> Plug</span>.</span>Router</span></span>
</span>
  plug </span>:</span>match</span></span>
  plug </span>:</span>dispatch</span></span>
</span>
  get </span>"</span>/</span>"</span> do</span></span>
    send_resp</span>(</span>conn</span>,</span> 200</span>, "</span>Hello World!</span>")</span></span>
  end</span></span>
end</span></span></code></pre>
I will also assume that we are using Mix release</a> named hello</code>
that we later copy to /opt/hello</code>.</p>
systemd unit file#</a>
</h3>
We have only one thing left, we need to define our hello.service</code></a>:</p>
[</span>Unit</span>]</span></span>
Description</span>=</span>Hello World service</span></span>
</span>
[</span>Service</span>]</span></span>
Environment</span>=</span>PORT</span>=</span>80</span></span>
ExecStart</span>=</span>/opt/hello/bin/hello start</span></span></code></pre>
Now you can create file with that content in
/usr/local/lib/systemd/system/hello.service</code> and then start it with:</p>
# systemctl start hello.service</span></span></code></pre>
This is the simplest service imaginable, however from the start we have few
issues there:</p>

It will run service as user running supervisor, so if it is run using global
supervisor, then it will run as root</code>. You do not want to run anything as
root</code>.</li>
On error it will produce (BEAM) core dump, which may contain sensitive data.</li>
It can read (and, due to being run as root</code>, write) everything in the system,
like private data of other processes.</li>
</ul>
Service readiness#</a>
</h2>
Erlang VM isn't really the best tool out there wrt the startup times. In
addition to that our application may need some preparation steps before it can
be marked as "ready". This is problem that I sometimes encounter in Docker,
where some containers do not really have any health check, and then I need to
have loop with check in some of the containers that depend on another one. This
"workaround" is frustrating, error prone, and can cause nasty Heisenbugs when
the timing will be wrong.</p>
Two possible solutions for this problem are:</p>

Readiness probe - another program that is ran after the main process is
started, that checks whether our application is ready to work.</li>
Notification system where our application uses some common protocol to inform
the supervisor that it finished setup and is ready for work.</li>
</ul>
systemd supports the second approach via sd_notify</code></a>. The approach
there is simple - we have NOTIFY_SOCKET</code> environment variable that contain path
to the Unix datagram socket, that we can use to send information about state of
our application. This socket accept set of different messages, but right now,
for our purposes, we will focus only on few of them:</p>

READY=1</code> - marks our service as ready, aka it is ready to do its work (for
example accept incoming HTTP connections in our example). It need to be sent
within given timespan after start of the VM, otherwise the process will be
killed and possibly restarted</li>
STATUS=name</code> - sets status of our application that can be checked via
systemctl status hello.service</code>, this allows us to have better insight into
what is the high level state without manually traversing through logs</li>
RELOADING=1</code> - marks, that our application is reloading, which in general may
mean a lot of things, but there it will be used to mark :init.restart/0</code>-like
behaviour (due to erlang/otp#4698</a> there is wrapper for that function in
systemd</code> library). The process need then to send READY=1</code> within given
timespan, or the process will be marked as a malfunctioning, and will be
forcefully killed and possibly restarted</li>
STOPPING=1</code> - marks, that our application began shutting down process, and
will be closing soon. If the process will not close within given timespan, it
will be forcefully killed</li>
</ul>
These messages provide us enough power to not only mark the service as ready,
but also provides additional information about system state, so even operator,
who knows a little about Erlang or our application runtime, will be able to
understand what is going on.</p>
The main thing is that systemd will wait with activation of the dependants of
our system as well as the systemctl start</code> and systemctl restart</code> commands
will wait until our service declare that it is ready.</p>
Usage of such feature is quite simple:</p>
[</span>Unit</span>]</span></span>
Description</span>=</span>Hello World service</span></span>
</span>
[</span>Service</span>]</span></span>
# Define `Type=` to `notify`</span></span>
Type</span>=</span>notify</span></span>
Environment</span>=</span>PORT</span>=</span>80</span></span>
ExecStart</span>=</span>/opt/hello/bin/hello start</span></span>
WatchdogSec</span>=</span>1min</span></span></code></pre>
And then in our supervisor tree we need add :systemd.ready()</code> after</strong> last
process needed for proper functioning of our application, in our simple example
it is after Plug.Cowboy</code>:</p>
# hello/application.ex</span></span>
defmodule</span> Hello</span>.</span>Application</span> do</span></span>
  use</span> Application</span></span>
</span>
  def</span> start</span>(</span>_type</span>,</span> _opts</span>)</span> do</span></span>
    children </span>=</span> [</span></span>
      {</span>Plug</span>.</span>Cowboy</span>, [</span>scheme</span>:</span> :</span>http</span>,</span> plug</span>:</span> Hello</span>.</span>Router</span>]</span> ++</span> cowboy_opts</span>()},</span></span>
      :systemd</span>.</span>ready</span>(),</span> # <-- it is function call, as it returns proper</span></span>
                        # `child_spec/0`</span></span>
      {</span>Plug</span>.</span>Cowboy</span>.</span>Drainer</span>,</span> refs</span>:</span> :</span>all</span>}</span></span>
    ]</span></span>
</span>
    Supervisor</span>.</span>start_link</span>(</span>children</span>,</span> strategy</span>:</span> :</span>one_for_one</span>)</span></span>
  end</span></span>
</span>
  defp</span> cowboy_opts</span> do</span></span>
    [</span></span>
      port</span>:</span> String</span>.</span>to_integer</span>(</span>System</span>.</span>get_env</span>("</span>PORT</span>", "</span>4000</span>"))</span></span>
    ]</span></span>
  end</span></span>
end</span></span></code></pre>
Now restarting our service will not finish immediately, but will wait until our
service will declare that it is ready.</p>
# systemctl restart hello.service</span></span></code></pre>
About STOPPING=1</code> - the magic thing is that the systemd</code> library takes care of
it for you. As soon as the system will be scheduled to shutdown this message
will be automatically sent, and the operator will be notified about this fact.</p>
We can also provide more information about state of our application. As you may
have already noticed, we have Plug.Cowboy.Drainer</code></a> there. It is process that
will delay shutdown of our application while there are still open connections.
This can take some time, so it would be handy if the operator would see that the
draining is in progress. We can easily achieve that by again changing our
supervision tree to:</p>
# hello/application.ex</span></span>
defmodule</span> Hello</span>.</span>Application</span> do</span></span>
  use</span> Application</span></span>
</span>
  def</span> start</span>(</span>_type</span>,</span> _opts</span>)</span> do</span></span>
    children </span>=</span> [</span></span>
      {</span>Plug</span>.</span>Cowboy</span>, [</span>scheme</span>:</span> :</span>http</span>,</span> plug</span>:</span> Hello</span>.</span>Router</span>]</span> ++</span> cowboy_opts</span>()},</span></span>
      :systemd</span>.</span>ready</span>(),</span></span>
      :systemd</span>.</span>set_status</span>(</span>down</span>: [</span>status</span>: "</span>drained</span>"]),</span></span>
      {</span>Plug</span>.</span>Cowboy</span>.</span>Drainer</span>,</span> refs</span>:</span> :</span>all</span>,</span> shutdown</span>:</span> 10_000</span>},</span></span>
      :systemd</span>.</span>set_status</span>(</span>down</span>: [</span>status</span>: "</span>draining</span>"])</span></span>
    ]</span></span>
</span>
    Supervisor</span>.</span>start_link</span>(</span>children</span>,</span> strategy</span>:</span> :</span>one_for_one</span>)</span></span>
  end</span></span>
</span>
  defp</span> cowboy_opts</span> do</span></span>
    [</span></span>
      port</span>:</span> String</span>.</span>to_integer</span>(</span>System</span>.</span>get_env</span>("</span>PORT</span>", "</span>4000</span>"))</span></span>
    ]</span></span>
  end</span></span>
end</span></span></code></pre>
Now when we will shutdown our application by:</p>
# systemctl stop hello.service</span></span></code></pre>
And we have some connections open to our service (you can simulate that with
wrk</code>) then when we ran systemctl status hello.service</code> in separate terminal
(previous will be blocked until our service shuts down) then you will be able to
see something like:</p>
● hello.service - Example Plug application</span></span>
     Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)</span></span>
          Active: deactivating (stop-sigterm) since Sat 2022-01-15 17:46:30 CET;</span></span>
          1s ago</span></span>
          Main PID: 1327 (beam.smp)</span></span>
          Status: "draining"</span></span>
          Tasks: 19 (limit: 1136)</span></span>
          Memory: 106.5M</span></span></code></pre>
You can notice that the Status</code> is set to "draining"</code>. As soon as all
connections will be drained it will change to "drained"</code> and then the
application will shut down and service will be marked as inactive</code>.</p>
Watchdog#</a>
</h2>
Watchdog allows us to monitor our application for responsiveness (as mentioned
above). It is simple feature that requires our application to ping systemd
within specified interval, otherwise the application will be forcibly shut down
as malfunctioning. Fortunately for us, the systemd</code> library that provides our
integration, have that feature out of the box, so all we need to do to achieve
expected result is set WatchdogSec=</code> option in our systemd.service</code> file:</p>
[</span>Unit</span>]</span></span>
Description</span>=</span>Hello World service</span></span>
</span>
[</span>Service</span>]</span></span>
Environment</span>=</span>PORT</span>=</span>80</span></span>
Type</span>=</span>notify</span></span>
ExecStart</span>=</span>/opt/hello/bin/hello start</span></span>
WatchdogSec</span>=</span>1min</span></span></code></pre>
This configuration says that if the VM will not send healthy message each 1
minute interval, then the service will be marked as malfunctioning. From the
application side we can manage state of the watchdog in several ways:</p>

By setting systemd.watchdog_check</code> configuration option we can configure the
function that will be called on each check, if that function return true</code>
then it mean that application is healthy and the systemd should be notified
with ping, if it returns false</code> or fail, then the check will be omitted.</li>
Manually sending trigger message in case of detected problems via
:systemd.watchdog(trigger)</code>, it will immediately mark service as
malfunctioning and will trigger action defined in service unit file (by
default it will restart application)</li>
Disabling built in watchdog process via :systemd.watchdog(:disable)</code> and then
manually sending :systemd.watchdog(:ping)</code> within expected intervals
(discouraged)</li>
</ul>
Security#</a>
</h2>
We should start with changing default user and group which is assigned to our
process. We can do so in 2 different ways:</p>

Use some existing user and group by defining User=</code> and Group=</code> directives
in our service definition; or</li>
Create ephemeral user on-demand before our service starts, by using directive
DynamicUser=true</code> in service definition.</li>
</ol>
I prefer second option, as it additionally provides a lot of other security
related options, like creating private /tmp</code> directory, making system
read-only, etc. This has also some disadvantages, like removing all of given
data on service shutdown, however there are options to keep some data between
launches.</p>
In addition to that we can add PrivateDevices=true</code> that will hide all
physical devices from /dev</code> leaving only pseudo devices like /dev/null</code> or
/dev/urandom</code> (so you will be able to use :crypto</code> and :ssl</code> modules without
problems).</p>
Next thing is that we can do, is to disable crash dumps generated by BEAM</a>.
While not strictly needed in this case, it is worth remembering, that it isn't
hard to achieve, just use Environment=ERL_CRASH_DUMP_SECONDS=0</code>.</p>
Our new, more secure, hello.service</code> will look like:</p>
[</span>Unit</span>]</span></span>
Description</span>=</span>Hello World service</span></span>
Requires</span>=</span>network.target</span></span>
</span>
[</span>Service</span>]</span></span>
Type</span>=</span>notify</span></span>
Environment</span>=</span>PORT</span>=</span>80</span></span>
ExecStart</span>=</span>/opt/hello/bin/hello start</span></span>
WatchdogSec</span>=</span>1min</span></span>
</span>
# We need to add capability to be able to bind on port 80</span></span>
CapabilityBoundingSet</span>=</span>CAP_NET_BIND_SERVICE</span></span>
</span>
# Hardening</span></span>
DynamicUser</span>=</span>true</span></span>
PrivateDevices</span>=</span>true</span></span>
Environment</span>=</span>ERL_CRASH_DUMP_SECONDS</span>=</span>0</span></span></code></pre>
The problem with that configuration is that our service is now capable on
binding any</strong> port under 1024, so for example, if there is some security
issue, then the malicious party can open any of the restricted ports and then
serve whatever data they want there. This can be quite problematic, and the
solution for that problem will be covered in Part 2, where we will cover socket
passing and socket activation for our service.</p>
With that we achieved quite basic level of isolation to what Docker (or other
container runtime) is providing, but it do not require overlayfs</code> or anything
more, than what you already have on your machine. That means, updates done by
your system package manager will be applied to all running services. With that
you do not need to rebuild all your containers when there is security patch
issued for any of your dependencies.</p>
Of course it only scratches the surface of what is possible with systemd wrt
the hardening of the services. More information can be found in RedHat
article</a> and in systemd-analyze security</code> command
output</a>. Possible features are:</p>

creation of the private networks for your services</li>
disallowing creation of socket connections that are outside of the specified
set of families</li>
make only some paths readable</li>
hide some paths from the process</li>
etc.</li>
</ul>
Coverage of just that topic is a little bit out of scope for this blog post, so
I encourage you to read the documentation of systemd.exec</code></a> and
articles mentioned above for more details.</p>
Summary#</a>
</h2>
This blog post is already quite lengthy, so I will split it into separate parts.
There probably will be 3 of them:</p>

Part 1 - Basics, security, and FD passing (this one)</a></li>
Part 2 - Socket activation</a></li>
Part 3 - Logging</li>
</ul>


Log all the things
2022-01-02T00:00:00+00:00
In Elixir 1.11 landed set of new features that allows for more powerful logging
by utilising Erlang's logger</code></a> features. Here I will try to describe
new possibilities and how You can use them to improve your logs.</p>
</span>
New log levels#</a>
</h2>
Elixir gained 4 new log levels to total 8 (from most verbose to least verbose):</p>

debug</li>
info</li>
notice</strong> *</li>
warning (renamed from warn)</li>
error</li>
critical</strong> *</li>
alert</strong> *</li>
emergency</strong> *</li>
</ul>
* new levels</small></p>
This allow to provide finer graded verbosity control, due to compatibility
reasons, in Elixir backends we need to translate these levels back to "old" set
of 4. The current table looks like:</p>
Call level</th> What Elixir backend will see</th></tr></thead>

debug</code></td> debug</code></td></tr>
info</code></td> info</code></td></tr>
notice</code></td> info</code></strong> *</td></tr>
warning</code> (or warn</code>)</td> warn</code></td></tr>
error</code></td> error</code></td></tr>
critical</code></td> error</code></strong> *</td></tr>
alert</code></td> error</code></strong> *</td></tr>
emergency</code></td> error</code></strong> *</td></tr>
</tbody></table>
* "translated" messages</small></p>
We can set verbosity to all levels. This may be confusing during the transition
period, but we cannot change the behaviour until Elixir 2 (which is not
happening any time soon).</p>
Usage of the new levels is "obvious":</p>
Logger</span>.</span>notice</span>("</span>Hello</span>")</span></span></code></pre>
Will produce message with notice</code> level of verbosity.</p>
Additionally the logger.level</code> option in configuration supports 2 additional
verbosity levels that you can use in your config:</p>

:all</code> - all messages will be logged, logically exactly the same as :debug</code></li>
:none</code> - no messages will be logged</li>
</ul>
Per module log level#</a>
</h2>
This is change that can be quite handy during debugging sessions. With this
change we have 4 new functions in Logger</code> module:</p>

get_module_level/1</code></a></li>
put_module_level/2</code></a></li>
delete_module_level/1</code></a></li>
delete_all_module_level/0</code></a></li>
</ul>
These allow us to manipulate verbosity level on per-module basis. What is
non-obvious and is super handy is that it allows both lowering and raising</strong>
verbosity for given module. This mean that:</p>
require</span> Logger</span></span>
</span>
Logger</span>.</span>configure</span>(</span>level</span>:</span> :</span>error</span>)</span></span>
</span>
defmodule</span> Foo</span> do</span></span>
  def</span> run</span> do</span></span>
    Logger</span>.</span>debug</span>("</span>I am still there</span>")</span></span>
  end</span></span>
end</span></span>
</span>
Foo</span>.</span>run</span>()</span> # Does not log anything</span></span>
</span>
# Set `debug` level for `Foo` module only</span></span>
Logger</span>.</span>put_module_level</span>(</span>Foo</span>,</span> :</span>debug</span>)</span></span>
Foo</span>.</span>run</span>()</span></span>
# `I am still there` is logged</span></span>
Logger</span>.</span>debug</span>("</span>I will not be printed</span>")</span></span>
# Nothing got logged as top-level verbositi is still set to `:error`</span></span></code></pre>
Of course it will not work if you decide to use compile time purging</a></p>
Logger handlers#</a>
</h2>

Warning!</strong> This is not fully implemented in both - Erlang and Elixir. Writing
your own handlers without additional knowledge can cause overload problems.</p>

Erlang together with their logging implementation needed to provide a way to
ingest these logs somehow. This is done via Erlang logger handlers (in this
article called handlers</em> in contrast to Elixir backends called backends</em>
there).</p>
Handlers are modules that export at least 1 function log/2</code> that takes 2
arguments:</p>

log_event</code> which is a map with 3 fields:

:level</code> - verbosity level</li>
:msg</code> - tuple describing message:

{:io.format(), [term()]}</code> - format string and list of terms that should
be passed to :io_lib.format/2</code> function</li>
{:report, map() | keyword()}</code> - report that can be formatted into string
by report_cb/{1,2}</code> set in metadata map (see below)</li>
{:string, :unicode.chardata()}</code> - raw string that should be printed as
a message</li>
</ul>
</li>
:meta</code> - map containing all metadata for given event. All keys should be
atoms and values can be anything. Some keys have special meaning, and some
of them will be populated automatically by the Logger</code> macros and functions.
These are:

:pid</code> - PID of the process that fired log event</li>
:gl</code> - group leader of the process that fired log event</li>
:mfa</code> - tuple in form of {module(), name :: atom(), arity :: non_neg_integer()}</code>
that describe function that fired log event</li>
:file</code> - filename of file that defines the code that fired log event</li>
:line</code> - line in the given file where the log event was fired</li>
:domain</code> - list of atoms that can be used to describe log events
hierarchy which then can be used for filtering. All events fired using
Logger</code> macros and functions will have :elixir</code> prepended to their
domain list.</li>
:report_cb</code> - function that will be used to format {:report, map() | keyword()}</code> messages. This can be either 1-ary function, that takes report
and returns {:io.format(), [term()]}</code> leaving truncation and further
formatting up to the main formatter, or 2-ary function that takes report
and configuration map %{depth: pos_integer() | :unlimited, chars_limit: pos_integer() | :unlimited, single_line: boolean()}</code> and returns already
formatted :unicode.chardata()</code>. More about it can be found in separate
section</a>.</li>
</ul>
</li>
</ul>
</li>
</ul>
Return value of this function is ignored. If there will be any exception raised
when calling this function, then it will be captured and failing handler will be
removed. This is important, as if such handler is the only one, then you can be
left without any logging handler and miss logs.</p>
The important thing about Erlang handlers and Elixir backends is that Erlang
handlers functions are called within caller process</strong> while Elixir backends
are called in separate process. This mean that wrongly written Erlang handler
can cause quite substantial load on application.</p>
To read on other, optional, callbacks that can be defined by Erlang handler, that
will not be covered there, I suggest looking into Erlang documentation</a>.</p>
Structured logging#</a>
</h2>
One of the biggest new features in the Elixir 1.11 is support for structured
logging. This mean that the log message do not need to be free-form string, but
instead we can pass structure, that can provide more machine-readable data for
processing in log aggregators. In Elixir 1.11 is simple as passing map as a
first argument to the Logger</code> macros:</p>
Logger</span>.</span>info</span>(</span>%</span>{</span></span>
  status</span>:</span> :</span>completed</span>,</span></span>
  response</span>:</span> :</span>ok</span></span>
})</span></span></code></pre>
This will produce message that looks like:</p>
14:08:46.849</span> [info]</span>  [response: :ok, status: :completed]</span></span></code></pre>
As we can see, the map (called report</em>) is formatted as a keyword list. This is
default way to present the report data. Unfortunately we cannot access the
metadata from the Elixir backends, but we have 2 ways to make these messages
more readable for the human operator:</p>

Utilise Logger</code>'s translators</a></li>
Using :report_cb</code> field in metadata</li>
</ol>
1st option is described quite good in Elixir documentation and is available
since Elixir 1.0 as it was used to translate error_logger</code> messages in old
Erlang versions. Here I will describe the 2nd option which provide way for
caller</strong> to define how report should be formatted into human-readable string.</p>
:report_cb</code> accepts 2 kind of functions as an argument:</p>

1-ary function, that takes report as an argument and should return tuple
in form of {:io.format(), [term()]}</code> that will be later formatted
respectively by the formatters.</li>
2-ary function that takes report and configuration map as an arguments and
should return formatted string.</li>
</ul>
1st option is much easier for most use cases, as it do not force you to worry
about handling width, depth, and multiline logs, as it will all be handled for
you.</p>
For example, instead of doing:</p>
Logger</span>.</span>info</span>("</span>Started HTTP server on http://localhost:8080</span>")</span></span></code></pre>
We can do:</p>
Logger</span>.</span>info</span>(</span></span>
  %</span>{</span></span>
    protocol</span>:</span> :</span>http</span>,</span></span>
    port</span>:</span> 8080</span>,</span></span>
    address</span>: "</span>localhost</span>",</span></span>
    endpoint</span>:</span> MyEndpoint</span>,</span></span>
    handler</span>:</span> Plug</span>.</span>Cowboy</span></span>
  },</span></span>
  report_cb</span>:</span> &</span>__MODULE__</span>.</span>report_cb</span>/</span>1</span></span>
)</span></span>
</span>
# …</span></span>
</span>
def</span> report_cb</span>(</span>%</span>{</span>protocol</span>:</span> protocol</span>,</span> port</span>:</span> port</span>,</span> address</span>:</span> address</span>})</span> do</span></span>
  {"</span>Started ~s server on ~s://~s:~B</span>", [</span>protocol</span>,</span> protocol</span>,</span> address</span>,</span> port</span>]}</span></span>
end</span></span></code></pre>
While the second entry seems much more verbose, with proper handler, it can
provide much more detailed output. Just imagine that we would have handler that
output JSON data and what information we could contain in such message:</p>
{</span></span>
  "</span>msg</span>": "</span>Started HTTP server on http://localhost:8080</span>",</span></span>
  "</span>metadata</span>": {</span></span>
    "</span>mfa</span>": "</span>MyMod.start/2</span>",</span></span>
    "</span>file</span>": "</span>foo.ex</span>",</span></span>
    "</span>line</span>":</span> 42</span></span>
  }</span></span>
}</span></span></code></pre>
Now our log aggregation service need to parse msg</code> field to extract all
information that is contained there, like port, address, and protocol. With
structured logging we can have that message available already there while
presenting the "human readable" form as well:</p>
{</span></span>
  "</span>text</span>": "</span>Started HTTP server on http://localhost:8080</span>",</span></span>
  "</span>msg</span>": {</span></span>
    "</span>address</span>": "</span>localhost</span>",</span></span>
    "</span>port</span>":</span> 8080</span>,</span></span>
    "</span>protocol</span>": "</span>http</span>",</span></span>
    "</span>endpoint</span>": "</span>MyEndpoint</span>",</span></span>
    "</span>handler</span>": "</span>Plug.Cowboy</span>"</span></span>
  },</span></span>
  "</span>metadata</span>": {</span></span>
    "</span>mfa</span>": "</span>MyMod.start/2</span>",</span></span>
    "</span>file</span>": "</span>foo.ex</span>",</span></span>
    "</span>line</span>":</span> 42</span></span>
  }</span></span>
}</span></span></code></pre>
You can see there that we can have more information available in the structured
log that would otherwise needed to be crammed somewhere into the text message,
even if it is not important in "regular" Ops observability.</p>
This can raise a question - why not use metadata for such functionality, like it
is available in LoggerJSON</code></a> or Ink</code></a>? The reason is that their reason
existence is different. Metadata meant for "meta" stuff like location, tracing
ID, but not for the information about the message itself. It is best shown on
example. For this use Elixir's implementation of GenServer</code> wrapper that
produces error log entry on unknown message handled by default handle_info/2</code>:</p>
Logger</span>.</span>error</span>(</span></span>
  # Report</span></span>
  %</span>{</span></span>
    label</span>: {</span>GenServer</span>,</span> :</span>no_handle_info</span>},</span></span>
    report</span>:</span> %</span>{</span></span>
      module</span>:</span> __MODULE__</span>,</span></span>
      message</span>:</span> msg</span>,</span></span>
      name</span>:</span> proc</span></span>
    }</span></span>
  },</span></span>
  # Metadata</span></span>
  %</span>{</span></span>
    error_logger</span>:</span> %</span>{</span>tag</span>:</span> :</span>error_msg</span>},</span></span>
    report_cb</span>:</span> &</span>GenServer</span>.</span>format_report</span>/</span>1</span></span>
  }</span></span>
)</span></span></code></pre>
As we can see there, the report contains information like:</p>

:label</code> - that describes type of the event</li>
:report</code> - content of the "main" event

:module</code> - module that created the event, it is important to notice, that
it is also present in metadata (as part of :mfa</code> key), but their meaning is
different. Module name here is meant for the operator to know the name of
the implementor that failed to handle message, while :mfa</code> is meant to
describe the location of the code that fired the event.</li>
:message</code> - the message itself that hasn't been handled. Notice, that it is
not stringified in any way there, it is simply passed "as is" to the
report. It is meant to be stringified later by the :report_cb</code> function.</li>
:name</code> - name of the process. Remember, similarly to :module</code>, the PID of
the current process is part of the metadata, so in theory we could use value
from there, but their meaning is different (additionally this one may be an
atom in case if the process is locally registered with name).</li>
</ul>
</li>
</ul>
Metadata on the other hand contains information that will be useful for
filtering or formatting of the event.</p>
The rule of thumb you can follow is:</p>

If it is thing that you will want to filter on, then it probably should be
part of the metadata. If you want to aggregate information or just display
them, it should be part of the message report.</p>
</blockquote>
Log filtering#</a>
</h2>
Finally we come to first feature that is not directly accessible from the Elixir
Logger</code> API (yet). Erlang's logger</code> have powerful functionality for filtering
log messages which allows us to dynamically decide which message should, or
should not be logged. These even can alter messages on the fly.</p>
Currently that functionality is available only via :logger</code> module. It can be
used like:</p>
defmodule</span> MyFilter</span> do</span></span>
  def</span> filter</span>(</span>log_event</span>,</span> opts</span>)</span> do</span></span>
    # …</span></span>
  end</span></span>
end</span></span>
</span>
:logger</span>.</span>add_primary_filter</span>(</span>:</span>my_filter</span>, {</span>&</span>MyFilter</span>.</span>filter</span>/</span>2</span>,</span> opts</span>})</span></span>
# Or</span></span>
:logger</span>.</span>add_handler_filter</span>(</span>handler_id</span>,</span> :</span>my_filter</span>, {</span>&</span>MyFilter</span>.</span>filter</span>/</span>2</span>,</span> opts</span>})</span></span></code></pre>
Few important things that need to be remembered when writing such filters:</p>

It is best practice to make such functions public and define filters using
remote function capture, like &__MODULE__.process_disabled/2</code> (so not
anonymous functions either). It will make such filter much easier for VM to
handle (it is bigger topic why it is that, I may to cover it in another post).</li>
Filters are ran within the same process that fired log event</strong>, so it is
important to make such filters as fast as possible, and do not do any heavy
work there.</li>
</ul>
Filters can be used for 2 different things:</p>

preventing some messages from being logged</li>
modifying a message</li>
</ul>
While the former is much more common, I will try to describe both use cases
there, as the latter is also quite useful.</p>
Filters are defined as 2-ary functions where 1st argument is log event, and
second argument is any term that can be used as a configuration for filter.
Filter should return one of these 3 values:</p>

:stop</code> - which will immediately discard message and do not run any additional
filters.</li>
:ignore</code> - which mean that given filter didn't recognise the given message
and leaves it up to other filters to decide on the action. If all filters
return :ignore</code> then :filter_default</code> option for the handler will be taken.
By default it is :log</code>, which mean that message will be logged, but default
handler has it set to :stop</code> by default, which mean, that non-matching
messages will be discarded.</li>
Just log event (possibly modified) that will cause next filter to be called
with altered message. The message returned by the last filter (or in case of
:ignore</code> return, previous filters) will be the message passed to handler.</li>
</ul>
Preventing some messages from being logged#</a>
</h3>
Most common use-case for filters will probably be rejecting messages that aren't
important for us. Erlang even prepared some useful filters</a>:</p>

domain</code> - allow filtering by metadata :domain</code> field (remember as I said
that metadata is for filtering?). It supports multiple possible relations
between the log domain and defined domain.</li>
level</code> - allow filtering (in or out) messages depending on their level, in
both directions. It will allow you to filter messages with higher level for
some handlers. Just remember, that it will not receive messages that will not
pass primary/module level.</li>
progress</code> - filters all reports from supervisor</code> and
application_controller</code>. Simply, reduces startup/process shutdown chatter
that often is meaningless for most time.</li>
remote_gl</code> - filters messages coming from group leader on another node.
Useful when you want to discard/log messages coming from other nodes in
cluster.</li>
</ul>
Modifying a message#</a>
</h3>
Sometimes there is need to alter messages in the system. For example we may need
to prevent sensitive information from being logged. When using "old" Elixir
approach you could abuse translators, but that was error prone, as first
successful translator was breaking pipeline, so you couldn't just smash one on
top and then keep rest working as is. With "new" approach and structured logging
you can just traverse the report and replace all occurrences of the unsafe data
with anonymised data. For example:</p>
def</span> filter_out_password</span>(</span>%</span>{</span>msg</span>: {</span>:</span>report</span>,</span> report</span>}}</span> =</span> event</span>,</span> _opts</span>)</span> do</span></span>
  %</span>{</span>event </span>|</span> msg</span>: {</span>:</span>report</span>,</span> replace</span>(</span>report</span>)}}</span></span>
end</span></span>
</span>
@</span>filtered</span> "</span>[FILTERED]</span>"</span></span>
</span>
defp</span> replace</span>(</span>%</span>{</span>password</span>:</span> _</span>}</span> =</span> map</span>)</span> do</span></span>
  for</span> {</span>k</span>,</span> v</span>}</span> <-</span> %</span>{</span>map </span>|</span> password</span>:</span> @</span>filtered</span>},</span> into</span>:</span> %</span>{}</span> do</span></span>
    {</span>k</span>,</span> replace</span>(</span>v</span>)}</span></span>
  end</span></span>
end</span></span>
</span>
defp</span> replace</span>(</span>%</span>{"</span>password</span>"</span> =></span> _</span>}</span> =</span> map</span>)</span> do</span></span>
  for</span> {</span>k</span>,</span> v</span>}</span> <-</span> %</span>{</span>map </span>|</span> "</span>password</span>"</span> =></span> @</span>filtered</span>},</span> into</span>:</span> %</span>{}</span> do</span></span>
    {</span>k</span>,</span> replace</span>(</span>v</span>)}</span></span>
  end</span></span>
end</span></span>
</span>
defp</span> replace</span>(</span>list</span>)</span> when</span> is_list</span>(</span>list</span>)</span> do</span></span>
  for</span> elem </span><-</span> list </span>do</span></span>
    case</span> elem </span>do</span></span>
      {</span>:</span>password</span>,</span> _</span>}</span> -></span> {</span>:</span>password</span>,</span> @</span>filtered</span>}</span></span>
      {"</span>password</span>",</span> _</span>}</span> -></span> {"</span>password</span>",</span> @</span>filtered</span>}</span></span>
      {</span>k</span>,</span> v</span>}</span> -></span> {</span>k</span>,</span> replace</span>(</span>v</span>)}</span></span>
      other </span>-></span> replace</span>(</span>other</span>)</span></span>
    end</span></span>
  end</span></span>
end</span></span>
</span>
defp</span> replace</span>(</span>other</span>),</span> do: other</span></span></code></pre>
This snippet will replace all occurrences of :password</code> or "password"</code> with
filtered out value.</p>
The disadvantage of such approach - it will make all messages with such fields
allowed in case if your filter has :filter_default</code> set to :stop</code>. That mean,
that if you want to make some of them rejected anyway, then you will need to
manually add additional step to reject messages that do not fit into your
patterns. Alternatively you can use filter_default: :log</code> and then use opt-out
logging. There currently is no way to alter the message and make other filters
decide whether log it or not (as of OTP 24).</p>
Summary#</a>
</h2>
New features and possibilities with relation to logging in Elixir 1.11 can be
overwhelming. Fortunately all of the new features are optional and provided in
addition to "good 'ol Logger.info("logging")</code>". But for the people who works on
the observability in BEAM (EEF Observability WG, Sentry, Logflare, etc.) it
brings a lot of new powerful capabilities.</p>
I am thrilled to see what will people create using all that power.</p>


Treachery of Representation
2020-04-27T00:00:00+00:00
</p>
I had that discussion on different chats few times already, but it is always
troublesome for people (not just programmers):</p>

Representation is not data</strong></p>
</blockquote>
And this concept is confused a lot</strong>.</p>
Almost any of you know that you can do ping 1.1.1.1</code>, however did you know that
you can also do ping 16843009</code> and the result will be the same? Magic? Not
really. What we commonly treat as an IPv4 address 1.1.1.1</code> is only textual
representation of the IPv4 which in reality is single 32-bit number which is
represented as dot separated 8-bit values just out of convenience. The same
applies to SHA-1 hashes, UUIDs, etc.</p>
We people are great in finding patterns and simplifications, so when one sees
something like:</p>
-----BEGIN PGP PUBLIC KEY BLOCK-----</span></span>
</span>
mQINBFd1CwMBEADTJG+QszZ9JK6G+IZpvK/g8mJicqKVTV1OmI1toIml5c/5r/70</span></span>
dBc1FuASTM58PPZYJoKkbIh489QIvTm+8UdYu9zNRkT33el7pW1lEe5A6sq2bvoY</span></span>
GE8DioHTewdoSDPf7KIaZQeRcIY4n93rUOwA6GEtNpyYGP6ss5Y7BGk4ltSioe7N</span></span>
o+4ZmWoQZwZzRon3t1eWVVP9tRMG5nLzL3XA5FK8TqoYuuIOR0Kw9+yK/CIJi6EV</span></span>
NGXidQEhRX/e6cDcqmo6LQuQluuhpwKjD3rVIq5P55kDcyUXHkcr9/RBIY1GCw0m</span></span>
r03H053xYhPib4S3fkcGMz1zrvgPR8M6T8VYXERZDjKzQbkkjKx2umP+MQb57N/m</span></span>
Ndr+vks5PoPOPTGS9DYPaWf0fTVWQz/Wr6SuqKYWOObw2Ql1fOFS9dlQ26VHqtdP</span></span>
CjCS/+kxTWP0SNIlvmICj68JNq0Elti8DNF+8K4U/zjqphqLUOMYo+bfxF3Xudlx</span></span>
…</span></span>
X5tXXRd2Jx3rkquLhlOaIfwx5ZKkc0jFvFq6L6iQ8M3w2osTTQ8wzLVrKcA/1E37</span></span>
vzFw0eLlAHxqKKx8AGXhlG+bu4z9</span></span>
=HbWE</span></span>
-----END PGP PUBLIC KEY BLOCK-----</span></span></code></pre>
Then they will automatically say that this is PGP public key, however it is not!
This is only armored (textual) representation of such. We can find infinite
amount of ways to represent this in different formats that will suit our needs.
We can use binary format, we can post it as huge integer:</p>
1657454081776872747238013876359322713234391537499703933766980145269184386317049</span></span>
8330933605635736383746482612365150874768089262930090931944711968773749521913699</span></span>
7027071169368463413472374786860753667060249108643195012586135134901393013828317</span></span>
1125513524511702912188470760919764295140961468869648430645347829208242125220011</span></span>
3151968877120814976862830922750002170809783899223077039784557147675345249970781</span></span>
5464553976008128226157133495317638249211710007765696991167161819945117066390428</span></span>
3919051957061561015867974004807638646034325543036054946820462087690124141551114</span></span>
9854678368254378572150652196681912690617688434615368858186497935662439355552496</span></span>
4729580502224487557084501045154245589928832642849687361389957940229955065758222</span></span>
6366651257907657297666835432894495612402178521246832570964915000759172307812244</span></span>
6160593540459285722105074344450899751799861113519463231780643100018160335021016</span></span>
1172711806124926227346822406879926575104066365513850591901167977031017773147553</span></span>
3494796072257580113875396010361380353105632085492892479603715336182682602428143</span></span>
5608690637230685977144014723666813959973503662453370415571851701179428151882882</span></span>
8889353256564968091064157105454996484998754991348566666564916601887263240881070</span></span>
4832198618558083182213501188928625277122728831796445123889707032846261305648630</span></span>
2527176048898518342591580941543224978208167594213111446673112848581061191825984</span></span>
1291021894031447958561527430552161385666081569185835507259834732917990092146589</span></span>
8117077934368522739843309368028814069423002395997449936891495583107309283509271</span></span>
1180077432800679370662483747523408021854844321836016009650358574989381886337852</span></span>
1722516883913145519094014205375128862340002341518180047810642843228568880540172</span></span>
3417922325879216322183112046755680152782225539937453280161300294135511699130848</span></span>
0976433759293246140554344092404212626999402181891603820290765820423321186458935</span></span>
1984485518225135337585187247603955326905920439923621172092130464391731498388744</span></span>
8750392578000721268878463072511675222925230106406594118598329633269324358794764</span></span>
4929066475821912729913815944439237132097525066761248367529760023296968812652711</span></span>
1354078624366444615698555226671150629567656072692570827367021777825853318746140</span></span>
2863318577339635525264674498700520998767957283494612638659115064541050609931927</span></span>
0626267703727694820552442716314957841578389580288169926927235290811137237887434</span></span>
4221841397313800123531757418475100660454200912589623058183943877051413097869194</span></span>
2339339754985429142461984170624136785051296102102697817632037910221364646864651</span></span>
9855796964298457614203439681372147302465919398126442059583983219983682993673878</span></span>
5195511144366725918343436684113783818943460018156359275075623210303933783990867</span></span>
1450770230403557029292899962005967013424432778295850913345133681904640414130225</span></span>
6074976712103846157531047532174675106877834562907257767751866399263004265826476</span></span>
2927779415738730155062074319337848630585696081256539500213340093725740025118088</span></span>
1549556332762444079547546812082703704589345493095057789148939194264110489919019</span></span>
4566170053142149254913384680974964028594308712215301647785051390937700375621904</span></span>
6850793920957951588471978130651986351908543266002688052663010508763853950782008</span></span>
2184664457189708083557342565742275344614046037371745586008320578024563750973078</span></span>
8918522749313602046891225282398255933496131418430638794803291508377033406843725</span></span>
5993318777198284973711860102116509775279126577240620025658230260626934855295524</span></span>
0163482197624118826096325983427230149324714688555788577159485966979794672252786</span></span>
2049579060682321390550407916111513505775215286804094663126348519235979240267931</span></span>
1393177415036576603095965158893137047832256486627193835346772464826381170364046</span></span>
9672727436992842647585191196834755720506018282532384722445185130057363544741494</span></span>
9620010167886013834746116804462363836560880757170533362435237472884909654085819</span></span>
2200315846789415790035979932125380804930427597237917426179754754825031469057587</span></span>
5408385218092192305802900158223031488228733997920493838052062060383233685427730</span></span>
2479303637471252558356068491271641497786414938837109276566129349124360527596986</span></span>
0939690673802611552820885948021359288569430897510414990828683956519429277901630</span></span>
3954037665434237643444138532861028612543250728043306643540240455527506302759025</span></span>
2547253114427400839942700506775082503198395916277602696346477632254520324291510</span></span>
7176423995375530679412205690580599798550778233049642580773735945645101772412041</span></span>
9770061447364487938863821369661007066965634140967634082532169232884947757170676</span></span>
4001778436464233464495522570739848976059303542657842980445747616892332657374307</span></span>
3505719849994753220744681385854225182109628156151628193674787620122409837550175</span></span>
0873904989000402627417710830371222663052469412680310145900108150056384479011904</span></span>
2835209300157229749112634620468892621559922849964862659975166458450846483410310</span></span>
9108533277021935459848755879666329563851612903497154909548344186653653532596030</span></span>
1909102262515819267424216040525600483418895125777391628629285735284199384249913</span></span>
4569616259150374342828948745370477195614375418738814522169402762983961841736104</span></span>
6795438095408475881223956725863549758305981513146677968273034768754628733774741</span></span>
5601175386054118189600876918455578756071642172440419960753296039010117418642333</span></span>
0700157761536698272554982837358908293635750455157297675798024638098958969267103</span></span>
6863406950021125691193872800648046146136177370993251603456335654259717488367827</span></span>
4221891453042458615303763101765455461453483141839094655158338636716818847773378</span></span>
3089958604524376415306549473187862340575705348685942443697841897636463859949926</span></span>
4444464568286807145435412305718175821967958934523535510530750957533464207224580</span></span>
2279373368828019548205439249352189164575083536583721094688224573324098240189090</span></span>
5336968277502282844656778779596324101840145138970262636945152077842421376473220</span></span>
7492641493346476621126619083363859115527821109445021807274717048079752479814069</span></span>
8489075707866445180273917549185493159488867602009975459154326516617446093785735</span></span>
7657409659738148136025968774346968795138906298743560267773358615395905430092434</span></span>
4090587183336112957335092481531071573600577415752552329335484584825448451281828</span></span>
8397267947223750920443272557916925997060835636847307503958963847413230458654999</span></span>
1510218368567378788057116023931994479266585942028846093606765504042151064449401</span></span>
7535053560571943565326163798733090077562063019270432774405336624206120162684878</span></span>
2795961016400131165092351347259003395336488326011264715606875560607985355101479</span></span>
1760498444803027184543196012361666512501664115266209446783611568980849759393366</span></span>
9004607206787763941830633066297736259437811557856982004496007136271945140295164</span></span>
1021123058841472126203195417354602182201593697351601555663719500088120144234523</span></span>
6489841188841869277921900098197497636848312736842577887524394082685476810652434</span></span>
4368023998666968889307055236194661569426524463555778731728379410827687126847841</span></span>
9767542567328134223554830928399001003746312827667717886760656045983730239481725</span></span>
0247192299979431368314503262526820359421826762698555934534502139292008724527340</span></span>
8566373287784944330515935635629564982235980178134875308234950470548865802766425</span></span>
3860142806266365958072884321048903006167833769100547151312160642686817573070175</span></span>
8517347856656124090595544002510378196552188147101332012713619529417880258338847</span></span>
9548595874229820385574539354556804227765871124740709294288382971201998678678558</span></span>
1135393032892116765181068789505132747404363007086649489491404624220840121961493</span></span>
9770980821410301887703418151100099801563662384204228166203260763728750184267283</span></span>
0131194615371938999731752172864595591147593535783229858980094812827904216900881</span></span>
0264484164635513605434976822336090709152508883040986176420243997392303278619543</span></span>
7132483818824284005791373572104029624554831023783619373272102143855002602772145</span></span>
1625555397693952027234647740913597061447733291018249879406776786602117249183252</span></span>
3028866932885668665387107476073897428999191152484269358567755332923524842064600</span></span>
3794929412192609471679420879441380627388660584277219132508113058600032222404868</span></span>
2912812185436951502936757180777020507704813293182422695970566889605944405493165</span></span>
1013050164683342445192514465075993459075480256309099680013258781889270979210377</span></span>
2978191200620386753171174166739274414356307635252359045515977252139725132495302</span></span>
4643451106675617568043015073444443099894159838269138387489278049925887086707284</span></span>
7829572513234800005657626866917894409850513692287942317924246061209211855433159</span></span>
9474862901923613493886199119281189221558415812534549155894713481042125093206748</span></span>
5327877884372067603958264162802943954426330867112852706462838411380034486572690</span></span>
0832825677844671600199639823373591008100835318576086939856769129801540454104086</span></span>
1677822021197754242364262145059312498290181267047980332959890135114462272477083</span></span>
2267001649763807453258389353505463199327476285470780889266423077199955133598911</span></span>
4228000626496272875548822896127419699372645929502724379809375313984715132685176</span></span>
4622127184084472238931211915326009413737074357833496656532089093339434182463301</span></span>
4308133856938907909683630708624133178764345576977957526210285792755588339502313</span></span>
8908965753207073692034844982785108219589789141504389489294521808974273153775683</span></span>
4710352841094853913320997396720648350856909009547717827126308937619267105900330</span></span>
5101845147776164022119091114165228791856934734405699522377750777712347632477472</span></span>
1009173135645423776920173241282009759190193451325534130721465706915799208352319</span></span>
7599436014849832502702741642253476812447287458898351283245611697635355626967353</span></span>
1666248190388798775893317097285333032144668959144551590316718826094121548655064</span></span>
7327190413743862383571166391385591854324294967408637687666499017919659934556270</span></span>
6341760473807628223322195796330107447627192021125501916257505545056806593020009</span></span>
7877518699192190569125611877551646125360935329152981694983591991442830972118425</span></span>
6024578420108611596786779391910078283535602332400676206290364979979566874130677</span></span>
6262721807685562884678614519635750114010818380727834704599313495363241399979019</span></span>
0070441245844791323697888362361156445150402411489057842369956860981561106349532</span></span>
0427329254749281230185541338204013332686802017845355478766281601177626930879718</span></span>
3924781109376832016663185405467845874107109473909888472781931298639743129405819</span></span>
9018728534014004893438844921156199392052668509822766055873031732621412819415997</span></span>
5106658073874599600482906898626458321908644717727705714148184312035554569461172</span></span>
2571014234210898150511238337764888383510750920318252617219954833323650772315822</span></span>
8694765635893382122506622111504457282650772215884112102663088591828929423478423</span></span>
4630228876316754232955789012778765153858505693804419173581459528959967049577393</span></span>
3203503330240909044864203529502068033769777504652970513017398216284525619584395</span></span>
7399576075789549294657904493999442658423657409794863027890740986637503143358200</span></span>
1934990177507763078658788020042351989886877390781804018450451154174423611071016</span></span>
3402098758156739497049080279046203857466871438452835345118677125325469128068133</span></span>
1366454054797295342986762057003055457075865166635455089017691661317421628642115</span></span>
4993805637263456481774330967345531754745117304815930559577694749501153002539548</span></span>
5404256140871044730263074060283878378820730250169561914787775689729320033889071</span></span>
6450665587233640719268830704525315924153018596256798187945369509492198616718735</span></span>
1727110697661735449130358956047142033543848955222885833760237898733698724067547</span></span>
8419101187420106748225373809326555034920972273032855852671672606413074751824780</span></span>
5154501659974029329400410688611596507780739224024571262805816781734162995302986</span></span>
6056046847929263719956354500493986529434154889061593903845247605955487265601755</span></span>
6043989537334852057279780609368927725387526205357590787648273036069019155497325</span></span>
7284579301234574132272242978804776467287642325689728908190752748762036032550553</span></span>
0297580281513471442798728664035839834832168788094160691038978903832691745390723</span></span>
8280774908923640913005731937007494753031663001263187085109076785504199996190411</span></span>
2645973159719700472616661742318567810513248327998862965177705803004760763074321</span></span>
2313603449759152065701241959565937761991117902298693865777609004090724789824238</span></span>
3972666226870968534310397976577839690259724502877323852210276007099716458281622</span></span>
0266833622323608118226700555948541874878778700732590694206338358355846043998302</span></span>
2076998169472117379362890485627714537422759747305703434242979589456473320420530</span></span>
3798100254154945537756689378818628238489975669085399495869198707118002332356201</span></span>
2860990261282434269797439143900692337984204234325905652408969984088094424603914</span></span>
1324281031601472803905094128246361871536795078781179969095901324385165589352908</span></span>
5570928042409387112284705341815718794836739937616493333506970623215543845017075</span></span>
9783614942442566404035374016694617857746139989520230332862060498894309587704609</span></span>
2307147218032013573247425767983972450327115251550658313094143782360088885333289</span></span>
5608157274415946459185147621933085094399365522400749862053761657227728398780469</span></span>
7624550738125029927864820017089079162293784666905532642099274666498693255031122</span></span>
2889026525933407361948759134200215183348933115884125348251449445229741605009136</span></span>
5213445919902677380093249284489832185317941999559970629694476644944502557922299</span></span>
4045467263054565580347599600913022630568314834909143526070011238471771326443583</span></span>
5999517514967645625805546197521789593944893957635299668589873242979729214565080</span></span>
2261124045260532526154306748833731566597485067054072337617187788422439611692535</span></span>
6816154352988676611143089545607089006494597803681742408872009589972348314218354</span></span>
5885756829028538416517815718956811889863727858335798002033148984814372442071252</span></span>
2067544365025698588181983841271622441543580184070308095038169545529105805899001</span></span>
5825565028451412722300055213104773432923209992505613530612435181227942384006400</span></span>
6523874291660768676182135558923684054785558197662748591717032047925055385422584</span></span>
8826587112169068650291428518318455696828296955217726852703371476729712500873714</span></span>
9472757285042561083989577043290348125840395005244750850424202067248954262318073</span></span>
3815815411200758900340511052777147951522351782525348580404031636340281256009099</span></span>
3622685663270098759448305092624146163465790024589793366278716865415624355235291</span></span>
3985878069536487297514629837101726802175517221106646456169044468305875210017956</span></span>
3762872046696988950422497705388225997904571624738961093738188347083184831994615</span></span>
5653340873908308859802165057177945265893576798033839250748977766722025452484918</span></span>
4969906442004769360577067228954632576483279588054785969942183381969384204862823</span></span>
5527295851917376805942967603231125591852023457112648669282453139677076502138626</span></span>
9831545895899997311194908866099892662200438185858903893802349550297108232343975</span></span>
7063807807</span></span></code></pre>
Before you ask - yes, this is my PGP public key as a single decimal integer.</p>
Next time you will see someone asking for "shorter UUID" you can point them
there and say - just pick shorter representation.</p>


Stop Spreading Crap at My `$HOME`
2020-04-15T12:00:19+02:00
Disclaimer:</strong> Yes, this is a rant. Yes, I am displeased with state of
software. And yes, I can use some harsh words. You were warned.</p>
For a some time now we have Filesystem Hierarchy Standard</a> which describes
which data goes where in your *nix installation. In fact we have that for 28
years, which is almost as long as I live. This is quite some time. But for
whatever reason we apply that only for the system hierarchy and we cannot have
the same for something much closer to our heart - our dear $HOME</code>.</p>
Common practice for storing all the stuff for different applications is to use
dot files stored in $HOME</code>. These are files or directories starting with dot in
their name which makes such files "hidden" on Unix systems. There is problem
though. First let me show you all the files starting with dot in my $HOME</code>:</p>
-r--------  1 hauleth staff     9 Mar 30  2019 .CFUserTextEncoding</span></span>
-rw-r--r--  1 hauleth staff 16388 Apr  9 11:46 .DS_Store</span></span>
drwx------  2 hauleth staff    64 Apr 13 20:54 .Trash</span></span>
-rw-------  1 hauleth staff 17835 Apr 10 21:17 .bash_history</span></span>
drwx------ 12 hauleth staff   384 Mar 30  2019 .bash_sessions</span></span>
drwxr-xr-x 13 hauleth staff   416 Mar 26 14:02 .cache</span></span>
lrwxr-xr-x  1 hauleth staff    40 Aug 21  2019 .chunkwmrc</span></span>
drwxr-xr-x 20 hauleth staff   640 Apr 15 21:59 .config</span></span>
drwx------  3 hauleth staff    96 Apr 20  2019 .cups</span></span>
lrwxr-xr-x  1 hauleth staff    39 Aug 21  2019 .curlrc</span></span>
lrwxr-xr-x  1 hauleth staff    43 Aug 21  2019 .dir_colors</span></span>
-rw-r--r--  1 hauleth staff  2587 Sep 10  2019 .direnvrc</span></span>
drwxr-xr-x  6 hauleth staff   192 Apr 14 18:27 .docker</span></span>
drwx------ 14 hauleth staff   448 Oct  1  2019 .dropbox</span></span>
-r--------  1 hauleth staff    20 Apr  1  2019 .erlang.cookie</span></span>
drwxr-xr-x  3 hauleth staff    96 Jul  1  2019 .gem</span></span>
-rw-r--r--  1 hauleth staff   518 Mar 25 22:44 .gitconfig</span></span>
drwx------ 14 hauleth staff   448 Apr 15 22:03 .gnupg</span></span>
drwxr-xr-x  4 hauleth staff   128 Apr  2  2019 .hammerspoon</span></span>
drwxr-xr-x  5 hauleth staff   160 Apr 14 19:39 .hex</span></span>
drwx------  3 hauleth staff    96 Apr 26  2019 .httpie</span></span>
-rw-r--r--  1 hauleth staff   165 Feb 22 13:50 .jlassetregistry.json</span></span>
drwxr-xr-x  3 hauleth staff    96 Nov 29 22:13 .jssc</span></span>
drwxr-xr-x  9 hauleth staff   288 Feb 21 17:46 .julia</span></span>
drwx------ 14 hauleth staff   448 Oct  2  2019 .keychain</span></span>
drwxr-x---  5 hauleth staff   160 Jan 24 12:20 .kube</span></span>
-rw-------  1 hauleth staff   720 Apr 15 16:29 .lesshst</span></span>
drwxr-x---  3 root    staff    96 Apr  3  2019 .lldb</span></span>
drwx------  4 hauleth staff   128 Dec 18 14:16 .local</span></span>
drwxr-xr-x  8 hauleth staff   256 Mar 20 12:29 .mitmproxy</span></span>
drwxr-xr-x 18 hauleth staff   576 Apr 13 14:36 .mix</span></span>
-rw-r--r--  1 hauleth staff   116 Apr 13 20:59 .nix-channels</span></span>
drwxr-xr-x  4 hauleth staff   128 Apr 14 13:47 .nix-defexpr</span></span>
lrwxr-xr-x  1 hauleth staff    46 Feb  1 22:13 .nix-profile</span></span>
drwxr-xr-x  3 hauleth staff    96 Sep 27  2019 .npm</span></span>
drwxr-xr-x  3 hauleth staff    96 Dec 19 14:05 .pex</span></span>
-rw-r--r--  1 hauleth staff   183 May  2  2019 .profile</span></span>
drwxr-xr-x  5 hauleth staff   160 Mar 20 16:27 .proxyman</span></span>
drwxr-xr-x  2 hauleth staff    64 Feb 17 14:09 .proxyman-data</span></span>
-rw-------  1 hauleth staff  1391 Jan 22 11:46 .psql_history</span></span>
-rw-------  1 hauleth staff  1950 Feb 19 12:42 .python_history</span></span>
lrwxr-xr-x  1 hauleth staff    37 Aug 21  2019 .skhdrc</span></span>
drwx------ 11 hauleth staff   352 Oct  2  2019 .ssh</span></span>
drwxr-xr-x  4 hauleth staff   128 May  8  2019 .terraform.d</span></span>
drwxr-xr-x  4 hauleth staff   128 Dec  1 21:52 .thumbnails</span></span>
drwxr-xr-x  9 hauleth staff   288 Mar  9 17:14 .vagrant.d</span></span>
-rw-------  1 hauleth staff 11124 Apr 13 20:21 .viminfo</span></span>
lrwxr-xr-x  1 hauleth staff    37 Aug 21  2019 .vimrc</span></span>
drwx------  4 hauleth staff   128 Nov  8 16:51 .w3m</span></span>
drwxr-xr-x 37 hauleth staff  1184 Apr 13 21:21 .weechat</span></span></code></pre>
As you can see, there is bunch of them, some are linked to my dotfiles
repository, but most are not. What is the problem there?</p>
The spread.</p>
All of the data is smeared between gazillion of different filed with different
types of data. This causes headaches because:</p>

I cannot easily backup all the configuration as I need to check each file
independently to check if this is configuration file or it is data</li>
I cannot exclude data from my backups in a uniform way. I need to check each
file independently and exclude it, remembering to do that for each new tool
that I add. Alternatively I can use whitelist instead of blacklist of paths
that I want to backup, but that pretty much defies the idea of having
automatic backup.</li>
Cleaning up the old caches and data is troublesome as user need to review all
the folders and know what data they see mean.</li>
</ul>
But what We can do? Well, the macOS and Windows got it somewhat right, these
OSes provide special paths for storing all configuration, caches, user-data,
etc. in special, dedicated, and well known locations within user directories. It
is like mentioned earlier FHS but for $HOME</code>.</p>
Unfortunately no other OS out there had:</p>

Power to force developers to behave reasonably</li>
Standard to which developers should adhere to</li>
</ol>
Unfortunately the 1st point is still true, but the 2nd one is somewhat resolved
in form of XDG Base Directory Specification</a>. This is pretty short,
simple, and straightforward spec where which data should go.</p>
This makes life so much easier:</p>

Want to have backup your configuration? Just copy $HOME/.config</code> where you
want.</li>
Want to reset your configuration to base one? Just delete $HOME/.config</code>.</li>
Want to cleanup caches? Remove $HOME/.cache</code>.</li>
Etc.</li>
</ul>
It makes your $HOME</code> much cleaner, much more manageable, much more like your
place, where you are the ruler, not as a manager of the bulletin board or the
storage house.</p>
At one point I was even considering to have approach similar to one taken by
Graham Christensen</a>, but that would make my life even more
miserable instead of forcing developers to fix their software.</p>
Just in case, this is not only mine view on the state of configuration files,
just see old Google+ post by Rob Pike</a> where he states exactly the same
thing that FreeDesktop team try to address. I may not agree on all the concepts
with Commander, but this one I vote all-fours.</p>
For these who want to stop that madness and fix at least some software out there
that is broken, you can check out my script</a> that tries to fix
(at least partially) non-conforming software.</p>

TL;DR</strong></p>
Please, stop leaving your c**p in my $HOME</code> and call it "installation". If
you want that, then I think you should find nearest Modern Art Gallery.</p>

You can comment it on Lobste.rs</a>
Reddit</a>
or @ me on Twitter.</p>


Writing Vim Plugin
2019-11-04T18:21:18+01:00
While there are many "tutorials" for writing plugins in Vim, I hope this one
will be a little bit different from what is out there, because it won't be
about writing plugin per se</em>. If you want to find information about that then
you should check out :h write-plugin</code></a>. I want this article to be about how
plugins come to life, using my own experience on writing
vim-backscratch</code></a> as an example.</p>
Problem#</a>
</h2>
All plugins should start with a problem. If there is no problem, then there
should be no code, as there is no better code than no code</a>. In this
case, my problem was pretty trivial: I wanted a temporary buffer that would let
me perform quick edits and view SQL queries while optimising them (and run them
from there with Tim Pope's dadbod</a>).</p>
"Simple obvious solution"#</a>
</h2>
Now that we have defined the problem, we need to try the first possible solution.
In our case, it is opening a new buffer in a new window, edit it, and then close it
when no longer needed. It is simple in Vim:</p>
:new</span></span>
" Edits</span></span>
:bd</span>!</span></span></code></pre>
Unfortunately this has bunch of problems:</p>

if we forgot to close that buffer, then it will hang there indefinitely,</li>
running :bd!</code> in the wrong buffer, can have unpleasant consequences,</li>
this buffer is still listed in :ls</code>, which is unneeded (as it is only
temporary).</li>
</ul>
Systematic solution in Vim#</a>
</h2>
Fortunately Vim has solutions for all of our problems:</p>

the "scratch" section in :h special-buffers</code></a>, which
solves the first two problems,</li>
:h unlisted-buffer</code></a>, which solves the third problem.</li>
</ul>
So now our solution looks like:</p>
:new</span></span>
:</span>setlocal</span> nobuflisted buftype</span>=</span>nofile </span>bufhidden</span>=</span>delete </span>noswapfile</span></span>
" Edits</span></span>
:bd</span></span></code></pre>
However that is a long chain of commands to write. Of course we could condense
the first two into a single one:</p>
:new </span>++</span>nobuflisted</span> ++</span>buftype</span>=</span>nofile </span>++</span>bufhidden</span>=</span>delete </span>++</span>noswapfile</span></span></code></pre>
But in reality this does not shorten anything.</p>
Create a command#</a>
</h2>
Fortunately we can create our own commands in Vim, so we can shorten that to a
single, easy to remember command:</p>
command!</span> Scratch new </span>++</span>nobuflisted</span> ++</span>buftype</span>=</span>nofile </span>++</span>bufhidden</span>=</span>delete </span>++</span>noswap</span></span></code></pre>
For better flexibility, I prefer it to be:</p>
command!</span> Scratchify </span>setlocal</span> nobuflisted buftype</span>=</span>nofile </span>bufhidden</span>=</span>delete noswap</span></span>
command!</span> Scratch new </span>+</span>Scratchify</span></span></code></pre>
We can also add a bunch of new commands to give us better control over our new
window's location:</p>
command!</span> VScratch vnew </span>+</span>Scratchify</span></span>
command!</span> TScratch tabnew </span>+</span>Scratchify</span></span></code></pre>
Those commands will open a new scratch buffer in a new vertical window, and
a new scratch buffer in a new tab page, respectively.</p>
Make it a more "vimmy" citizen#</a>
</h2>
While our commands :Scratch</code>, :VScratch</code>, and :TScratch</code> are nice, they are
still not flexible enough. In Vim we can use modifiers like :h :aboveleft</code></a> to define exactly where we want new windows to appear
and our current commands do not respect that. To fix this problem, we can
simply squash all the commands into one:</p>
command!</span> Scratch </span><</span>mods</span>></span>new </span>+</span>Scratchify</span></span></code></pre>
And we can remove :VScratch</code> and :TScratch</code> as these can be now done via
:vert Scratch</code> and :tab Scratch</code> (of course you can keep them if you like, I
just wanted the UX to be minimal).</p>
Make it powerful#</a>
</h2>
This has been in my $MYVIMRC</code> for some time in the form described above until
I found out Romain Lafourcade's snippet</a> that provided one additional
feature: it allowed to open a scratch buffer with the output of a Vim or shell
command. My first thought was - hey, I know that, but I know I can make it
better! So we can write a simple VimL function (which is mostly copied from
romainl's snippet, with a few improvements):</p>
function!</span> s:scratch</span>(</span>mods</span>,</span> cmd</span>)</span> abort</span></span>
    if</span> a:cmd</span> is</span>#</span> ''</span></span>
        let</span> l:output</span> =</span> []</span></span>
    elseif</span> a:cmd[</span>0</span>] </span>is</span>#</span> '!'</span></span>
        let</span> l:cmd</span> =</span> a:cmd</span> =~</span>' %'</span> ?</span> substitute</span>(</span>a:cmd</span>,</span> ' %'</span>,</span> ' '</span> . </span>expand</span>(</span>'%:p'</span>),</span> ''</span>)</span> : </span>a:cmd</span></span>
        let</span> l:output</span> =</span> systemlist</span>(</span>matchstr</span>(</span>l:cmd</span>,</span> '^!\zs.*'</span>))</span></span>
    else</span></span>
        let</span> l:output</span> =</span> split</span>(</span>execute</span>(</span>a:cmd</span>),</span> "\n"</span>)</span></span>
    endif</span></span>
</span>
    execute</span> a:mods</span> . </span>' new'</span></span>
    Scratchify</span></span>
    call</span> setline</span>(</span>1</span>,</span> l:output</span>)</span></span>
endfunction</span></span>
</span>
command!</span> Scratchify </span>setlocal</span> nobuflisted noswapfile buftype</span>=</span>nofile </span>bufhidden</span>=</span>delete</span></span>
command! -nargs=1 -complete=command</span> Scratch </span>call</span> <</span>SID</span>></span>scratch</span>(</span>'<mods>'</span>, <</span>q-args</span>>)</span></span></code></pre>
The main differences are:</p>

special case for empty command, it will just open an empty buffer,</li>
use of is#</code> instead of ==</code>,</li>
use of :h execute()</code> instead of :redir</code>.</li>
</ul>
As it is quite self-contained and (let's be honest) too specific for $MYVIMRC</code>
we can can extract it to its own location in plugin/scratch.vim</code>, but to do so properly we need
one additional thing, a command to prevent the script from being loaded twice:</p>
if</span> exists</span>(</span>'g:loaded_scratch'</span>)</span></span>
    finish</span></span>
endif</span></span>
let</span> g:loaded_scratch</span> =</span> 1</span></span>
</span>
function!</span> s:scratch</span>(</span>mods</span>,</span> cmd</span>)</span> abort</span></span>
    if</span> a:cmd</span> is</span>#</span> ''</span></span>
        let</span> l:output</span> =</span> []</span></span>
    elseif</span> a:cmd[</span>0</span>] </span>is</span>#</span> '!'</span></span>
        let</span> l:cmd</span> =</span> a:cmd</span> =~</span>' %'</span> ?</span> substitute</span>(</span>a:cmd</span>,</span> ' %'</span>,</span> ' '</span> . </span>expand</span>(</span>'%:p'</span>),</span> ''</span>)</span> : </span>a:cmd</span></span>
        let</span> l:output</span> =</span> systemlist</span>(</span>matchstr</span>(</span>l:cmd</span>,</span> '^!\zs.*'</span>))</span></span>
    else</span></span>
        let</span> l:output</span> =</span> split</span>(</span>execute</span>(</span>a:cmd</span>),</span> "\n"</span>)</span></span>
    endif</span></span>
</span>
    execute</span> a:mods</span> . </span>' new'</span></span>
    Scratchify</span></span>
    call</span> setline</span>(</span>1</span>,</span> l:output</span>)</span></span>
endfunction</span></span>
</span>
command!</span> Scratchify </span>setlocal</span> nobuflisted noswapfile buftype</span>=</span>nofile </span>bufhidden</span>=</span>delete</span></span>
command! -nargs=1 -complete=command</span> Scratch </span>call</span> <</span>SID</span>></span>scratch</span>(<</span>q-mods</span>>, <</span>q-args</span>>)</span></span></code></pre>To boldly go…#</a>
</h2>
Now my idea was, hey, I use Vim macros from time to time, and these are just
simple lists of keystrokes stored in Vim registers. Maybe it would be nice to have
access to that as well in our command. So we will just add a new condition that
checks if a:cmd</code> begins with the @</code> sign and has a length of two. If so, then
set l:output</code> to the spliced content of the register:</p>
function!</span> s:scratch</span>(</span>mods</span>,</span> cmd</span>)</span> abort</span></span>
    if</span> a:cmd</span> is</span>#</span> ''</span></span>
        let</span> l:output</span> =</span> ''</span></span>
    elseif</span> a:cmd[</span>0</span>] </span>is</span>#</span> '@'</span></span>
        if</span> strlen</span>(</span>a:cmd</span>)</span> is</span>#</span> 2</span></span>
            let</span> l:output</span> =</span> getreg</span>(</span>a:cmd[</span>1</span>]</span>,</span> 1</span>,</span> v:</span>true</span>)</span></span>
        else</span></span>
            throw</span> 'Invalid register'</span></span>
        endif</span></span>
    elseif</span> a:cmd[</span>0</span>] </span>is</span>#</span> '!'</span></span>
        let</span> l:cmd</span> =</span> a:cmd</span> =~</span>' %'</span> ?</span> substitute</span>(</span>a:cmd</span>,</span> ' %'</span>,</span> ' '</span> . </span>expand</span>(</span>'%:p'</span>),</span> ''</span>)</span> : </span>a:cmd</span></span>
        let</span> l:output</span> =</span> systemlist</span>(</span>matchstr</span>(</span>l:cmd</span>,</span> '^!\zs.*'</span>))</span></span>
    else</span></span>
        let</span> l:output</span> =</span> split</span>(</span>execute</span>(</span>a:cmd</span>),</span> "\n"</span>)</span></span>
    endif</span></span>
</span>
    execute</span> a:mods</span> . </span>' new'</span></span>
    Scratchify</span></span>
    call</span> setline</span>(</span>1</span>,</span> l:output</span>)</span></span>
endfunction</span></span></code></pre>
This gives us a pretty powerful solution where we can use :Scratch @a</code> to open
a new scratch buffer with the content of register A</code>, edit it, and yank it
back via "ayy</code>.</p>
Pluginize#</a>
</h2>
Now, it would be a shame to keep such a useful tool for ourselves so
let's share it with the big world. In this case we need:</p>

a proper project structure,</li>
documentation,</li>
a good catchy name.</li>
</ul>
You can find help on the two first topics in :h write-plugin</code></a> and :h write-local-help</code></a>
or in any of the bazillion tutorials on the internet.</p>
Finding a good name is something I can't help you with. I have picked
vim-backscratch</code>, because I like back scratches (everyone likes them) and, as
a nice coincidence, because it contains the word "scratch".</p>
Summary#</a>
</h2>
Creating plugins for Vim is easy, but not every functionality needs to be
a plugin from day one. Start easy and small. If something can be done with
a simple command/mapping, then it should be done with a simple command/mapping
at first. If you find your solution really useful, then, and only then, you
should think about turning it into a plugin. The whole process described in this
article didn't happen in a week or two. It took me about a year to reach the step
Make it a more "vimmy" citizen</em>, when I heard about romainl's script on IRC.
I didn't need anything more, so take your time.</p>
Additional pro-tips:</p>

make it small, big plugins will require a lot of maintenance, small plugins
are much simpler to maintain,</li>
if something can be done via a command then it should be made as a command,
do not force your mappings on users.</li>
</ul>


Let's talk about `application/0`
2019-07-26T00:00:00+00:00
When you start your new Elixir project via mix new my_awesome_project</code> you will
end with something like this in mix.exs</code>:</p>
defmodule</span> MyAwesomeProject</span>.</span>Mixfile</span> do</span></span>
  use</span> Mix</span>.</span>Project</span></span>
</span>
  def</span> config</span> do</span></span>
    [</span></span>
      name</span>:</span> :</span>my_awesome_project</span>,</span></span>
      # …</span></span>
      deps</span>:</span> deps</span>()</span></span>
    ]</span></span>
  end</span></span>
</span>
  def</span> application</span> do</span></span>
    [</span></span>
      extra_applications</span>: [</span>:</span>logger</span>]</span></span>
    ]</span></span>
  end</span></span>
</span>
  defp</span> deps</span> do</span></span>
    [</span></span>
      # …</span></span>
    ]</span></span>
  end</span></span>
end</span></span></code></pre>
And in most cases you will focus on deps/0</code>, sometimes on config/0</code>, but the
application/0</code> you will almost never touch, and if you do, you probably will only
need it to add new entry in :extra_applications</code> or sometimes
:included_applications</code>. Except for, that this function interns are terra
incognita, a place where you never look unless you are forced to, like "Read
it later" list in Safari. This is sad, as it is quite powerful and useful
piece of code.</p>
However first things first.</p>
What application/0</code> is for?#</a>
</h2>
In Erlang the running system is built from applications. Those applications are
like processes in your system managed by your system supervisor (SysV, systemd,
OpenRC, launchd, etc.). They are launched either during VM startup or on
direct user request via Application.start/1-2</code> and its family. Starting
this application is described in my_awesome_app.app</code> file which is commonly
generated by the build system from the template. In Rebar3 projects this
template is src/appname.app.src</code> and in Elixir it's the return value of the
named application/0</code> function. This generated file is known as Application
Resource File</a>. It is just tuple in form of {:application, :my_awesome_app, opts}</code> where opts</code> is a keyword list of additional properties
(to be exact it's output of the application/0</code> function).</p>
Two of those optional fields are quite known in the Elixir community:</p>

:mod</code> which is a tuple {module(), term()}</code> containing the starting point
of our application; in most cases this is the module that will return main
supervisor of the application.</li>
:applications</code> contains all applications required by our application; younger
Elixir developers possibly never seen that as since version 1.5 this field
is automatically filled by parsing :deps</code>, though we still can add entries
there via :extra_applications</code></li>
</ul>
There are also few Elixir specific fields, sometimes used in larger projects:</p>

:extra_applications</code> - those should be included in the release
and automatically started before running current application; but aren't in the
dependencies list because, for example, are in default distribution, for
example logger</code> or inets</code>.</li>
:included_applications</code> - applications which should be included in the
release, but not automatically started on boot.</li>
</ul>
Wait, there is more!</p>
Unfortunately not all of the highly useful keys are used/known in the community.</p>
Application environment#</a>
</h2>
For some reason everyone calls it configuration, so if you are familiar with
config/config.exs</code> and for some reason you decided to go against the Library
Guidelines</em></a> and you have decided to use application environment for
configuring your code (there are reason to do so, even when you publish your
code as a "library", see lager</code> or opencensus</code>) then you will soon find that
putting configuration in your's library config/config.exs</code> do not matter much
in it's dependants. You have 2 possibilities how to solve that:</p>

Use Application.get_env/3</code> and define your "default" as a 3rd argument.</li>
Use :env</code> to set data that should be loaded to application environment by
default.</li>
</ul>
Be wary that the second option works only  for current application, so you
cannot configure other applications (for example logger</code>) there. But what is
the point?</em> You may ask, and I found one. If you want to use default
sys.config</code> file for configuring your application then sometimes few pointless
configuration option can land there, like :ecto_repos</code> variable which truly
doesn't matter much in production as it's only used by Ecto's Mix tasks. What
I do is to add new entry in application/0</code> with:</p>
env</span>: [</span></span>
  ecto_repos</span>: [</span>MyAwesomeApp</span>.</span>Repo</span>]</span></span>
]</span></span></code></pre>
And call it a day. Now I can focus on keeping real</strong> configuration options in
config/config.exs</code> and remove unneeded fields from there. But remember that you
still can override these values by setting them in sys.config</code> if needed, so
this is pure win for me.</p>
Start phases#</a>
</h2>
Application configuration also allows you to define additional pieces of code to
be run after your application started. For example imagine situation when you
want to send Slack notification that given node started and is ready to work.
You can do it via temporary task in Supervisor.init/2</code> by defining child list
like:</p>
[</span></span>
  MyApp</span>.</span>Repo</span>,</span></span>
  MyApp</span>.</span>Worker</span>,</span></span>
  {</span>Task</span>,</span> &send_slack_notification</span>/</span>0</span>}</span></span>
]</span></span></code></pre>
Alternatively you can use :start_phases</code> in application/0</code>:</p>
start_phases</span>: [</span></span>
  notify</span>: []</span></span>
]</span></span></code></pre>
And then define in your Application</code> module function start_phase/3</code>:</p>
def</span> start_phase</span>(</span>:</span>notify</span>,</span> :</span>normal</span>,</span> opts</span>)</span> do</span></span>
  :</span>ok</span> =</span> send_slack_notification</span>()</span></span>
end</span></span></code></pre>
Where 1st argument will be the name of the phase, 2nd will be start type the
same as in Application.start/2</code> callback</a>,
and 3rd is the value passed in :start_phases</code>.</p>
The awesome part there is that start_phase/3</code> is called not only for current
application, but all of it's dependencies as well.</p>
Registered names#</a>
</h2>
This is one of the things that had more sense in Erlang world than in Elixir,
but by being good citizen we should use it as well. This is a nice solution for
lack of namespacing in Erlang - it allowed release tools to detect collisions in
named processes. This is simple list of atoms that contain all names that are
globally registered by this application. Example form Elixir's
Logger</a>:</p>
registered</span>: [</span>Logger</span>,</span> Logger</span>.</span>BackendSupervisor</span>,</span> Logger</span>.</span>Supervisor</span>,</span> Logger</span>.</span>Watcher</span>]</span></span></code></pre>
Unfortunately Phoenix do not use this field itself and do not suggest using one
in it's default project generator. But in general it's good practise.</p>
Summary#</a>
</h2>
There is much more in application configuration to what most Elixir code is
using, it is worth sometimes to read how your application is defined and ran.
For more information about generating application description file you can check
out mix help compile.app</code></a>.</p>


Common Test for Elixir developers
2019-07-15T00:00:00+00:00
In my new job I have opportunity to work a little bit more with Erlang code and
Erlang related tools than earlier.  That forced me a little to learn and use
Common Test</a> library which is the</strong> testing library in Erlang world.
When after that I needed to work back it hit me hard how much stuff I am
missing in Elixir.  In this article I will try to present Common Test library
from the viewpoint of long standing Elixir guy and present how it compares to
the ExUnit.</p>
Unit testing vs integration testing#</a>
</h2>
The main difference between these two is their intended usage.  In it's core
ExUnit is intended as a library for unit testing (no surprise there, as it is
literally in it's name), on the other hand Common Test is more about integration
testing and looking on the system as a whole.  So the discussion there would not
be fair, as it isn't apples to apples, but for second let's check what makes
each of them great for their intended uses:</p>
ExUnit for unit testing:</p>

Simple way to setup and teardown tests via setup</code> and setup_all</code> macros</li>
Built in testing of documentation examples via doctest</code></li>
Immutability of test environment (at least that is what you should try to
achieve)</li>
Very simple way of running different test modules in parallel (tests within
single module are always sync)</li>
Each test run in separate process, which prevents sharing data between
processes via process dictionary</li>
Grouping tests into describe</code> blocks</li>
In overall simplicity of the library as a whole</li>
</ul>
Common Test for integration testing:</p>

Persistent test results - logs, results, "generated garbage" is stored between
runs in the Common Test, which allows to track what and when happened, and if
needed trace the bug in the logs/leftovers</li>
Generating "private directory" per suite</li>
Grouping tests while allowing single test to be reused in different groups</li>
Extensive control about order and parallelization of tests within groups</li>
Built in very extensive and detailed reports (HTML and JUnit)</li>
Distributed testing</li>
Define dependencies between tests</li>
Set of helpers for working with different network protocols (SSH, Telnet,
SNMP, FTP, Erlang RPC, netconf)</li>
Built in support for data fixtures</li>
Built in support for complex configuration of test runs via test
specifications</a></li>
Built in logging (we will cover that later)</li>
</ul>
So as you can see, the Common Test is much broader and complex, but at the same
time it provides a lot of utilities that are really helpful while writing
integration tests, that can span multiple applications.</p>
Writing Common Test suite#</a>
</h2>
While CT provides enormous power in hand of the experienced tester/developer
it's API isn't the top of the class (which happen a lot in Erlang tools, whoever
wanted to integrate them with anything then they will know what I mean).  But in
fact writing simple test suite in CT is fairly easy:</p>
-</span>module</span>(</span>example_SUITE</span>).</span> % The naming convention (with uppercase _SUITE) Erlang</span></span>
                        % convention which allow ct to find test suites.</span></span>
                        % Something like ExUnit _test.exs naming convention</span></span>
</span>
-</span>export</span>([</span>all</span>/</span>0</span>]).</span></span>
</span>
-</span>export</span>([</span>test_function_name</span>/</span>1</span>]).</span></span>
</span>
all</span>()</span> -></span></span>
    [</span>test_function_name</span>]</span>.</span></span>
</span>
test_function_name</span>(</span>_Config</span>)</span> -></span></span>
    2</span> =</span> 1</span> +</span> 1</span>.</span></span></code></pre>
It is pretty easy and understandable format, but with due CT power it quickly
can grow to be "less" readable</a>.
Taming that power can be problematic and can cause few headaches for newcomers.</p>
Reporting#</a>
</h2>
Another part, and the one that I miss the most in ExUnit, is reporting in Common
Test.  This alone is in my humble opinion THE</strong> best part of the library.  To
present you how this work assume that we have above test suite which we will run
with ct_run -dir .</code>. This will compile and run our tests (obviously), but in
addition to that it will generate hell lot of cruft files, some JQuery (yes,
this is still used in 2019), some CSS files, some HTML and other.</p>
Just check this out</a>. This is example report
generated by the Common Test. As you can see it contains a lot of information in
quite readable format. Not only it contains information about current run, but
all previous runs as well, which is really handy when tracking regressions.</p>
But can we store even more information there? Yes, as CT includes simple
logging facility it is completely possible to log your own information during
tests, for example, lets modify our test to log some information:</p>
test_function_name</span>(</span>_Config</span>)</span> -></span></span>
    ct</span>:</span>log</span>("</span>Example message</span>"),</span></span>
    2</span> =</span> 1</span> +</span> 1</span>.</span></span></code></pre>
Now when we run tests again, then we will see more information (even coloured)
in our test log</a>:</p>
</p>
Additionally there is support Surefire XML output (commonly known as JUnit XML)
via hook that is distributed with Common Test</a>.
This is very useful, as many CI services (for sure Jenkins, Circle CI, and
GitLab CI) support such files to provide better CI results than raw stdout.</p>
Comparison to ExUnit#</a>
</h2>
ExUnit is great tool, which is very flexible while being relatively simple.
Unfortunately that simplicity makes it very lacking when comparing with "more
established" solutions like Common Test which shines when we "grow out" of the
ExUnit tests. It provides a lot great tools that really help with finding bugs
(for example when I was joining the project and I was trying to run tests
locally I have some issues due to misconfiguration, it was much easier to send
tarball of the test results instead of sending wall of text captured from
stdout).</p>
Don't get me wrong, ExUnit is great and powerful, and for sure you should use
it. It is great evolution over EUnit, but again, it would be worth of having
built in Surefire XML output</a>
due to it's popularity and support in many CI services.</p>
Future#</a>
</h2>
But is everything lost and we need to fall back to the Erlang code (which can be
problematic for many people) to use Common Test? Not exactly, Comcast (yes, that
Comcast) created simple wrapper</a> for Elixir (unfortunately seems pretty
dead to me) and I have started Commoner</a> library that is meant to
provide API more familiar and easier to use:</p>
defmodule</span> Commoner</span>.</span>ExampleSuite</span> do</span></span>
  use</span> CommonTest</span>.</span>Suite</span></span>
</span>
  test </span>"</span>foo bar</span>"</span> do</span></span>
    assert</span> (</span>1</span> +</span> 1</span>)</span> in</span> [</span>2</span>]</span></span>
  end</span></span>
</span>
  test </span>"</span>bar baz</span>"</span> do</span></span>
    CommonTest</span>.</span>log</span>("</span>Hello</span>")</span></span>
  end</span></span>
end</span></span></code></pre>
I am very interested about your feelings about Common Test usage in Elixir and
about proposed API for Commoner. You can ping me:</p>

On Elixir forum thread about Commoner</a></li>
On Twitter via @hauleth</code></li>
</ul>


Here be (owned) books
2019-07-14T17:38:48+01:00
One of Rust's biggest pros is its unique ownership system. Unfortunately, it is
also one of the hardest things to learn. In this article I will try to explain
it the same way I had learned it and how I introduce it to people.</p>
Disclaimer</strong>: If you do not find this article helpful try to search for another.
People are different and different things zing</em> them.</p>
Let's have a book#</a>
</h2>
Ownership becomes simple and natural if you just acknowledge it as an
application of real world relationships. For example, imagine types in Rust as
a kind of written note. We have different types of notes and based on that, each
of them will be handled differently.</p>

short ones, like phone no. of the hot waiter/waitress</li>
longer ones, like this article</li>
longest ones, like a Lord of the Rings</em></li>
</ul>
Using this analogy let me try to introduce you, dear reader, to the amazing
world of Rust's ownership.</p>
One can own the book#</a>
</h2>
Each note, no matter what size it is, can have one owner. Me, you, anyone, it
doesn't matter, but there will be only one owner. You can do whatever you want
with such note but with that power comes, not so great, responsibility: after
you are done with this book you will need to get rid of it. Since you are a law
abiding citizen you will recycle the note in the appropriate receptacle, but it
is your responsibility to do it. Of course this is not the only way to deal with
a note. You can also give it to someone and then it will be hers or his
responsibility.</p>
To rephrase it in the Rust way, it would look like this:</p>
struct</span> Note</span>;</span></span>
</span>
fn</span> john</span>() {</span></span>
    let</span> book</span> =</span> Note</span>;</span> // john creates the book and he owns it</span></span>
</span>
    // here he can do whatever he want with our `book`</span></span>
}</span> // at the end of his life john will destroy all his belongings</span></span>
</span>
fn</span> steve</span>() {</span></span>
    let</span> book</span> =</span> Note</span>;</span> // new book</span></span>
</span>
    // he can do whatever he wants to do with his book</span></span>
</span>
    sally</span>(</span>book</span>);</span></span>
    // steve gives the book to `sally`,</span></span>
    // Sally has the responsibility to destroy it</span></span>
</span>
    // now steve cannot do anything with this book,</span></span>
    // as it is not his personal belonging anymore</span></span>
}</span></span></code></pre>One can borrow the book#</a>
</h2>
When we don't want to give someone a book (we like that one), we can also lend
them one. And there are two ways to borrow one book:</p>

We can edit that book (ex. it is our personal dairy) and we lend it to someone
to check our spelling. We trust that person and we explicitly allow her to
edit our notes in place. We call it mutable borrow</strong>.</li>
We do not trust someone and we lend our beloved book with no permission to edit
it. Even more, that person knows, that writing something in that book will
make us go rampage and destroy the whole universe. It will be an immutable
borrow</strong>.</li>
</ul>
Of course if we borrow something from someone else, then we can lend it further
with the same rules that were applied to us.</p>
Rust also ensures that mutable borrow</strong> is unique. There will never be more
than one person that will be allowed to edit the book. We can still create a chain
of trust - like when I find someone who is better at English than me, I would
allow this person to correct an article written by me or my friend who has
entrusted me with correcting his text.</p>
Immutable borrows</strong> aren't exclusive. I can lend my books as many times as I
want with one exception: I cannot lend a book that is still borrowed by someone
who can change its content.</p>
In Rust it would look like that:</p>
fn</span> my</span>() {</span></span>
    let mut</span> book</span> =</span> Note</span>;</span></span>
</span>
    spelling_corrector</span>(</span>&mut</span> book</span>);</span></span>
    // we must explicitly mention that we lend the book</span></span>
    // and we don't give it away</span></span>
</span>
    reader</span>(</span>&</span>book</span>);</span></span>
}</span></span>
</span>
fn</span> spelling_corrector</span>(</span>book</span>: &mut</span> Note</span>) {</span></span>
    // correct spelling in place</span></span>
}</span></span>
</span>
fn</span> reader</span>(</span>book</span>: &</span>Note</span>) {</span></span>
    // read a book</span></span>
}</span></span></code></pre>Not all notes are worth borrowing#</a>
</h2>
Sometimes this whole process of lending and then receiving a note back is much
more complicated then just cloning the whole note for someone else. Imagine that
you are in school and friend wants to copy your homework. What you do is lend
your homework to him, and with caution he can clone it on his own. This is what
Rust's Clone</code> trait provides - a method to clone content of struct without moving
its ownership.</p>
#[derive(</span>Clone</span>)]</span></span>
struct</span> Homework</span>;</span></span>
</span>
fn</span> my</span>() {</span></span>
    let</span> homework</span> =</span> Homework</span>;</span></span>
</span>
    friend</span>(</span>&</span>homework</span>);</span></span>
}</span></span>
</span>
fn</span> friend</span>(</span>work</span>: &</span>Homework</span>) {</span> // we lend it immutably</span></span>
    let mut</span> homework</span>:</span> Homework</span> =</span> work</span>.</span>clone</span>();</span></span>
    // your friend now has his own modifiable copy</span></span>
}</span></span></code></pre>
But some notes are even shorter than that. They are so short and easy to clone
that it is much easier to clone them every time, instead of explicitly
calling the method. Like when you give your phone number to a hot girl at the
bar, the Copy</code> trait automatically clones your note so the other has their own copy.
Again, this is for small types that can be mechanically copied each time when needed.</p>
#[derive(</span>Copy</span>,</span> Clone</span>)]</span></span>
// everything that is `Copy` must be also `Clone`</span></span>
struct</span> PhoneNo</span>;</span></span>
</span>
fn</span> my</span>() {</span></span>
    let</span> no</span> =</span> PhoneNo</span>;</span></span>
</span>
    hot_stuff</span>(</span>no</span>);</span></span>
}</span></span>
</span>
fn</span> hot_stuff</span>(</span>no</span>:</span> PhoneNo</span>) {</span></span>
    // fingers crossed</span></span>
}</span></span></code></pre>Conclusion#</a>
</h2>
There is more to learn, but these are the basic laws of ownership in Rust.
Everything else is based on this. If you understand this, it will become much
easier for you to understand how other types behave and, more importantly, why
they work the way they do.</p>


Dumb Elixir VIsual (and iMproved) editor
2019-04-13T21:40:05+02:00

Earlier published on Medium</a></p>
</blockquote>
</p>
I am quite orthodox Vim user and I like to know everything that is happening in
my editor configuration. By no means I am "minimal" Vim user, I currently have
48 plugins installed, I just very carefully pick my plugins (few of them I have
written myself) to use only plugins that I understand and I can fix on my own in
case of problems. This results with configuration I understand and control in
all means.</p>
What this article is not#</a>
</h2>
This isn't "introduction into Vim" article neither "how to configure Vim for
Elixir development". Here I will try to describe how I use Vim for Elixir and
Erlang development, you can find some nice ideas and tips that may help you, but
by any means I do not mean that this is configuration you should or should not
use. You should use whatever you like (even if I do not like what you use) as
long as you understand what you use and why you use that.</p>

Any sufficiently complicated set of Vim plugins contains an ad hoc,
informally-specified, bug-ridden, slow implementation of half of Vim's
features.</em>
-- robertmeta's tenth rule.</p>
</blockquote>
Now we can start.</p>
Basics#</a>
</h2>
I am using NeoVim 0.3.4, but almost everything I will describe there should work
in Vim 8.1+ as well. If you will encounter any problems, then please let me
know.</p>
Vim doesn't (as 13.04.2019) support Elixir out of the box, so what we
need is to install vim-elixir</code></a> plugin which will provide niceties like syntax
colouring and indentation. Even if syntax colouring isn't your thing then I
would still recommend installing it as it provides other things that it provides
few other niceties that I will describe later.</p>
But how to install it? In truly minimal setup you can just create
pack/elixir/start</code> directory within your ~/.vim</code> folder
($XDG_CONFIG_DIR/nvim</code> in case of NeoVim) and clone given repository there,
however I am using vim-packager</code></a> which is based on minpac</code></a> and is
truly minimal package manager (it is important distinction from vim-plug</code></a>
or others that also manages plugin loading, these plugins only manage fetching)
which even do not need to be loaded during "normal" runtime, only when you are
updating plugins.</p>
Project navigation#</a>
</h2>
A lot of people, when come from other editors, often install NERDTree to have
"project drawer" functionality within Vim, because this is what they are used
to. Unfortunately "split windows and the project drawer go together like oil
and vinegar</a>" and it can result in painful experience or, which
is even worse, avoiding built in functionalities, because these do not mix well.
Vim comes with built in NetRW plugin for working with remote files and directory
tries. However for me this plugin is bloated as well and I would love to get rid
of it (unfortunately it is not currently possible as few functionalities relies
on it, namely dictionaries) so I replaced everything with dirvish</code></a>.
Additionally I often use fuzzy finder, which in my case is vim-picker</code></a> with
fzy</code></a> which for me is much faster and more precise than popular FZF.</p>
These tools are great when we are navigating in tree that is new to us or do not
have explicit structure. When we are working on Elixir projects then we know
before hand that there will be some commonly shared structure, like
lib/<name>.ex</code> will contain source code, test/<name>_test.exs</code> will contain
test files, etc. What is more we know that <name></code> part will be shared between
file and its unit tests. This is very powerful assumption, as this allow us to
use vim-projectionist</code></a> with ease. This plugin provide 3 main
functionalities (for me):</p>

Jumping to the files basing on their path, so for example I can use :Elib foo</code> to automatically jump to file lib/foo.ex</code>. It doesn't seems like much,
but it also provides fuzzy finding, and allows me to define more specific
matches, like :Econtroller foo</code> will open
lib/app_web/controllers/foo_controller.ex</code> (not exactly that as I use
different project layout, but that is topic on another article).</li>
File templates, so when I start editing file (doesn't matter how I opened it,
so this do not require to use above :Elib</code> command), so when I start editing
test file it automatically add scaffold which I can configure per project.</li>
Alternate files which in short are "related" files. For example when I edit
file lib/foo/bar/baz.ex</code> and I run :A</code> it will create (if not exist) and
jump to the file test/foo/bar/baz_test.exs</code> which will be already scaffolded
by the earlier functionality. Recently it even became possible to have
multiple alternates.</li>
</ul>
Whole plugin is configured by .projections.json</code> file, but it would be
infeasible to add this file to each project you work for. Fortunately there is
solution for that, we can define "heuristics" that will try to match for given
project structure and provide such features "globally". My configuration for
that looks like this:</p>
let</span> g:projectionist_heuristics[</span>'mix.exs'</span>] </span>=</span> {</span></span>
            \</span> 'apps/*/mix.exs'</span>: { </span>'type'</span>: </span>'app'</span> }</span>,</span></span>
            \</span> 'lib/*.ex'</span>: {</span></span>
            \</span>   'type'</span>: </span>'lib'</span>,</span></span>
            \</span>   'alternate'</span>: </span>'test/{}_test.exs'</span>,</span></span>
            \</span>   'template'</span>: [</span>'defmodule {camelcase|capitalize|dot} do'</span>,</span> 'end'</span>]</span>,</span></span>
            \</span> }</span>,</span></span>
            \</span> 'test/*_test.exs'</span>: {</span></span>
            \</span>   'type'</span>: </span>'test'</span>,</span></span>
            \</span>   'alternate'</span>: </span>'lib/{}.ex'</span>,</span></span>
            \</span>   'template'</span>: [</span></span>
            \</span>       'defmodule {camelcase|capitalize|dot}Test do'</span>,</span></span>
            \</span>       '  use ExUnit.Case'</span>,</span></span>
            \</span>       ''</span>,</span></span>
            \</span>       '  alias {camelcase|capitalize|dot}, as: Subject'</span>,</span></span>
            \</span>       ''</span>,</span></span>
            \</span>       '  doctest Subject'</span>,</span></span>
            \</span>       'end'</span></span>
            \</span>   ]</span>,</span></span>
            \</span> }</span>,</span></span>
            \</span> 'mix.exs'</span>: { </span>'type'</span>: </span>'mix'</span> }</span>,</span></span>
            \</span> 'config/*.exs'</span>: { </span>'type'</span>: </span>'config'</span> }</span>,</span></span>
            \</span> '*.ex'</span>: {</span></span>
            \</span>   'makery'</span>: {</span></span>
            \</span>     'lint'</span>: { </span>'compiler'</span>: </span>'credo'</span> }</span>,</span></span>
            \</span>     'test'</span>: { </span>'compiler'</span>: </span>'exunit'</span> }</span>,</span></span>
            \</span>     'build'</span>: { </span>'compiler'</span>: </span>'mix'</span> }</span></span>
            \</span>   }</span></span>
            \</span> }</span>,</span></span>
            \</span> '*.exs'</span>: {</span></span>
            \</span>   'makery'</span>: {</span></span>
            \</span>     'lint'</span>: { </span>'compiler'</span>: </span>'credo'</span> }</span>,</span></span>
            \</span>     'test'</span>: { </span>'compiler'</span>: </span>'exunit'</span> }</span>,</span></span>
            \</span>     'build'</span>: { </span>'compiler'</span>: </span>'mix'</span> }</span></span>
            \</span>   }</span></span>
            \</span> }</span></span>
            \</span> }</span></span>
</span>
let</span> g:projectionist_heuristics[</span>'rebar.config'</span>] </span>=</span> {</span></span>
            \</span> '*.erl'</span>: {</span></span>
            \</span>   'template'</span>: [</span>'-module({basename}).'</span>,</span> ''</span>,</span> '-export([]).'</span>,</span> ''</span>]</span>,</span></span>
            \</span> }</span>,</span></span>
            \</span> 'src/*.app.src'</span>: { </span>'type'</span>: </span>'app'</span> }</span>,</span></span>
            \</span> 'src/*.erl'</span>: {</span></span>
            \</span>   'type'</span>: </span>'src'</span>,</span></span>
            \</span>   'alternate'</span>: </span>'test/{}_SUITE.erl'</span>,</span></span>
            \</span> }</span>,</span></span>
            \</span> 'test/*_SUITE.erl'</span>: {</span></span>
            \</span>   'type'</span>: </span>'test'</span>,</span></span>
            \</span>   'alternate'</span>: </span>'src/{}.erl'</span>,</span></span>
            \</span> }</span>,</span></span>
            \</span> 'rebar.config'</span>: { </span>'type'</span>: </span>'rebar'</span> }</span></span>
            \</span> }</span></span></code></pre>This will provide:#</a>
</h3>
For Elixir:#</a>
</h4>

lib</code> that will contain project source files which will be already filled with
module named Foo.BarBaz</code> for file named lib/foo/bar_baz.ex</code> (jump by :Elib foo/bar_baz</code>)</li>
test</code> for test files which will be instantiated with module named
Foo.BarBazTest</code> for file test/foo/bar_baz_test.exs</code> that will already use
ExUnit.Case</code> (you can jump via :Etest foo/bar_baz</code>), will define
alias Foo.BarBaz, as: Subject</code>, and will run doctests for that module</li>
config</code> for configuration files</li>
mix</code> for mix.exs</code></li>
</ul>
It will also define test files as default alternates for each source file (and
vice versa, because alternate files do not need to be symmetric), so if you
run :A</code> in file lib/foo/bar_baz.ex</code> it will automatically jump to the
test/foo/bar_baz_test.exs</code>.</p>
For Erlang:#</a>
</h4>

src</code> for source files</li>
app</code> for *.app.src</code> files</li>
test</code> for common test suites</li>
rebar</code> for rebar.config</code> file</li>
</ul>
The relation between source files and test files is exactly the same as in
Elixir projects.</p>
One thing can bring your attention, why the hell I define helpers for mix.exs</code>
and rebar.config</code> as you can simply use :e <file></code>. The answer is simple, :e</code>
will work for files in Vim working directory while :E</code> will work form the
projectionist root, aka directory where is .projections.json</code> file defined (or
in case of heuristics, from the directory that matched files). This mean that
when I edit files in umbrella application I can use :Emix</code> (or :Erebar</code>) to
edit current sub-project config file and :e mix.exs</code> to edit global one.</p>
Completion and language server#</a>
</h2>
For completion and code formatting I use vim-lsp</code></a>. I have tried most of the
language server clients out there, but I always come back to this one for a few
reasons:</p>

It is implemented only in VimL which mean that I am not forced to installing
any compatibility layers or fighting with different runtimes.</li>
It is simple enough that I can easily dig into it, and fix problems that I
have encountered.</li>
It doesn't override any built in Vim functionality and instead provide set of
commands that you can then bind to whatever mappings you want.</li>
It do not force me to use autocompletion, which I do not use at all. At the
same time it provides seamless integration with built-in Vim functionality of
omnicompletion and user completion by providing lsp#complete</code> function.</li>
</ul>
This approach of not providing default mappings is really nice for power users,
as this allow us to define everything on our own. For example some of plugins
use <C-]></kbd> for jumping to definition, which I often use (it is
jump to tag definition) and shadowing it would be problematic for me. So in the
end I have created my own set of mappings, that have additional feature of being
present only if there is any server that supports them:</p>
func!</span> s:setup_ls</span>(</span>...</span>)</span> abort</span></span>
    let</span> l:servers</span> =</span> lsp#get_whitelisted_servers</span>()</span></span>
</span>
    for</span> l:server</span> in</span> l:servers</span></span>
        let</span> l:cap</span> =</span> lsp#get_server_capabilities</span>(</span>l:server</span>)</span></span>
</span>
        if</span> has_key</span>(</span>l:cap</span>,</span> 'completionProvider'</span>)</span></span>
            setlocal</span> omnifunc</span>=</span>lsp</span>#</span>complete</span></span>
        endif</span></span>
</span>
        if</span> has_key</span>(</span>l:cap</span>,</span> 'hoverProvider'</span>)</span></span>
            setlocal</span> keywordprg</span>=</span>:LspHover</span></span>
        endif</span></span>
</span>
        if</span> has_key</span>(</span>l:cap</span>,</span> 'definitionProvider'</span>)</span></span>
            nmap</span> <</span>silent</span>> <</span>buffer</span>></span> gd</span> <</span>plug</span>>(</span>lsp</span>-definition</span>)</span></span>
        endif</span></span>
</span>
        if</span> has_key</span>(</span>l:cap</span>,</span> 'referencesProvider'</span>)</span></span>
            nmap</span> <</span>silent</span>> <</span>buffer</span>></span> gr </span><</span>plug</span>>(</span>lsp</span>-references</span>)</span></span>
        endif</span></span>
    endfor</span></span>
endfunc</span></span>
</span>
augroup</span> LSC</span></span>
    autocmd!</span></span>
    autocmd</span> User</span> lsp_setup </span>call</span> lsp#register_server</span>(</span>{</span></span>
                \</span> 'name'</span>: </span>'ElixirLS'</span>,</span></span>
                \</span> 'cmd'</span>: {_->[</span>'elixir-ls'</span>]}</span>,</span></span>
                \</span> 'whitelist'</span>: [</span>'elixir'</span>,</span> 'eelixir'</span>]</span></span>
                \</span>}</span>)</span></span>
    autocmd</span> User</span> lsp_setup </span>call</span> lsp#register_server</span>(</span>{</span></span>
                \</span> 'name'</span>: </span>'RLS'</span>,</span></span>
                \</span> 'cmd'</span>: {_->[</span>'rls'</span>]}</span>,</span></span>
                \</span> 'whitelist'</span>: [</span>'rust'</span>]</span></span>
                \</span>}</span>)</span></span>
    autocmd</span> User</span> lsp_setup </span>call</span> lsp#register_server</span>(</span>{</span></span>
                \</span> 'name'</span>: </span>'solargraph'</span>,</span></span>
                \</span> 'cmd'</span>: {server_info->[</span>'solargraph'</span>,</span> 'stdio'</span>]}</span>,</span></span>
                \</span> 'initialization_options'</span>: {</span>"diagnostics"</span>: </span>"true"</span>}</span>,</span></span>
                \</span> 'whitelist'</span>: [</span>'ruby'</span>]</span>,</span></span>
                \</span> }</span>)</span></span>
    autocmd</span> User</span> lsp_setup </span>call</span> lsp#register_server</span>(</span>{</span></span>
                \</span> 'name'</span>: </span>'dot'</span>,</span></span>
                \</span> 'cmd'</span>: {server_info->[</span>'dot-language-server'</span>,</span> '--stdio'</span>]}</span>,</span></span>
                \</span> 'whitelist'</span>: [</span>'dot'</span>]</span>,</span></span>
                \</span> }</span>)</span></span>
</span>
    autocmd</span> User</span> lsp_server_init </span>call</span> <</span>SID</span>></span>setup_ls</span>()</span></span>
    autocmd</span> BufEnter</span> * </span>call</span> <</span>SID</span>></span>setup_ls</span>()</span></span>
augroup</span> END</span></span></code></pre>Running tasks and linting#</a>
</h2>
A lot of people "cannot live" without lint-as-you-type feature, but I think,
that not using such functionality makes me a less sloppy and better programmer.
It makes me to think when I write and do not rely on some magical friend that
will always watch over my shoulder. However when the problem happens in my code
I would like to know where and quickly jump to the place where error occurred.
Additionally I would like to run tasks in the background without interruption to
my current work. All of it became possible with introduction of asynchronous
tasks in NeoVim and Vim 8. So I have created plugin asyncdo.vim</code></a> that
allows me to easily implement :Make</code> command that works exactly like built
in :make</code></a>, but do not halt my normal work. Together with vim-makery</code></a>
(which nicely integrates with vim-projectionsit</code>) and built in functionality of
:compiler</code></a>, which is supported by vim-elixir</code>, it allows me to easily run all
sorts of commands very easily. If you look into projectionist heuristics above
you will see that there is "makery"</code> key defined for *.ex</code> and *.exs</code> files.
That allows me to run :Mlint %</code> to run Credo on current file and the results
will be present within QuickFix window which together with my qfx.vim</code></a> will
mark lines with errors using signs. In the same manner I can run :Mtest</code> to run
tests for whole project and have failed tasks visible in QuickFix window.</p>
Other utilities#</a>
</h2>
There is bunch of other plugins that are quite helpful when it comes to working
on Elixir projects and do not interfere with Vim features, ex.:</p>

vim-dadbod</code></a> which allows you to run SQL queries from within Vim, and I
have written integration with Ecto</a> which is provided with
vim-elixir</code> by default. So if you are working on Elixir application that has
MyApp.Repo</code> Ecto repository then you can run :DB MyApp.Repo</code> and Vim will
open your DB client within separate terminal that will be connected to your DB</li>
vim-endwise</code></a> that will automatically add end to your do blocks</li>
direnv.vim</code></a> simplify management of environment variables in per-directory
manner</li>
vim-editorconfig</code></a> (sgur</a>'s one, not official one) - pure VimL support
for EditorConfig</a> files</li>
</ul>
Summary#</a>
</h2>
I hope that you find some nice ideas within this article that will help in
improving your own Vim configuration without adding much clutter.</p>
No, I will not publish my own vimrc</code> in fear that some of you will copy it as
is (also not that this is particularly troublesome for anyone who is aware of
Google to find it). Instead I highly suggest You to dig into your own
configuration and for each line ask yourself:</p>

Do I know what this line does</strong>?</li>
Do I really need</strong> this line?</li>
</ul>
And if answer for any of these questions is no</strong> then remove such line. In the
end you either learn what for it was, or that you never needed it.</p>}}
Summary#</a> </h2>
I know that it took quite while since the last post (ca. 1.5 years), but I hope that I will be able to write the final part much sooner than this.</p>

Part 1 - Basics, security, and FD passing</a></li>
Part 2 - Socket activation (this one)</a></li>
Part 3 - Logging</li> </ul>

Who Watches Watchmen? - Part 1

`Summary#</a> </h2> This blog post is already quite lengthy, so I will split it into separate parts. There probably will be 3 of them:</p> Part 1 - Basics, security, and FD passing (this one)</a></li> Part 2 - Socket activation</a></li> Part 3 - Logging</li> </ul>`

Log all the things

Treachery of Representation

Stop Spreading Crap at My `$HOME`

Writing Vim Plugin

Let's talk about `application/0`

Summary #</a> </h2>
There is much more in application configuration to what most Elixir code is using, it is worth sometimes to read how your application is defined and ran. For more information about generating application description file you can check out `mix help compile.app</code></a>.</p>`

Common Test for Elixir developers

Here be (owned) books

Conclusion #</a> </h2>
There is more to learn, but these are the basic laws of ownership in Rust. Everything else is based on this. If you understand this, it will become much easier for you to understand how other types behave and, more importantly, why they work the way they do.</p>

Dumb Elixir VIsual (and iMproved) editor

Hauleth's blog

Jeopardy! world

Who Watches Watchmen? - Part 2

How much memory is needed to run 1M Erlang processes?

Hauleth's blog

Jeopardy! world

Who Watches Watchmen? - Part 2

Multiple ports#</a> </h3> The question may arise - how to allow our service to listen on more than one port, for example you want to have your website available as HTTPS alongside "regular" HTTP. This means that our application needs to listen on two restricted ports:</p> 80 - for HTTP</li>

Summary#</a> </h2> I know that it took quite while since the last post (ca. 1.5 years), but I hope that I will be able to write the final part much sooner than this.</p> Part 1 - Basics, security, and FD passing</a></li> Part 2 - Socket activation (this one)</a></li> Part 3 - Logging</li> </ul>

How much memory is needed to run 1M Erlang processes?

Summary#</a> </h2> First of all:</p> Please, do not use ChatGPT for writing code for microbenchmarks.</p> </blockquote>

Who Watches Watchmen? - Part 1

Security#</a> </h2> We should start with changing default user and group which is assigned to our process. We can do so in 2 different ways:</p> Use some existing user and group by defining User=</code> and Group=</code> directives in our service definition; or</li>

Log all the things

Treachery of Representation

Stop Spreading Crap at My `$HOME`

Writing Vim Plugin

Pluginize#</a> </h2> Now, it would be a shame to keep such a useful tool for ourselves so let's share it with the big world. In this case we need:</p> a proper project structure,</li> documentation,</li>

Let's talk about `application/0`

Common Test for Elixir developers

Here be (owned) books

Dumb Elixir VIsual (and iMproved) editor

Summary#</a> </h2>
I know that it took quite while since the last post (ca. 1.5 years), but I hope that I will be able to write the final part much sooner than this.</p>

Part 1 - Basics, security, and FD passing</a></li>
Part 2 - Socket activation (this one)</a></li>
Part 3 - Logging</li> </ul>
