Backgrounding of `NotificationsManager`for Fault Tolerance

[amydevs]

Specification

Currently the sending of notifications is entirely synchronous. The NotificationsManager.sendNotifcationcurrently blocks until the Notification has either been sent or failed to be sent.

In order to make the NotificationManager more fault tolerant, the sending of notifications needs to be assigned to a TaskManager task instead. This task needs to reschedule itself with a delay that doubles each retry. This would require a refactor of the notifications domain to abstract over the TaskManager in queuing notifications, but it would give us an outbox/inbox system similar to emails.

Expanded RPC Handlers

The RPC handlers need to be expanded to add ability to both clear the notifications of the outbox and remove a pending notification from the outbox.

NotificationIds

NotifcationIds need to be exposed on the notifications themselves, much like how AuditEventIds are exposed on the AuditEvents. The reason for this is because the user should be able to delete Notifications / cancel a pending Notification in the outbox. This was already discussed as an improvement during MatrixAI/Polykey-CLI#50, but there is an actual need for it now. The Encoded version of the NotificationId will be available both in the Notifications returned by read*Notificaitons, but not in the database itself. The reason for this is because the database already stores the ID as the key, which is then filled in as a field in the Notificaiton returned by read*Notifications methods.

Additional context

Tasks

1. Implement non-blocking NotificaitonsManager.sendNotification
2. Implement exposing NotificationIds on Notifications
3. Implement readOutboxNotifications, removeOutboxNotification, and clearOutboxNotifications
4. Rename readNotifications, removeNotification, and clearNotifications to readInboxxNotifications, removeInboxNotification, and clearInboxNotifications respectively

[linear]

ENG-299 Backgrounding of `NotificationsManager`for Fault Tolerance

[CMCDragonkai]

Technically not fault tolerance. It's "network partitioning". But even better is to realise that this just a "delay tolerant network". https://en.wikipedia.org/wiki/Delay-tolerant_networking - which is required for P2P networks - especially Polykey style.

@faithveloro09 another technology concept.