Sometimes the easiest anwser is the most janky anwser. In my journey to deploy Frappe’s Helpdesk solution, I hit an obscure problem with how Frappe handles Message IDs. Since SMTP providers today typically generate a Message ID themselves when a message goes out for their own bounce/complaint tracking, any application that uses it’s own generated Message ID won’t work correctly. And primarily where you’re deploying a help desk / ticketing solution, this will cause all kinds of heartache.

In summary, when a reply goes out from Frappe, it generates a Message ID and then stores that so when a reply comes in and is processed by the application, the Message ID can be used to look up the previous thread and then attach it back to the original ticket. In my case, this just appeared as if we were constantly getting new messages without anything to match on and inflating our ticket queue dramatically.

While trying to troubleshoot this, I tried a variety of means with limited success.

  1. Patching the Message Matching process to use the Ticket Number in the subject line. This failed simply because there was too much variety in the ticketing systems outbound emails. Some messages sent with (#ticketno) and some sent at #ticketno - meaning that any regex wouldn’t always work. There is also a security issue if a customer just replies with a number, they could hop into another customers tickets. (Not great!)
  2. Creating a Module / Extension for Frappe to Hook into the Message Send Process This should have been the obvious way to make this work. We deploy Frappe’s Helpdesk via a Docker compose file and so the containers themselves would need to deploy on boot this patch. When installing a handmade extension, I found too many problems trying to get Frappe to find and boot the module. At certain points, it would report the module wasn’t avaialble to Gunicorn and would crash on startup. (In testing, I could find the package deployed and some mechanisms showed it was working and others it did not).
  3. Catch the email via a SMTP middleman and update Frappe via REST API This turned out to be the easist, least painful of the solutions. Since I was running out of time to debug the module and having had limited exposure to Frappe’s plugin framework, I needed a solution to get us up and running as quickly as possible. Getting a quick Python SMTP server up and running is trival, and having it on the front end act as SMTP to Frappe and then on the backend talk to AWS’s SES API directly seemed to do the trick. When we used the SES API’s, we get the SES Message ID and can temporarily store it. And with that, we can make two requests back to Frappe to update the Communications and Email Queue tables and set the correct Message ID on both.

Now, when a message goes out from Frappe, it gets updated as the email is sent with the new Message ID, email clients can reply, and on response, I get matched tickets and the emails connect together! No more inadvertent new tickets and hopefully a lot happier teammates and customers.

In the future, I want to directly get a patch back into Frappe for this somehow, but in the SMTP standard, there is too many cases where a server could respond with something other than the Message ID with a 250 success message. It might get queued, or other reasons and so trying to parse those messages might not scale across providers. At worse, even just adding a AWS SES option and allowing it to write back the Message IDs would work, but maybe someone with better experience with Frappe’s core might get further than I did.

For now, this middeman adds a bit of complexity, but at the price of a good night sleep, I’ll take it!