528: Goldfish Chunks
Transcript from 528: Goldfish Chunks with Tyler Hoffman, Christopher White, and Elecia White.
EW (00:00:07):
Hello and welcome to Embedded. I am Elecia White, here with Christopher White. Our guest this week is Tyler Hoffman, who has graciously returned to answer our many, many questions.
CW (00:00:22):
About Memfault and other things. Hi Tyler. Welcome.
TH (00:00:26):
Hi both of you. Yeah, I am very excited. This is I guess the third recording we have done together.
CW (00:00:30):
Is it? Okay. We have been doing this a long time.
TH (00:00:34):
Yeah, we did a part one and part two last time.
EW (00:00:38):
Could you tell us about yourself, as if we met at an Embedded Systems Conference?
TH (00:00:46):
For sure. The intro that I always give, basically, no matter who I meet. Yeah. I am Tyler. For the people who know the space, co-founder of Memfault, if you ever heard of it. If you have not, probably heard of Interrupt, which is one of our blogs that we write a bunch of stuff to.
(00:01:02):
But I got my start at Pebble, a smartwatch company back in 2014. They ran a Cortex-M3 and I just wrote a bunch of firmware there. Found my way getting tired of rebuilding a bunch of developer tools made for firmware engineers that solved a lot of issues that they had.
(00:01:21):
Memfault was kind of the final attempt at building a set of tools that I did not want to keep rebuilding at other companies. That got acquired by Nordic, and now I am working at Nordic Semiconductor.
EW (00:01:34):
It was acquired a year ago. Have things settled down? Have things changed?
TH (00:01:41):
I think we are pretty much in a- Well, it can always be better I think with acquisitions, especially into a large company. It has pretty much hit a steady state, where every product- We can go down a rabbit hole here.
(00:01:55):
Most products they release and that we release, we do it in lockstep. Which I think is a big win, because it means anything they release has Memfault baked in, or has it kind of a turnkey solution. And then anything we do on our end, has the Nordic chips on the top of mind, or at least built in as well.
EW (00:02:18):
We are going to do lightning round. You know the rules to this, so let us just get started.
TH (00:02:25):
Great.
CW (00:02:25):
I do not know why I am asking this. Favorite backyard theater play?
TH (00:02:30):
It is hilarious. "The Princess Bride". It was the one my wife wrote, but this is my wife's hobby.
CW (00:02:37):
Okay, cool.
EW (00:02:38):
Favorite dinosaur?
TH (00:02:41):
T-Rex, because they are ridiculous. They are huge. Go ahead.
CW (00:02:45):
Favorite bug, either software or entomological?
TH (00:02:50):
Oh man! Pass. I do not have one.
EW (00:02:54):
Do T-Rexes have feathers?
TH (00:02:57):
Not to my knowledge.
CW (00:02:58):
Do you like to complete one project, or start a dozen?
TH (00:03:02):
I love to complete a project.
EW (00:03:05):
When someone finds out what you do, like on an airplane, what questions do they always ask you?
TH (00:03:12):
It is always something like, "Can you fix my son's computer?" Or, "That is what my son does." Or, "Can you fix my phone?" just immediately.
CW (00:03:22):
If you could teach a college course, what would you want to teach?
TH (00:03:25):
I thought about this one, because I cheated. This was one of the ones you gave me. I would like to teach a course on cooking, or sustaining yourself through non-processed foods.
CW (00:03:37):
Nice.
EW (00:03:38):
Do giraffes make sense?
TH (00:03:40):
No. No idea how they work.
CW (00:03:43):
I do not know if we have asked you this one in the previous shows. Probably we have, but we are going to ask it again, and then we will refer back to the other shows to see if your answer has change. Favorite fictional robot?
TH (00:03:53):
WALL-E. I think that one still stands.
CW (00:03:55):
All right. Yeah, it sounds right.
TH (00:03:55):
From previous ones.
EW (00:03:57):
Have you ever licked a nine volt battery?
TH (00:04:00):
No, I have not.
EW (00:04:01):
What?
TH (00:04:03):
Also- Well, I was going to say my daughter definitely has, because she puts everything in her mouth. But I do not think we really interface with batteries quite yet.
EW (00:04:11):
Well, I mean, nine volts are special. They are spicy.
CW (00:04:14):
That is how you test them.
EW (00:04:16):
Yeah!
TH (00:04:17):
Did you see they came out with a coating for coin cell battery?
CW (00:04:21):
Yes.
TH (00:04:21):
Yeah. A special one, that is brand new. Anyways.
CW (00:04:24):
Bad tasting. Yeah. Batteries taste bad now. I do not know if they do this in nine volts, so might have to test that. See if it shocks you, and tastes bad.
EW (00:04:33):
It is just a little shock.
CW (00:04:35):
A fully charged one is pretty good. Depends on if you have had a lot of salty crackers <laugh>.
TH (00:04:43):
And how old it is. The fresh ones are the best.
CW (00:04:46):
Right. Right. Favorite acronym?
TH (00:04:48):
"In case you missed it," because I think it took me so long. ICY- I cannot even do it. But I just did not know it for the longest time. But now I use it.
EW (00:04:59):
I saw "If you know you know," as an acronym. All the Ks and the Ys, I was just toast for...
CW (00:05:05):
<laugh>
TH (00:05:07):
I have never seen that one in real life, so I am looking forward to it, the day that I come across it.
EW (00:05:13):
Do you have a tip everyone should know?
TH (00:05:16):
Many. I think one of the ones that I have been realizing recently, is you can blend any sort of green into pesto, and put it in pasta. Speaking of food.
EW (00:05:29):
So like kale and linguini.
TH (00:05:33):
Let us do it. Yeah. You should probably put some garlic in it. But yes, I do it all the time.
CW (00:05:37):
Cilantro
TH (00:05:37):
<laugh>
EW (00:05:37):
<laugh>
TH (00:05:40):
I got you. Any sort of green, huh? All right. So when I buy the big box of Costco greens and I am like, "There is no way I am going to use all of this before it goes bad," it goes into pesto, and we consume it as a family meal in pesto form. Yeah. Kids and all.
EW (00:05:58):
Is cooking a new passion for you? So we talked in 2021.
TH (00:06:06):
Four years ago. Four or five. Yeah.
CW (00:06:08):
2021 was not five years ago. Come on!
EW (00:06:12):
<laugh> Just keep saying that.
TH (00:06:16):
Is it a new passion? I would say, for sure growing. I also see it as a necessity. I have two kids now that are both eating solids. So one is almost a year, and one is two years. It is born out of we all need to eat, and I do not want to eat out every single day, or buy frozen meals.
(00:06:36):
I buy all the stuff. I procure it all. I cook it all. I know exactly the inventory and everything like that. It has just become my role in the family.
EW (00:06:52):
That does bring up the other question of, is there anything new going on in the last five years that has changed? You are still at Memfault. What could possibly have changed?
CW (00:07:00):
<laugh> Two kids.
EW (00:07:00):
<laugh>
TH (00:07:01):
So many things. Basically everything. In a lot of ways. Yeah, I think the standard family items have all kind of happened. Yeah, I got married, had two kids. We moved probably into our lifelong home in San Francisco, which we are very excited about.
(00:07:22):
The company sold. Think those are the big ones.
CW (00:07:27):
Those are the big ones. Yes.
EW (00:07:30):
Seems like a big list.
TH (00:07:32):
Yeah. It all happened last year- Or, a lot of them happened last year. But yeah.
EW (00:07:38):
It was a surprise to me that Memfault sold to Nordic. But kind of cool, for the reasons you said, that Nordic is one of the chip vendors that you would really want to use this on, because they are internet enabled.
(00:07:53):
But I want to say, was it a surprise to you? I mean, I hope not in the end, but...
TH (00:08:02):
No, it was not. I guess I am biased. It is one of the coolest semiconductor companies. They are so focused on software. They contribute to Zephyr. They have literally baked it into their SDK. They are the largest upstream contributors. And they have such a suite of products that are working better and better together every year, and every release that they produce.
(00:08:27):
But they- I guess twofold. One, realized one of the blockers to being successful with their products, was the end users actually successfully writing software and making it bug free. And not having to spend all of Nordic's FAE time to help the customers write bug-free code. There was always back and forth, "Is it your bug? Is it our bug? Who knows?"
(00:08:55):
And then secondarily, I think the semiconductor companies are realizing that it is far better to make annual revenue on licenses or something you can sell their customer, rather than selling them a chip and then hoping they come back in a few years and buy more chips from you. So that was another reason I think Nordic was excited by the SaaS business model.
EW (00:09:22):
It evens out the revenue.
TH (00:09:24):
Yeah.
EW (00:09:24):
Otherwise it is very lumpy.
TH (00:09:27):
Mm-hmm. Yeah. For us, it was not necessarily a surprise. I think we got this advice over the many years of having the company, is your acquirer or potential acquirers will always be the ones that you partner with, and have been working with for a while.
(00:09:41):
I think we had been working with Nordic for five or so years. We had integration and the integration kept getting better and better. Memfault was an automatic Kconfig flag in their SDK that you could enable. And we had a bunch of built-in metrics, that you could capture LTE metrics and performance, and Bluetooth as well.
(00:10:03):
They wanted us to spend even more time on the integration. They wanted obviously the revenue. And from our side, we are like, "Finally! Great! We finally have access to your engineers and the teams that we have been trying to get a hold of, to help improve our integration with the Nordic chip sets. Because it can improve the way we sell, and how easy it is to sell into Nordic chips."
EW (00:10:31):
Has the acquisition changed your job significantly?
TH (00:10:35):
Yes. I think probably the primary way that it has changed my job, is we are now trying to merge. So Nordic had an existing cloud product called "nRF Cloud," that- It was very much on the leading edge, which makes sense.
(00:10:57):
Like you buy a chip or you are thinking about buying a chip, and you need a few things. You need security, provisioning, a way to contact the device just in general, and you need OTA. That is what nRF Cloud was.
(00:11:12):
I think Memfault actually took more of a ladder role. It was when you realize you have problems, or when you are scaling. You are going to run into issues around debugging, gathering analytics, doing large scale analysis on all of the devices in the fleet.
(00:11:30):
So the goal now is to merge both nRF Cloud and Memfault into one product, that you may sell at different times to different companies, but it feels the same. Of course, in the cloud backend, it will be ten more years until it feels like the same software project. But that is my goal now, is helping and working on that transition/merging.
CW (00:11:57):
That is why I have to log into nRF Cloud.
TH (00:12:00):
Yes. I hope many of those experiences will continue to improve over the coming months, and they will. That is the goal. But yes, there is still a little bit of, "Log into nRF Cloud, redirect to Memfault. Log into Memfault, redirect to nRF Cloud."
CW (00:12:13):
Yeah.
EW (00:12:13):
Both Christopher and I are putting Memfault in client devices.
CW (00:12:19):
Have put. I finished.
EW (00:12:22):
It was an average. I am in will put.
CW (00:12:24):
Okay <laugh>.
EW (00:12:24):
What do we need to know? And what do we need to tell our clients about what Memfault is?
TH (00:12:35):
That is such a huge question.
EW (00:12:39):
Yeah, because it is sunny outside. I wanted to go hang out with the dog outside, while you just record this part.
TH (00:12:45):
<laugh>
EW (00:12:45):
I will be back in about ten minutes, okay?
TH (00:12:49):
Yeah, exactly. What do you need to know? I will start with that. Try and take the easy- Well, it is easiest on, I guess, chip sets that have more of an easy mode. Zephyr, ESP32, Nordic chipsets, they are quite easy to integrate on.
(00:13:09):
If you are just integrating into a bare Cortex-M sort of firmware, there is probably more that you will need to do, to help out the Memfault SDK integration, because it is firmware after all. You have to modify linker scripts manually. And Windows is hard.
(00:13:27):
But all of it is in the documentation and honestly, I am sure you have talked about it a lot in the other podcast, but the AI tools are really good. We now have a little AI helper bot, chatbot, on our docs page, which I think actually helps quite a bit as well. It continues to improve as well with the new models.
(00:13:48):
And then what do your customers need to know?
EW (00:13:53):
What do our clients need to know?
TH (00:13:54):
Clients. Clients. Sorry. Thank you.
EW (00:13:56):
Not the customers for the devices, but the device vendors.
CW (00:14:01):
The people we write software for.
EW (00:14:02):
That we are working for.
CW (00:14:03):
Yeah.
TH (00:14:04):
In what sense? Is this how they should use Memfault? Or, why they need to use Memfault? Or?
CW (00:14:09):
I think a little bit of why, and a little bit of-
EW (00:14:11):
How.
CW (00:14:12):
What does it do for- Does it do- We know this, but there is sometimes some confusion about, "Oh, does it do OTA for us? Oh, is it going to consume our customer data, or store our customer data for us? Or will this find-" I keep getting questions like, "Anytime there is a bug, will Memfault help with this?" Sometimes yes, but many times no, because it is like, "That is not the kind of bug we are going to be-"
TH (00:14:39):
Totally. And it is not a magic wand.
CW (00:14:41):
Right.
TH (00:14:41):
As much as I wish for the thing to be magical, I continue to believe-
CW (00:14:46):
Yeah. I get tons of questions like, "Oh, we are having Bluetooth performance issues, where sometimes it disconnects from the app and we cannot tell why and blah, blah, blah." And I am like, "I do not think that is really going to help. I think we need to look at our detailed logs and catch it."
(00:14:58):
This is more for fleet performance and things. But it is a difficult question to answer as an engineer sometimes, because it is a fine line. It will find specific bugs if there are crashes in asserts and things too.
TH (00:15:12):
And to your point about Bluetooth performance, we are trying to do both, I would say. We have gotten pretty good at the LTE side of it. Because that was honestly the first project we were working on with Nordic for the last two or three years, is they were trying to optimize and fix a lot of issues with the nRF91 series.
(00:15:30):
So now you can actually capture a lot of the metrics from the modem. Which tower is it connected to? Which country is it in? All these sorts of stats, from the actual cell modem, which is great. Most people do not actually know how to capture those.
(00:15:46):
And then Nordic has a feature where you can actually capture an LTE modem trace. Then you can download that, and throw that into Wireshark, and actually see exactly what is happening.
(00:15:57):
That is really how Nordic debugs. They are like, "If you have any issues, please capture a modem trace." Most people do not know how to do that in general, and then definitely do not have remote diagnostic or remote capabilities to grab that. But Memfault has that baked in.
(00:16:09):
You can just say, "Grab a modem trace." Memfault will pull it over and then you can download it from the Memfault interface. So that is very cool. Hopefully we do the same for Bluetooth soon.
(00:16:19):
But what do your end clients need to know? It is not a magic wand. I think the effort that you put into it, is what you will get out of it. I think this is something that I have always believed in, is you can write software in a way that is trying to surface as many bugs as early as possible.
(00:16:40):
You can also write software in a way that you are trying to hide or sink as many bugs as possible into the system, and not raise them.
(00:16:50):
But the more you raise issues, the more you- Not necessarily log, but I mean, I like asserts if it is still in development mode or it is on a small subset of devices. The more you assert, the more you will catch them in Memfault. Because then you will get the exact stack trace, and memory stack of what is actually going on.
EW (00:17:08):
That is a good point. I do not think I have articulated it as well as you have. Which is there are times when folks- When I have written software to expose as many bugs as possible, as early as possible.
(00:17:26):
And there have been a few times where I want to hide any remaining bugs. I want the failures to fail very quietly, and just sort themselves out and reboot or whatever. I do not need to know about them. I just need them to go away.
TH (00:17:46):
Because also they may not be essential, right? It may be like if it is every minute you are trying to capture a temperature sensor, you are just like, "We will capture it the next minute." It is totally fine.
EW (00:17:55):
I was actually thinking about times when I had big demos, and I did not want to see the bugs in the next week. I just need things to work as well as they are going to work, and we will go back to bug fighting mode after.
TH (00:18:10):
Mm-hmm.
EW (00:18:10):
So on the, "What is it that Memfault does?" I looked at the memfault.com website. I have done it before, and I have sent other people there before.
(00:18:23):
When I sent a colleague there for my current project, he came back and he was just like, "I do not know what that does. But we need to do firmware update, we need to do security, we need to do a bit of monitoring on the devices and we need to handle hard faults." And I do not think they do any of those things.
TH (00:18:44):
What?
CW (00:18:44):
<laugh>
EW (00:18:50):
Your website is a little perplexing. Like maybe it is written in CEO speak?
CW (00:18:59):
Well, now I have to go look.
TH (00:19:04):
Honestly, I think it is very difficult to write to everyone as well. If you do need to write to CEOs and you need to write to firmware engineers on the same website, I would say that is just challenging in general.
(00:19:15):
We do not A, B, C, D, E test, depending on your persona logging in, based on tracking of sorts. We do not actually do that for a marketing website. The ultimate goal is to get somebody to write in and say, "Hi, I would love to have a call."
(00:19:33):
So ultimately it does not need to explain it as well enough to turn that person away to be like, "I actually do not need some of these five things," or something like that. And then they will make the decision themselves. We would love for them to just say, "Hi."
CW (00:19:49):
Are you sure they spelled it correctly? Because this website says a lot.
EW (00:19:53):
"Resolve quality escapes to improve customer satisfaction."
CW (00:19:56):
Well, that is just the first line. You got to scroll down. It says-
TH (00:19:58):
It also sounds like that person may not have read all that much on the website. I totally agree with that, that the first two lines may not exactly spell it out. But I think some of the subpages actually are very explicit now.
EW (00:20:12):
Oh, I think so too. And you have a section that is for embedded engineers, and they did not really get to that part. I am pretty sure what they wanted to do was say, "No, that does not work. We have to build our own." And I just went back to let us just-
TH (00:20:26):
They were just trying to prove it, prove it for themselves.
EW (00:20:28):
Talk it over what we are doing here.
CW (00:20:31):
Never build it again.
EW (00:20:33):
What we actually came up with was, "We can do it ourselves. But let us use the same interface and framework that Memfault uses, in case we ever decide to change our minds."
CW (00:20:42):
Sure.
EW (00:20:42):
And then three days later, "We should just use Memfault's," was the response. Like, "Yes! Thank you."
TH (00:20:49):
Yes! We got them. We had a talk about this before. It is just never worth anybody's time, especially early on, to build out a whole suite of tools.
CW (00:20:59):
"What product are we making?"
TH (00:21:02):
Exactly. "What does the thing actually have to do?" "Oh, it has to-" Like the Yoto, "It has to play music for kids." "Great. Let us build your bugging tools." I do not think that would ever really happen.
(00:21:10):
But it also depends on scale, just how many devices they are trying to ship. If they only want to ship ten, then do not use Memfault.
EW (00:21:20):
That is actually a good question. Have we really defined- Okay, so let me define what Memfault does, and then you can tell me how wrong I am, which is great.
TH (00:21:33):
Amazing. Love this exercise.
EW (00:21:36):
With Memfault, if you have a device that has a connection to the internet-
CW (00:21:45):
Even by proxy.
EW (00:21:46):
BLE, smartphone, or Raspberry Pi with Wi-Fi-
CW (00:21:54):
Or somebody watching the serial tracing, cut and pasting, and then pasting them to the website. That works too.
TH (00:22:00):
Mm-hmm. It does work.
CW (00:22:00):
<laugh>
EW (00:22:02):
All right. That was not what I planned to do.
CW (00:22:04):
It is the worst way, but it is possible.
TH (00:22:07):
It is the way we prototype. Yes. It is the first gate, step one.
EW (00:22:13):
Okay, you have something that connects to the internet, and you want to build one of them. That is easy. You just do that yourself. It becomes your pet. It lives in your house. You check on it whenever you need to check on it.
(00:22:24):
Now your friends and family want the same device, so you build it for them. You realize you are constantly calling Aunt Agatha and telling her that her goldfish is about to die, unless its bowl is cleaned. Whatever thing that this device does, you have to monitor it constantly.
(00:22:44):
But hopefully you do not also have to monitor that Uncle Freddy's bowl has not checked in in three weeks, so you do not even know if his fish is dead. Clearly we are building internet enabled goldfish again. Goldfish bowl, not goldfish.
(00:23:03):
Memfault separates the idea of the data that is related to your product, whether or not the fish is alive, to the data that is related to any internet enabled product. Which is, does it check in? Does it have a heartbeat? Has it crashed? Is its battery doing well? How do you do firmware update? How do you connect the user to the device? Provisioning.
(00:23:30):
It takes those common IoT device problems and puts them in their own little box called "Memfault." And then this data that is important to you about the goldfish status and the water health, that goes to your own servers. Memfault does not care about your data.
(00:23:50):
Memfault wants to talk about the operation of the device, not what the device is doing. The way you accomplish that is you put some pieces of hooks in your code, and those send up special messages to Memfault, that provide all the data.
(00:24:11):
Memfault will host your firmware update bytes. Through secure magic means it will send the data down, and you will have to firmware update locally, which is as it should be, of course.
(00:24:26):
Okay. What did I get wrong?
CW (00:24:29):
<laugh>
TH (00:24:30):
Wow. Pretty good actually. The one thing that I really enjoyed, and maybe it is a feature gap or not, but I do think I really appreciated your distinction between the data Memfault collects, and the data that your product and backend and the data pipeline that your company owns, should collect.
(00:24:50):
Meaning we do believe there should be a separation between the diagnostics and observability, data and systems, and separately the thing that makes your product work. Because ultimately you may cancel Memfault.
(00:25:04):
Or, you may realize you are not going to make any more money on this device, but you want to keep these devices operational for the most minimal costs. You may not want to have active monitoring on a bunch of devices anymore. And so you may want to cancel the subscription. You do not want the main data pipe to go away either.
(00:25:25):
But yeah, that is the goal, is build reusable pieces and components that people can plug into their firmware. That is we are talking about probably MCU specific stuff, since that is what this podcast is about. But we also build things for Linux and Android. Having all that data plumb into one single system, regardless of what hardware you are building, is pretty cool, I would say.
(00:25:49):
I think the other just high level Aha! moment, was we do not want you to have to hire cloud and data engineers, just to get the observability data into a system. You may have to have a backend engineer to build your actual product, but hopefully they are not working on helping the firmware team debug. That should just be a firmware only solution. Memfault can help firmware engineers just do that alone.
EW (00:26:21):
That becomes really important if your internet enabled goldfish bowl goes from ten friends and family, to a thousand or 10,000. Where you could spend all of your time just trying to figure out why one of them crashed, when what you really need to do is figure out this other crash that half of them experience.
TH (00:26:43):
Correct.
EW (00:26:43):
But unless you can see which is which, you can only respond to the one that cries the most.
TH (00:26:55):
The customers that I think are great matches from Memfault, they have already started experiencing, or they already assume that there will always be, devices in the field that are failing.
(00:27:07):
Like, if you have 10,000, one per day will have some sort of hardware failure. Whether that is a permanent one, or a temporary one, or a chip out of spec. They are just wanting to track these things down. Maybe they have a queue of bugs, and they will write a workaround.
(00:27:24):
Or, they will ship that customer a new device. Or, they will email them proactively and be like, "Hey, you have a really bad battery. We will go ahead and replace your device."
EW (00:27:34):
No, no, I did that. I did that. I emailed, early days in Fitbit, before Memfault. I said, "Your device seems to have had some sort of catastrophic error that I do not understand," at 3:00 PM on Sunday.
(00:27:50):
The person who I emailed was an internal beta user, was also really creeped out when they said, "I do not know how you are finding these things out, but that was when I dried my Fitbit-"
CW (00:28:03):
"In the clothes dryer."
EW (00:28:05):
"In the clothes dryer. It took a swim and then it went for dry. That was a laundry day. Please do not monitor my activities. I do not know why you need to know that I am laundering."
CW (00:28:13):
<laugh>
TH (00:28:17):
"We just want to improve your experience."
EW (00:28:19):
"We just want to-" Yes.
TH (00:28:21):
Yeah.
EW (00:28:23):
Okay. So what do you call the data that goes to Memfault? You said, "Diagnostics and observability." Chris said, "Telemetry," earlier.
CW (00:28:31):
That is because I am old.
TH (00:28:33):
Also a good word too. I think people use that all the time.
EW (00:28:35):
I would have said, "Operations data."
TH (00:28:38):
Also works. None of these things are wrong, I would say.
EW (00:28:42):
Well, now I feel like that is a challenge. I wished we had come up with wrong names.
CW (00:28:47):
"Memfault chunks" is the official name.
TH (00:28:50):
The other core principle that we believe in in Memfault, and it is maybe just a conversation point, is we generally believe the device is always in control, rather than the cloud is in control.
(00:29:02):
So I think for devices where the brain is on the device, they decide when to firmware update. The devices decide when they are in a bad state. You put as much logic into the device. I think that is a great match from Memfault.
(00:29:15):
If the device is truly dumb- I think this is like AWS Greengrass, where all of the logic is in the cloud, and the device just basically gets told when to make operations. That is not a great fit, because there are no bugs happening on the device itself. The bugs are probably all on the cloud or... Yeah, anyways.
EW (00:29:40):
So with firmware update, the device should not be commanded, "Hey, you need to go firmware update." The device should instead be smart enough to say, "You know, it is about midnight my time. I have not seen any action lately. I am plugged in. Let me go see if there is firmware." So it is responsible for checking.
TH (00:30:04):
I think it is ultimately responsible, yes. I think there are optimizations where the cloud may- I think a few of our products are deployed into buildings. It has a lock on every unit in a building or access control. And so they want to update all the devices on Sunday at three in the morning, because-
CW (00:30:25):
Which building is this?
EW (00:30:25):
<laugh>
TH (00:30:27):
Exactly, right? Go break into it. There is obviously some cloud control from that. But yes, the device is ultimately going to know, is it being used? Like you said, Elecia, is it plugged in? Is it connected to Wi-Fi? What is its battery life state? Does it have enough charge to firmware update?
(00:30:48):
All these things, I think the device is in more control and has more knowledge, to be able to make the right decision.
CW (00:30:57):
We are talking mostly about Wi-Fi devices, but Bluetooth only devices, there is a little bit more work to do. Because they obviously cannot go hit a web REST API endpoint and say, "Where is my firmware?" They have to talk to the phone, and the phone has to do some of this for it.
(00:31:13):
As well, the Memfault chunks, which I am going to keep saying, because I love it-
TH (00:31:18):
Great name.
CW (00:31:21):
You have to get from the device to the proxy that does have connection to the internet. So there is more infrastructure there. That is usually more work for the firmware developer, not- That is the stuff that is like, "You got to figure this out, because we do not have internet access." Right?
TH (00:31:37):
Generally. Yeah. The connectivity path that we call it is "eventual connectivity," is what these devices generally need to have, because the devices- All of the Memfault data is almost exported. The only thing that is input into a device is the OTA payload, and a little bit of configuration.
(00:32:00):
That is all generally commanded by a Wi-Fi device, or honestly BLE via- Like, Nordic has a couple of standard protocols, whether that- We call it "Memfault Diagnostic Service." We call it "MDS," but then there are a few other protocols that the nRF apps and nRF Util and their BLE libraries will use, which now speak a little bit more Memfault-like data and can ask for OTAs.
(00:32:27):
But ultimately, the gateway device is going to be responsible for querying for which OTA, for which device, given which software version. And then pull that down and then pass that to the device, in whatever way it transfers data.
(00:32:43):
That made sense.
CW (00:32:45):
Mm-hmm. Yeah. I ended up using the Memfault RAM buffer and then flushing that to flash. I have a bigger buffer on flash, and anytime the phone connects, it says, "Is there any Memfault data for me?" And the device says, "Yeah, here is 10K of stuff."
EW (00:33:01):
Does the app talk to the company servers alone? Or to the Memfault servers as well? I mean, my Goldfish chunks-
CW (00:33:08):
<laugh>
EW (00:33:11):
Sorry, I just want that to be the title. Do the Memfault Goldfish chunks go to my company's server and then to Memfault? Or does the app send them straight to Memfault?
TH (00:33:25):
You can do either. The easiest is directly to Memfault, at any point in time in the process. So whether that is a Wi-Fi device or an LTE device, it can go directly to Memfault. That is, I would say the easiest, it requires less engineers within the company to allocate to us.
(00:33:43):
However, and I think it is a good thing ultimately, mini IoT devices are single secure channel to the company's backend, and so it has to go to their backend. It is the only way the device can communicate to anything on the internet. Usually it is AWS 90% of the time.
(00:34:03):
So for every single MQTT packet or CoAP packet that AWS receives, it will just trigger a little function or a post request to the Memfault cloud saying, "I got a new chunk. Here is a chunk." And then we will reassemble all that data on our backend.
(00:34:17):
It is more expensive, because AWS makes all of its money on data transfer. So that is why we honestly advocate for a lot of companies to send it to us directly. That is why many companies ultimately switch and bypass AWS, but they can do it.
EW (00:34:36):
But if your plan is to replace Memfault when you have enough users that that makes sense, then you plan for the, "I am sending all my data to my home, and forwarding what I need for now."
TH (00:34:47):
It is one way to think about it. Yeah. The data that Memfault's SDK exports is not, I guess, generic enough, or rather it is not a standard protocol. We do not use OpenTelemetry, for example.
(00:35:00):
I think it is another thing people are coming across. It is not really built yet, or small enough, for embedded devices. So we built a lot of protocols and data transports ourselves.
(00:35:10):
But yes, that is a thing people are always thinking about. It is like, "How do I-" And it is not something we are preventing. It is, "How do I eventually plan for the time or the emergency, in which I would have to redirect data from Memfault elsewhere?"
(00:35:25):
Most of the time for us, it is like create a domain name that is essentially an alias for a Memfault domain. So if you are going to send data to devicedata.goldfishbowls.com, have that be an alias for a Memfault domain, which is the ingress URL, so that you can ultimately redirect it somewhere else.
EW (00:35:48):
Oh, okay. I was thinking now I would have to do a firmware update, or app update at least.
TH (00:35:53):
Multiple things. But yes, firmware update is the way the firmware engineers think about it. If you are a cloud engineer, you think about everything in DNS routes and domains.
CW (00:36:02):
And the data is really compact. When I was testing, I was like, "I will leave this device for-" It is like, "Oh, you have accumulated 800 bytes in the last eight hours." Like, "Okay."
TH (00:36:13):
It is super cool.
CW (00:36:14):
That is pretty amazing.
TH (00:36:15):
The fact that a core dump can be a few kilobytes of data, to get the stack traces. The metric payloads are as little as five bytes per metric value, because it is CBOR. It is pretty cool.
EW (00:36:27):
So the analytic tracking and the error tracking is data that comes to you. Firmware update, which- Everyone is scared of firmware update. I find that so hilarious, because it is so much easier than it was.
CW (00:36:45):
Yeah, you should have been scared 15 years ago.
EW (00:36:47):
Oh my God.
TH (00:36:48):
It is true.
EW (00:36:48):
Yes. You should have been terrified 15 years ago.
TH (00:36:52):
Why are they scared? Why are they scared? What makes them scared?
EW (00:36:55):
The things that they should have been scared all along. The security, bricking devices.
CW (00:36:59):
I think the bricking devices is the big one. Everybody still is afraid of that.
EW (00:37:03):
It is like, "No, we do not have to do that anymore. We have plenty of flash. We will just put a lifeboat on there."
CW (00:37:10):
Security, reading it- All that stuff is-
EW (00:37:13):
Reading out the firmware.
CW (00:37:14):
Oh, somebody is going to get our precious software.
EW (00:37:16):
Someone is going to botnet our devices, as though it was worth it.
TH (00:37:20):
Do you believe that it is an issue if the OTA is not encrypted on flash? Or that somebody is going to read the firmware out? People always come to us with that question. I have worked for companies where that is- Fitbit was a little bit more cautious about it. But Pebble was like, "Read our firmware. You are going to get to it anyways at some point."
CW (00:37:41):
I personally do not believe that is an issue, unless you are doing something you should not be doing. Like putting really important credentials in there, that should not be there.
TH (00:37:51):
Which you should not be putting in there.
CW (00:37:52):
Right. But my general feeling about devices that you sell to people, is once you do not have physical control over it anymore, and even if you do, somebody can get that stuff if they are motivated enough. So you have to assume that that is going to happen, even if you encrypt it. There are all these power attacks and things now.
TH (00:38:11):
Yeah, exactly.
EW (00:38:11):
Power attacks, sanding off the ROM-
CW (00:38:15):
Yeah, looking at it under a microscope.
EW (00:38:16):
Looking at it under a microscope. It is really hard to actually hide your bits these days. You just have to make it not worth it.
TH (00:38:25):
Yeah. Just do not be successful. Do not make the security breachers want to look at your payloads.
CW (00:38:33):
The fallback is we make it very hard to put arbitrary firmware on the device.
TH (00:38:37):
Exactly.
CW (00:38:37):
We do not make it impossible to read out the company's firmware. I think that is the more important thing, is putting arbitrary firmware on a device.
TH (00:38:46):
Yeah. That is where my recommendations end for the most part, is sign the firmware, make sure no one else can install the firmware. And put your keys somewhere, where they are designed to be held.
CW (00:38:58):
Yeah. GitHub.
EW (00:38:58):
<laugh> Not GitHub. Do not put your keys in GitHub.
TH (00:39:05):
What feedback do you have for Memfault SDK, or the experience?
CW (00:39:09):
Well, mine is going to be- All of your answers to any feedback I have, are going to be, "Update your Nordic SDK from 2.7."
TH (00:39:18):
Oh, man.
CW (00:39:19):
Yeah. We are on a really old- We are about to ship, so we do not want to have to do anything.
TH (00:39:22):
Well, I hope you at least updated the Memfault SDK, because we still support old SDKs. You just have to not use the built-in one.
CW (00:39:31):
I did not. I am using the old one, for now. We have it on the list to update everything, as soon as things calm down. But it is working fine. There are some limitations.
(00:39:40):
Like, it was not easy with the old, old, old SDK to put arbitrary- I wanted to do something arbitrarily when the heartbeats happened, and there was not an easy hook to do that, without rewriting a function. I figured it out, but it was not like, "Here is the hook to do something every heartbeat."
(00:40:03):
Yeah, I think it would be helpful- This is a general thing. Helpful to provide more examples to people, of how to get stuff off the device when they do not have Wi-Fi. I did have to scratch my head a little bit. We have our own methods for communicating for our data. I just eventually built on top of that, which is fine. It was probably the appropriate thing to do. But for a couple of days, I was like, "How do I do this, and how often?"
(00:40:32):
I think for me, there is a little bit of confusion still about how sampling works. Like, "Okay, we have-" We do not, but say we have 150,000 devices. Not all 150,000 devices are checking in every day and sending stuff to Memfault, because that would be insane.
(00:40:46):
So how do those get partitioned? How do we control that? How does that feedback into how much it costs? That kind of stuff was a little bit confusing.
EW (00:40:55):
Do you control that? Does the client company control that? Or does Memfault control that?
CW (00:40:58):
I do not believe we have control over that. We will report, but Memfault will toss them. Correct me.
TH (00:41:05):
Yeah. With SaaS- Our costs come from processing data. So yes, the increasing device count will cost us more, and will cost the customer more. For MCU devices, yes, we do exactly that. We basically toss the data if they send it to us.
(00:41:27):
For higher level software, Android and Linux, they actually download the config data or not, and then it is controlled on the device.
(00:41:33):
But MCU, it is honestly just so difficult to get data down to the device. OTA is a special one that people will build out. But getting that config down for the device, for it to know whether it should send data or not, usually gets built by the customer.
CW (00:41:49):
And nobody wants to do rate limiting and things on their own software. Time is hard.
TH (00:41:56):
But they usually build a flag that turns on and off our SDK itself.
CW (00:42:00):
Yeah. Okay.
EW (00:42:03):
What was the other issue?
CW (00:42:04):
Oh, time. But that is not their issue. It is just you have to have the clock set for things to work correctly.
TH (00:42:08):
Oh my gosh. The clock. I know. I was hoping that would be a solved problem these days, and it is still not a solved problem. Many devices are still being built today that do not have time. I am just like, "Oh no."
EW (00:42:19):
Why do the devices not get the time from BLE?
CW (00:42:22):
Well, they do, but you have to write that.
EW (00:42:25):
Oh.
TH (00:42:26):
Then you have to keep it and persist it across reboots, and that is not dependable, and... Yeah.
CW (00:42:30):
Yeah. We are having a lot of issues with that.
EW (00:42:33):
Seems like a Zephyr feature that should exist.
CW (00:42:36):
I am sure it does. It is just config_bluetooth_clock_NTP something, something, something, and it is only available in- I do not know. Sorry. We could talk about Zephyr in a few minutes. <laugh>
EW (00:42:51):
<laugh>
CW (00:42:51):
But yeah, that is how this can scale, is eventually you end up with some statistical sampling of the devices that are in the field, not every single device, something telemetry all day long.
TH (00:43:03):
Yeah. My philosophy is you want to capture the bugs that happen one of every 10,000, 100,000 hours, and you need to know the number of devices and number of hours those devices are actually running real software. And you need that many devices running at any given time. So there is an equation to run.
CW (00:43:27):
Okay.
TH (00:43:27):
Then it also depends on whether those devices are absolutely critical or not. I would argue that we have a customer wearable company, and every user pays them per month. And if a device does not work for a customer, that customer gets mad and wants to stop paying them. So they actually really care how all their devices are working. So they actually track most of the devices and their data.
(00:43:55):
Some companies sell a device for a hundred bucks, and never make any more money on those customers. And so they are like, "We can sample."
EW (00:44:04):
What about FDA devices? Do you deal with those? Are those a we do not sample, we have to track everyone?
TH (00:44:12):
All those customers are generally tracking every single device.
EW (00:44:15):
Yeah. Okay.
TH (00:44:15):
But also the medical device space, as you know, is just such a hairy place. We only work with a few of those companies. And usually it is during internal testing, or very close relations with their end customer. That is like, "Can we track the data? Is the firewall open? Can it send data to Memfault?" All these sorts of things. And then also is it actually like a Class I, Class II, Class III device?
CW (00:44:42):
Okay.
TH (00:44:43):
But as we have discussed, it is just observability. It is telemetry data.
EW (00:44:48):
It is not HIPAA data.
TH (00:44:49):
It is not.
CW (00:44:51):
I mean, you could do something stupid, because you can add custom metrics.
EW (00:44:55):
Oh. You could absolutely do something moronic.
CW (00:44:57):
That is one thing I wanted to mention. When you say the company data and the Memfault data is separate, it is under your power to keep those separate. We do have some things that are specific to our device that we log in Memfault. Like, "This device did not manage to collect any customer data for the last hour," that we log because we want to know.
(00:45:18):
But that does not say which customer. It does not say what kind of data. It just says, "Data collects failed," or "Data collect succeeded," or "Something timed out." But you could log whatever you want. It is just a metric. So you could put customer ID in there, or social security number. Do not do that.
EW (00:45:36):
Do not do that.
TH (00:45:37):
Just with any sort of observability tool, you can log whatever you want to it. We usually push that on the customer. We have a lot of hooks and recommendations on how to capture things.
(00:45:48):
For core dumps, it is basically do not grab those regions of RAM. And, put all your private and sensitive stuff in a certain section, and do not capture it. For logging, do not send it. And for metrics, usually it is just numbers and so it is honestly not that bad.
EW (00:46:05):
One of the questions a listener posed- Mark posed. Was, "When should people not use your product?"
TH (00:46:14):
Super low bandwidth. There are a few things, right? As we have grown, we continue to find places where we should not use it. Super low bandwidth. LoRa is really difficult, because also LoRa it is really hard to firmware update. But it is also just hard to get data out in some sort of reasonable capacity. So that is really tough.
(00:46:34):
If the device requires 100% uptime, it is just another nuance thing. Where it is like once you scale to ten to 100,000 devices, you are at those numbers expecting that many devices fail, and you are just trying to mitigate those.
(00:46:50):
We have customers that come to us and say, "We have 10,000 machines on the factory floor, and if any one goes down, we are hosed. Can you help us?" And that actually is their bread and butter, as an observability system that allows them to do that. That would actually be a differentiator for them.
(00:47:09):
We can help them. We still do sign them. But it is more of a supplementary thing.
CW (00:47:15):
Yeah, because that is a timeliness issue, right? Memfault is not going to give you instant-
TH (00:47:19):
They need to be alerted within five seconds, if something is going down or wrong.
(00:47:22):
Memfault is built, or just has been built over time, for our sampling. Like we said, "You want to sample some of the devices, not all the devices." So we have made some compromises along the way, or optimizations, to where we are not trying to track second resolution of data and build that alerting product.
EW (00:47:43):
Right. When I think about systems like this that I have worked on, there are some systems where you want to know right away. But for large parts of wearables or other industrial things, it is pretty much just, "Oh, I just walked into work. Did anything catastrophic happen last night, that I needed to add to my list today?" Not, "Wake me up at 3:00 AM when things go wrong."
TH (00:48:09):
Can you imagine being woken up at 3:00 in the morning, when a Fitbit went in the dryer?
CW (00:48:15):
<laugh>
EW (00:48:15):
<laugh>
CW (00:48:15):
First of all, who is doing laundry at 3:00 in the morning?
TH (00:48:17):
Yeah. Exactly.
EW (00:48:18):
There were days when it felt like that.
CW (00:48:20):
But yeah, but that is where that observability, that alarm, is part of the product, right?
TH (00:48:27):
It is part of the product-
CW (00:48:28):
That is a feature.
TH (00:48:28):
Exactly. It will either require tons of money to observe it, or they will charge their downstream customers tons of money.
(00:48:38):
Honestly, those customers should probably use something more similar to Datadog. Because Datadog charges $10 per device per month. But they will process and store that much data and have built a product that is more along those lines. But that will bankrupt any company that is trying to build something and sell it for a 100 bucks.
CW (00:49:00):
Yeah, our $99 wearable, $10 a month per- Oop.
EW (00:49:05):
That is for faster data, where you need alarms and very timely response. And real-time-ish.
TH (00:49:12):
Also Datadog is not built- They are sending data in JSON and binary blobs. It is going to be megs of data per minute, is their threshold. We are built for a different system.
EW (00:49:28):
Oh. Christopher is telling me I cannot type and talk at the same time. I do not know what he is talking about.
CW (00:49:34):
I do have a support question for you, while you are here.
TH (00:49:37):
Great. I am available, apparently.
CW (00:49:38):
<laugh>
EW (00:49:38):
<laugh>
CW (00:49:39):
I do not understand sessions, and where I would use them, or what they do.
TH (00:49:44):
Yep, that is a good question.
EW (00:49:47):
You have been validated.
CW (00:49:47):
Yay.
TH (00:49:48):
Yeah.
EW (00:49:49):
Okay. Well, it has been a good show.
CW (00:49:50):
<laugh>
EW (00:49:50):
<laugh>
CW (00:49:52):
I agree. You do not understand.
TH (00:49:57):
There are devices that are operating all the time. For those devices we, and it is built in, capture heartbeat metrics. Heartbeats are typically captured every hour or two hours, which is just a summary of what has happened over the last hour. How long was your Bluetooth connected? How many packets did you not send? How many bytes were written to flash? All these sorts of things.
(00:50:17):
And then there are devices that are used sporadically, like a camera. Or like a personal camera and you wanted to take a session. Or a playback session for a media player. All those are you are trying to track a summary of what has happened during that session. Which may be a minute or an hour or a few days.
(00:50:39):
That is when we guide customers towards sessions. There is nothing by default built in, because we do not know what your product does. I think one session we honestly should implement, but we have not yet, because it can lead to abuse. Is like a Wi-Fi session, or a Bluetooth session. When that connects and then disconnects, give us a summary of what happened over that session.
CW (00:51:03):
Okay. Okay.
TH (00:51:05):
But sometimes devices just get into loops, or they are meant to disconnect and connect all the time. So that would just lead to a bunch of noise.
CW (00:51:13):
It seems like you could get in trouble with too much data, with that paradigm.
TH (00:51:19):
Too much data. And for some companies, a kilobyte of data here and there is actually a lot. We are talking to companies that are deploying-
EW (00:51:27):
Cell modems. Or even worse, deep sea modems.
CW (00:51:32):
Deep sea modems. Some of them are so bad you are-
EW (00:51:34):
They are so slow!
TH (00:51:37):
<laugh> I am glad you are working on those, or at least have understanding.
EW (00:51:39):
<laugh> Yes.
CW (00:51:43):
Deep sea modems are just acoustic couplers separated by a kilometer, right?
EW (00:51:48):
Let us go with yes, because wow they were loud.
CW (00:51:51):
I listened to you testing it in your office. That is what it sounded like.
EW (00:51:57):
But all my nostalgia for how modems used to sound. Now I could just have it be louder.
(00:52:03):
I was not quite done with when we should not use your product. I get the idea that very small systems, like my ten person family goldfish bowl would not work, because I would not want to pay for it.
(00:52:24):
At what point does Memfault recommend we start thinking about integrating? Is it 15, or 1500?
TH (00:52:38):
That is a good question. I would say it is more along the lines of around a thousand.
(00:52:45):
It is like I guess for nRF Cloud, the provisioning, security, you are buying nRF91s and you need them to work over LTE. That sort of division of the product, a hundred or so. You probably should be using nRF Cloud, rather than trying to roll your own solution.
(00:53:06):
And then for the diagnostics piece, I would say a thousand. I would say we do not talk to too many companies, or it is not worth the effort- I will expand upon that. Unless the companies are trying to shoot for a thousand, and then hopefully more. We rarely talk to customers that are like, "We only want to bill a thousand and that is it forever." Because it is a tough sell in that sense.
(00:53:30):
By the effort, is like we put forth a lot of effort to help our companies instrument their firmware. There is only so much Memfault can provide out of the box, because every product is ultimately different. The way people think about things. How do you want to track either its product analytics or firmware? Or, what signals a broken device?
(00:53:55):
We will try to help them with that. We will run sessions every week or so about how to do that. Sometimes they come to us with a new MCU, or a new version of an RTOS that we have never worked with. We will expand the SDK support and help them along the way.
(00:54:13):
Some companies are not worth talking to, because they do not have the appetite. And yeah, I could go on.
EW (00:54:23):
At what point is somebody too big to go to Memfault?
TH (00:54:28):
Ooh. Good question. We have a few companies in the millions of devices threshold. Anything beyond that, it is a really tough sell, because they do start getting into the cost. It is cost prohibitive basically.
(00:54:47):
But those companies are starting to look into the sampling. They are like, "We do not actually need-" Also, if they are selling a million devices, they do not need all million to work every single day, basically. Or they are not as critical.
(00:54:58):
They also assume that many are going to fail. It is just a, "How quickly can we drive these customers through customer support and replace these products?" which I guess Memfault does help with to an extent.
(00:55:13):
Any company that has many more than those devices, they probably also have many more engineers. And they can allocate a few engineers and a cloud engineer and a data engineer, to build something that is more tailored to exactly their use case.
(00:55:27):
We have done our best to make a generalizable solution that works for most customers. But if you are trying to, for instance, look at your backend and mobile app and firmware logs and how it pertains to a couple other devices in an ecosystem, and you want to all view that in one single pane of glass, Memfault does not build for all of that.
(00:55:54):
If companies are needing that and requiring that, then we can hook into that. We can help you with API integrations. But maybe they should just go off and build their own.
EW (00:56:07):
There is a point at which it is worth going off to build your own.
TH (00:56:14):
For sure. It is all about money. It is all about money, and do they have the resources and firmware engineers sitting on their hands enough, to honestly go do the thing. We talk to a lot of companies that just are out of resources. They are like, "We have no time to debug our products. We are completely under water and support requests. Can you help?"
CW (00:56:33):
Yeah. And there are a lot of companies that just would not do this, period. The company I am consulting for, there are two firmware engineers, me and the other guy. We would not be doing observability for the two of us, because we can barely keep up with the work that we have. So doing this was-
EW (00:56:51):
But you still had to do firmware update.
CW (00:56:53):
Firmware update already existed.
EW (00:56:55):
Oh, okay.
CW (00:56:56):
Well, we just did Nordic's thing. Yeah, we had to do firmware update, but there was a lot of stuff already there.
TH (00:57:05):
Yeah. Thankfully, the hard stuff on firmware update is, like you said, it is much better than it once was. The bootloaders are more or less figured out. You have A/B partitioning to pull down new firmware and flip over to the new one.
(00:57:18):
The Memfault solution is a lot of, "Help me build the backend to firmware update, so that I can roll it out to certain devices, based on certain rules." Which again, requires a backend engineer for this diagnostic piece, which we hope no one has to hire themselves, because they are very expensive and just very single use case.
CW (00:57:43):
And this client had already done the OTA piece on the backend, so we will be replacing that. We will just be switching over to get it from you instead of S3, plus a lot of features. So it is nice.
EW (00:57:55):
The other question that Mark had, which was very good question is, "What do you wish customers would ask, before they start a serious engagement?"
TH (00:58:05):
That is a good question. I guess it is a little bit of an ask, but it is like ask themselves, "What is this worth?" The reason is because I think there are many- Our equivalent in the cloud space are Century, it is Datadog, BugSnag, Crashlytics.
(00:58:24):
A lot of these tools are very good at starting from zero. One engineer can instrument their backend application, their Python application, and get value out of it. They can generally scale to an enterprise scale, if they are working for a company. Firmware is less plug and play, than a pip install library.
(00:58:53):
I would love for even the Nordic stuff to be even simpler. But ultimately people rearrange their memory maps differently, and they need to store a bunch of data, and a lot of that stuff transports itself, just make it much more difficult.
(00:59:05):
So we spend my time. I spend hours with customers, trying to help them. My fees and my hourly rate is not $5 an hour. I wish the customer would come to us and be like, "Is this problem worth an eighth of an engineer's salary a year? And if so, let us use Memfault."
(00:59:28):
If my budget is $1,000 a year, then I can barely hop on a call with you to solve your problem, or talk about your problem basically. It is just always a surprise. It is always a sticker shock. And I know the pricing- I think we actually are pretty explicit about the price on our website, or at least ballpark wise.
CW (00:59:50):
Yeah. It was pretty quick to find when I looked for it.
TH (00:59:52):
I wish it was cheaper. I wish it was $100 a month. But the number of companies that would be successful without having a lot of our time spent with them, pair programming with them, it just would not work out ultimately.
EW (01:00:08):
So you do have to do a fair amount of hand-holding. You have not gotten to a crank solution where you say, "Here is our website," and you never really talked to them.
CW (01:00:20):
I did not talk to them.
TH (01:00:21):
There is a spectrum, because I have never talked to either of you about Memfault, except for this session maybe.
(01:00:26):
There are engineers we have worked with in the past, who have used us through multiple different companies. They are pretty much self-serve. They actually will come to us. They will be like, "I need an account. I am scaling to 10,000. What is the price? Great. I have budget-"
(01:00:41):
Truly, this is maybe tens of companies that have done that, but it is because engineers have traversed through different ones. There are very smart engineers, who would never want to talk to a salesperson in their life. Who are committed. Who will do it themselves and will ultimately just give us a call or a ring when they have a hundred devices and they are like, "Cool, I am blocked now."
(01:01:03):
I would say the average engineer almost always needs a good amount of time from us, and especially to convert the problem into the solution. I would say some can do the integration, some can capture a core dump and can see it.
(01:01:18):
But now how do they write code in a way that once they have this new ability, how do they now make use of it? That is where I think you require my eight years of thinking about it, or a lot of our teams.
EW (01:01:36):
The problem space is hard if you have never seen it. I did toys and there was no internet connectivity for most of my toys. Even with that, I still had to think about scalability and bug fixes.
(01:01:55):
Then with ShotSpotter and we had hundreds to thousands of units in the field. I needed to know which ones were healthy, which ones were not, and how that would affect my entire distributed system. So we built Excel sheets that did what Memfault does. And we spent a lot of time on firmware update.
(01:02:18):
So when I got to Fitbit, we were looking at problems of scaling, and how to know that systems were healthy and all of the telemetry and diagnostics. I am like, "Oh, well, I am definitely not using Excel again. Let us just go to Python." I thought I was so cool for that.
(01:02:38):
Of course, my login for that lasted way too long. Which you do not really want your engineers to have access to the customer data, or even the diagnostics, unless that is what they are doing. But since I was the only one who really understood the Python script, I had access a lot longer than I should.
(01:02:58):
But the whole process of understanding how these things work together, it does not come up when you are building the internet enabled goldfish bowl. It does not feel like a core product feature. It is not part of the MVP. The minimum viable product is, did the fish die?
(01:03:26):
You are right, you need someone to spend a lot of time thinking about this. Because most people will only be able to see firmware update, because that is something we all talk about.
(01:03:38):
They will not be able to see the observability, or the horrific things that happen when you have a one in a million bug, but you also have a million devices out there. In which case your one in a million bug happens all the time.
TH (01:03:56):
We started the company when, of course, it was not at the very beginning, but many companies had never thought about any of these problems before.
EW (01:04:05):
They still do not.
TH (01:04:08):
Truly. Truly they still do not. Many still do not. Many people have still never searched for OTA yet in Google to find us, or have never heard of Memfault. So it was a lot of education. We spent a lot of time educating customers on why they should care.
(01:04:24):
I think that was still something that we are continuing to do all of the time. Which is why we do these podcasts, and why we have our whole podcast series as well at Memfault, is sheerly education.
EW (01:04:39):
What is the name of your podcast series? Which I should know, because we did not episode with you.
TH (01:04:43):
Coredump. The Memfault Coredump series. It is usually with François and Chris.
(01:04:46):
But yeah, the people are never going to- And I understand it. The people are never going to understand the pain, until they have felt it. Which is why we have so many repeat customers, that have moved from one company to another. So that is a very common, honestly, path for inbound.
(01:05:07):
Sometimes is not the right time or the right moment. Which is why we try to sell them the security provisioning earlier in the life cycle of their product. Because those are things that they absolutely need, whether that is for legislation and requirements. Then later we can sell them the observability piece, when that feels more on fire to them.
EW (01:05:33):
Yes, because that is when they will need it, is when the fires break out.
TH (01:05:37):
Yeah. I wish companies would put it on ten devices. Because I think at Pebble, our critical piece to get working on- We had a dev board basically. It was like, "Cool. Dev boards. Get OTA working, and get core dumps working. Because otherwise we are not going to move fast."
(01:05:55):
I wish more companies would think that way. But I understand it when they do not prioritize that work.
EW (01:06:01):
No, because that work is not the work that is the features.
TH (01:06:08):
Correct. Exactly.
CW (01:06:09):
"Show me on this spreadsheet the dollars coming from debugging." Nowhere.
EW (01:06:12):
Well, it sounds like Nordic is a good thing for Memfault. But I have to say I am quite disappointed selfishly.
TH (01:06:29):
So here is the thing.
CW (01:06:30):
What?
TH (01:06:30):
One, we were a startup after all. We had to continue raising money. I guess at the growth expectations, we were never going to last forever. Or it would probably turn into a little bit of a different company, in terms of what we could support. Maybe we would have to cut a platform, maybe we would cut Android or something like that.
(01:06:52):
With Nordic support though, we can do a lot more. And they want us to do a lot more. One thing that they love, and that we will not stop doing, is supporting every chip and every hardware platform out there.
(01:07:03):
Because if Nordic, God forbid, does not make the sale on a Bluetooth chip or a Wi-Fi chip, they can still get some revenue from other companies' chips, by having them on Memfault. So that is something that we will continue to do forever.
EW (01:07:21):
I am sorry, Tyler. I was totally teeing up a joke, and you ruined it!
CW (01:07:25):
<laugh>
TH (01:07:26):
I did. Sorry. But you-
CW (01:07:32):
Why are you disappointed, Elecia?
EW (01:07:33):
<laugh>
TH (01:07:36):
I was triggered, because every single company that talked to us for six months after the acquisition, came to that as the first question and be like, "Are you going to support NXP chips? Should we just stop talking right now?"
EW (01:07:51):
My personal beef with this, is that Memfault was a sponsor of the show. Nordic was a sponsor of the show. And then you guys got together, and now neither one of you is sponsoring the show.
CW (01:08:04):
<laugh>
TH (01:08:05):
Oh. I did not know that actually. So not a big deal.
EW (01:08:09):
I just wanted to hassle you. I really did not mean to trigger you.
TH (01:08:17):
Well, it is good, because I was probably not going to say my little spiel, had that not come up.
EW (01:08:22):
Tyler, it has been really good to talk to you, but we should go out and enjoy that sunshine. Do you have any thoughts you would like to leave us with?
TH (01:08:31):
I do. I thought about this one. One of my favorite things in the last few years since we last chatted, was we met a lot of our neighbors. It has turned into a wonderful community, both of families and friends. A lot of different people and a lot of different times of their life, which is amazing. I hope everyone else prioritizes it, because it has really changed our lives, my wife and I.
EW (01:08:59):
Meet your neighbors.
TH (01:09:01):
Meet your neighbors.
EW (01:09:02):
It is one of the pieces of advice you would get if you joined your Community Emergency Response Team. The folks who withstand emergencies in the community, fires, hurricanes, earthquakes, whatnot, are the communities who actually can say more than hello to their neighbors, who know just a little bit about their neighbors. It really is effective for disaster situations, but it is also just nice.
(01:09:30):
Our neighbor is moving, and we are bummed.
TH (01:09:34):
Oh, man.
EW (01:09:35):
Hopefully we will get another one. Maybe a cute family.
(01:09:37):
Our guest has been Tyler Hoffman, co-founder of Memfault.
CW (01:09:43):
Thanks, Tyler.
TH (01:09:45):
Thank you both for having me. It was a pleasure.
EW (01:09:48):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon Listener Slack group for questions, and Ko-fi supporters for coffee. And thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm the website.
(01:10:10):
Now a quote to leave you with, from Jane Goodall. "What we do makes a difference, and you have to decide what kind of difference you want to make."
