Disclaimer: The article was written in October 2021, but was only published in December 2024. Therefore some of the information might be outdated

MQTT is a popular choice of protocol, especially for IoT systems. It is a perfect for connecting constraint embedded devices to the cloud because of its light footprint, publish-subscribe model, bi-directional connection, and support to secure connection with TLS.

In general, an embedded device connects to an MQTT broker to publish data — mostly sensor readings and subscribes to a topic to listen for commands from the cloud. The user’s web or app client can send these commands directly to the broker, or the server (on behalf of the user) could send it. This post covers the latter method in detail; it is an essential piece of the system, especially when you want non-MQTT clients (like OAuth application) to access the IoT-enabled device.

                       ┌──────────────────────────────┐
                       │                        CLOUD │
            ┌──────┐   │  ┌──────┐     ┌────────────┐ │
            │DEVICE│◄──┼─►│      │     │            │ │
            │  A   │   │  │      │     │            │ │
            └──────┘   │  │      │     │            │ │
            ┌──────┐   │  │      │     │            │ │
            │DEVICE│◄──┼─►│ MQTT |◄───►| WEB SERVER │ │
            │  B   │   │  │BROKER│     │            │ │
            └──────┘   │  │      │     │            │ │
            ┌──────┐   │  │      │     │            │ │
            │DEVICE│◄──┼─►│      │     │            │ │
            │  C   │   │  │      │     │            │ │
            └──────┘   │  └───▲──┘     └──────▲─────┘ │
                       └──────┼───────────────┼───────┘
                              │               │
                          ┌───▼───────────────▼───┐
                          │         USER          │
                          ├──────┬──────┬─────────┤
                          │MOBILE│ WEB  │3rd PARTY│
                          │CLIENT│CLIENT│OAUTH APP│
                          └──────┴──────┴─────────┘

Since our cloud application is in Python, all the implementation details mentioned here are for it, but the same approaches could be taken in other languages/stacks. Also, this post is limited to MQTT v3.1 and v3.1.1.

A little background

As mentioned above, we had a similar setup in our IoT project — devices connecting to MQTT broker and users (via mobile apps) connecting to our web server. We wanted to send “commands” to the devices from the web server. The commands could either — be fire-and-forget or request-response types, executed in-process (of web request) or by the background workers, sent to one or multiple devices at a time.

Considering those use-cases, we were able to jot down the core utilities required for it:

• a simple blocking function to publish any message
• a simple blocking function to subscribe to topics, wait for n number of messages or until timeout, returning the messages received
• a simple blocking function to achieve request-response kind of functionality — could be a wrapper of the above two

After that, we started looking for an MQTT client to connect the webserver to the broker. We decided to go with paho-mqtt — the python module part of the Eclipse Paho project, which also maintains MQTT libraries for other popular languages. The module is stable, feature-rich as well as well designed. However, it took us some time and iteration to work with it.

This article goes through that journey of making MQTT work with our web server like a charm — it puts forward various iterations, learnings, and know-hows we had during a year with this project.

I write this post as a series of actions, events, and decisions to give a complete picture. If you want some code snippets to work with, you can find the link to the code snippet below.

MQTT helper functions and classes in Python. ($2183967) · Snippets · Snippets

[v1] Simple wrapper functions

We thought this would be easy — as paho mostly did the heavy lifting and had the blocking APIs we needed(well, almost, you will see!).

Publish function was simple. It was a “wrapper” around the paho.mqtt.publish.single function.

def publish_to_mqtt(
        topic: str, payload: str, qos: int = 0, retain: bool = False
) -> MQTTMessageInfo:
    return publish.single(
        ...
    )

Similarly, we found paho.mqtt.subscribe.simple, but this one was not enough. It did not have any “timeout”, meaning it will block forever if no message on that topic is received. We decided to copy the code of it and add a check for timeout (the bold part below). That did the job.

def subscribe_to_mqtt(
        topics: Union[str, List[str]],
        qos: int = 0,
        msg_count: int = 1,
        retained: bool = False,
        timeout: int = 5,
        userdata: dict = None
) -> Union[MQTTMessage, List[MQTTMessage]]:
    """
    modified paho.mqtt.subscribe.simple() method to support timeout
    """

    # callback for on_connect
    def _on_connect(..., userdata: dict, ...):
        # raise Exception if did not connect
        ...
        # subscribe to topics in userdata
        ...
        

    # callback for on_message
    def _on_message_simple(..., userdata: dict, message: MQTTMessage):
        # ignore if msg_count already 0
        ...
        # ignore if message is "retained" and user does not want it
        ...
        # decrement msg_count and set/add it to messages in userdata
        ...
      

    # validate msg_count and qos
    ...
    # Set ourselves up to return a single message if msg_count == 1, or a list if > 1
    ...
    # make userdata
    ...
    # create client instance and set callbacks and user credentials
    ...
    # connect to broker, mark start_time and start "threaded network loop" with client
    ...
    # wait till all message received or timeout
    while True:
        should_timeout = time.time() > start_time + timeout
        if userdata["msg_count"] == 0 or should_timeout:
            client.loop_stop()
            client.disconnect()
            break
        time.sleep(0.2)
    # raise error if no or less messages found
    messages = userdata["messages"]
    if not messages or len(messages) < msg_count:
        raise TimeoutError(...)
    return messages

We naively thought that combining the above two functions would do the job for request-response. But that did not work very well, as client instances were created and connected twice, making it really slow and practically useless — as most of the time, our devices will send responses before the server even subscribes.

So instead of using the publish_to_mqtt function, we modified the above subscribe_to_mqtt such that the “publish arguments” are passed as userdata. When the client is connected, it publishes this data after subscribing. This optimization made sure that the publish happens in the same connection as subscribe, and also subscription is done before publishing so that the response is not lost.

def subscribe_to_mqtt(...):
    ...
    # callback for on_connect
    def _on_connect(client: MQTTClient, userdata: dict, flags: int, rc: int):
        ...
        # subscribe to topics in userdata
        ...
        # publish data based on pub_on_connect in userdata
        if "pub_on_connect" in userdata:
        ...

def request_response_to_mqtt(
        topic: str,
        payload: str,
        response_topic: str,
        qos: int = 0,
        response_msg_count: int = 1,
        response_timeout: int = 5,
        retained: bool = False
) -> Union[MQTTMessage, List[MQTTMessage]]:
    # create userdata with `pub_on_connect`
    userdata = {
        "pub_on_connect": {
            "topic": topic, "payload": payload, "qos": qos
        }
    }
    # subscribe with created userdata and
    # wait for messages on response_topic
    return subscribe_to_mqtt(
        userdata=userdata,
        ...
    )

You can find the full code for this under mqtt_v1.py file in the code snippet.

>>> # simple demo
>>> for i in range(5):
...     start=time.time(); rpc(device, "RPC.Ping"); end=time.time()
...     print(f"[{i}] Time taken: {end - start}")
... 

RPC: RPC.Ping responses: b'{"result": "pong"}'
[0] Time taken: 1.220142126083374
RPC: RPC.Ping responses: b'{"result": "pong"}'
[1] Time taken: 0.8168301582336426
RPC: RPC.Ping responses: b'{"result": "pong"}'
[2] Time taken: 1.0094208717346191
RPC: RPC.Ping responses: b'{"result": "pong"}'
[3] Time taken: 1.0069520473480225
RPC: RPC.Ping responses: b'{"result": "pong"}'
[4] Time taken: 1.003695011138916

And that’s it. We were up and running as it passed the basic tests, and that was enough until our users started complaining about latency — more about that in the next section.

[v2] Singleton that re-uses the connection

Once after rolling out the system in beta with about 20–25 devices and its users, we got some feedback from them. They were experiencing noticeable latency (more than 3 seconds) when running commands to control their devices.

After doing some basic profiling, it was clear that the bulk of the latency is because of the TLS handshake and connection establishment with the broker. Each time the sever interacts with the broker, it has to establish a new TCP+TLS connection. We decided to optimize this in the same way Database connections in web servers usually do — persisiting the connection and re-using it. Since most webservers (at least the WSGI once) re-use the processes created for serving web requests, it is possible to have an MQTT connection that can outlive the HTTP request-response cycle.

To accomplish this, we decided to use a Singleton class that holds the client and its connection. This class would then expose those utilities and under the hood use the persisted connection. Most of the work was to re-organize the existing code from the previous version to support this pattern. The singleton class (we called it MQTTAgent) looked something like the below.

class MQTTAgent:
    _instance: "MQTTAgent" = None
    _client: Optional[MQTTClient] = None

    @staticmethod
    def get_instance() -> "MQTTAgent":
        """ Static access method. """
        ...

    def __init__(self):
        """ Virtually private constructor. """
        ...

    def clean_up(self):
        """ Cleanup network connection and client. """
        ...

    def _connect_async(self):
        """ Setup client and connection asynchronously. """
        ...

    def connect(self, timeout: int = 5):
        """ Connect synchronously with timeout. """
        ...

    def disconnect(self):
        """ Disconnect from broker synchronously. """
        ...

    def __on_connect(self,
                 client: MQTTClient,
                 userdata: dict,
                 flags: int,
                 rc: int):
        """ Callback called when client connects. ""
        ...


    def __on_message(self,
                 client: MQTTClient,
                 userdata: dict,
                 message: MQTTMessage):
        """ Callback called when message received. """
        ...

    def is_connected(self) -> bool:
        """ Check if connected to broker. """
        ...

    def publish(self,
            topic: str,
            payload: str,
            qos: int = 1,
            retain: bool = False) -> MQTTMessageInfo:
        """ Publish message synchronously. """
        ...

    def subscribe(self,
              topics: _Topics,
              qos: int = 1,
              retained: bool = False,
              msg_count: int = 1,
              timeout: int = 5) -> _Messages:
        ...

    def request_response(self,
                     topic: str,
                     payload: str,
                     response_topics: _Topics,
                     qos: int = 1,
                     response_msg_count: int = 1,
                     response_timeout: int = 5,
                     retained=False) -> _Messages:
        ...


# convenient global instance that could be
# directly imported and used.
mqtt_agent = MQTTAgent.get_instance()

You can find the full code for this under mqtt_v2.py file in the code snippet.

>>> # simple demo
>>> for i in range(5):
...     start=time.time(); device.rpc("RPC.Ping"); end=time.time()
...     print(f"[{i}] Time taken: {end - start}")
... 

Connection Accepted.[0] mqtt client connected to mqtt-broker:1883.
RPC: RPC.Ping responses: b'{"result": "pong"}'
[0] Time taken: 1.0129196643829346
RPC: RPC.Ping responses: b'{"result": "pong"}'
[1] Time taken: 0.10076355934143066
RPC: RPC.Ping responses: b'{"result": "pong"}'
[2] Time taken: 0.10155487060546875
RPC: RPC.Ping responses: b'{"result": "pong"}'
[3] Time taken: 0.10111713409423828
RPC: RPC.Ping responses: b'{"result": "pong"}'
[4] Time taken: 0.1007685661315918

The MQTT utility is now clean and elegant to use as well as efficient. The latency improved by a factor of 10 (check the demo code output above). Users noticed the difference and appreciated it. A win for the day!

[v3] Making it thread-safe

When we got the requirement to make concurrent command calls to multiple devices, we had two options — execute them in various threads or use a distributed task queue system (we already had a Celery setup with our server).

The former looked like the apt solution but would require us to make the v2 solution “thread-safe”, as mqtt_agent.subscribe(and therefore mqtt_agent.request_response) was not designed keeping concurrency in mind. Instead of doing that, we lazily went with the “task queue” approach. It would schedule the concurrent command calls and hands them to distributed background workers for execution. It did the job, but because the tasks were “queued” and “distributed-ly executed”, the performance was not consistent. It would take more than expected time — when the queue is long, or workers are busy. It often happened as the number of devices per user increased. It looked like we had no other option but to make MQTTAgent thread-safe.

In v2, the userdata(set as an attribute of MQTTClient and passed to on_message handler) keeps the “subscription state”; the key to making MQTTAgent thread-safe is to make it thread-safe. The idea is to have a “store” of all “subscription states”, and when a message is received, it checks the store, and if any “subscription” had requested to receive messages from that topic, then we add this message to that “subscription state”. We also need to handle subscriptions to the topics smartly — if two thread subscribes to the same topic then internally it should only subscribe once and give the same message to both.

Firstly, we created _MQTTSubscription class for storing “subscription state” instead of using dict. Also moved the logic of adding new messages inside the class.

class _MQTTSubscription:
    def __init__(self,
                 topics: _Topics,
                 msg_count: int = 1,
                 qos: int = 1,
                 retained: bool = False):
        self.thread_id: int = current_thread().ident
        self.topics: List[str] = topics if isinstance(topics, list) else [topics]
        self.messages: _Messages = list() if msg_count > 1 else None
        self.pending_msg_count: int = msg_count
        self._msg_count: int = msg_count
        self.qos: int = qos
        self.retained: bool = retained

    def add_message(self, message: MQTTMessage):
        """ Handle new message for this subscription. """
        # ignore - pending_msg_count <= 0 or message is retained but the request is not for retained
        ...
        # decrement pending_msg_count and add it to messages
        ...

After that, we added the “subscription store” and “topic reference count” attributes to MQTTAgent. The “subscription store” is a set of all subscription instances that are waiting for messages. The “topic reference count” is a map of the topic and the number of subscription instances subscribed to it.

class MQTTAgent:
    ...
    _subscriptions: Set[_MQTTSubscription] = set()
    _topics_count: Dict[str, int] = dict()

    ...
    
    def _disconnect(self)
        ...
        # reset _subscriptions, _topic_count
        self._subscriptions = set()
        self._topics_count = dict()
        ...

When a new “subscription” is requested, a subscription instance is created and added to _subscriptions set, “topic reference count” is incremented for those topics, the _client subscribes to the topics that are not already subscribed (with reference count as 1). The reverse happens when the subscription needs to be “closed or unsubscribed”, the reference count is decremented, and the _client unsubscribes to the topics that are not referenced anymore (with reference count as 0). In __on_message callback, we check all _subscriptions and call subscription.add_message for the once where the subscription topic and message topic match.

class MQTTAgent:
    ...

    def __on_message(self, ...):
        ...
        # iterate through subscriptions and find any that has this topic
        for subscription in tuple(self._subscriptions):
            ...
            # check if any topic matches for this subscription
            ...
            # add message to subscription
            subscription.add_message(message=message)

        ...
    ...

def _subscribe(self, subscription: _MQTTSubscription):
        """ Subscribe based on subscription. """
        # put it in _subscriptions
        ...
        # increment topic count - make list of topics to subscribe (once that are not yet subscribed)
        ...
        # subscribe to those topics
        ...

    def _unsubscribe(self, subscription: _MQTTSubscription):
        """ Unsubscribe based on subscription. """
        # pop subscription from _subscriptions
        ...
        # decrement topic count - make list of topics to unsubscribe (once that are not needed)
        ...
        # unsubscribe to those topics
        ...

    def subscribe(self, ...) -> _Messages:
        ...
        # connect to client
        ...
        # validate  msg_count and qos
        ...
        # make subscription object
        subscription = _MQTTSubscription(
             topics=topics,
             msg_count=msg_count,
             qos=qos,
             retained=retained
        )
        # subscribe
        self._subscribe(subscription=subscription)
        # wait till all message received or timeout
        start_time = time.time()
        while True:
            should_timeout = time.time() > start_time + timeout
            if subscription.pending_msg_count==0 or should_timeout:
                # unsubscribe
                self._unsubscribe(subscription=subscription)
                break
            # sleep
            time.sleep(0.1)
        return subscription.messages
   ...

You can find the full code for this under mqtt_v3.py file in the code snippet.

>>> # simple demo
>>> print(f"Calling RPC to {len(devices)} device concurrently.")
Calling RPC to 5 device concurrently.
>>> with ThreadPoolExecutor(len(devices)) as executor:
...     start = time.time()
...     # make sure mqtt_agent is connected
...     mqtt_agent.connect() 
...     tuple(executor.map(
...         lambda device: rpc(device, "RPC.Ping"),
...         devices
...     ))
...     end = time.time()
...     print(f"Total time taken: {end - start}")
... 
Connection Accepted.[0] mqtt client connected to mqtt-broker:1883.
RPC: RPC.Ping responses: b'{"result": "pong"}'
[140217424803584] Time taken: 0.14078974723815918
RPC: RPC.Ping responses: b'{"result": "pong"}'
[140217441588992] Time taken: 0.1967012882232666
RPC: RPC.Ping responses: b'{"result": "pong"}'
[140217433196288] Time taken: 0.19601941108703613
RPC: RPC.Ping responses: b'{"result": "pong"}'
[140217416410880] Time taken: 0.19498801231384277
RPC: RPC.Ping responses: b'{"result": "pong"}'
[140217449981696] Time taken: 0.6972782611846924
Total time taken: 1.314225673675537

It was now possible to call MQTTAgent functionalities from multiple threads safely. We started calling commands to multiple-device concurrently with the help of Python ThreadPoolExecutor(as shown in the above demo code) instead of distributing it to background workers. It has not just made it faster and efficient, but we could now scale it to make simultaneous calls to a higher number of devices (tested with 100 devices at a time).

What next?

We have some ideas that could take the utility to the next level. Some of them are as follows.

• “Iter/Stream” like API for getting messages on subscription as they are received. Currently, all messages are “collected” and then returned.
• “Future/Promise” like API for having both async and sync options at the user level. Currently, all functionality is blocking.
• Multiple Broker support.

The current implementation is enough for our use cases, so we might not work on it. But if we get good responses to this post, we could think of working on it and releasing it as an open-source python package. While Eclipse Paho is solid and stable, it requires an “elegance layer” on top of it to make it work like a charm, like how therequests package does it for urllib3. This package could exactly be that — MQTT for Humans™!

In the initial stages of developing the SmartBoard at Next Home, decisions were like navigating through the fog-unclear and shrouded in uncertainty for the future. Some choices felt right at the moment but left us questioning their viability in the long run, and vice versa. Connecting with other hardware enthusiasts in the Hardware Mafia community and our customers at Makenica showed me that our challenges were shared.

Challenges like this inspired me to write about the intricacies of hardware startup struggles. Building on my previous article, The Hard Things About Building a Hardware Startup, let’s now explore the details of various stages of product development.

The Hard Things About Building a Hardware Startup | Harsh Bhikadia

Every stage in hardware development comes with its unique set of demands. Transitioning from one phase to the next requires not just a change in gears but a fundamental shift in mindset. Although a lot of aspects of these stages are shared with “pure software” products, I will be looking at them in terms of a hardware product.

Proof of Concept (PoC) Stage

This is the phase when you have an idea and want to quickly test it.

• Be as scrappy as possible — build with whatever you got
• Don’t take too long to build; this is not the stage where you want to spend too much time
• Always start with mockups rather than building the actual thing
• Be honest to yourself, in terms of validating the idea
• Don’t fixate on the product or solution yet; experiment with various technologies, form-factor, etc.

Prototyping Stage

Here, you iterate and move closer to your product.

• Start showcasing to your potential customers, and let their insights shape product features.
• Understand the problem space better, and build deeper insights regarding it
• Optimize for iteration time and not unit cost, at this stage you are developing and not trying to make a profit
• Maintain the specification of the product that you have in mind, each iteration should solidifying it
• Each iteration should be a step towards the ultimate product

Pilot Stage

At this stage, you think you have a product that can sell.

• Match production-grade parameters as much as possible-in terms of quality, cost, performance, etc
• Mitigate failure cases, and minimize risks associated with it
• Have product quality checks and standards in place
• Validate your product in real-world scenarios
• Meet customer expectations, as this phase is a critical bridge between development and full-scale production
• Finalize Bill of Material (BoM), manufacturing processes, material, etc

Production Stage

At this point, you are producing replicas as fast and cheaply as possible.

• Optimize Cost of goods sold (COGS), reduce the number of custom parts, assembly time, etc. go for anything that can reduce the cost without comprising quality
• Plan for contingencies; ensure adaptability to diverse suppliers and components.
• Have contracts and deals in place to secure better prices and ensure stable supplies.
• Build solid partnerships with various manufacturers and suppliers for your product
• Minimize failure rates during the manufacturing process.

Developing hardware products is like a journey with different stages, each telling its own story. Priorities and challenges change, requiring a constant dance of adaptation. Remember, success comes from moving forward in the development process, not getting stuck at any one stage for longer than is required. The ability to make correct decisions regarding it is what separates the better product builders.

Originally published at https://bhikadia.com.

Harsh Bhikadia

A few months back I wrote about What is a personal website?, while I was pondering on the point of having a personal website. Basically I threw the old inactive website and used my Medium profile as my website’s landing page. From then on I managed to write more articles, averaging 1 every month.

What is a personal website? | Harsh Bhikadia

I have been also re-skilling myself with modern web development, The last time I did web was way back in 2015! Primarily doing it as an effort to revamp the Makenica website. To put my skill to the test I decided on having a new website again. Apart from that I am also taking this opportunity as a way to express myself, not getting restricted to what Medium had to offer. So, keep an eye on custom MDX components I drop in various blogs!

The Stack

I picked up the Profoliox template, that came with the bundle I purchased from LEXINGTON. I was really impressed with the look and feel of the template — quite minimalist but at the same time also expandable. The template is built on Astro and Tailwind CSS, excellent choices for building a static site that is composed of “Components”.

That was about the developer stack, for editing I will be using Obsidian. It is something with which I am quite familiar with. I really like the simplicity it brings when it comes to writing and reading Markdown files. You can expand the basic functionality with various excellent plugins developed by the community.

The whole stack is meant to perform really fast on the user’s browser, shipping minimum javascript, along with excellent editing experience for the editor.

The Hacking

After finalising the stack, it was time to start building it. Starting from a professional template saved me a lot of time. The UI is almost untouched by the template.

After the basic setup was done, I wrote the content for the Home and Work pages. Once that was done, it was time to start importing the articles I published on my Medium. This is what took the maximum time. In the process, I had to learn about MDX and had to build a component for embedding links inside the article (check the top of the article, I have embedded the link of an old post).

Few touches and finalisation here and there, and it was done. In all it took me a weekend to complete the website.

I really like the website — the way it looks, and the technology behind it. Also thought would be apt to write about it as well, and hence this post. I have a few more ideas that I would like to incorporate in future.

Originally published at bhikadia.com.

Harsh Bhikadia

He followed what Eric Ries of ‘The Lean Startup’ fame said about talking to potential customers to validate his idea. He discovered that his product addressed a significant pain point, and there exists no solution for it either. Encouraged by the positive feedback, he started putting hours in developing the product. He also created captivating renders to showcase its potential. He got overwhelming response, customers where eager to get their hands on that device.

Months after working on it, when he started to delve deeper into the manufacturing process, reality hit him hard. He realised that his designs are not manufacturable at commercially viable cost. He had fallen into the trap that countless entrepreneurs face: the lack of early engagement with manufacturers. It is probably as important as talking to customers, but is often overlooked by a lot of Hardware entrepreneurs.

How would it help?

Engaging with manufacturers early in the product development process provides a multitude of benefits that can save entrepreneurs from costly mistakes and ensure a smoother journey towards success.

Help you from taking any “dead end” paths

Manufacturers possess a wealth of knowledge and experience. They have witnessed countless product development cycles and are familiar with the pitfalls and mirages that often trap enthusiastic entrepreneurs. By involving manufacturers from the outset, they can help identify potential roadblocks, technological limitations, and design constraints, allowing you to navigate around them.

Get to know the viability of your product soon

Manufacturers can offer invaluable insights into the feasibility of your product. They can evaluate the proposed design and determine whether it can be efficiently manufactured within the desired timeframe and budget. Their expertise enables them to identify any potential manufacturing challenges early on, enabling you to make informed decisions about product modifications or pivots before investing significant resources.

Get an estimate of the cost in producing the goods

One of the critical factors in any product’s success is its cost-effectiveness. Manufacturers can provide accurate cost estimates based on their knowledge of materials, production processes, and economies of scale. By understanding the production costs upfront, entrepreneurs can make informed decisions about pricing, profitability, and the overall market viability of their product.

Why people don’t do it?

Despite the numerous benefits, entrepreneurs often hesitate to engage with manufacturers early in their product development journey due to a few common misconceptions.

Fear of manufacturers stealing their idea

Entrepreneurs may fear that sharing their idea with manufacturers exposes them to the risk of intellectual property theft. While it is essential to protect your intellectual property rights, engaging with manufacturers does not necessarily entail revealing the full details of your invention. By signing non-disclosure agreements (NDAs) and carefully selecting trusted manufacturers, entrepreneurs can mitigate the risk while still benefiting from their expertise.

Belief that it’s not the right time to talk to manufacturers

Some entrepreneurs mistakenly believe that engaging with manufacturers should only occur once their product design is finalized or closer to production. However, involving manufacturers early can prevent costly redesigns or rework later in the process. Even if you haven’t committed to a particular manufacturer, seeking their advice and insights can help shape your design decisions and ensure manufacturability from the start.

At Makenica, we have witnessed numerous startups and founders embark on their product development journeys, we strongly advocate for early engagement with manufacturers. It will be super valuable. Their expertise can guide you towards a more viable and manufacturable product, saving you time, money, and headaches along the way. So, take the leap and start engaging with those who can help turn your product dreams into reality. Happy Making!

Hello🙏, readers! This blog is going to discuss the concept of “Value”. How you as an entrepreneur (or any professional) should consider it as the real measure of your work instead of anything else.

Often entrepreneurs measure success by the amount of business they’re doing. But for me, it is just a side-effect of something more profound — Value.

When I say “Value,” I’m not talking about vague paper valuations or mystical calculations. I’m referring to the value proposition your product or service brings to the table. It’s about your role in the grand scheme of things — the market, the supply chain, the lives of your customers. In simple words, it’s about how darn useful your offering is.

Think about it this way: your product or service should be so packed with “value” that it’s on par (or even more) with the cash it’s exchanged for. We’re getting into some deep economics waters here, and if you’re curious, you can venture into the vast realm of the Theory of Value.

But hey, let’s keep it real. I’m no economics guru, and I’m guessing you’re not looking for a lecture either! Instead, I’m talking from my eight years of experience running multiple startups. Trust me, I’ve seen the real magic happen when you focus on delivering value rather than just chasing numbers.

I would want to share a couple of concepts around the idea of Value, that has stayed in my head. They are also good signals for indicating that you’re generating Value or holding a key role in the Value chain:

“Shut Up and Take My Money” Moments

Have you seen the above meme before? Well, it’s more than just a meme; it’s a profound expression of value. Think about those products or services that are so darn valuable that when you’re pitching them, your potential customers are practically throwing money at you. It’s that moment when they can’t wait to get their hands on what you’re offering because they see its immense worth. I think this meme aptly represents the idea of Product-Market Fit as well.

The Double “Thank You” Moments

Something that I picked up from Amit Varma’s podcast — “The Seen and the Unseen”, originally stated by John Stossel. Imagine a transaction where both you and your customer walk away thanking each other. It’s not just about the product changing hands; it’s about mutual appreciation for the value exchanged. You’ve probably experienced this when you’ve finally found that perfect item you’ve been searching for, and the seller genuinely thanks you for choosing their store. “Win-win”, “poisitive-sum” are other ways of explaining the same phenomena.

So, dear entrepreneurs, let’s shift our focus from merely doing business to creating value that resonates. Remember, when your product or service is so incredibly valuable that people can’t help but thank you (and themselves) for it, that’s when you know you have made it as an entrepreneur.

Stay valuable, and stay awesome! 🚀

🙏Namaste Pythonistas! Let me ask you something — how do you usually maintain your requirements.txt file? If you’re like me, you must be using good ol’ pip freeze > requirements.txt command. Sure, it’s quick and easy, but have you ever found yourself looking at the file later and wondering what on earth cffi==1.15.1 refers to? Have you struggled to figure out if it was installed directly or as a dependency of another package? The ideal way would be to “hand write” it, but who has the time for it!?

Well, let me tell you about pip-chill. It’s like a smarter version of pip freeze. Instead of simply dumping all installed packages into the requirements file, it shows only the ones you directly installed.

To get started, just install pip-chill using pip itself:

pip install pip-chill

After that here is how I generate the requirements.txt file now:

pip-chill --no-chill -v > requirements.txt

• --no-chill : does not include pip-chill itself in the output
• -v : outputs the indirect dependencies as well, but commented. Therefore it is easy to keep track of them.

It generates the following for a basic Django project.

django==4.2.3
psycopg2-binary==2.9.6
requests==2.31.0
# asgiref==3.7.2 # Installed as dependency for django
# certifi==2023.5.7 # Installed as dependency for requests
# charset-normalizer==3.2.0 # Installed as dependency for requests
# idna==3.4 # Installed as dependency for requests
# sqlparse==0.4.4 # Installed as dependency for django
# typing-extensions==4.7.1 # Installed as dependency for asgiref
# urllib3==2.0.4 # Installed as dependency for requests
# urllib3==1.26.16 # Installed as dependency for botocore, requests

See how pretty it looks. It just works and solves my problem of properly maintaining requirements.txt. Now, I can focus on coding and building cool stuff, leaving the hassle of maintaining the requirements file to pip-chill.

So just, take a chill pill with pip-chill and chillax!

Happy coding 👋.

We have been building our consumer hardware startup for the past 4–5 years. Currently, it is in the “pre-production” phase where we are waiting for our first production batch to come from manufacturers. When we started it, we massively underestimated the amount of effort and time it will take to just develop the product.

Our previous venture before this was an app tech business which we managed to scale to more than 1 million registered users in a matter of 1 year. The app was launched within a month after the ideation. And all of this was done with 4 people team out of an apartment.

When we decided to do hardware, we thought the journey would be similar — will develop our MVP in a few months, launch it and then build a pipeline of products around the same problem space. We thought that the fundamentals of any startup are the same — building products that customers need. While the statement remains true, the “stuff” you are dealing with is completely different. Therefore the blueprint we had, had to go out of the window. We had to adjust, adapt and learn on the job while doing it.

While we were building it we had pondered multiple times about — How is it different from a regular software-tech startup? How is it similar? Why do people keep saying it is hard? What challenges, if solved would reduce the parity in building a hardware startup?

Now that we have reached so far with our startup, and thinking about these things — I have compiled a list of things that make building a Hardware startup harder.

1. Iterations are costly

Developing any product requires iterations. You need to iterate a lot, especially if you are building something that has never been built before. In general, the cost of iterating each time is many-fold higher than just doing software.

Take “build time” for example — it is the time taken to generate an “artefact” from an abstract design — it is usually PCB from its design, a plastic part from CAD or drawing file and binary from source code. While for the latter it is just a matter of a few minutes. For both the former once, it could take weeks if not months to get your designs in hand.

Not just cost, a short iteration cycle also reduces the risk associated with developing a new product by a huge factor. It makes reverting from a wrong product decision much easier.

2. Can’t fix things on the fly

With software, things can be fixed over the air, it works so well and is so cost-effective (practically no cost) that it is taken for granted and is not much appreciated by that industry.

While on the other hand, the best case for any hardware issue is fixing it on the field or calling back devices for rework, the worst could be building new units from scratch. All of this costs money, and it badly demotivates people working on it. They start doubting themselves, the product, etc — while such issues are just part and parcel of developing a product — no new product is without its issue.

Fulfilling product guarantees is what kills most companies that reach that stage.

3. Dealing with physical things

Physical products come with physical limitations or problems. Things like delivering products without damaging, stocking or producing them in the right batch size, keeping a dust-free environment, having the right toolsets, etc.

Imagine you negotiated a small paid pilot of 20–25 units of your product, and you started building them, while screwing you snap your screwdriver multiple times causing scratch marks on the plastic part, each time that happens you dissemble and assemble it back with a new part, finally, you manage to complete the batch and ship it. It reaches the customer, he opens the box and 3 devices are broken and 2 are malfunctioning — although you did thorough testing before shipping.

This has happened to us and many other startups who took these things lightly.

4. Scaling is not just replicating more

From 1 unit to even 100 — you will have to re-write your designs, and change processes, components and vendors multiple times. That is how it is. There is no auto-scaling or server-less equivalent here that will work at all scales.

Also, you cannot take shortcuts and skip any steps in scaling, doing that would be fatalistic. You have to build first build 1 working unit to be able to make 10 and from there to 100s. Until you reach a stable point of production, you will have to re-configure and re-tool your process each time.

5. Multi-disciplinary problems

Most hardware startups today can be summarised as “custom software written for custom electronics, wrapped in custom plastic”. They are almost always multi-disciplinary. Although such projects are a lot of fun to work on, it also comes with its problems.

To start with assembling such a team is hard. Then once you have it, the teams usually tussle with each other to mark their territories — to decide what feature or spec is whose responsibility. If not managed well this could become toxic and derail the project itself.

And good luck with debugging “multi-disciplinary issues”! It is not usually obvious where the issue lies. Reproducing the issue to debug on first place is harder.

6. Lacking ecosystem

Lastly, all of the above could be less painful if a support system for such a startup existed. No startup is 100% built on its own, it requires ecosystem players, partners and enablers to work for them, in the case of hardware you need it locally too. Along with that access to people who have “been there, done that” is also required. Such a community is non-existing in India — everyone is building in silos.

For example take China(particularly Schenzen), today if you want to build a small accessory product (a plugin or extension equivalent of software), you can build, test, succeed or fail fast in that ecosystem. That is why you see “funky gadgets” coming from there. It is a good sign. It indicates there is room for anyone to come up with any wild idea and at least deliver a minimal viable product for it.

Without such an ecosystem, people will take safe bets — build “me too” products for which the market already exists — because they know doing anything different would be like swimming against the tide.

Noticing a vacuum for an enabler in the ecosystem that can empower hardware startups and pave their way for success, we also started Makenica.com. With it, we want to revolutionise how innovative hardware ideas come to life.

I firmly believe that while hardware presents unique challenges, it is certainly within reach. It also requires its own playbook, it can’t be done in the same way as an app or SaaS product.

The Hard Things About Building a Hardware Startup was originally published in Startup Stash on Medium, where people are continuing the conversation by highlighting and responding to this story.

A trip down memory lane

The earlier site I mentioned was set up during my college days. Back in 2015, it was my final year and it was college placements season. Although I was never truly interested in taking a placement from college or a job in general, my parents wanted me to sit for it. “This can be your backup if you change your mind” — they said.

So, the first task of the “placements” is to make and submit your resume, but I went a step further and decided to set up a website. I always wanted to have a site, don’t know why! I couldn’t resist when I saw that the domain bhikadia.com (that's my surname) was available. I always wanted to give my non-Gmail email to people and here I had harsh@bhikadia.com and in my head, it was the coolest thing I owned. Long story short — I picked a template (went with Sustain) — rewrote my resume in Markdown— and published it.

I graduated and never took any job, continued with my startup. The website remained as it is. The email address I was excited about — I never checked the inbox of it. The only update to the website I did was to remove my address and phone number, when I got a call from a random guy seeking guidance on career, he mentioned he got my number from the site.

So apart from accidentally breaching my privacy, the website was useless to me and I never worked on it from then.

Back to present

Now, the bug of having-a-cool-website has bitten me again! This time I wanted to start clean, so I decided to just have the Medium profile. This time equally unclear about what to do with it — came the genesis of this very meta article.

So, Why am I doing it?

• Without sounding narcissistic, I realized I do like talking about my work and things I am interested in with people in my private circle. But why do I hesitate to do the same in public?
• I fear people’s judgment and what to fight that rakshasa within. I want to make this website — the weapon against it.
• For a year or so — I have been writing in my private journal and started to like it. I think some of the ideas and writings would interest people in general.

To conclude, I am still determining where this (writing blogs) will take me. This might just wither like has happened in past, or for a change I make it bloom this time around.

Before 2019, all my projects were traditionally deployed on VMs — managed manually by SSH-ing into them — replicated using images or snapshotting features provided by the cloud provider. While that was enough when the projects were small, and updates were not often, some tasks were difficult or impossible to do with that setup — e.g., upgrading OS, migrating across cloud providers, automatic scaling, etc. On the other hand, I hosted projects on PaaS platforms like Google App Engine and Heroku — which had an excellent one-command deployment process. But these platforms had many limitations in covering many use cases and were tightly coupled with cloud providers.

Meanwhile, I have followed developments around Containerization, Kubernetes, etc. I saw these technologies as providing PaaS like experience on IaaS platforms. But back then, it seemed very overwhelming — unable to conceptualize the complete picture and unsure of where to start, especially for my brown-field projects. I took a step-by-step approach towards it — figuring out what technology or tool to choose from at each step. It took about 2–3 months to migrate fully. This article is about that journey — it outlines a blueprint for anyone wanting to do the same. I have tried to keep it agnostic to any framework, stack, or cloud provider. To keep it concise, I have relied on links to other good detailed resources — encourage you to go through them.

0. Prepare your application

Before starting anything, you should prepare your application to be a good citizen of Cloud Native land. You need to familiarize yourself with “The Twelve-Factor App” — a methodology for how to write Software-as-a-Service applications and deploy them.

Most good backend frameworks follow some or all principles mentioned. If the framework you use does not cover some, you will find good plugins/packages/libraries to work around it. In most cases, you are just a google search away from knowing about it.

These are generally good to follow principles, irrespective of Cloud Native. So, your efforts will not go in vain, even if you don’t go all the way.

1. Containerize your application

The first step towards this journey is to containerize your application — package it with all its required dependencies — so that it can run as an isolated “container” independent of any environment or host OS. Containerization (to run “container”) is similar to Virtualization (to run “VM”). But the abstraction is at different levels — it does not include the entire OS — therefore, the size and start time are significantly smaller.

Docker is the de-facto containerization tool to build, run and remotely store images. The following are some excellent resources that will help you in it:

• Docker Official Getting Started Guide
• What is Docker Used For? A Docker Container Tutorial for Beginners
• The Docker Handbook
• How to Choose a Container Registry: The Top 9 Picks
• How to Reduce Docker Image Size: 6 Optimization Methods
• How to Secure Docker for Production Environment?

For any framework-specific guide, a simple google search should help you out.

2. Write scripts for each workflow step

You will need to write scripts, command line tools, aliases, etc., to streamline all or most developer processes — like running static analysis, building applications, tests, etc. These will also come in handy when you write CI/CD pipelines in the future. You could also integrate these scripts with the IDE your team uses.

You can start with converting your README guide to an executable script that could bootstrap the dev environment for you. An automated, streamlined and repeatable process is the key to having a pleasant onboarding for developers.

You could also use the docker-compose tool to spin up all services — your application with its dependent services like DB, Caching, etc. with one command. It allows you to write your multi-container stack as a YAML file. It can be used in production as well, for simple use cases.

3. Get comfortable with k8s

Kubernetes (aka k8s) is the de-facto platform to run containerized applications in the cloud at scale. It is impossible to cover all k8s concepts here, but I will skim through various key ones.

At its core, it is a controller and an orchestrator that tries to match the running state with the desired state. The desired state is provided in the form of YAML files. If you have gone through anything related to k8s, you might have come across them. It, in general, has the following format with required fields:

apiVersion: v1
kind: <resource kind>
metadata:
  name: <resource name>
  labels:
    <arbitrary key>: <arbitrary value>
spec:
  <spec based on resource kind>

The unit of compute in k8s is the Pod — a group of one or more tightly-coupled containers that share network and storage. Pods are rarely created on their own but are instead deployed using various workload resource types — based on the use case. These are some of the workload types:

• Deployment — to keep replicas of pods running and do rolling updates of them
• StatefulSet — manage “stateful” applications that require persistence
• DaemonSet — to your pods on each Node of the cluster
• Job — to run pods until it “completes”
• CronJob — to run pods periodically

Along with it, to manage configuration for the Pods, use ConfigMap and Secret resources.

These Pods are ephemeral. Therefore addressing them with IP to communicate with each other is impractical. Here, K8s’ Service resource comes to the rescue — it exposes (privately or publicly) a set of Pods as a single DNS name.

All K8s workloads are namespaced; this allows the isolation of resources within a cluster. You can deploy the same resources in different namespaces — e.g., to deploy staging and production in the same cluster.

As of now, try and stay away from running “stateful” applications inside the k8s — as it is a bit advanced. I suggest only going that route once you have been running a stable cluster for a while. Use managed services if offered or if you decide to self-host it — then do it in a standalone VM outside of k8s.

For development, to run k8s on your desktop, use minikube, MicroK8s, Docker For Desktop. Although they are good to start with, they are not full-fledged, and I would suggest using a cloud-based k8s cluster to get real experience.

Various cloud providers have managed k8s offering, here is a comparison guide. As Kubernetes is standardized, that makes it also agnostic to the provider. Therefore migrating or replicating clusters across these vendors is seamless (at least for stateless applications), making your application free of any vendor lock-ins.

4. Improve your k8s experience

Now that you have kubectl apply some YAML files to a cluster, it’s time to make this process less painful and less prone to typos!

I suggest using the following tools to manage a k8s cluster:

• k9s — It is an excellent command line GUI that makes k8s monitoring and managing a cakewalk. There is no need to write long commands or be precise with pod names. It is feature-rich, and I highly recommend switching to it.

• Helm — It is the de facto package manager for your k8s cluster. Essentially it is a template engine for k8s resources — simplifying deployments with no or less configuration. Along with that, it also manages the install-update-uninstall cycle of these resources. Use ArtifactHub, to find out various open source or community-managed Helm charts for services like — PostgreSQL, Redis, RabbitMQ, etc.

Some more tips:

• Use the kubectl port-forward (or k9s) to access admin/internal services without exposing it to the outer world.
• Write a Helm chart for your application. This way, your workloads are packaged and can be configured for different environments. It also makes installing staging and production with the same workloads but differing configs (like DB name, hostname, etc.). The chart generated by the helm create command is a good starting point.

5. Expose your service to the Internet

While you could directly expose your K8s service with the type LoadBalancer, the preferred way is via Ingress resource and an IngressController installed in the cluster. With it, you can use a single LoadBalancer (saving cost) for multiple services instead of using one for each service.

An Ingress resource defines the HTTP(S) route from outside the cluster to the service within. It describes various ingress rules based on parameters like domain, path prefix, TLS certificates, etc.

An Ingress resource does nothing on itself; your cluster requires an IngressContoller installed to make it work. In most cases, ingress-nginx will meet your requirements, but there are many options here. I found this comprehensive comparison [sheet] beneficial.

6. Git repo for your k8s cluster

Now that you have familiarized yourself with k8s and understand how it works, it’s time to “write” and version-control your k8s cluster that you will deploy in production. This practice is called Infrastructure-as-Code (IaC), allowing you to use other source code tools like Git, Pull Request, CI, etc., with your infrastructure. We need one or more tools to achieve this, to read the code and figure out what workloads should be spun up, updated, or torn down.

Since Helm releases are the unit of our deployment, I suggest using Helmfile as the primary tool for managing them. It allows you to “write” Helm releases, have a different config for each environment, and provides convenient commands like — helmfile apply and helmfile sync. Although not a full-fledged IaC tool, i.e., it can not increase or decrease a node in a cluster but is enough to take care of our k8s needs. For that, you can use Terraform, but I found it overkill for my requirements.

Since all our configurations will be in a Git repo, we must manage secrets properly. Helmfile does this for you along with helm-secrets plugin — it encrypts these secrets so that it is safe to put them in a repo.

The file structure of my repo looks like the following:

local/
  values.yaml
  secrets.yaml.dec
staging/
  values.yaml
  secrets.yaml
production/
  values.yaml
  secrets.yaml
values/
  my-app.yaml
  rabbitmq.yaml
  redis.yaml.gotmpl
  ...
helmfile.yaml
values.yaml
secrets.yaml
.sops.yaml

7. Install some must-have k8s utilities

Now, as most things are ready, some of the following must-have utilities will come in handy.

• prometheus-stack — It is the monitoring, alerting, and observability stack for your k8s cluster. It installs Prometheus and Grafana to the cluster and starts collecting vitals of all Pods right out of the box. Encourage you to learn more about it and use it to monitor your services.
• cert-manager — It will automatically manage the issuance and renewal of all your TLS certificates. It works with all issuing mechanisms or providers — self-signed, CA based, ACME server based (like Let’s Encrypt), and various thers.
• ExternalDNS — It will synchronize all K8s services and ingress with DNS providers. It works with various DNS providers.

8. CI/CD plumbing

Now let’s start triggering various builds and deployments by pushing them to our git repo. You must pick one of the CI/CD servers to run these jobs. I usually use GitLab CI as I host my git repo there.

One of the prerequisites of doing CI is to have a good automated test suite. You can execute these steps manually if you are not doing it or are not confident with it. However, I encourage you to start moving towards having a reliable automated test suite.

I configure my pipeline in the following way:

• Continuous Integration — This pipeline will execute tasks like static checks, unit tests, docker build, and push. Every commit push to any branch will trigger it.
• Continuous Deployment — This pipeline will execute all CI tasks, along with version bumps, push Helm chart, automatically deploy (with Helmfile) to staging and wait for manual approval for production deployment. Every tag push will trigger it.

These pipeline configurations are not universal. Feel free to be imaginative with it. Ultimately it is an automated process to streamline the integration of software and its deployment.

9. Continue improving

Congratulations! You should be able to deploy your production application and various dependent services in a Cloud Native way. While this may be enough, you have a long way to go when it comes to it.

I encourage you to go through various advanced concepts and patterns within k8s to take your game to the next level:

• Pod Probes
• Pod Lifecycle
• Horizontal Pod Autoscaling
• Pod Security
• Pod Affinity and Anti-affinity
• Sidecar container pattern
• Running stateful applications
• Operator pattern
• Custom Resource Definitions (CRDs)

Lets not waste time and get started with the code.

add dependecy:

dependencies:
   backdrop: ">=0.0.0 <0.1.0"

run get package:

$ flutter packages get

drop in BackdropScaffold widget in your code:

import 'package:backdrop/backdrop.dart';
import 'package:flutter/material.dart';

void main() => runApp(new MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return new MaterialApp(
      title: 'Backdrop Demo',
      home: BackdropScaffold(
        title: Text("Backdrop Example"),
        backpanel: Center(
          child: Text("Backpanel"),
        ),
        body: Center(
          child: Text("Body"),
        ),
      ),
    );
  }
}

this is how it should look:

I have writter a more detailed explaination about backdroppackage and BackdropScaffold widget in another article. Have a look if want to know more options.

Quickly implement backdrop with flutter