Before taking on the direct responsibility of implementing a Bug Bounty program from scratch at Unico IDtech, I frequently dealt with intrusive thoughts about the risks of failure. The risks were tangible: the possibility of poor implementation, inadequate prioritization of vulnerabilities by internal teams, the consequent disengagement of researchers, and as a result, the negative impact on the company’s reputation and the security professionals involved.

These concerns were not about the security of our products, but rather about the operational complexity that a Bug Bounty program demands. I had seen programs fail in other organizations, not due to lack of technical security, but because they underestimated the management, communication, and processes required.

But there was something different at Unico: a genuine concern for information security and leadership willing to invest adequately in the program.

Today, with the maturity gained throughout this journey, I can confidently say that with proper planning, the advantages far outweigh the challenges in any company minimally concerned with information security matters.

I say this because Bug Bounty elevates your company’s security to a level that no internal security team could achieve alone. It allows ethical researchers to explore vulnerabilities and breaches across our entire infrastructure and application ecosystem. This provides a holistic view of your environment, evaluated by highly qualified professionals from Bug Bounty platforms.

The Specific Challenge of Biometrics in Brazil

As a company focused on biometrics and digital identity, we face a particular challenge: it is extremely difficult to find researchers who specialize in this field. We’re not just talking about traditional web application vulnerabilities. We’re talking about sophisticated attack vectors focused on biometric fraud, spoofing, deepfakes, and liveness detection bypass.

Identifying potential attack vectors in biometric verification flows requires very specific knowledge that few researchers possess. And this in a country where fraud risk is alarming.

According to recent data, Brazil is among the 15 countries with the highest fraud risk in the world. When you understand that Unico is embedded in a large part of the Brazilian market, processing millions of biometric validations daily for banks, fintechs, telecommunications companies, and other critical sectors, the responsibility becomes even greater.

The goal of our Bug Bounty is precisely to combat this fraud scenario, bringing the world’s best security researchers to challenge our protection systems.

Strategic Choices: Building the Foundations

We made the strategic decision to start with a private program. We wanted to have control over who was testing our systems, calibrate our rewards appropriately based on market benchmarks, and most importantly, structure robust internal processes before scaling to a public program.

This approach allowed us to validate and optimize the entire program operation: triage workflows, response SLAs, integration with development teams, and severity criteria. Throughout this journey, we gradually expanded the scopes, moving from specific endpoints to wildcard scenarios (e.g., *.unico.io), reaching the robust scope we have today.

Internally, we did our homework. We conducted several disclosure sessions with our Security Champions program and with stakeholders of critical products. This internal awareness was fundamental to ensure that identified vulnerabilities would receive proper attention and prioritization.

Innovation Through Collaboration

One of the differentiators of our approach was recognizing that Liveness Detection is too complex to be tested without adequate context. Our technology has multiple layers of protection, anti-debugging mechanisms, and manipulation detection that, while effective against attackers, can also be barriers for legitimate researchers trying to assess security.

Understanding this challenge, we decided to promote collaborative sessions where researchers could:

• Share insights from their initial SDK analyses
• Discuss technical challenges encountered with anti-debugging measures
• Better understand the security architecture and protection objectives
• Identify which resources and contexts would be most useful for effective testing

This transparency created something rare: mutual trust. Researchers were not adversaries trying to break our systems. They were partners helping us build more secure products.

It was through these sessions that we matured what should be delivered to researchers. We understood that we needed to provide not just documentation, but real testing environments and integrated mechanisms that simulated production conditions.

With this learning, we created focused campaigns specifically for the Liveness scope, offering amplified rewards (reaching up to US$ 20,000 for critical vulnerabilities). More importantly, we provided researchers with integrated testing mechanisms on the same platforms our clients use in production: Web, Android, and Flutter.

The result of these campaigns validated our security posture: no critical vulnerability was reported. This demonstrated that the implemented protection layers were being effective against sophisticated bypass attempts carried out by experienced and well-equipped researchers.

Building Operational Excellence: Numbers and Learnings

Since the launch in April 2024, our private program has achieved metrics that reflect not only volume, but mainly engagement quality and operational efficiency:

• ~380 researchers invited and active in the program
• ~100 reports analyzed, 37 validated as security issues
• +US$ 31,000 in rewards paid to the security community

But numbers alone don’t tell the whole story. What really matters is what we built throughout this process.

We learned to calibrate expectations, both internal and from researchers. We learned that transparency and clear communication make all the difference. We learned that it’s not just any reported vulnerability that generates engagement, but those well-documented, evaluated, and clearly explained about the exploitation risk. A complete proof of concept that demonstrates the real impact of a specific vulnerability has convincing power that no presentation slides can achieve with development teams.

And most importantly, we learned that Bug Bounty is not about paying for vulnerabilities. It’s about building a sustainable partnership with the security community.

The Next Chapter: Opening the Doors

After structuring robust processes, validating our operation, and building a solid foundation of relationship with the community, the time has come to expand the program’s reach.

As of January 13, 2026, Unico IDtech’s Bug Bounty program will be public.

We want to reach a larger number of people interested in testing our Liveness. We want researchers and security enthusiasts from around the world to be able to identify weaknesses and try to bypass Unico’s liveness detection mechanisms. We want to further scale our security through the collective wisdom of the global community.

Our public scope will include:

• Complete wildcards: *.unico.io and *.acesso.io
• Complete By Unico collection + Credentials
• Special focus on Liveness Detection, our core technology

The private program fulfilled its purpose: it allowed us to build the necessary foundations to operate a public program in a sustainable and professional manner.

If you are a security researcher, bug hunter, or just starting in this field, I invite you to check out our program when it goes public on January 13:

hackerone.com/unico_idtech

About Unico

Unico is the world’s largest identity verification network and a trust link in the digital society. With solutions based on facial biometrics, machine learning, and reinforced security layers, Unico validates with 100% certainty who is carrying out a transaction and the identity risks involved. In this way, it combats fraud, protects data, and promotes trust between people, companies, and governments, collaborating to build a safer and less bureaucratic world. Globally renowned funds such as SoftBank, General Atlantic, and Goldman Sachs trust and invest in Unico.

Bug Bounty: Unico IDtech’s Journey So Far was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

When 90% of software organizations are rushing to adopt AI coding assistants, there’s one critical question most aren’t asking: What are we amplifying?

The 2025 DORA report, which surveyed nearly 5,000 technology professionals worldwide, found something surprising and concerning. While AI adoption has exploded across the software industry, the results are decidedly mixed. Individual developer effectiveness is up, code quality has improved, and throughput has increased. But instability has also risen. Friction remains unchanged. Developer burnout persists at the same troubling levels.

The report’s most important insight cuts through the hype: “AI’s primary role in software development is that of an amplifier. It magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones.”

At Unico, Brazil’s leading digital identity platform, we proved this thesis by doing something counterintuitive: we spent a long time building our platform engineering foundation before we scaled AI adoption. This is the story of why that sequence mattered, and what happened when we finally gave our developers AI tools.

What We Were Amplifying

When I joined Unico in October 2022 as Senior Tech Lead, I found a company managing over twenty microservices with high-reliability requirements but operating without platform consistency. Our technology landscape was what I came to call a “fruit salad”: ten different technology stacks across teams, multiple databases requiring duplicate solutions, thousands of dollars wasted on redundant cloud services. Teams made isolated decisions with no cross-communication. Conway’s Law was in full effect: disconnected teams had created disconnected architectures.

The math was brutal. If you have N dependencies and M technology choices, your total maintenance cost grows as N × M². We were spending 61% of our developer effort on maintenance rather than innovation, drowning in the complexity of our own making.

The risk was clear: if we had adopted AI in that state, we would have amplified the chaos. AI would have helped us write inconsistent code faster, generate incompatible architectures more quickly, and create technical debt more quickly. We needed to build the foundation first.

Building the Foundation: Dev Prime

Over the next twelve months, we built what we called Dev Prime, Unico’s platform engineering practice. It established our approved technology stack, transformed our Architecture Review process into a genuine quality gate, and introduced what we call the “Classics” philosophy: instead of endless rewrite proposals, we focused on making existing systems more reliable through SLOs, modernized libraries, feature flags, and better deployment processes.

I won’t rehash the details of how we built Dev Prime; we’ve documented that journey here. What matters for understanding our AI current position is what Dev Prime gave us: organizational alignment through a Tech Lead Council, engineering consistency through approved standards, and a culture shift from “rewrite everything” to “improve iteratively.”

But Dev Prime alone wasn’t enough. The game-changer came from how we measured success.

The User Journey Revolution

In mid-2024, we made a fundamental shift in how we thought about reliability. We stopped asking “Is the server up?” and started asking “Can users complete their critical journeys?”

We implemented over 200 individual Service Level Indicators across 23 services, all tracked via Infrastructure as Code with Terraform. But here’s what made it transformative: every single SLI was mapped to an actual user goal, not an infrastructure metric (see more details here).

We didn’t measure “API response time” — we measured “User can complete payment transaction in a few seconds or less.” We didn’t track “Service X is available” — we tracked “Users can validate biometrics with a certain success rate.” We organized everything around actual user journeys.

Our payment platform alone has over twenty-five SLIs. Our identity verification service has 10+ user journeys. Each journey has multiple SLIs that track availability and latency at different levels of complexity.

We also implemented a sophisticated four-tier alerting system designed around user impact. A fast burn — six times the normal error rate — triggered an immediate incident because customers were already experiencing problems. A slow burn that sustained twice the error rate for 20 minutes escalated to a complete incident response. But here’s the innovation: when we consumed 70% of our error budget, we didn’t wait for disaster. We brought together SREs, engineers, and product managers to create a recovery plan before users were affected.

For example, a 98.5% SLO gives us about eleven hours of error budget per twenty-eight days. Consuming 70% of that budget — about 7.5 hours — became our trigger for proactive intervention. We shifted from reactive firefighting to predictive maintenance.

This user-journey focus became critical to our AI success. It gave us immediate feedback on whether AI-generated code was helping or hurting the things that actually mattered. Every release was validated against real user success metrics within minutes. There is no “hope and pray” deployment cycle at Unico.

The Seven Capabilities

By mid-2025, before we scaled AI adoption, we had built all seven organizational capabilities identified by DORA as critical for AI success.

We had a clear AI stance, with documented policies on approved tools and when to use AI rather than human judgment. We maintained healthy data ecosystems with clean, versioned ML training data and production data pipelines with quality controls at ingestion. Our internal documentation in Confluence was comprehensive and AI-accessible, using standard templates for Architecture Review documents. Our monorepo was well-structured with clear README files.

Dev Prime required strong version control with frequent, atomic commits. We mandated feature flags for risky changes and built rollback capabilities into our deployment pipeline. Our deployment metrics showed the discipline.

Those two hundred SLIs tracking user journeys demonstrated our user-centric focus. We measured what mattered to users, not what was easy to measure. And Dev Prime itself was our quality internal platform.

We had built the foundation. Now we could amplify it.

The AI Implementation

In July 2025, we rolled out AI coding assistants across the organization. We implemented AI Reviewer using Gemini for automated code reviews, with Dev Prime rules codified in the repo, and adopted Claude Code for development.

We also made a crucial decision about measurement. Our SRE manager leading the project, Luiz Casali, put it perfectly: “From the beginning, we decided not to go down the path of monitoring usage, but rather to look at how much more we’re delivering.”

We could have tracked how many lines of code each developer wrote, which AI tool each person preferred, how many AI suggestions were accepted, or whether commits were co-authored with Claude. But all of that would have created a surveillance culture and focused attention on the wrong things. It would have optimized for outputs — lines of code written — rather than outcomes — business value delivered.

Instead, we asked a straightforward question: Are we shipping more value?

The Results: A Thirty-One Percent Shift

The numbers told a clear story. In June 2025, our baseline month before AI, we merged 488 pull requests across 64 releases. That worked out to 7.6 PRs per release. Our work distribution showed 39% features, 39% maintenance chores, 20% bug fixes, and about 2.4% refactoring.

That distribution revealed the problem: developers were spending 61% of their effort on non-feature work. We had a stable platform with good deployment practices, but we weren’t optimized for innovation velocity. More than half of every developer’s time went to keeping the lights on rather than building new value.

By August 2025, after AI had reached stable adoption, the picture had fundamentally transformed. We merged 538 pull requests — a sustained 10.2% growth — across 67 releases. We maintained our cadence of one release every 2.5 days, with 8.0 PRs per release, showing we hadn’t sacrificed our small-batch discipline.

But the work distribution had shifted dramatically: 51% features, 30% maintenance, and refactoring still at 2.4%, proving we maintained our architectural discipline even as we moved faster.

Before AI, we spent 39% of our effort on features and 61% on other work. After AI, we spent 51% on features and 49% on other work. That’s a 31% relative increase in feature work. With the same team size, we gained roughly 20% more innovation capacity. AI hadn’t just made developers faster — it had freed them from toil so they could focus on strategy and creativity.

The 10% volume increase was nice, but the 31% shift was transformative. It meant that approximately one-third more of our engineering capacity was now dedicated to building new value for users rather than maintaining existing systems.

And here’s the critical difference from what DORA found in most organizations: our stability didn’t degrade. In fact, it improved. We maintained our small batch deployments at 8.0 PRs per release. We kept our release frequency high at every 2.5 days. Our SLO compliance remained strong across all two hundred user journey indicators. Our incident rates didn’t increase—if anything, they decreased slightly because our comprehensive quality gates validated AI-generated code before it reached production.

Why We Were Different

Most organizations in the DORA study saw AI increase throughput but also increase instability. We saw the opposite. The reason comes down to the foundation we built and how it interacted with AI.

Because we already had all seven DORA capabilities in place, the system had built-in checks and balances. When AI-generated code ran faster, our Architecture Review process caught design issues. When AI suggested changes, our automated testing validated them. When AI created more pull requests, our two hundred SLIs immediately caught any quality degradation. There was no hope for the best. The system revealed problems within minutes of deployment.

By codifying Dev Prime into Gemini’s style guide, we created a self-reinforcing quality system. AI checked AI. The automated code review ran in 5 minutes instead of days, catching style violations, inconsistent patterns, and deviations from our approved tech stack before any human reviewed the code. This meant human reviewers could focus on the aspects that actually required human judgment: architectural soundness, design elegance, and strategic alignment with business goals.

Because we measured business value rather than vanity metrics, we kept ourselves honest. We didn’t care if someone wrote 10,000 lines of code if those lines didn’t ship features users wanted. We didn’t track tool adoption rates because that would have created internal competition between AI tools, rather than focusing on outcomes. We cared about one thing: the percentage of our work that delivered business value. That metric told us everything we needed to know.

Because we had real user-journey visibility through two hundred SLIs, we had immediate feedback on what mattered. If AI-generated code introduced a subtle bug that degraded payment processing success rates by a small margin, our alerts caught it within 10 minutes. If a code change slowed down biometric verification, our user journey SLOs flagged it before users complained. The feedback loop was immediate and tied to actual user experience rather than abstract technical metrics.

This combination created something powerful: AI could move fast because the guardrails were automatic and comprehensive. Developers felt confident accepting AI suggestions because they knew the system would catch problems. The fear of “what if AI breaks something critical” evaporated because we had built a platform that made breaking things difficult and detecting breaks immediate.

A Call for More Research

These are Unico’s initial results after just a few months of AI adoption at scale. They’re promising, and they align with what DORA’s research predicted. But they’re also observations from a single company in a specific context: a Brazilian digital identity platform managing twenty-plus microservices with high reliability requirements.

One company’s success doesn’t constitute proof. It’s a data point, not a dataset.

The software engineering community needs more rigorous research on AI coding tools. We need clear, standardized metrics that go beyond vanity measurements like lines of code written or suggestion acceptance rates. We need longitudinal studies that track not just immediate productivity gains but long-term impacts on code quality, system stability, technical debt accumulation, and developer well-being.

We need to understand which organizational capabilities truly matter for AI success and which are merely correlated with it. We need to know whether the 31% shift toward feature work we observed is replicable across different company sizes, different domains, and different engineering cultures. We need to understand the failure modes: what happens when companies adopt AI without the proper foundation, and how bad does it get?

Most importantly, we need research that measures what actually matters to businesses and users, not what’s easy to instrument. Developer velocity is meaningless if it comes at the cost of system reliability. Feature delivery is hollow if those features don’t solve real problems. Productivity gains are pyrrhic if they lead to burnout three months later.

If you’re leading an engineering organization and experimenting with AI coding tools, I encourage you to measure carefully, document thoroughly, and share your findings. If you’re a researcher studying software engineering productivity, this is a critical moment to establish rigorous methodologies before the industry commits to approaches that may or may not actually work.

The AI coding revolution is happening whether we’re ready or not. The question is whether we’ll navigate it with data and discipline, or with hype and hope.

This document was written with the help of Claude

AI as an Amplifier: Build Your Platform Before Scaling AI was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

In an increasingly digital world susceptible to cyber threats, Unico recognized the necessity to address the security of its solutions with seriousness and proactivity. Integrating security practices into software development has become a priority to ensure the protection of our customers’ data and the reliability of our products. However, facing this challenge often resulted in difficulties in measuring and quantifying the effectiveness of preventive actions, such as implementing the “shift left” concept. So, we decided to adopt the OWASP Software Assurance Maturity Model (SAMM). This framework provides us with a structured basis to assess the maturity of our security practices and develop concrete actions to promote them. Based on the assessment provided by SAMM, we implemented individualized actions focused on each product, ensuring that security measures were adapted to the specific needs of each solution. This customized approach allowed us to address vulnerabilities in a targeted way, improving security on various fronts. With SAMM, we seek not only to increase security in our products but also to cultivate a culture of collaboration between application security and engineering teams, transforming security into a strategic ally in innovation.

What is OWASP SAMM?

OWASP SAMM is a maturity model that allows companies to assess and improve their software security practices. It is structured in five essential areas: Governance, Design, Implementation, Verification, and Operation, each one focused on different aspects of the Software Development Life Cycle (SDLC). In addition, SAMM organizes the evolution of security into three maturity levels, allowing organizations to advance progressively and structurally.

The Five Areas of SAMM

1. ​Governance​: Establishes security policies and ensures compliance with regulations.
2. ​Design​: Integrates security practices from the initial phases of development.
3. ​Implementation​: Applies techniques and tools to test software security.
4. ​Verification​: Implements processes to monitor and respond to security events.
5. ​Operation​: Ensures the team is aware of security best practices.

Why Implement SAMM?

Until early 2024, the only way we had to assess the security of our products was the number of vulnerabilities. With the aim of being more proactive, we decided to scale our security efforts and use OWASP SAMM as a guide. With this, we seek to improve our practices and ensure that each product undergoes an individualized assessment.

Expected Benefits

Implementing SAMM brought several benefits to Unico:

• ​Performance Indicators​: Provided valuable insights into areas that needed specific improvements.
• ​Compliance​: Ensured that our practices were aligned with relevant legislation.
• ​Continuous Improvement​: Conducted periodic assessments to monitor our progress and adjust our security strategies.

Results Achieved

Before the implementation of OWASP SAMM, our average product security maturity was still in an initial stage, which reflected the urgent need for improvements. We conducted a benchmarking, based on data provided by OWASP SAMM, which revealed that the average maturity of the companies analyzed was at a higher level than ours. Based on this data, we set challenging goals and developed a concrete action plan in the areas outlined by the framework, aiming to raise our security level and align it with the best market practices.

The actions included:

• ​Governance​: Adjusted processes and disclosed security policies.
• ​Design​: Established secure development standards.
• ​Implementation​: Ensured security in the pipelines.
• ​Verification​: Conducted architecture analysis and exploitation tests.
• ​Operation​: Integrated incident management with development teams.

The good news? We exceeded our expectations and reached a maturity level that puts us very close to the second stage of OWASP SAMM, which is divided into three levels. While the first level represents initial and ad-hoc practices, the second stage is characterized by more consistent and repeatable processes. With our recent result, we are practically at level 2, demonstrating that our security maturity is above the benchmark provided by OWASP. This reflects the engagement and commitment of our security and engineering teams, who have advanced significantly in the assessment of our security practices and reaffirm our commitment to continuously raising the protection of our products.

Conclusion

Implementing OWASP SAMM was a significant step in Unico’s journey towards a robust and efficient security culture. The results obtained reflect our commitment to ensuring that each product we develop is aligned with the best market practices. We will continue to improve, always seeking maximum protection and the trust of our customers. Security here is not just a priority; it is the essence of our work.

Originally published at https://fernando-silva.medium.com on April 11, 2025.

Implementing OWASP SAMM at Unico: Towards More Secure Products was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

Original author: Leandro Ikeda Santos and Pedro Henrique Spagiari.

Until recently, Unico, like many tech companies, tracked the health of its services with systemic metrics: CPU, memory, disk, network, etc. Efficient? Yes. Sufficient? Not so much. The gap between technical monitoring and user experience became a growing problem. And the answer, a radical shift, came in the form of SLOs (Service Level Objectives).

“We have the CPU and memory graphs, but we didn’t know if the user could log in,” explains an engineer at Unico. The problem was clear: the monitoring did not reflect the real user experience. Incidents occurred, impacts were felt, but the response was often reactive, based on infrastructure alerts that arrived too late. This model, common in many companies, had a cost: the user’s perception of service quality was left in the background.

Noticing this gap, Unico decided it was time for a radical change. It was necessary to have a way to measure and ensure service quality from the perspective of who really matters: the user. The answer came in the form of Service Level Objectives (SLOs).

The strategy was to stop looking only at the servers and start mapping and measuring the critical user journeys — the flows that deliver real value. From there, we defined SLOs, i.e., goals, for the latency and availability of these journeys, creating an alert system focused on what the user really felt.

The implementation journey, its challenges and achievements

Adopting SLOs, however, was much more than a simple technical change; it was the beginning of a deep and complex cultural and procedural transformation. We knew this would require careful orchestration, involving not only SREs, but also software engineers and product teams, breaking down silos and aligning everyone around the common goal of improving resilience and user experience. This required us to overcome several challenges:

1. Promoting a new culture
   The first major challenge was cultural. The introduction of SLOs was much more than adopting new metrics; it implied a profound cultural change throughout the organization. The concept, although clear to SREs, was not intuitively obvious to everyone. We encountered natural resistance to change, teams accustomed to traditional monitoring and, sometimes, initial disinterest due to the abstractness of the concept. It was crucial to translate the importance of SLOs into an accessible language, demonstrating their direct value to the business and the customer. Promoting essential collaboration between SRE, Development, and Product, overcoming different priorities and ways of working, was fundamental.
2. Mapping what really matters: the critical user journeys (CUJ)
   The next step was to identify and map the critical journeys (CUJs). This was not a simple exercise of drawing flowcharts, but a deep dive to understand what the user wanted to accomplish and where their pain points were. We assemble cross-functional teams (SRE, SWE, PMs) to capture the complexity of these journeys. The challenges encountered were related to the inherent complexity of the flows themselves, the need to align different perspectives on what was “critical,” and ensuring the scalability of the process. We overcame this by detailing the journeys with diagrams, creating runbooks, standardizing documents and establishing periodic reviews.
3. Translating experiences into numbers: defining SLIs and SLOs
   Once the journeys have been mapped out, the next step was to measure their success. We defined Service Level Indicators (SLIs) focused on the experience (login success rate, transaction response time, etc.), instead of just infrastructure metrics. We created SLIs at different levels of granularity to allow for precise diagnostics and an overview of service health. From the SLIs, we established the Service Level Objectives (SLOs), the goals that represented our promise of reliability for each journey.
4. Creating a new incident flow
   The implementation of SLOs was not just a matter of adopting new metrics. It required completely rethinking the incident management flow, with nuances that go beyond merely adapting existing processes. This involved:
   Intelligent prioritization: Creating a system that classified severity based on the real impact on valid user requests.
   Effective escalation: Defining clear levels and automating escalation (using tools like OpsGenie) to ensure a fast and appropriate response to severity, potentially reaching the CTO in minutes for critical cases. Ensuring the responsibility and preparedness of teams to respond at any time was crucial.
   Multichannel communication: Structuring internal communication (via Slack) and external communication (via status page), with integrations to optimize the flow.
   Automation: Integrating tools (New Relic, OpsGenie, Jira, Slack) to automate ticket creation, notifications, and information sharing, streamlining the entire process.
   Holistic view: Connecting information about request for change (RFC), dependencies between services, and the progress of remediation tasks to the context of incidents, consolidating everything in dashboards for a complete view of service health.
   Continuous training: Training all teams on their roles, responsibilities, the new tools and processes. Monitoring and adjusting the flow constantly based on feedback and performance analysis.
5. Learning and continuously improving: the postmortem
   We transformed postmortems into learning tools. We adopted a blameless culture, focused on understanding the system and not blaming individuals. Standardizing the root cause document, using techniques like the “5 Whys” for in-depth analysis, and ensuring that each postmortem generated clear, traceable and time-bound actions, sharing the learnings widely.
6. Balancing innovation and stability: SRE and the Error Budget rite
   We implemented the concept of Error Budget, the “budget” of errors that each service could consume within a period. We defined automatic triggers and real-time monitoring. When the budget was exhausted, freezing policies were triggered, interrupting non-essential deployments to prioritize stability. This required a new rite and adjustment to the cadence of the teams. Fundamentally, this gave SREs the necessary empowerment and authority, based on data, to make decisions that impacted development (such as pausing releases), always with the goal of protecting the user experience. Overcoming initial apprehension and building the necessary trust between SRE, Development, and Product was essential for this model to work.

Impact without beating around the bush: a leap in availability

The transformation yielded concrete results. The global SLO of Unico’s products, which was 95.57% in January 2024, jumped to 99.92% in December of the same year, remaining above 99.90% in 2025.

“We identified more incidents because we monitored more things, but the response time dropped drastically” says the engineer. Although the greater granularity of monitoring led us to identify more incidents, the mean time to repair (MTTR) decreased drastically. We solved problems much faster, minimizing the impact on the user.

Lessons for the future

Our journey is not unique. The need to monitor services with a focus on user experience is a growing trend. We learned that clear communication during incidents and post-incident analysis are crucial, and that the Error Budget is a powerful tool to balance innovation with resilience. The implementation of SLOs was much more than a technical project. It was a process that transformed the way the company views reliability, prioritizing the user experience and fostering a culture of collaboration and continuous learning.

Unico continues to evolve its incident management, seeking to improve its processes and ensure the best possible experience for its customers. The SLO shift was a turning point, a reminder that, in the end, what really defines our success is the experience we deliver to our users.

Originally published at https://medium.com on April 28, 2025.

From Systemic Monitoring to Excellence in SLOs was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

This document is part of Unico’s Dev Prime series.

Nowadays, it’s very common to have an extensive system divided into multiple parts, whether microservices, modular monoliths, Cloud Functions, or even a combination of these.

Let’s imagine, for example, a typical e-commerce:

The customer places an order, and from there, the e-commerce backend needs to perform two operations: save the order in the orders table and notify the billing system to process the payment via credit card. Let’s assume that the notification involves sending a message to a message broker like PubSub, RabbitMQ, Amazon SNS, etc.

This is what we call dual writing: you need to simultaneously modify information in two (or more) distinct systems.

The three most commonly used ways to implement dual writing, considering our example, are:

Send a message to the payment process system
Start a transaction in the database
  Update the Orders table
Commit the transaction in the database

Start a transaction in the database
  Update the Orders table
  Send a message to the payment process system
Commit the transaction in the database

Start a transaction in the database
  Update the Orders table
Commit the transaction in the database
Send a message to the billing system4

The problem arises when data is updated in one system but not correctly updated in the other:

• In the first case, we might successfully send the message, but there’s no guarantee that the order will be saved in the database.

• In the second case, the situation is similar: during the commit, an error might occur, but by that point, the message has already been sent to the broker.

• In the last case, we might successfully save the order in the database, but an error could occur when sending the notification to the billing system.

In all three situations, inconsistencies in the process may arise. Inconsistency issues are among the hardest to detect and recover from because they occur infrequently and are not easily monitored. However, depending on a company’s scale, they can have drastic consequences, such as data loss and all the resulting repercussions (e.g., financial loss, loss of trust, etc.).

Some patterns like 2PC (Two-Phase Commit) are used to mitigate such situations.

2PC is Good But Not Always the Best Solution

Two-phase commit (2PC) is a classic pattern for distributed transactions. It ensures that a transaction is carried out in two stages:

1. Pre-commit: A coordinator informs all participants that a transaction is about to occur and waits for all of them to respond affirmatively.

2. Commit: The coordinator informs the participants that the transaction should be finalized. If any participant reports an error, the coordinator instructs all others to roll back the transaction.

This is undoubtedly a relatively safe pattern. However, it can present some challenges:

a) Not all systems support it.

In the example above, we would need to develop a custom code to act as the participant responsible for recording in the database. Adaptation can be costly. As highlighted in this link, it’s a relatively complex protocol.

b) Coordination loss can lead to indefinite delays.

If there’s a loss of coordination between the main node and the others, the system must assume that the transaction was not committed. However, the coordination may end up “waiting” indefinitely for confirmation.

While 2PC offers strong consistency guarantees, these drawbacks can make it impractical or overly expensive in specific scenarios.

Transactional Outbox to the Rescue

The Outbox Pattern is an approach to ensure that business logic is executed reliably, even in failure scenarios. It provides a way to ensure that business events are recorded in a reliable database and that their processing occurs asynchronously.

Databases have evolved over the years to enhance reliability through mechanisms like replicas, backups, and other redundancies, effectively minimizing data loss. Even if the transport mechanism (in our case, Debezium or GCP Pub/Sub) fails, the original data will always be stored somewhere.

Unlike 2PC, implementing the Outbox Pattern is much simpler. Additionally, adapting existing systems is relatively straightforward since it primarily requires creating a consumer for the chosen messaging system in the nodes that need to replicate the information.

Continuing with our example, we no longer publish the event directly to the messaging system alongside the insertion into the orders table. Instead, we save the event in an “outbox” table within the same database transaction that saves the order. Being part of the same transaction ensures consistency during replication—either everything is saved or nothing is saved.

Once the information is saved in the outbox table, it can be replicated to other systems using various mechanisms.

The Outbox Table

The Outbox Pattern doesn’t detail how the outbox table should be designed. There are several options for its format and the number of tables needed. However, at Unico, the recommended approach is as follows:

A Single Outbox Table

If the total number of events generated in a short time frame is vast or if routing events from a single table proves difficult, the ideal approach might be to have one Outbox table per event type.

However, at Unico, we currently do not face this scenario. Our solution becomes more straightforward if we maintain a single table that captures all events. This doesn’t mean we’ll have one table for the entire company. While having one table per system/product is advisable, other granularities can also be applied. Just avoid creating a rule that requires one table per event.

What Should Be Stored in the Outbox Table?

The system’s architecture and consumer needs will largely dictate this choice.

• In a system adhering to DDD principles, storing the state of an Aggregate or Entity might make sense.

• In other systems, simply replicating the data from the source table could be the better option.

The most critical point when implementing the Outbox Pattern is the risk of creating tight coupling between the event sender and receiver. Such coupling is not always desirable. Changes in the source could break the consumers. To address this, we use Protocol Buffers (protobuf) to enable schema evolution without breaking contracts.

A third approach to what is inserted into the Outbox table is sending just a signal that a specific record was created or updated in the source table. In this case, only the record ID is sent, and the system receiving the event from the Outbox table must query the source table to retrieve the required information.

This approach introduces a race condition: if a second change occurs after the Outbox table entry is created but before the destination system queries the source table, only the latest “snapshot” of the information is captured. In other words, the details of the first event could be lost.

As a rule of thumb, never notify that a record was updated.

Suggested Structure for the Outbox Table

The most commonly recommended structure for the Outbox table includes the following fields:

• id: A unique identifier for the event. While GUIDs are not typically recommended as primary keys, we won’t create relationships with other tables or perform frequent searches on them in this case. Additionally, PostgreSQL’s UUID type stores this information efficiently.

• aggregatetype: The type of the Aggregate, Entity, Event, etc. Debezium uses this field to determine the topic to which the event will be routed.

• aggregateid: The ID of the source data. This can be a transactionID, a process number, or something similar.

• payload: The information being emitted as the event.

• created_at: A field storing the record’s insertion timestamp. Although this field is not propagated, it is helpful for periodic cleanup of the Outbox table.

This structure provides a solid foundation for implementing the Outbox Pattern while maintaining simplicity and flexibility for future scalability.

PostgreSQL Table Creation Command

create table OutboxEvents(
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregatetype character varying(100),
    aggregateid character varying(50),
    payload jsonb,
    created_at timestamp WITH TIME ZONE DEFAULT NOW()
);

Although Debezium allows for a free-form JSON structure in the payload field, the recommendation is to use the spec here.md format. This market standard ensures the inclusion of some essential fields in every event.

Attributes to Be Included in the Payload

• specversion: The specification version. Fixed value: “1.0”.
• id: A unique identifier for the emitted event. The event must be re-emitted with the same ID if an error occurs during sending. Event subscribers should be prepared to handle duplicate events. A UUID (RFC 4122) is recommended for this field.
• time: A timestamp indicating when the event occurred. Follow the RFC 3339 standard. The timezone must always be UTC, so no offset specification is required.
  Example: 2022–04–12T23:20:50.52Z
• type: The description/name of the event. This is one of the most important attributes as it identifies the topic in PubSub.
  The event name should include a noun + past tense verb and begin with the product/squad name.
  Examples: ecommerce.order.received
• source: The originator of the event. The CloudEvents specification allows for flexibility in naming the originator. Follow the same convention as the event name: squad/product + service name.
• subject (optional): While this attribute is optional in the specification, it is recommended for associating all events of an operation.
• datacontenttype: This must always be “application/protobuf” at Unico.
• dataschema: The URI for the payload schema. It must be a Protobuf schema.
• data: Contains the event’s data.

Message sending and receiving must be ensured using Protobuf. However, since PostgreSQL’s JSONB fields do not natively support Protobuf, the recommended approach is to serialize the Protobuf data into JSON before storing it in the database.

Here’s the base proto file that should serve as a template for all events:

syntax = "proto3";
import "google/protobuf/timestamp.proto";
import "google/protobuf/any.proto";  
message OutboxEvent {
  string specversion = 1;  
  string type = 2;  
  string source = 3;  
  string subject = 4;  
  string id = 5;  
  google.protobuf.Timestamp time = 6;  
  string datacontenttype = 7;  
  string dataschema = 8;  
  google.protobuf.Any data = 9;
}

Example

To illustrate, let’s consider an event named someevent. This event contains three data fields: transactionId, doc, and image.

The proto file for this event would be:

syntax = "proto3";
message SomeEvent {
  string transactionId = 1;  
  string doc  = 2;  
  float image_id = 3;
}

Considering the base proto file above, an event with the “SomeEvent” proto would be serialized as follows:

{
  "specversion": "1.0",
  "type": "someevent",
  "source": "integration",
  "subject": "1ec07712-79b7-485a-a0e2-0a1c33fd1016",
  "time": "2020-04-30T04:00:00Z",
  "datacontenttype": "application/json",
  "dataschema": "http://<schemapath>",
  "data": {
    "transactionId": "1ec07712-79b7-485a-a0e2-0a1c33fd1016",
    "doc": "123.123.123-00",
    "image_id": "ea02254f-28f4-4b31-99a5-957bb024f78d"
  }
}

Outbox Pattern was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

This document is part of Unico’s Dev Prime series.

tl;dr;

Whenever we encounter a challenge not directly tied to Unico’s core business, we consistently turn to an external dependency, whether a library or a cloud service.

Background

Maintenance costs are crucial yet often underestimated in the software development journey. Every production code has a maintenance price tag that usually overshadows the initial coding expenses. Some research suggests that maintenance can consume as much as 70% of total costs, while others claim it might soar to 90%. In real-world scenarios, these costs often lead to a greater need for engineering staff to keep systems operational, pulling focus away from driving the company’s core business forward. In the worst cases, this can result in a system that becomes too daunting to modify. The choice to rewrite this “legacy” code is often made in error, compounding the financial burden.

At Unico, we strive to sidestep this costly situation. This document aims to outline a strategy for integrating external dependencies. A good starting point is to ask: “Is the system I’m building aligned with Unico’s mission?” If the answer is no, there’s a strong chance that leveraging an external dependency would be more economical. For example, we wouldn’t dream of creating a database management system (DBMS) from scratch; instead, we’d tailor PostgreSQL on GoogleSQL to meet our requirements. If this logic holds for a DBMS, why wouldn’t it apply to any other code that doesn’t serve our core business goals?

Don’t code, buy

The key is to consistently rely on external dependencies for Horizontal Problems. While not directly linked to Unico’s core business, these challenges are crucial for running our infrastructure smoothly.

These dependencies can come in the form of cloud services or open-source libraries. Here are a few examples:

• We use PostgreSQL hosted on Google Cloud for our database management system (DBMS).
• For continuous delivery (CD) tools, Argo CD is our go-to.
• LaunchDarkly serves our feature flag needs.
• For documentation, we turn to Confluence.
• When new challenges arise, like scanning files in Cloud Storage, we seek a Software as a Service (SaaS) solution that meets our needs.
• This also involves setting up environment variables and using regular expression libraries.

The truth is that many Cross Problems have already been tackled elsewhere, and trying to solve them internally can waste valuable time and resources. It’s unrealistic to think we can navigate complex issues with little investment. Tapping into the expertise of companies specializing in these solutions is often the more economical choice.

While the examples may seem obvious, Unico has plenty of gray areas. Take our messaging system, for instance; it handles both email and SMS and manages redundancy across various platforms to reduce the risk of extended outages. It’s more cost-effective to choose a service with a robust Service Level Agreement (SLA) and hold our vendor accountable than to create our messaging redundancy. This logic applies to messaging and deployment pipelines, file scanning tools, and database access management.

Another way to look at it is that each system could quickly become an independent startup. So, do we want to build a new code infrastructure startup or focus on strengthening our core identity provider business?

How to choose an external service

1. Understand the Problem’s Criticality: The first step is to assess how critical the problem is for Unico’s business. We need to determine whether we will offer the service as a product or if it will serve solely as an internal tool. If it’s the latter, there’s a good chance that a solution already exists.
2. Prefer Native Solutions: We prefer native solutions from Google Cloud. In the latter half of 2022, we established Google Cloud as our primary provider. If a native solution exists that addresses our specific problem, we will choose it.
3. Leverage Internal Resources: Before selecting a new external tool, it’s crucial to investigate whether there is already a tribe within Unico that has tackled a similar issue. Since Unico comprises several mergers and acquisitions, challenges often recur across tribes. We should utilize existing internal solutions whenever possible.
4. Careful Documentation Review: When considering a new dependency, it’s essential to read the documentation thoroughly and evaluate whether it genuinely resolves the issue. While there may be a tendency to rush into creating a proof of concept (POC), we must be mindful of the risks of adding a dependency without a comprehensive understanding.
5. Choose Popular Tools: Always opt for the most popular tool rather than the cheapest. Popular tools typically have more robust support and investment in availability, as their teams are experienced in managing high-traffic volumes.
6. Document and Review: Finally, document the solution and undergo a review process. This review aims to understand trade-offs, identify risks, and unify Unico’s development processes. Remember: any new microservice or external dependency introduction must go through an Architecture Review.
7. Move to Implementation: We can proceed to the actual implementation with a thorough understanding and documentation.

Hypothetical Problems

In many discussions about introducing dependencies, the question often arises: “Are we creating this service to solve a problem that we believe will arise in the future, and the external dependency will not solve that?” The M&A argument is quite common at Unico: “We are an M&A company, and we need to protect ourselves for future acquisitions.” As human beings, we tend to worry about problems that may happen. However, this concern often serves only as a defense mechanism. Many factors can change before a situation becomes relevant, making these concerns most often merely hypothetical.

The concept of “You Are Not Gonna Need It” (YANGI) is the best approach to adopt here. We should focus on solving our current problems, and only when a hypothetical problem materializes should we seek a solution. When we start to worry excessively about what might happen, we risk entering a state where everything seems to have exaggerated importance, which can hinder our progress.

It is worth remembering that YANGI also involves a trade-off. Avoid “quick win” solutions that meet immediate needs but may result in significant rework in the future. When planning, it is crucial to consider how the software can evolve.

Another hypothetical problem arises when we believe these dependencies do not solve the problems exactly as we expect. However, considering maintenance costs, adapting to an external solution (including adjustments in the problem definition) is often more viable than developing new infrastructure to cover that tiny gap.

Risks

Although the benefits mentioned earlier are significant, adding a dependency on an external library or a cloud service brings some risks. This article, written by Russ Cox, clearly addresses the dangers of introducing a dependency without due caution. It is essential to remember that adding a dependency is not intrinsically evil; having the necessary attention matters.

We need to create a mental model for integrating a new dependency. Davi Reis wrote an article that describes a model: Dependency Management, where he summarizes the idea in the formula below:

The formula suggests that the benefits of adding a dependency grow linearly (f(x)). At the same time, the costs increase exponentially (g(x)) with the number of touch points it generates in the codebase. The touch points can be seen as connections with the code or the number of people in the organization who need to familiarize themselves with this dependency. A clear example of this formula is introducing a new programming language: the benefits of using an existing language instead of creating a new one are evident. However, when we introduce several programming languages in an organization without discussing the benefits, the costs grow exponentially. Now you understand why your CTO doesn’t let you code in your favorite language. :)

This same logic applies to frameworks, configuration libraries, and regular expression parsers. Therefore, it is vital to deeply understand what other teams do when introducing a new dependency. Never decide in isolation within your tribe. The most efficient solution is to maintain a single dependency for each type of problem. This article discusses a similar concept, calling it “embracing the most boring technology.” The formula mentioned that the perspective presented also connects with Russ Cox’s article. He recommends that this be our approach, as the simple is always the best.

The formula and perspective mentioned also connect with Russ Cox’s article. He recommends that we should isolate dependencies. When we have a dependency in a restricted scope (for example, a PDF parser in Unico Sign), the exponential cost decreases because we limit the touch points to the scope of a single application.

References:

• Unico Dev Prime
• Our Software Dependency Problem
• Dependency Management
• Dan McKinley :: Choose Boring Technology

Dependencies — when you should write code was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2023, Unico’s the React JS design system component library was created with Material UI, named Genoma UI. This article describes the motivations and learnings from this initiative.

What is Genoma?

Genoma (Genome) is the complete set of genetic material (DNA or RNA) of an organism, containing all the genes necessary for its development, functioning, and reproduction. The name “Genoma” is an interesting choice for a design system because, just as the genome contains all the information necessary for the development and functioning of an organism, the design system brings together all the components, guidelines, and standards that ensure the consistency and visual cohesion of a digital product. It is the “DNA” of design, providing the foundation for creating interfaces, just as the genome guides the formation and functioning of a living being.

What is a Design System?

A design system is a set of guidelines, reusable components, and visual and behavioral standards that promote the consistent creation of user interfaces (UI) across different products or platforms. It encompasses:

• Component library: buttons, icons, forms, tables, etc.

• Style guide: rules for colors, typography, spacing, and how to use these elements cohesively.
• UX/UI standards: best practices for navigation, visual feedback, and consistent interactions.
• Documentation: guidelines on how to use each component and apply design principles.

The goal is to ensure consistency and efficiency in product development, allowing teams to work collaboratively with a common foundation.

What is Material UI?

Material UI (MUI) is an open-source React component library that implements Google’s Material Design. MUI provides a collection of pre-styled and customizable components, making it easier to create modern and responsive interfaces in React.

Before Genoma UI

Genoma has existed since 2020, as a set of style guides, best practices, and documentation provided by the design team to guide developers in maintaining visual consistency in web applications (React and Angular). We did not have a design system component library.

React applications represent about 90% of web applications, so the solution was prioritized for this platform. Components were implemented in a decentralized manner and without standardization. For example, in styling, different approaches were used: SCSS, Styled Components, or CSS Modules. If a team had two applications, the same button would likely be implemented in different ways, evolutions did not happen simultaneously, which compromised visual consistency. The main difficulties included:

• Need to recreate components;
• Changes required adjustments in all applications;
• High development costs when creating and adjusting components;

There have already been several changes in visual components at Unico, and the development cost was high, given that adjustments had to happen in multiple repositories and applications.

The team I was part of needed to work on a new application, I suggested and led the creation of the reusable component library, since that we would again have to create components in a decentralized way, it was an opportunity to change the scenario.

In discussions with Bruno Fonseca, design and product team, we raised alternatives for creating the library:

• Code a component library from scratch using CSS/JSX.
• Use a market component library.

We opted for an experiment with Material UI, seeking agility in component coding, in addition to having extensive documentation, predefined themes and accessibility support. Bruno Fonseca was already familiar with MUI and brought this alternative.

Design System Library

We conducted the experiment with the component library in a new application and validated the screens with the design and product team.

After the experiment:

• Design Doc: A technical document was prepared proposing the availability of the component library with MUI as a npm package, accessible for other React applications.
• Approval and Development: The document was approved, and the Genoma UI library was developed.
• Restriction of Customizations: To ensure agility, we limited the creation of new components to those existing in MUI, avoiding excessive customizations. To meet what was designed, when we have the need for a new component that combines several others, the combination must be made in the application, so as not to create components that do not exist in the MUI and lose control.
• Collaboration: The development of the library is collaborative, without a responsible team. The team that needs a component is responsible for adding it to the library.
• Guardians: Some people were named “guardians” to review the code, ensure quality, and clarify doubts. We have people from engineering and design in this role.

These are the steps to contribute:

• Figma: Is a tool for interface prototyping. The Material UI for Figma consists of representations of the React components from the library in Figma so that designers and developers can communicate and iterate more efficiently.
• GitHub: Version control and collaboration on the code of the components.
• Storybook: Tool used by engineers for documentation and testing of UI components.

We created Genoma UI to centralize the theme and MUI dependency. The theme defines the design properties of the application: color palette, typography, spacing, etc. When we make changes, all applications that use Genoma UI benefit.

The library is already in use in several applications. There was no effort to replace old components, new applications are already born using the component library, and existing ones use it as needed for adjustments or the creation of new components. In some cases, it was necessary to correct unit tests that validated the HTML, once the HTML is changed.

Conclusion

There was a need to align expectations, especially with engineering, reinforcing that everyone is responsible for the evolution and success of the project. In addition to the challenge of stabilizing the library version.

We are in a much better position. There are +5 applications using Genoma UI, one of them using it fully. +40 components available. We have greater visual consistency, increased component reuse, reduced development costs, and a more collaborative process.

If you find yourself in the same scenario, create a design system. I’m not saying to make all components available and migrate in all applications, create the design system and build components when you need to use or adjust them; in the long run, it will certainly have its benefits.

I would like to thank everyone who contributed in some way to the success of this initiative ❤️.

If you want to know more about Unico’s technologies, visit: Approved Technologies at Unico and Unico Dev Prime.

Genoma, Unico Design System was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

Finalizamos no último dia 22 de agosto o Programa Security Champions 2024 da Unico ID Tech, e a palavra que resume a jornada que passou é SUCESSO! A empresa formou 23 pessoas que se tornaram disseminadores de segurança da informação dentro dos times de desenvolvimento e ainda por cima conseguiu conquistar excelentes indicadores de mitigação de risco em seus produtos, trazendo para seus clientes soluções ainda mais seguras.

O DevOps promete um ciclo de vida de desenvolvimento de software que seja colaborativo e eficiente. Mas, na maioria das vezes, cobrindo os aspectos técnicos do problema, com muitas ferramentas e entre elas algumas que podem ser usadas para reforçar a segurança.

Em geral, as metodologias de desenvolvimento ágil criam um modelo desequilibrado nas organizações: geralmente 1 ou 2 especialistas em segurança para aproximadamente 100 desenvolvedores. Neste cenário, o poder de fogo da equipe de segurança é insignificante comparado ao ritmo de implantação.

Foi no intuito de equilibrar essa balança que aplicamos o programa Security Champions, um programa que ajuda as empresas a escalarem seus processos e conhecimentos de segurança, diminuindo os silos, aumentando a conscientização, além de apoiar as empresas na busca por sistemas e produtos mais confiáveis. Isso é muito a cara da Unico, não é mesmo?

O programa de Security Champion exerce um importante papel na integração de práticas de desenvolvimento nas operações diárias das empresas, ajudando a moldar um ambiente mais seguro e consciente. A ideia deste programa de conscientização começou a ganhar destaque na última década, mais especificamente em 2010 como parte de um movimento mais amplo, o DevSecOps.

Afinal, o que são Security Champions?

O termo “Security Champions” é bastante intuitivo. “Champions” (campeões) sugere indivíduos que defendem ou lideram uma causa dentro de um grupo maior — neste caso, a causa é a segurança de software dentro das equipes de desenvolvimento.

Grandes organizações de tecnologia como Google, Microsoft e Facebook possuem programas robustos de Security Champions. Estas empresas utilizam o programa para garantir que as práticas de segurança sejam uma parte integrante do desenvolvimento de seus produtos, muitas vezes em uma escala global.

Além das gigantes de tecnologia, muitas instituições financeiras, da área da saúde e de serviços também adotaram o conceito de Security Champions para proteger suas operações contra crescentes ameaças cibernéticas. Aqui no Brasil Itaú, Banco BV, C6, PagSeguro, SiDi e agora a Unico ID Tech são exemplos de companhias que possuem programas Security Champions reconhecidos.

Security champions são membros da equipe de desenvolvimento de aplicações, quality assurance, SRE, operações ou lideranças técnicas que possuem atração pela temática de segurança e entendimento dos aspectos técnicos e de negócio dentro da organização. Esses indivíduos recebem treinamento específico em desenvolvimento seguro e atuam como disseminadores do mindset de segurança nas squads de desenvolvimento, podendo responder questões sobre risco em aplicações, recomendar as melhores práticas de segurança, replicar treinamentos e atuar como uma ponte com os especialistas em segurança.

A busca por mudança de paradigmas

Do ponto de vista do desenvolvimento de produtos, a segurança da informação foi estereotipada com o passar dos anos como “o gargalo” ou “o departamento do NÃO!”. Essa visão aumentou com o crescente número de lançamentos de produtos, features e deploys diários de software que geram carga adicional de trabalho às equipes de segurança que redobraram sua atenção devido a processos adicionais de revisão de segurança.

Durante meus primeiros meses na Unico, adotei uma abordagem de cautela e observação, para aprender o máximo possível sobre a empresa na qual eu havia ingressado recentemente. Naquele momento, o modelo Security by Design estava longe de ser adotado e como explicado no parágrafo anterior havia um medo por parte das equipes de desenvolvimento em levar temas para o conhecimento do time de Application Security (AppSec), feedbacks ao formato de trabalho de AppSec, e seu aprofundamento nas atividades do dia a dia dos produtos. Com o passar do tempo percebi que esse cenário culminou na dificuldade que tínhamos do lado de cá (Application Security) em se aproximar dos Engenheiros de Software. Nesse mar de desafios e incertezas como seria possível reverter essa imagem?

Escrevi este artigo com objetivo para compartilhar o case de sucesso que a Unico IDTech teve por meio de um programa Security Champions estruturado, eficaz e de fácil medição, que possibilitou reduzir a fricção entre o time de segurança de aplicações e as squads de desenvolvimento e escalar a temática de desenvolvimento seguro em seus produtos.

O Security Champions traz um série de benefícios para as organizações, entre eles:

• Estabelecer uma rede de compartilhamento regular entre o Champion e os AppSecs.
• Construir uma biblioteca de conhecimento e vídeos para acelerar o processo de onboarding de novos devs, ou o trabalho de conscientização do AppSec.
• Manter os desenvolvedores atualizados por meio de distribuição de newsletters com tópicos atualizados da comunidade, últimas notícias, vídeos, artigos, podcasts, dicas e recursos compartilhados em canais de comunicação (slack, teams, whatsapp, email, etc,).
• Identificar e remediar vulnerabilidades no início do ciclo de desenvolvimento, o que diminui a probabilidade de falhas de segurança em produção.
• Gerar parceria com a squad de desenvolvimento e alcançar resultados positivos na disciplina de AppSec.
• Obter recursos de segurança especializados sem investimento adicional.
• Descentralizar a segurança distribuindo o conhecimento e as responsabilidades de segurança para toda a empresa.
• Criar a visibilidade de uma instituição que leva segurança de aplicações realmente a sério.

Percebemos que uma cultura de segurança sólida reduz significativamente os débitos técnicos e o risco de incidentes de segurança, o que pode economizar milhões em custos com esforço de pessoal em correções ou de recuperação e reconstrução em caso de incidentes.

E agora, quem poderá nos ajudar?

Parece bordão de seriado mexicano, não é?

Trouxe para Unico minha experiência de condução do Security Champions em outras casas, elaborei um backlog de ações principais e me ofereci para conduzir o programa.

Destaco o quão difícil foi argumentar sobre esta iniciativa com as lideranças, pois tínhamos 2 fortes pontos contra:

1. O projeto estava parado há um bom tempo sem ter alguém que assumisse papel de líder.
2. O Security Champion precisa ter uma porcentagem de tempo reservada de sua posição de desenvolvedor para gastar no cumprimento desta nova função.

Para minha surpresa, apresentei o plano para a liderança de SI que de imediato comprou a ideia, reservou budget e deu autonomia para seguir. Apresentamos o projeto também para lideranças de engenharia, que também compraram a ideia e 1 diretor sênior se disponibilizou para bater um papo com os participantes sobre os desafios que enfrentamos na segurança das aplicações e as expectativas que a empresa tem na formação de desenvolvedores com esse skill.

Aqui já podemos falar que demos o primeiro passo para o sucesso do programa. Sem o compromisso e a adesão da liderança o programa Security Champions seria apenas uma boa ideia no papel.

Promoção do programa

Após time e liderança de segurança entenderem claramente a definição do sucesso do programa, foi levado às equipes-alvo a existência da iniciativa para começar a colher os benefícios dele.

Como parte do processo de recrutamento dos participantes do programa, apresentamos em uma agenda com aproximadamente 100 Seres de tecnologia as diretrizes do Security Champions da Unico, bem como os temas dos treinamentos, premiação e regulamento. Destacamos pontos positivos da iniciativa aos participantes, tais como:

• Impulsionador de carreira
• Visibilidade e reconhecimento
• Aprendizado contínuo
• Networking
• Oportunidade de liderança e mentoria

No final da agenda abrimos oficialmente as inscrições e para surpresa do time de Segurança, 36 Seres se disponibilizaram a participar do programa e agregar ainda mais habilidades nas suas carreiras, a partir dali tínhamos um desafio para selecionar quem realmente deveria participar, havíamos planejado um programa para 25 participantes, definitivamente nosso objetivo sempre foi buscar indicadores qualitativos e não quantitativos.

Enfim, tínhamos nosso time de Champions selecionados, foi aí que passamos para as estratégias de engajamento, para tanto alguns ações específicas foram importante para alcançar este objetivo:

• Criar e compartilhar fundos de tela para reuniões em toda a empresa
• Criar máscara para foto de perfil e fundo de página para LinkedIn
• Participar como parte das reuniões contínuas de revisão de segurança e arquitetura
• Fornecemos mimos para a aula inaugural dos selecionados, distribuímos para os participantes camisetas, canecas, webcam cover e café gourmet, então, nada mais normal que iniciarmos os treinamento com uma talk denominada: “Café com desenvolvimento seguro”, não é mesmo?

Qual o perfil de um Security Champion?

Como verdadeiros campeões os escolhidos possuem habilidades multidisciplinares. Da fácil compreensão de práticas de codificação ao domínio de estruturas de segurança, estas habilidades permitem ao Ser ter uma visão holística do ciclo de vida do desenvolvimento de software e senso crítico acerca dos aspectos de segurança da empresa.

Outras skills também identificadas nos Champions escolhidos:

• Ajuda outros membros da equipe enquanto investe em si mesmo.
• Cria uma onda de entusiasmo que aumenta as práticas de segurança.
• Possui influência dentro de suas equipes.
• Consegue disseminar novas formas de trabalhar na equipe.
• Identifica risco de segurança para o negócio e traz possíveis soluções.
• Fornece feedback à equipe do projeto e ao AppSec.
• Reduz a pressão sobre a equipe de segurança por meio de um envolvimento ativo e contínuo.

Seções de treinamento

Foram ao todo 7 sessões de treinamentos com temas diversificados, mas que se conectam diretamente com o objetivo da área de Engenharia de Segurança da companhia em promover o ShiftLeft e Security by Design. Os tópicos abordados durante o treinamento foram: SSDLC e ferramentas para o ShiftLeft, Arquitetura Segura, Modelagem de ameaças em privacidade, Análise de vulnerabilidades em código, Segurança em APIs, Segurança em componentes de terceiro e Metodologias para evitar bypass em mecanismos de segurança mobile.

Trouxemos para aula inaugural palestrante de renome para falar sobre sua vivência com desenvolvimento seguro e os desafios encontrados por organizações globais em implementar uma cultura de segurança em código, chamamos o treinamento de “Café com Desenvolvimento Seguro”. Marcamos este evento para o início do dia e enviamos para cada Champions um kit com caneca e café gourmet para tornar a apresentação mais intimista e confortável para o público.

Os demais eventos foram conduzidos pela área de segurança e privacidade da Unico.

Atividades executadas pelos Champions

A formação dos Security Champions na Unico não se baseou apenas na bateria de treinamentos em desenvolvimento seguro que foram realizados, o time de AppSec se aproximou dos participantes e pôde delegar atividades conforme observaram a evolução do Ser em determinado assunto. Durante os 6 meses de treinamentos notou-se que os Champions as seguintes tarefas:

• Treinamentos internos para repasse de conteúdo
• Colaboração em integração, triagem e correção de SAST (Static Application Security Testing).
• Colaboração na integração, triagem e correção de SCA (Análise de Composição de Software).
• Redução do backlog de riscos de segurança em geral
• Atividades para medir a maturidade de desenvolvimento seguro com OWASP SAMM
• Modelagem de Ameaças
• Atuação em Pentest e correção de findings encontrados

Utilizamos uma ferramenta para monitoramento de tarefas específico para o programa, nela os Champions e AppSec podiam acompanhar projetos e garantir o gerenciamento das atividades.

Métricas de eficácia do Security Champions

Capacitação e engajamento foram as chaves do sucesso: Um desafio significativo em qualquer tipo de iniciativa de cultura é garantir que os participantes sejam adequadamente treinados, mantendo-os motivados e engajados, especialmente em organizações onde a segurança pode ser vista como um obstáculo ao desenvolvimento ágil.

Aqui na Unico conseguimos vencer esse desafio, aproximando o time de AppSec do Champion. Criamos agendas de repasse de conhecimento entre eles e determinamos que esses encontros, bem como os alinhamentos dados para corrigir vulnerabilidade nesta agenda geram pontuação para o ranking final do evento, afinal, quem não queria ganhar R$ de bônus em seu cartão iFood para gastar como quiser? Isso mesmo, outro ponto de motivação para o programa foram os prêmios oferecidos aos 3 primeiros Security Champions que mais apoiassem em correção de vulnerabilidades. Shazam!!! Eis que temos métricas de engajamento e ShiftLeft. Em geral, os AppSecs tiveram agendas quinzenais para repasses e compartilhamento de conhecimento sobre assuntos específicos do produto.

Anteriormente citamos o uso de ferramenta para monitoramento das ações desenvolvidas pelos champions, este recurso permitiu a Unico gerenciar todas atividades relacionadas a segurança que foram produzidas pelos participantes e classificá-las por natureza. A partir daí conseguimos consolidar num Dashboard as métricas que foi apresentado regularmente para os participantes e patrocinadores da iniciativa, conforme imagem abaixo:

Encerramento e Reconhecimento

Lembra das atividades executadas e registradas pelo Champions na ferramenta que citamos? Pois é, a partir dela criamos um ranking onde apresentamos a atualização no início de cada treinamento quem era o Ser mais engajado. Isso estimulou uma disputa saudável ponto-a-ponto até o final da iniciativa. Após uma jornada de 6 meses reconhecemos os Seres que contribuíram para disseminação da cultura de desenvolvimento seguro e mitigação de riscos cibernéticos.

Chegou a hora de celebrar!!! 🙌

Reunimos os participantes do programa presencialmente na sede da Unico para assistirem uma aula ao vivo e premiarmos aqueles que mais se destacaram nessa corrida. Participaram também deste encontro a Diretora de Segurança da informação da Unico e todo staff da área.

E como ninguém é de ferro, no final do dia confraternizamos em um agradável happy hour.

Conclusão

Em face de ameaças cibernéticas persistentes e sofisticadas, o papel dos Security Champions se tornou indispensável. Eles são nossos heróis que trabalham incansavelmente para fortalecer as defesas da companhia. Adotar o papel de um campeão de segurança não é apenas uma escolha de carreira; é um compromisso de construir um futuro digital seguro para sua empresa e para a área de atuação como um todo. Então, você está pronto para se tornar o defensor de segurança que sua empresa precisa?

Unico Security Champions was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Unico, we rely on external dependencies for tasks outside our core business focus. We do acknowledge the risks associated with incorporating new dependencies. Here, we detail the technologies that are approved for implementation at Unico. It’s crucial to keep in mind that any introduction of new technologies in particular areas (or their inclusion in this documentation) needs a thorough discussion during the Architecture Review.

Classic Systems

At Unico, we do not use the term legacy system; instead, we use the term Classical Systems. Classical Systems are those that were built before the definition of the rules described here. Many of them may use a technology different from the one listed in this document.

Never rewrite a Classical System to be compatible with the technologies described here.

This also applies to systems acquired through M&As. A rewrite is not needed to be compatible with Unico’s tech stack. Software rewriting involves a significant cost, so we prefer a conservative approach and maintain redundant solutions in smaller scopes.

A reflection on rewrites can be found at this link.

Cloud

Unico’s official cloud solution is Google Cloud. For new applications, always use Google Cloud. We understand that there are scopes of AWS applications (Auto and MakroSystem). In these cases, a migration is not necessary.

Kubernetes

At Unico, Kubernetes serves as the primary tool for managing services. Our setup includes GKE in conjunction with the Enterprise Cluster (GKEE), a solution offered by the SRE team to centralize clusters, guaranteeing enhanced control and standardized functionalities. For services running in production, it is mandatory to allocate their workloads to the Enterprise Cluster (GKEE).

Regarding applications utilizing AWS (Auto), we opt for EKS. However, the Golden Path we establish on Google Cloud does not achieve full compatibility.

Programming Languages

Unico supports three approved programming languages: Golang, C# (.Net), and Python (further details outlined below). It is important to highlight that .Net is exclusively utilized as a Linux application operating within Kubernetes containers. In the realm of web applications, Unico relies on ReactJS.

Python is commonly used for specific purposes such as data analysis, data science, and data engineering. Nevertheless, it is not recommended to deploy new services using Python code that directly impacts user pathways.

We opt not to utilize Typescript for web applications. While we recognize that Javascript is encompassed within Typescript, signifying that all Javascript code is compatible with Typescript, we harbor reservations regarding the cognitive load and code legibility associated with switching between the two languages.

It is understood that various applications are developed using diverse programming languages. Unless a compelling business rationale exists, there is no imperative to redevelop these applications. Nonetheless, the introduction of a new service in a programming language distinct from those delineated in this segment necessitates validation in the Architect Review.

GitHub Repository

Avoid creating separate repositories for new applications; instead, they should be housed in the respective mono repo:

1. Go
2. .Net
3. React

An exception has been made within the Unico organization regarding the creation of new repositories. This decision stems from the necessity for enhanced control. Currently, the company hosts over 1000 repositories, marking a 100% increase in the past year.

In instances where creating a new service is not warranted, explore options to store required items in an existing repository. This may include infrastructure files, decouplings, and other relevant components.

Database

Utilize PostgreSQL on Google Cloud for transactional tasks while ensuring your application accounts for eventual consistency. Direct queries towards non-writing read replicas.

Currently, our applications are deployed on Spanner, MongoDB, SQLServer, and MySQL. However, we are moving away from utilizing these databases for new transactional processes.

In particular, with respect to SQLServer, our. NET-based applications are integrated with this database. Nevertheless, since PostgreSQL is now supported, opting for SQLServer would incur unnecessary expenses and should not be considered.

At present, ElasticSearch is employed by Auto, People, and Sign for search functionalities. Moving forward, ElasticSearch will be exclusively reserved for the existent use case. Any other potential use cases must first be approved solely for current scenarios and discussed during the Architect Review for further consideration.

We have a rule that every productive environment must have high availability flags and at least one available replica (N+1).

Asynchronous Communication Between Services

Utilize the Outbox pattern for asynchronous communication among services. The Outbox Pattern stands out as the most straightforward method to guarantee atomicity between database writes and event publication. When operating within Google Cloud, opt for PubSub (Kafka is not utilized in new applications). However, refrain from directly sending data to PubSub for inter-service communication; instead, persist it in the database initially. For the implementation of the Outbox Pattern, leverage Debezium.

About asynchronous patterns

It’s important to consider the implications before implementing SAGA Patterns, CQRS, or any other queue orchestration pattern.

Many individuals expect an asynchronous system to boost application performance, but this isn’t always the outcome.

When it comes to CRUD operations, choosing CQRS can sometimes be counterproductive. It may delay processing requirements and increase the complexity of overarching concerns such as error management, user engagement, and application observability without a significant reduction in CPU cycles for task processing.

This concept also applies to applications with limited direct user interaction. System observability is improved when communications are synchronous, allowing errors to be quickly identified when a task begins.

Queues can be useful in specific situations: for tasks that aren’t critical to the application’s main path or when meeting the expected SLO is challenging.

Nevertheless, exercising caution is crucial in both scenarios mentioned above. When we have full control over the Unico stack, it’s more advantageous to thoroughly understand the system’s bottlenecks instead of relying on queuing mechanisms to cover up sluggishness.

Synchronous Communication Between Services

Use Google GRPC. If necessary, perform GRPC to Rest transcoding at the Istio layer for Rest communication.

Feature flags

We utilize Launchdarkly as the designated Feature Flags tool at Unico. It is mandatory for all new services to implement Feature Flags to ensure that service behavior remains consistent during deployments.

Within the Go mono repo, we have an existing library tailored for Launchdarkly integration, leveraging Uber FX. Uber FX is a Dependency Injection solution employed by Unico.

Data Cache

Utilize Redis solely for temporary data storage purposes, avoiding its use as a permanent storage solution (i.e., refraining from treating it as a database). If your service heavily relies on Redis, treating it as a primary dependency, it is crucial to implement the High Availability option to ensure continuous operation.

App Monitoring

Use NewRelic to monitor your applications effectively. Ensure that you have configured at least the golden signals and scheduled analysis in your system’s deployment routine.

For error tracking, opt for Sentry. It is crucial not to conceal errors or exceptions originating from your application.

Application Logs

Application logs are directed to Google’s CloudLogging for Google-based applications. Conversely, for applications hosted on AWS, CloudWatch is utilized.

In both scenarios, logging to stdout is standard practice, while synchronous libraries for log transmission are strictly avoided. Adhering to the principles outlined in the The Twelve-Factor App is crucial.

It is emphasized to refrain from using a Database for error logging within your application.

Release / Deployment

The software development lifecycle (SDLC) at unico is divided into 2 main tools for workflow and pipeline control:

• Github Actions
• Spinnaker

Github Actions

We utilize Github Actions as a comprehensive workflow solution for Build, Test, and Publish operations. Presently, there are two approaches to leverage this tool:

• Monorepo: Employ the designated workflow provided by the repository associated with the monorepo's programming language.
• Multirepo: Opt for Reusable Workflows as foundational components when constructing your workflow. This approach delegates the responsibility of maintaining and updating these steps to the SRE team.

Spinnaker

We exclusively rely on Spinnaker as the deployment pipeline at Unico. The utilization of Spinnaker is entirely dependent on Official Templates developed by the SRE team to uphold uniformity and consistency across services.

No other tools apart from the ones specified are utilized. The utilization of Google Cloud Build has been halted, and services are not deployed using Github Actions.

Design System

Teams can streamline their workflow by utilizing Genoma (Unico's Design System) as a React JS + Material UI (MUI) library for web applications, eliminating the need to create visual elements in separate repositories.

Key Benefits:

• Enhanced component reusability
• Ensured visual consistency across various solutions
• Lowered development expenses
• Encouraged collaboration among team members

Given that all components are MUI-based, incorporating new elements for shared purposes is a seamless process. While everyone is encouraged to contribute, it is essential to seek validation from the Design team before introducing new additions.

It's important to note that this library exclusively caters to React applications and is not compatible with Angular, .NET Framework, or similar platforms.

User Behavior Analysis

We utilize Google Analytics 4 (GA4) and Google Tag Manager (GTM) for behavior analysis primarily due to cost efficiency, given that both tools are available for free.

Human access to the database

All human access to the database must be conducted indirectly through hoop.dev. In case of database debugging or bug fixes, please reach out to the SREs for hoop.dev configuration assistance. Regular access to Unico's databases is currently deactivated.

Approved Technologies at Unico was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Dev Prime is a collective documentation initiative aimed at unifying Unico’s software engineering culture and addressing challenges many companies in Brazil face. Through Dev Prime, we established a shared foundation for best software engineering practices and consistency, fostering alignment across teams and preventing fragmented decisions. This effort led to simple but practical changes, such as forming a Tech Leads group to improve communication, stopping calling system billion dollar systems as legacy but modernizing them with a “classic” concept, and enhancing the architectural design review process so everyone in the organization can contribute, all bringing to a more cohesive and maintainable development process. From time to time, we will publish some raw texts of what we have built inside, hoping that they help others or even invite other engineers to know that there is an excellent place to build identity products. Enjoy the “consistency journey”!

The (not) 90-day rules

When I joined Unico in October 2022, I was entrusted with “standardizing the development process.” The company had an established process called Architecture Review, where senior engineers would review architectural designs and provide feedback based on a structured design document. However, we were unhappy with the process as it didn’t result in any technical advancements in the proposed architectures. The discussions frequently resulted in bikeshedding and groupthink, the root causes of the lack of quality improvement.

During my first Architecture Review, I was aware of the importance of the 90-day rule, which meant I had to observe and learn as much as possible about the organization I had recently joined. However, despite my best intentions to take in information, I couldn’t help but offer my input. I noticed that a specific design relied excessively on event-based communication via Pubsub, which concerned me because the actual use cases were unclear. Moreover, although the infrastructure had been rewritten into a microservice-based architecture, it had a weak error-handling strategy. Given my limited context, I did my best to help the design owner clarify their thoughts.

After my first encounter, I realized the issue wasn’t the absence of dissent during the Architectural Review session. Instead, the team appeared to have an innate desire to build a robust microservice architecture without fully considering the advantages, disadvantages, and mainly cross-cutting concerns. I comprehend the rationale behind overengineering. The company transitioned from a phase lacking design validation to a phase where each alteration required scrutiny, marking a significant milestone in the company’s development. However, it became evident that further progression was necessary. The inconsistency in design choices prompted me to delve deeper into the level of standardization in software architecture across engineering teams. This exploration ultimately led me to investigate the root of this intention.

Risk of failing

Before we proceed, let me share a glimpse of Unico’s engineering culture. It can be likened to a fruit salad with diverse pieces representing different aspects of software engineering. I believe Unico was a blend of various engineering flavors resulting from the company’s multiple acquisitions, each with a unique engineering culture.

Senior leadership requested the creation of more reusable components, such as a standard authorization system that various products could utilize. Developing such features is crucial for a company’s growth and expansion, as demonstrated by the success of AWS at Amazon. However, some junior engineers misinterpreted this directive, assuming they needed to completely rewrite their systems to make them more compatible with others. This misinterpretation could lead to an endless cycle of rewrites, and teams begin creating their generic components without trying to rebuild their products to benefit from the shared infrastructure.

I also saw teams adopting new technology without considering its long-term sustainability. For example, some teams may choose Spanner because it offers global consistency. However, Unico had a user base only in Brazil by then. Other teams created an Elasticsearch index without considering its future maintainability. Other teams may select MongoDB only to explore a different type of database. I’ve observed that these situations occur when teams work independently without discussing the pros and cons of adopting different technologies within the company.

I faced the risk of failure when attempting to unite the development process. According to Conway’s law, the independent structure of teams often leads to isolated design decisions. However, creating unity within an organization requires rethinking the communication channels between different groups. It was no surprise that Unico had a team topology like the figure below:

Teams barely talked to each other, and communication flow was hierarchical through Engineering Directors and centralized in the CTO. As a result, I took it upon myself to bridge the gap between disconnected teams and address the issues mentioned above.

This scenario is quite interesting because I spoke with people from several startups in Brazil and learned that Unico is one of many startups facing these challenges. Therefore, this write-up could be helpful to many others experiencing similar issues.

Tech Leads

The amount of code a software engineering organization produces or its ability to deliver excellent features to users are only some of the crucial aspects of company culture. Instead, what matters most is what I call collective knowledge within a company, which leads to the outcomes mentioned above. Large tech companies like Google excel in this area by promoting knowledge sharing through documentation, shared codebase, and openness at every level of their organization. However, when teams are fragmented, individuals or groups may go in different directions, and the resulting common wisdom is not shared among peers.

To solve the team isolation, we built a group of the most relevant individual contributors from each significant area of the company. We called this group Tech Leads, acknowledging it has a different meaning from what is mainly used in the industry. But they have two distinct roles:

1. Represent their teams in the discussions, bringing the team demands to the larger group and communicating the technical decisions made to their teams.
2. Collectively build the engineering vision for the company that is well thought out and discussed.

Just by creating this group, we straightened up new communication bridges centralized on my figure and the close relation created with the members of the group:

More recently, members of the tech lead group suggested a six-month rotation program within the group. This program made the discussions more interesting by fostering a culture of bringing different engineering perspectives to the group. More importantly, this group created strong cohesion within the engineering organization. We made communication bridges that didn’t exist before and created opportunities for new leaders to emerge in the company.

Finally, as the output of the Tech Lead meeting, we started documenting our best engineering practices in the company that we called Dev Prime.

Build your Champions and start small.

When new ideas reach out to an organization, naturally, a group of promoters and another group of detractors will born. In a group of detractors, new ideas usually take people out of their comfort zone, or it is common for people to ask: “The company has reached its current state without the changes you are proposing; there’s no need for that.” On the other hand, promoters take the opportunity to relate to their career objectives, such as making progress on the company ladder, acquiring new knowledge, or simply building more reliable products to let them sleep during long weekends.

My strategy is to find champions. At the initial stage, champions are neither detractors nor pure promoters. They are people who want to listen to new ideas but have questions about them. When someone asks hard questions, it is your opportunity to improve your ideas and guarantee they will have a place in your company.

Adopting a mono repo was one of the debatable themes when I joined Unico. As an ex-Googler, I’m very fond of the Google3 concept, a centralized code repository where employees can access most lines of code running in production. I also believe that a company’s most important asset is not only the product it sells but the knowledge it produces throughout time. This is valid for tech companies and every area — think of a restaurant and the knowledge a reputed chef brings to your table. As a software engineer, I also believe the knowledge materializes in the code (until GenAI takes it). The culture of multi-repo fragments the knowledge sharing and puts me miles away from that amazing experience I had at Google.

On the other hand, Google historically invested a lot of energy into improving its repo technology by building different tools to scale build time, run tests faster, and reduce development time (come on, Google, this should go to Google Cloud). People complained about that. But when you have champions, you invite them to be part of the journey. They might need to grasp the idea initially but will like the trip. They will learn to start small, understand the tradeoffs, and see the improvements until they spread new ideas within and outside the company.

The annoyance of multiple options

It annoys me the number of multiple options that all the different software engineering solutions can provide. Take, for example, the options available for general data storage — from classical relational databases, K/V with fragile transaction control, document databases optimized for search, or very robust databases spanning consistency across the planet. Each has a different use case, but the correct usage is difficult to grasp in an immature engineering culture. Engineers like to experiment with new, shiny technology just because they want to.

It’s not only about technology but also about development culture. The decision of what to build or what to buy from a SAAS provider is never trivial. Few would create a database from scratch, but many would build a layer to monitor internal user access to comply with privacy laws. Why should you make the last but not the first, as both are unrelated to your core business?

This kind of discussion happens at many levels of an engineering organization. Examples at Unico are choosing between plain Rest APIs and GRPC, how to do good code reviews, how to handle errors in an application, how to guarantee atomic operations in distributed systems, what technology to use and not to use, and how to release software, among others. Suppose you don’t want to end up in the fruit salad situation. In that case, it is impractical for an engineering organization to delegate these topics to isolated groups making decisions independently. It would be best to have someone looking across the board, either the CTO or someone who can build these best practices.

Cross-cutting concerns, maintainability, and the joy of consistency

The Wikipedia defines cross-cutting concerns as:

Aspects of a program that affect several modules without the possibility of being encapsulated in any of them. These concerns often cannot be cleanly decomposed from the rest of the system in both the design and implementation. They can result in either scattering (code duplication), tangling (significant dependencies between systems), or both.”

Although neglected, most of the time is spent on development organizations (especially in the hyper-growth stage), discussing and fixing cross-cutting concerns. Unico’s primary business is building identification, but everything related, such as logging, user authentication, authorization, data encryption, or microservice communication, can be considered cross-cutting concerns. Engineers working on biometric engines should not worry about them, but I’d rather have them focus on improving the models so they bring more value to the company.

Another neglected aspect is the capability to maintain the software a company builds in the long term. Some papers say 70% of software development costs are in maintenance, while others say 90%. The term big ball of mud became popular with the group that loves microservices, clean architecture, and modularity. Unfortunately, maintainability became a common excuse for system rewrites where the main culprit might be the weak engineering culture.

I prefer to think of cross-cutting concerns and maintainability as a result of a single word: consistency.

Even in a diverse environment like Unico, all software engineering organizations should seek consistency. There will be a payoff in the future. In software development, consistency is the capacity to define a single piece of technology to solve a problem. A company is inconsistent when it uses PostgreSQL and MySQL for a relational database (even when it mixes relational databases with no-SQL solutions). It is the same when a company allows every team to choose their favorite programming language to write microservices — this company is inconsistent.

Lack of consistency brings a cost that cannot be neglected by the person overseeing the engineering organization. I took from this article by Davi Reis what I like to call the consistency equation (he didn’t name it this way):

The excerpt below describes the formula and why lack of consistency has a price:

Assume you have a frontend codebase using vanilla js (i.e., no libraries or frameworks). You decide, very correctly, according to our model, to save a lot of work by adopting AngularJS. This is one of the most popular and well-maintained frameworks, so it hits home on “choosing a library guide.” That increases the first component of our equation and has little impact on the second since you use no other libraries. Now, rinse and repeat, and add another great library to your codebase, ReactJS. Equally popular and well maintained. Now you have a more productive codebase right? Well, not. The left side of the equation continued to increase, but the right side increased faster since it is super hard to combine ReactJS and AngularJS, and now you have the worst nightmare for many developers.

This exact example happened at Unico. We had both AngularJS (10% of our codebase) and ReactJS (90%). Unfortunately, when we decided to introduce an external library for the design system, we spent most of our time discussing incorporating the new library into our Angular applications (although the business impact could have been more minimal). Another example is that we must apply CDC solutions to Postgres and SQLServer.

Most decisions to adopt external libraries are in the open-source space. However, the lack of consistency is even more tragic when financial costs are involved. More recently, when reviewing where we are spending our money on cloud services, we realized that we were throwing away millions of dollars for similar solutions to address the same purpose just because someone made an expensive local team decision in the past.

Classics — stop rewriting what brought you here.

Unico’s largest product, Unico Check, has a stack code of more than ten years, which is what some might consider a deprecated technology. Given that perception, it is straightforward to think that everything is wrong and that the company should put all its energy into rewriting this system. But should Unico rewrite a system that has been successfully running for over ten years and growing yearly? Rewriting, per si, is the wrong answer — the codebase that made your business a billion-dollar company has its value.

Unfortunately, our industry has a strong culture of deprecated systems. Everything hard to maintain is called deprecated, and people start endless rewrite projects that paralyze profitability. I’ve heard engineers saying that every piece of software has a limited lifecycle, but I ask them, “ Do you have the budget to rebuild something that does the same thing?” I don’t, and neither do my CEO.

Our solution is simple: we stopped calling systems deprecated and called them Classics. Classics are systems created before the new rules of consistency were created. Systems from acquired companies are also considered Classics. They don’t need to be rewritten to fit in Unico’s stack.

This simple naming change greatly impacted the company culture; instead of having thoughts of rewrites, people began to make changes in the sense of a more reliable system (e.g., defining proper SLOs, modernizing libraries, introducing feature flags, better deployment process, fixing security issues) and primarily important — engineers focus on core problems of the company, instead of meta technology discussions.

The idea of Classic might sound contradictory to the consistency argument, and we are increasing the exponential maintenance costs of the consistency formula. In the initial stages of a company, you should be consistent when it grows. However, in the stage of Unico, we are increasing the maintenance costs, but with the condition that the business growth (the product demands) has precedence over the engineering perceived quality of the system.

Consistency is always a target for the future. The hypothesis is that new business demands will require changes in the software, and developing these demands will move the company into more modern, maintainable, and consistent technologies — and we will construct new business models on top of these new systems. This hypothesis is promising; new products like IDUnico and IDPay have been built on new stacks while we keep Unico Check better daily.

It is also important to remember the concept of composition in software engineering, which corroborates the hypothesis of the slow evolution of the stack. You don’t want to keep rebooting your company codebase from scratch when better tech appears. Usually, you want to compose different stacks you built to provide solutions for new problems arising from your user base.

High effort in the design phase

Building software is complex and requires a lot of time from specialized professionals. On the other hand, the market is eager for rapid results. To solve this dichotomy, the software industry coined the term Minimal Viable Product (MVP) to understand that they need to build the minimum necessary for an initial set of customers and iterate over the received feedback. It is a beautiful concept that got lost in translation when it reached out to many engineering organizations. I’ve seen many engineers translating the MVP concept as “I will build a throwaway code and pay the technical debt in a future rewrite.” The MVP is precisely the opposite; it is minimal but with the requirement to iterate fast over the same codebase to compose new features to grow the user base (or whatever other success metric a company might have). Communicate to any CEO that the very successful MVP you built needs to freeze because the team has to rewrite it. Even worse, explain why a product with no market fit must be rewritten because you chose the wrong architecture or the tech lead doesn’t like the programming language. Then, ask how they feel about your work.

We changed the Architectural Review process to put a lot of effort into the design phase, to ensure consistency among projects, and to build an infrastructure that starts small but is reliable and evolves when new features are added.

To put some incentives on reuse and consistency, we have some important rules when an Architectural Review is required:

• A new product (internal or external) is created.
• The design needs a new microservice in the cluster.
• There is a need to introduce a new external SAAS in the stack.
• There is a need to use a new tech that is not on the approved technologies list.
• The project needs a new database instance.
• When the group of TLs understands that more eyes on the design will help the team.

To avoid the groupthink behavior described at the beginning of this document, we built a process for someone to be an official approver of an architectural review. Start with the small group of champions and then evolve from this group. If someone wants to be a reviewer, the first step is actively participating in the design proposals, commenting, asking questions, looking for consistency, and helping the author improve their design. When someone feels active enough, they can propose to the current reviewers that they are ready to be approvers.

The approved tech stack

To create an approved stack in a diverse tech environment like Unico, we had to review the use cases of each technology and find common ground where everyone is minimally aligned. You will have to make controversial decisions, like choosing Javascript over Typescript, just because most of the codebase is JS, even though most frontend engineers are keen to use Typescript. Be patient; only new projects will follow the latest tech stack, and not every existing project will comply with your view of a service, but in the medium term, when products evolve, they will align with the new stack. And remember, don’t rewrite classics to comply with the tech stack (a rewrite takes forever and never ends).

I’ve included a copy of Unico’s approved technology internal document: https://medium.com/@bsoneca/approved-technologies-at-unico-c0d2ec9ab5b2

Also, please remember that you must ban those design decisions that have caused you trouble. The banned decisions usually relate to async jobs with long and concurrent transactions in the database (avoid them at any cost).

The Dev Prime

The Dev Prime is a collective documentation of the construction of Unico’s software engineering culture. I began to create a common ground of software engineering culture to be a source of consulting when someone has doubts about the best practice to follow. I also noticed it needed to be finished when I released the first text. It is continuous work for everyone in the organization to contribute.

However, software development became more evident within the company after its creation. People started to mention they need to “comply with the dev prime,” but not as something that is strictly required but as something we are building together. More champions were raised within the teams, and the culture became more uniform.

I understand that companies in more advanced stages already have a strong engineering culture. But companies in the initial stage or not very mature need someone overlooking to avoid local decisions. If every engineer chooses the technology that fits them best, they will leave the company with some really “legacy” code. Building this unit, engineers will go, but the culture will remain — the companies with some significant progress have proven that consistency is one of the keys to their success. And we at Unico understand that other companies are in the same situation as we were. From time to time, we will publish some raw texts of what we have built inside, hoping that it helps others or even invites other engineers to know that there is an excellent place to build identity products (instead of rising and repeating on building architecture).

Unico Dev Prime was originally published in Unico on Medium, where people are continuing the conversation by highlighting and responding to this story.