Language Resources

DZone's Featured Languages Resources

The Window Closes: Why I Switched to Kotlin Before the Codebase Got Too Big

I moved a project from Java to Kotlin in a single pass, back when the codebase was small enough that doing so cost almost nothing. The system I work on now is a different animal: hundreds of thousands of lines of Java, a monolith rather than a set of services. Converting it wholesale is the kind of gamble nobody signs off on. The option still exists technically, but the cost is now measured in quarters of engineering time rather than days. How much code existed when someone first asked about Kotlin decided everything that followed. I have spent more than a decade building and scaling high-load distributed systems on the JVM, across fintech, security, and media infrastructure, and the pattern holds: you pay for a language migration at the start, well before anyone puts it on a roadmap. The Easy Migration The small project was a handful of Spring Boot microservices with JPA and PostgreSQL, still short of production at a few thousand lines of code. A colleague who spent his evenings in the Kotlin community suggested we try it on this one. The team was small and curious, so we agreed within a day or two. Spring compatibility was the thing we worried about, and it never became a problem. Because nothing had shipped yet, we converted the entire codebase at once. IntelliJ IDEA can rewrite a Java file to Kotlin with a single keystroke, and the converter handles about 80 percent of the conversion on its own. The rest of the week went into manual cleanup. Adding nullability annotations stripped out the platform types, and the compiler immediately flagged a dozen possible null-pointer exceptions that Java had been hiding inside Optional. We moved from field injection to constructor injection, which fits Kotlin far better. Data classes replaced our POJOs and trimmed about a quarter of the boilerplate, making the model easier to read. Static members needed more attention. After conversion, our public static final constants ended up inside a companion object, which changed the bytecode that Java callers expected, so we annotated them with @JvmStatic and @JvmField to keep the modules we hadn't touched yet compiling. By the end of the week, the tests were green, and Spring Boot loaded the new classes without complaint. The whole exercise took one week because the codebase was small, which is the entire point: the same cleanup across a large system is measured in quarters, not days. The Parser That Sold Me The message parser is what convinced me the move was worth defending to the rest of the team. Our services received data, validated it against an Avro schema, and returned a result. Parsing under load burns resources, and many Java libraries rely on runtime reflection during serialization and deserialization. Jackson can match the performance without reflection, to be fair: its Afterburner and Blackbird modules generate bytecode at runtime for field and method access, which is a real optimization. The cost shows up per format. JSON, XML, YAML, TOML, and Avro protocols each want their own modules and manual wiring, and a new format usually means finding a library or writing one. Kotlinx.serialization works differently. Its compiler plugin generates every serialization routine at build time and never calls reflection. We standardized most formats around Kotlinx.serialization, using it as the common model layer for JSON, XML, YAML, and TOML, then bolted on Avro as a separate plugin module. The schemas were plain Kotlin classes with the @Serializable annotation, and code generation handled the rest. We had to implement Avro's default values in the plugin by hand, but only once. A smaller dependency surface, typing that held end to end, one approach instead of five. That early switch also bought time to question the rest of the stack while it stayed light: jOOQ for queries that grew complicated, smaller web frameworks for lightweight services, and test tooling with more expressive assertions. Experiments are cheap on a compact codebase, and you can settle choices that would turn into prolonged arguments once the system has weight and a dozen people depending on it. The Monolith, and What to Do About Yours The same migration on the monolith I work on now is a different proposition, and I would push back on anyone who claimed it could happen in one pass. The window there has thinned to a slot rather than shut. Java and Kotlin still compile side by side in a single project, leaving an incremental approach open. You can write new tests in Kotlin against the existing Java, peel bounded contexts off into separate Kotlin services, and drop the idea that migration has to happen all at once. None of that demands the big-bang rewrite that scares management off, and each piece keeps the window from closing further. Postponing the conversation makes it more expensive every quarter, which is the cost the easy migration above never had to pay. If your codebase is still small, bring it up now, while the call is for engineers, not a budget committee. The migration window rarely closes dramatically. One quarter, it simply stops being practical. It just gets heavier to lift, and in one planning session, you realize the topic stopped coming up a while ago. More

Should Developers Switch From SQLite to DuckDB?

By Rakshath Naik

If you are a data scientist, data analyst, or anyone who works with big data, you have likely used SQLite, or even use it daily. Introduced in 2000, SQLite quickly became a popular choice for database-related tasks. Whether you used it in an introductory database module, built a local data pipeline, or managed local metadata in production, SQLite has long been the gold standard for a zero-configuration database, with more than 25 years of widespread use. But in recent years, DuckDB has emerged as a more powerful alternative, designed specifically for analytical workloads. For tasks involving Data analysis, analytics, and ETL (Extract, Transform, and Load), DuckDB is quietly gaining traction. The developer community today is increasingly migrating to DuckDB, especially when running analytical aggregations across gigabytes of local Parquet or CSV files, where it shows its speed advantage. So the question that arises is: Should DuckDB Replace SQLite? In this article, we will explore the fundamental design differences between the two libraries and determine whether DuckDB should replace SQLite. The Evolution of SQLite and DuckDB Evolutionary timeline comparing the hardware eras, workload requirements, and engine architectures of SQLite and DuckDB from 2000 to the present. (Image by the Author) SQLite was created by American software engineer D. Richard Hipp, whose original motivation for developing the library stemmed from his work with U.S. Navy destroyers. While working on Navy software, he wanted a database that could work reliably without a complex client-server system. As a result, SQLite was designed with the Navy’s specific requirements in mind. It was easy to use, with zero administrative overhead, and had excellent fault tolerance by eliminating the client-server architecture, providing continuous uptime. However, as datasets scaled to millions of records and multicore CPUs became the norm, some of SQLite’s original design choices became its core bottlenecks. Modern CPUs have multiple cores for better speed, but SQLite processes data row by row on a single thread. Moreover, it requires users to manually import files before querying them, which can result in slow import and export processes. Then came DuckDB in 2019, designed for fast analytical processing on local data. Although it was not intended to be a competitor or direct replacement for SQLite, it effectively solved the problem of single-core processing, prompting developers to use it more frequently for building data pipelines. The Developer Interface: Similar Syntax, Different Engines When it comes to the interface, there is no noticeable difference between the two libraries. If you are already familiar with SQL, then querying data with DuckDB requires virtually no learning curve compared to SQLite. SQLite Python import sqlite3 conn = sqlite3.connect("sales.db") cursor = conn.cursor() cursor.execute( "SELECT region, SUM(sales) FROM transactions WHERE sales > 1000 GROUP BY" " region" ) results = cursor.fetchall() DuckDB Python import duckdb # Querying local disk files directly with standard SQL results = duckdb.sql( "SELECT region, SUM(sales) FROM 'sales.csv' WHERE sales > 1000 GROUP BY" " region" ).fetchall() They may look similar at first glance, but the differences become clear when you look deeper into their architectures. Architectural comparison between SQLite’s sequential row-store processing and DuckDB’s optimized columnar vector execution pipeline. (Image by Author) Query Engines Processing Across CPU Cores SQLite uses the Volcano Iterator Model, where a single core processes queries sequentially on a single CPU thread. If you sort 10 million rows, a single CPU thread handles the entire task while the rest of the CPU sits idle, increasing processing time. On the other hand, DuckDB uses morsel-driven parallelism, partitioning the dataset into smaller chunks and processing them across available CPU threads through independent task queues. A large operation is split into uniform data chunks called morsels. Idle worker threads grab any available morsels from the queue, keeping the CPU cores fully utilized until execution finishes. Deferred Execution and Logical Plan Optimization SQLite executes queries in an eager streaming fashion where the rows are read immediately from disk or B-Trees and pushed through the expression evaluator step without global transformations. DuckDB enforces a deferred pipeline with an explicit optimization pass before the data is even touched. [ Raw SQL Query ] →[ Unoptimized Logical Plan ] → [Optimizer Pass ] → [ Physical Execution Plan ] DuckDB optimizes the query to eliminate non-essential work. During optimization, unused columns are dropped entirely, and conditions like WHERE sales > 1000 are pushed to the file reader level, skipping non-matching data blocks. Additionally, various query operations are arranged dynamically based on cardinalities to reduce intermediate memory allocation. Memory Efficiency and Data Storage Structures At last, we will look at the memory architecture, as the data transfer between RAM and CPU cache lines is frequently the primary bottleneck in data applications. Both libraries use fundamentally different data storage structures. SQLite uses a row-oriented storage layout, where it stores every feature of a record in a single row. Combined with an extremely minimal baseline footprint of just 2–5 MB, this design is optimal for retrieving an individual row (eg: loading a single user’s profile), but not for running an aggregation like AVG(sales), as it forces SQLite to load every unused attribute into cache along with the target values. DuckDB, on the other hand, uses a columnar format. Each feature is stored in a separate column, with the column data stored as a contiguous block in memory. This approach is preferred for analytical queries because DuckDB reads only the specific columns, maximizing CPU cache efficiency. The columnar structure also enables zero-copy data sharing with tools such as Apache Arrow, PyTorch, and Polars without expensive memory serialization steps. To maintain its multi-threaded vector channels, DuckDB operates with a higher baseline memory footprint of 20–30 MB. Final Thoughts: Should DuckDB Replace SQLite? After comparing the architectures of the two libraries, DuckDB may appear to be the better alternative. So, coming to our final question: Should DuckDB replace SQLite? The answer is: No, not quite. After looking at the architecture of the two database engines, the core trade-offs become clear: SQLite uses a row-oriented layout and a single-threaded engine to maintain a minimal memory footprint of just 2–5 MB. It is optimized for transactional (OLTP) operations, securing its position as a universal component across mobile and desktop devices. Conversely, DuckDB’s columnar architecture with multi-core parallel processing makes it a great choice for analytical (OLAP) queries and scanning external .csv, .parquet, or .json files directly without any manual imports. It turns out to be a better option for modern data science and AI pipelines. SQLite was engineered at a time when embedded storage was expected to handle lightweight applications and reliable local commits. In contrast, DuckDB was built to function as an analytics engine for large datasets. So which library you use for what task depends entirely on your use case. Choose SQLite if you need a zero-maintenance, embedded database to store app configuration or handle fast single-row lookups with minimal RAM overhead. Choose DuckDB if you are building a professional data or AI pipeline, running aggregations (GROUP BY, SUM, AVG) across millions of records, extracting features from .csv files, or processing local data in Python notebooks without a complex setup. Overall, it is better to see both as libraries designed for different purposes rather than as competitors. If you like the article, connect with me on https://medium.com/@rakshathnaik62 for more articles. I mainly write about Science, math, and AI. More

Stop Writing Database Migration Scripts: Declarative Schema Sync for Java

By Eugene Naoumov

Building a Reddit Scraper in Go: JSON Listings, Pagination, Rate Limits, and Where It Breaks

By Viktor Hendelmann

Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop

By Shreyamsh Kamate

How to Migrate From LangChainGo to Genkit Go

If you built a Go AI service in the last two years, there is a good chance it runs on LangChainGo. It was the obvious choice: the name was familiar, the concepts came straight from Python, and it had the biggest provider catalog in the Go ecosystem. That choice has aged badly. LangChainGo's last release was v0.1.14 in October 2025, its last commit to main was January 2026, and it is still on 0.1.x after three and a half years. Meanwhile, the ecosystem moved: models were retired, SDKs were replaced, and APIs changed shape. LangChainGo did not move with them, and parts of it are broken today as a result. Genkit Go is where that workload belongs now. It is GA, it is on v1.13.1, and it is actively maintained by Google. This guide is the complete migration path: every LangChainGo concept mapped to its Genkit equivalent, with before and after code for each one. Follow it top to bottom, and you can move a real service in an afternoon. If you want the case for why rather than the how, read the companion article: LangChainGo vs Genkit Go: where Genkit shines. Why Migrate Now Three things in LangChainGo are broken today, and none of them will be fixed, because nobody is upstream to fix them. 1. The default model no longer exists. DefaultOptions() in llms/googleai/option.go hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) with no options fails on the first call: Plain Text googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements. 2. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, which Google replaced with google.golang.org/genai. Its README is explicit: End-of-Life Date: All support for this repository (including bug fixes) will permanently end on November 30, 2025. 3. Streaming is broken on every current Gemini model. Because of that dead SDK, any streaming call through the Google provider fails: Plain Text error in stream mode: invalid character ']' looking for beginning of value This fails on gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash. Non-streaming calls still work, and the OpenAI provider streams fine, so this is specific to Google on the legacy SDK. The Trap: Observability Breaks Your Agents This one catches people out, so it is worth knowing before you debug it the hard way. LangChainGo turns on streaming automatically whenever you attach a callbacks handler. From chains/options.go: Go if opts.StreamingFunc == nil && opts.CallbackHandler != nil { opts.StreamingFunc = func(ctx context.Context, chunk []byte) error { opts.CallbackHandler.HandleStreamingFunc(ctx, chunk) return nil } } agents/mrkl.go does the same thing. Since streaming is broken, attaching the only supported observability hook is also what breaks your agent. Same agent, same tool, one line different: Plain Text WITHOUT callbacks handler: err=<nil> out="The current year is 2026." WITH callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value On LangChainGo with Gemini today, you can have working agents, or you can have instrumentation. Not both. What You Need Go 1.24+ (this guide was written against Go 1.27)A Google AI API key, free from Google AI StudioOptionally, an OpenAI API key, if you want cross-vendor fallbackThe Genkit CLI, for the Developer UI: Shell npm install -g genkit Add Genkit to your module: Shell go get github.com/firebase/genkit/go@latest The Concept Map This is the whole translation table. Keep it open while you work. LangChainGoGenkit GoWhat changesllms.Model + provider New()plugin + genkit.InitModels become string references like googleai/gemini-3.5-flashllms.GenerateFromSinglePromptgenkit.GenerateTextDirect swapprompts.PromptTemplateai.WithPrompt or a .prompt fileDotprompt files keep prompts outside the binarychains.LLMChain + chains.Rungenkit.DefineFlowA typed Go function instead of a map[string]anychains.NewSequentialChainordinary Go codeJust call the next functionoutputparser.NewDefined[T]genkit.GenerateData[T]Real schema instead of prompt-and-parsetools.Tool interfacegenkit.DefineTool[In, Out]Typed arguments with a JSON schemaagents.NewOneShotAgent (ReAct)ai.WithTools, or genkitx.DefineAgentNative function calling instead of text parsing. Genkit also has a real agent primitive with sessionsmemory.NewConversationBufferresp.History(), or an agent session storeExplicit history, or aix.WithSessionStore if you want the buffer backembeddings + vectorstoreslocalvec or a vector store pluginGenkit has a local store, LangChainGo does notcallbacks.Handlerbuilt-in tracingDelete the handler entirelynothingmiddleware.Retry, middleware.FallbackRetry and fallback stop being your code Step 1: Set Up Genkit In LangChainGo, each provider is a concrete struct you construct and hold: Go llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.5-flash"), googleai.WithDefaultEmbeddingModel("gemini-embedding-001")) if err != nil { log.Fatal(err) } defer llm.Close() In Genkit, plugins register with one runtime object and models become string references: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-3.5-flash")) That difference matters more than it looks. Registering a second vendor is one more plugin: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, )) Now both vendors live in the same registry, and a model is just a name, which is what makes swapping and falling back a configuration change later on. Step 2: Migrate Your Model Calls A plain text generation is close to a direct swap. Before: Go out, err := llms.GenerateFromSinglePrompt(ctx, llm, "In one sentence: what is Go's error handling philosophy?") After: Go out, err := genkit.GenerateText(ctx, g, ai.WithPrompt("In one sentence: what is Go's error handling philosophy?")) If you need the full response rather than just the text, use genkit.Generate, which gives you resp.Text(), resp.History() and usage data. Step 3: Turn Chains Into Flows A LangChainGo chain wraps a prompt template and takes a map[string]any: Before: Go prompt := prompts.NewPromptTemplate( "Triage this support ticket.\n{{.format}\n\nTicket: {{.ticket}", []string{"format", "ticket"}, ) chain := chains.NewLLMChain(llm, prompt) raw, err := chains.Predict(ctx, chain, map[string]any{ "format": parser.GetFormatInstructions(), "ticket": ticket, }) A Genkit flow is a typed Go function. Your input and output types are the contract, and the compiler enforces them: After: Go triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) { out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithPrompt("Triage this support ticket: %s", ticket)) return out, err }) out, err := triage.Run(ctx, "I was charged twice for order A-1029.") This step deletes the most code, because most of what a chain does is pass a map around. Go does not need that. Sequential chains have no equivalent, and do not need one. chains.NewSequentialChain becomes calling the next function: Go summary, err := summarize.Run(ctx, doc) if err != nil { return nil, err } return classify.Run(ctx, summary) Step 4: Migrate Structured Output Prioritize this step. In LangChainGo, structured output is prompt engineering. In Genkit, it is a schema. Before: Generate format instructions from your struct, paste them into the prompt, parse the text back. Go type Ticket struct { Category string `json:"category" describe:"one of: billing, bug, feature, other"` Severity int `json:"severity" describe:"1 (low) to 5 (critical)"` Summary string `json:"summary" describe:"one sentence"` } parser, err := outputparser.NewDefined(Ticket{}) // ...put parser.GetFormatInstructions() into the prompt... out, err := parser.Parse(raw) After: One call, with the schema sent to the model as a real response schema. Go type Ticket struct { Category string `json:"category" jsonschema:"enum=billing,enum=bug,enum=feature,enum=other"` Severity int `json:"severity" jsonschema_description:"1 (low) to 5 (critical)"` Summary string `json:"summary" jsonschema_description:"one sentence"` } out, _, err := genkit.GenerateData[Ticket](ctx, g, ai.WithPrompt("Triage this support ticket: %s", ticket)) Note the tag change: describe: becomes jsonschema_description:, and enums move into a jsonschema: tag where the model actually enforces them. Step 5: Migrate Your Tools The LangChainGo tool interface is Call(ctx context.Context, input string) (string, error). One string in, one string out, no schema. A tool that needs typed arguments has to describe its schema in prose and unmarshal by hand: Before: Go func (RefundTool) Name() string { return "issue_refund" } func (RefundTool) Description() string { return `Issues a refund. Input MUST be a JSON object string with exactly these keys: {"order_id": string, "amount_cents": integer}. Example: {"order_id":"A-1","amount_cents":2500}` } func (RefundTool) Call(ctx context.Context, input string) (string, error) { input = strings.TrimSpace(input) input = strings.TrimPrefix(input, "```json") input = strings.TrimPrefix(input, "```") input = strings.TrimSuffix(input, "```") var a RefundArgs if err := json.Unmarshal([]byte(strings.TrimSpace(input)), &a); err != nil { return fmt.Sprintf("error: input was not valid JSON (%v). Got: %s", err, input), nil } return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil } In Genkit, the tool function is generic over its argument type, and the schema comes from the struct: After: Go type RefundArgs struct { OrderID string `json:"order_id"` AmountCents int `json:"amount_cents"` } refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.", func(ctx *ai.ToolContext, a RefundArgs) (string, error) { return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil }) Delete the fence stripping, the unmarshalling, and the prose schema. The model receives a real JSON schema, and the arguments arrive typed. Step 6: Replace the Agent A LangChainGo ReAct agent drives tools by parsing the model's text output for Action: and Action Input: lines, wrapped in an executor with an iteration cap: Before: Go agent := agents.NewOneShotAgent(llm, []tools.Tool{RefundTool{}, agents.WithMaxIterations(6)) out, err := chains.Run(ctx, agents.NewExecutor(agent), "Refund order A-1029 for 25 dollars and 50 cents.") In Genkit, native function calling means a one-shot agent is just a generate call with tools attached: After: Go out, err := genkit.GenerateText(ctx, g, ai.WithPrompt("Refund order A-1029 for 25 dollars and 50 cents."), ai.WithTools(refund)) No executor, no iteration cap, no text parsing. Genkit runs the tool loop for you. When You Want a Real Agent, Genkit Has One The generate call above replaces a OneShotAgent. For anything that has to hold a conversation, Genkit Go ships an actual agent primitive with sessions and persisted state, which LangChainGo has no equivalent for: Go import ( aix "github.com/firebase/genkit/go/ai/exp" "github.com/firebase/genkit/go/ai/exp/localstore" genkitx "github.com/firebase/genkit/go/genkit/exp" ) g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-2.5-flash"), genkit.WithExperimental()) store := localstore.NewInMemorySessionStore[State]() agent := genkitx.DefineAgent(g, "taskAgent", aix.InlinePrompt{ ai.WithSystem("Manage a task list. Use tools when changing tasks."), ai.WithTools(addTask), }, aix.WithSessionStore(store), aix.WithDescription[State]("Task management assistant"), ) Three constructors are available: DefineAgent for an inline prompt, DefinePromptAgent to wrap a prompt already in the registry, and DefineCustomAgent to replace the prompt loop with your own code. Options cover a session store, WithStateTransform to reshape state on the way out to a client, and WithStreamTransform for chunks. You drive one with Run, RunText, or Connect for a streaming connection. These live in the exp packages. Agents are experimental in Genkit Go v1.13.1, and calling genkitx.DefineAgent without genkit.WithExperimental() on genkit.Init panics with a message telling you so. The API may change between minor releases, so weigh that before putting one in production. Step 7: Migrate RAG Check this step before planning your timeline, because it has an infrastructure consequence. LangChainGo ships fifteen vector stores: AlloyDB, Azure AI Search, Bedrock Knowledge Bases, Chroma, CloudSQL, Dolt, MariaDB, Milvus, MongoDB, OpenSearch, pgvector, Pinecone, Qdrant, Redis and Weaviate. Every one of them needs an external service running. There is no in-memory or local option, and the pull request adding one has been open since February 2024. If you needed a local store, you wrote it yourself. Genkit ships localvec, a file-backed local store, so the same thing is three calls: After: Go embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001") store, retriever, err := localvec.DefineRetriever(g, "policy", localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil) if err != nil { return err } // Index your documents once. localvec.Index(ctx, docs, store) // Retrieve at request time. found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket)) Then hand the documents straight to the model: Go out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithDocs(found.Documents...), ai.WithPrompt("Triage this ticket using the policy documents: %s", ticket)) If you were on a hosted store, Genkit has Pinecone, Weaviate, Postgres, and AlloyDB plugins. Step 8: Migrate Memory LangChainGo keeps conversation history in a buffer attached to the chain: Before: Go c := chains.NewConversation(llm, memory.NewConversationBuffer()) chains.Run(ctx, c, "My name is Xavi and I work in Go.") chains.Run(ctx, c, "What is my name and what language do I use?") Genkit passes history explicitly: After: Go resp, err := genkit.Generate(ctx, g, ai.WithPrompt("My name is Xavi and I work in Go.")) if err != nil { return err } resp, err = genkit.Generate(ctx, g, ai.WithMessages(resp.History()...), ai.WithPrompt("What is my name and what language do I use?")) For a plain generate call, this is the one mapping where LangChainGo is less typing. The trade-off is that Genkit has no hidden buffer to keep in sync, which is what you want the moment your conversation state lives in Redis or Postgres and your handlers are stateless. If you want the buffer back, that is what an agent's session store is for. Give the agent a store and pass a session ID, and the history is the framework's problem again: Go store := localstore.NewInMemorySessionStore[State]() agent := genkitx.DefineAgent(g, "taskAgent", aix.InlinePrompt{ ai.WithSystem("Manage a task list. Use tools when changing tasks. Be terse."), ai.WithTools(addTask), }, aix.WithSessionStore(store)) for _, turn := range []string{ "Add 'buy milk' to my list.", "What did I just ask you to add?", } { out, err := agent.RunText(ctx, turn, aix.WithSessionID[State]("demo-session")) if err != nil { return err } fmt.Printf("> %s\n%s\n\n", turn, out.Message.Text()) } Plain Text > Add 'buy milk' to my list. Task added. > What did I just ask you to add? You asked me to add 'buy milk' to your list. Nothing in that loop passes the first turn into the second. The session ID does it. localstore.NewInMemorySessionStore is the one that ships for local development; the SessionStore interface is what you implement to put sessions in Redis or Postgres. So the honest version of this row is: Genkit gives you both ends. Explicit resp.History() when you want stateless handlers, and a store-backed session when you want the buffer. Step 9: Delete Your Resilience Code This is the step that usually surprises people with how much it removes. LangChainGo has no middleware layer. callbacks.Handler is seventeen methods that all return nothing: Go HandleText(ctx, text) HandleLLMGenerateContentStart(ctx, ms) HandleLLMGenerateContentEnd(ctx, res) HandleToolStart(ctx, input) // ...and thirteen more You can watch a call. You cannot wrap it, retry it, rewrite its request, or substitute a model, because there is no next(). There is no retry, no fallback, and no OpenTelemetry anywhere in the library, so if your service has those, you wrote them. In Genkit, they are configuration: After: Go out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)), ai.WithDocs(found.Documents...), ai.WithTools(refund), ai.WithPrompt("Triage this ticket and act on it: %s", ticket), ai.WithUse( &middleware.Retry{MaxRetries: 2}, &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}, ), ) Register the middleware plugin at init, and those two lines are live: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{}, )) Point the primary at a dead model, and the request heals onto the other vendor on its own: Plain Text WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" {Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued full refund of $25.50 (2550 cents) for the duplicate charge on order A-1029.} Gemini went down, OpenAI picked it up, the tool still ran, and the typed struct came back intact. Delete your callbacks handler too. Genkit traces flows, generate calls, tool calls, and retrieval automatically. Run the Developer UI to see them: Shell genkit start -- go run . Step 10: Wrap the Entrypoints Finally, expose your flows over HTTP. Where LangChainGo left you writing net/http handlers around chains, Genkit turns a flow into an endpoint: Go mux := http.NewServeMux() mux.HandleFunc("POST /triage", genkit.Handler(triage)) log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux)) The flow's input and output types define the request and response shapes, so the whole path is typed end to end. Before You Commit The migration is not free. Check these three things first: Provider and vector store catalog. LangChainGo has seventeen model providers and fifteen vector stores. Genkit Go covers Google AI, Vertex, OpenAI, Anthropic, Ollama, and the OpenAI-compatible vendors, plus Pinecone, Weaviate, Postgres, and AlloyDB. If you are on Dolt, MariaDB, or Bedrock Knowledge Bases as a vector store, confirm you have a path. Conversation buffers. There is no drop-in ConversationBuffer. You pass history explicitly, as in Step 8. Specialized chains. Helpers like chains.NewSQLDatabaseChain and the map-reduce summarisation chains have no one-to-one equivalent. In Genkit, they are Go functions you write, which is usually shorter than the chain was, but it is work. Conclusion The migration is smaller than it looks, because most of a LangChainGo program is plumbing that Go does not need: maps passed between chains, format instructions pasted into prompts, JSON unmarshalled out of tool strings, and a resilience layer you wrote yourself. Typed flows, typed tools, schema-backed output, and middleware delete all four. The reason to start now is not ergonomics. It is that the library your service depends on has a retired default model, an end-of-life SDK underneath its Google provider, broken streaming, and an observability hook that breaks the agents it is supposed to observe. Those are not problems you can wait out. For the side-by-side numbers and the full picture, read the companion: LangChainGo vs Genkit Go: where Genkit shines. Further reading: Genkit documentationGenkit Go pluginsLangChainGo repositoryGoogle Gen AI SDK for Go, the replacement for the SDK LangChainGo still usesStop using Python for GenAI, use Genkit GoMastering Genkit: Go Edition

By Xavier Portilla Edo

CORE

Introduction to Error Prone: How to Catch Bugs at Compile Time

The Java compiler’s job is to check whether code follows the rules of the language. Trying to remove an element from a collection that can never contain it, or using the wrong letter in a date pattern, does not break those rules, so nothing stops the code from compiling. In short, any code that follows the rules compiles, even when it does not do what its author intended. What Is Error Prone? Error Prone is an open-source static analysis tool that Google built to fill this gap. Unlike most static analysis tools, it does not run as a separate step; it plugs into javac as a plugin. Using the information the compiler produces while analyzing the code, it looks for patterns that follow the rules of the language but are most likely wrong, and reports them as compiler warnings or errors. The idea is simple: a bug costs the least to fix when the code is written. Error Prone catches bugs during compilation, before they reach code review, CI, or production. Google has used the tool on its huge Java codebase for years, but its use is not limited to Google. NullAway, the null safety tool developed by Uber, runs as an Error Prone plugin. Spring Framework 7 checks its own codebase with NullAway during the build. Popular libraries such as Caffeine also use Error Prone in their builds. The examples in this article, which introduces you to Error Prone, are available in this repository. How It Differs From Other Static Analysis Tools The Java world has well-established tools such as SpotBugs, PMD, Checkstyle, and SonarQube. Three key features set Error Prone apart from them: It runs inside the compiler. It needs no separate build step or server. Because it is a plugin for javac, it works with any build tool that calls javac (Maven, Gradle, Bazel, Ant).It knows everything the compiler knows. The analysis runs on the AST that javac builds, after the type attribution and flow analysis phases. This gives Error Prone access to all symbol and type information. For example, it knows exactly what the generic type argument of an expression is, or which overload was chosen.It does not stop at a diagnosis; it suggests a fix. Most checks come with a suggested fix, which can be applied to the source code with a single command. How Does Error Prone Help Developers? The strength of Error Prone is that it can stop code that compiles but behaves incorrectly, right at compile time. Let’s take a closer look at the following example from the official documentation: Java Set<Short> s = new HashSet<>(); for (short i = 0; i < 100; i++) { s.add(i); s.remove(i - 1); } System.out.println(s.size()); This code seems to remove the previous element at each step, so you would expect it to print 1. It prints 100 instead, because the expression i - 1 is an int and is boxed to Integer. Since the Set<Short> contains no Integer, remove removes nothing. And because the remove(Object) signature accepts any type, the compiler does not complain either. Error Prone, however, refuses to compile this code: Plain Text [ERROR] ShortSet.java:[22,21] [CollectionIncompatibleType] Argument 'i - 1' should not be passed to this method; its type int is not compatible with its collection's type argument Short Now let’s look at the following code: Java private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("YYYY-MM-dd"); String reportName(LocalDate date) { return "report-" + FORMATTER.format(date) + ".csv"; } The code looks right when you read it, and it even works correctly for most of the year. However, Y in the pattern does not mean the calendar year; it means the week-based year. The calendar year is written with a lowercase y. When the last days of the year fall into the first week of the next year, the two values diverge. For example, for LocalDate.of(2025, 12, 29) the result is report-2026-12-29.csv, so the date jumps a year ahead. When the bug shows up also depends on the locale. The same code starts producing the wrong year on December 28 in the en-US locale, and on December 29 in the tr-TR locale. For the rest of the year the tests pass, and in code review the difference between YYYY and yyyy is easy to miss. Error Prone recognizes this pattern, stops the compilation and suggests the fix as well: Plain Text [ERROR] ReportDate.java:[16,83] [MisusedWeekYear] Use of "YYYY" (week year) in a date pattern without "ww" (week in year). You probably meant to use "yyyy" (year) instead. Did you mean 'private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");'? Note that in both examples the code is valid Java and compiles without any problem. At best, your IDE shows a warning; the build does not break. Since the code can also pass the tests, the bug can live unnoticed for a long time. With Error Prone, you do not have to rely on anyone’s attention to catch these subtle bugs: the build fails, the error message explains the problem, and it often suggests the fix as well. Extending Error Prone Error Prone is not a closed set of rules. Where the built-in checks fall short, you can add your own rules on the same infrastructure they use. There are two ways to do this: Writing transformation rules with Refaster. Rules of the form “when you see this code, replace it with that” are written as plain Java templates.Writing your own BugChecker. Team-specific bug patterns are defined with the same API that Error Prone uses for its own checks. Refaster Refaster is a refactoring tool that is part of Error Prone, built for large-scale transformations across big codebases. You do not write the rule against an AST API, but in plain Java: “when you see this, replace it with that.” A Refaster rule is an ordinary Java class with at least two methods. @BeforeTemplate defines the pattern to look for, and @AfterTemplate defines the code to put in its place: Java class StringIsEmpty { @BeforeTemplate boolean before(String s) { return s.length() == 0; } @AfterTemplate boolean after(String s) { return s.isEmpty(); } } The template is matched with type information, not as text. That is why both name.length() == 0 and user.getEmail().length() == 0 are caught, while the same expression on a type other than String that also has a length() method is not. By writing more than one @BeforeTemplate, you can combine different patterns that lead to the same result in a single rule. Using a rule takes two steps: The rule class is compiled with the RefasterRuleCompiler plugin, which produces a .refaster file. The plugin lives in the error_prone_refaster artifact and is passed to the compiler with the -Xplugin:RefasterRuleCompiler --out rules.refaster argument.The generated .refaster file is passed to Error Prone with -XepPatchChecks:refaster:<file>. Error Prone writes the matches to a patch file, or applies them directly to the source code with -XepPatchLocation:IN_PLACE. Refaster is ideal for tasks such as migrating from one API to another, removing a deprecated method, or standardizing an idiom within a team. Picnic’s open-source Error Prone Support project also offers a way to run Refaster rules on every build, like a regular Error Prone check. Custom BugCheckers Error Prone’s built-in checks do not cover every rule. Alongside widely shared rules such as “sensitive data like national ID numbers, credit card numbers and email addresses must not be written to logs” or “do not use new BigDecimal(double) for monetary calculations", each team may also have its own rules, such as "this internal API must no longer be called". Error Prone lets you write these rules on the same infrastructure as its built-in checks. A custom check is a Java class that extends BugChecker. It declares which AST nodes it is interested in through the *TreeMatcher interfaces it implements. Error Prone calls the check for each such node, and the check returns a match and an optional fix. The following example implements the sensitive data rule. Although the rule is general, only the team knows which data is sensitive. That is why the team marks fields such as national ID numbers or email addresses with its own @SensitiveData annotation: Java public record Customer( String name, @SensitiveData String nationalId, @SensitiveData String email, @SensitiveData String cardNumber) {} The check reports an error when a log call is given a marked field, or an object whose toString() would print marked fields: Java @AutoService(BugChecker.class) @BugPattern(summary = "Sensitive data must not be written to logs", severity = ERROR) public final class SensitiveDataLogging extends BugChecker implements MethodInvocationTreeMatcher { private static final String SENSITIVE_DATA = "org.jugistanbul.SensitiveData"; private static final Matcher<ExpressionTree> LOGGER_CALL = instanceMethod() .onDescendantOfAny( "java.lang.System.Logger", "java.util.logging.Logger", "org.slf4j.Logger"); @Override public Description matchMethodInvocation(MethodInvocationTree tree, VisitorState state) { if (!LOGGER_CALL.matches(tree, state)) { return Description.NO_MATCH; } tree.getArguments().stream() .flatMap(argument -> stringifiedValues(argument, state)) // "..." + x → "...", x .filter(value -> isSensitive(value, state)) .forEach(value -> state.reportMatch( buildDescription(value) .setMessage("'%s' contains sensitive data and must not be written to logs" .formatted(state.getSourceForNode(value))) .build())); return Description.NO_MATCH; } private static boolean isSensitive(ExpressionTree value, VisitorState state) { // customer.email(): the field or accessor itself is annotated var symbol = ASTHelpers.getSymbol(value); if (symbol != null && ASTHelpers.hasAnnotation(symbol, SENSITIVE_DATA, state)) { return true; } // customer: the class, or one of its fields, is annotated, so toString() leaks it var type = ASTHelpers.getType(value); if (type == null || type.isPrimitive() || !(type.tsym instanceof ClassSymbol valueClass)) { return false; } return ASTHelpers.hasAnnotation(valueClass, SENSITIVE_DATA, state) || valueClass.getEnclosedElements().stream() .anyMatch(member -> member.getKind() == ElementKind.FIELD && ASTHelpers.hasAnnotation(member, SENSITIVE_DATA, state)); } // stringifiedValues(...) splits a string concatenation into its operands; see the repo } In the demo’s CheckoutService, the first line passes because it only logs the customer's name, while the other two do not compile: Java LOG.log(Level.INFO, "Order " + orderId + " placed by " + customer.name()); LOG.log(Level.DEBUG, "Order " + orderId + " placed by " + customer); LOG.log(Level.INFO, "Receipt for order " + orderId + " sent to " + customer.email()); Plain Text [ERROR] CheckoutService.java:[25,67] [SensitiveDataLogging] 'customer' contains sensitive data and must not be written to logs [ERROR] CheckoutService.java:[28,90] [SensitiveDataLogging] 'customer.email()' contains sensitive data and must not be written to logs The check looks at the expression passed to the logger, so a value first copied into a local variable (String email = customer.email();) is not tracked; following it would need data-flow analysis. This short class shows all the parts of a custom check: @BugPattern: Defines the rule's name, description, and severity (ERROR or WARNING). The rule can be turned off, or its severity changed, by its class name (-Xep:SensitiveDataLogging:WARN), and it can be suppressed with @SuppressWarnings.Matchers: Ready-made building blocks such as MethodMatchers express conditions like "any method of a System.Logger, java.util.logging or SLF4J logger" with type information, in a single expression.Type and symbol information: ASTHelpers gives access to the symbol and type of the logged value. Annotations are resolved by their fully qualified names, so an annotation with the same name in another package does not get mixed in. A text search cannot do this.@AutoService: Makes the check discoverable through ServiceLoader. Adding the jar that contains the checks to the annotation processor path is enough. This check only reports the error, but checks can also suggest fixes with SuggestedFix. BigDecimalDoubleConstructor in the repository is a short example that suggests BigDecimal.valueOf(double) instead of new BigDecimal(double). Checks are tested with CompilationTestHelper. // BUG: Diagnostic contains: comments in the test source mark the line where an error is expected. For checks that suggest fixes, BugCheckerRefactoringTestHelper compares the output of the fix with the expected source code; BigDecimalDoubleConstructorTest in the repository is an example. Running the Examples All the examples in this article are available in runnable form in the error-prone-in-action repository: ShortSet and ReportDate, which trigger the built-in checks, three custom checks with their tests, and the StringIsEmpty Refaster rule. The repository's README explains which command runs each example. Conclusion The Java compiler checks whether code follows the language rules, not what its author intended. Error Prone fills this gap from inside the compiler. Using javac’s type information, it catches code that compiles but is most likely wrong, and often suggests the fix as well. Where the built-in checks fall short, you can write transformation rules with Refaster and team-specific checks with the BugChecker API. In my view, this approach is more valuable today than ever. As AI coding assistants speed up the pace of writing code, tools that catch bugs at their cheapest moment, at compile time, are turning from a luxury into a necessity.

By Hüseyin Akdoğan

CORE

Giving AI Agents Safe Access to Legacy Systems: Introducing legacy2mcp

The Problem: Agents Are Ready, Legacy Systems Aren't AI agents are getting good at reasoning and tool use. The systems they need to reach, though, are often 10-20 years old: SOAP services, relational databases, message queues. These systems hold the claims, orders, accounts, and policies that businesses run on. Most enterprises can't rewrite them, and most can't safely expose them to an LLM either. Model Context Protocol (MCP) gives agents a standard way to discover and call tools. But someone still has to write an MCP server for every legacy system by hand, and get the security right each time. Every new server means new code for validation, logging, and permissions. That repetition is where projects stall: the agent demo works in a week, and the security review takes months. What legacy2mcp Does legacy2mcp is an Apache 2.0 Python toolkit that auto-generates MCP servers from legacy systems. Instead of hand-writing tool definitions, you point the toolkit at the system's own description of itself and get a working, policy-enforced MCP server. Today: a WSDL/SOAP adapter. Point it at a WSDL, and it exposes the operations as MCP tools.Planned: database adapters (Postgres, SQL Server, Oracle) and queue adapters (Kafka, RabbitMQ, SQS).Extensible: a small adapter interface, so new legacy-system types can be added without touching the core. Design Principles Letting an LLM call into a system of record is a risk, so the defaults are conservative. Safe by default. No arbitrary SQL and no write operations unless you explicitly opt in. An agent can't do anything you didn't deliberately allow, and a misconfigured prompt can't turn a read-only tool into a destructive one.Validated inputs. Every tool's input is checked against a JSON Schema before it reaches the legacy system. Malformed or unexpected arguments are rejected early, which protects fragile older systems that were never designed to handle LLM-generated input.Audit logging. Every call is recorded: who, what, with which arguments. When someone asks what the agent did last Tuesday, you have an answer.Adapter interface. One contract for all backends, so safety behavior stays consistent whether the source is a SOAP service, a database, or a queue. How It Works Plain Text WSDL / DB / Queue -> Adapter -> Tool definitions + JSON Schemas -> MCP server -> Agent (safety policy + audit log applied here) The adapter introspects the source. For SOAP, that means reading the WSDL.It generates MCP tool definitions and JSON Schemas from the discovered operations and types, so the agent sees clear names, descriptions, and typed parameters.The generated server applies the safety policy and audit logging on every call. These aren't optional add-ons; they sit in the call path.Any MCP-compatible agent can discover and use the tools, with no custom client code. Quick Start Shell # TODO: install command pip install legacy2mcp # TODO: generate a server from a WSDL legacy2mcp generate --wsdl https://example.com/service?wsdl (Replace with the actual CLI usage from the README.) Example: An Agent Calling a SOAP Service Picture a claims system that has exposed a SOAP operation for looking up claim status for a decade. An agent asked, "What's the status of claim 12345?" discovers the generated get_claim_status tool, sees from its schema that it needs a claim ID, and calls it. The server validates the input, forwards the request to the SOAP service, logs the call, and returns the result. Nobody wrote an MCP server by hand, and the agent can't reach any operation that wasn't exposed. Plain Text TODO: short transcript or screenshot of an agent discovering and calling a generated tool. Why Not Just Write the MCP Server by Hand? You can, and for one service it's fine. At ten services, you're copying the same validation, logging, and permission logic ten times, and each copy can drift. One team tightens its input checks, another forgets to log arguments, a third leaves a write operation exposed. Generating servers from the source definition keeps behavior uniform and makes safety a property of the toolkit rather than of each developer's diligence. It also means that when the legacy service changes, you regenerate instead of re-editing by hand. Roadmap DB adapters (Postgres, SQL Server, Oracle), read-only by defaultQueue adapters (Kafka, RabbitMQ, SQS)More granular per-tool permission policiesCommunity-contributed adapters Get Involved Repo: https://github.com/bvenkata/legacy2mcpStar it, try it on a service you have, and open issues for the adapters you need most.Adapter contributions are welcome, especially for systems that are painful to integrate today. Takeaway Agents don't need legacy systems replaced. They need a safe, standard way in. legacy2mcp aims to make that a generated artifact, not a custom project.

By Balaji Venkatasubramaniyar

CORE

PWA on iOS, Part 2: HTTPS, Manifest and Service Worker

This is Part 2 of the PWA on iOS series, and it’s the most practical one. There will be almost no history or corporate politics here, but there is groundwork you need for the rest of the series to make sense. You’ll learn why a PWA works on a laptop but not on an iPhone, and how HTTPS, the manifest, the Service Worker lifecycle, and two caches decide that. The typical story starts at a desk. The frontend runs on a laptop, the phone is on the same Wi-Fi network, Safari on the iPhone has an address like http://192.168.1.20:5173 open, and everything looks fine: the interface loads, the buttons work, and our PWA's icon sits next to the others on the phone's Home Screen. Then someone turns on airplane mode, taps the icon, and instead of the app, sees a message saying the page can't be opened. Meanwhile, everything is green in Chrome on the laptop: the worker is registered, the cache is full, and the team spends the first half hour hunting for a bug in the manifest, even though the phone has never seen that worker. No official tool will give you the answer, either. In iOS 26, Apple removed the last formal installation requirements, and any site added to the Home Screen now opens as a web app by default. The WebKit blog put it bluntly: "Put simply, Safari now has no requirements at all for installation." Google, for its part, stopped using Lighthouse as a PWA checker. Until Lighthouse 12, it had a separate PWA category: a set of automated checks that produced a score and showed whether an app met the criteria of an installable progressive web app. Lighthouse 12.0, released in April 2024, removed the PWA category, following Chrome's relaxation of its installability criteria. Even earlier, the Chrome blog acknowledged that the service worker check wasn't producing the intended result: sites were adding a worker with an empty fetch handler just to pass it. The green "this is a PWA" checkmark is gone on both platforms, and there is now only one way to tell whether you have an app or just a website with an icon: pick up a phone and test. Behind every web app icon on your phone's screen are three things: HTTPS, a service worker, and a manifest file. People usually think of HTTPS as the padlock in the address bar, but here it's needed for something else: without it, the browser simply won't agree to run code that acts as the bridge between the page and the network. The manifest is the simplest of the three: a small JSON file linked from <head> that records the app's name, start page, display mode, scope and icon. And the most interesting layer, the Service Worker, is a separate script the browser runs alongside the page but without access to its interface. The app's network requests pass through it, and it decides whether to go to the network, serve a saved copy, or show a fallback page when there's no connection. Why a Service Worker Needs HTTPS and a Secure Context A service worker is dangerous precisely because it's long-lived: once installed, it keeps running on later visits to the site and can intercept page requests within its scope. That's why registration is allowed only over HTTPS. The worker effectively sits between the web app, the browser, and the network, and it can alter requests and responses. If sw.js could be registered over plain HTTP, an attacker on public Wi-Fi could swap that file just once. A malicious worker would then stay in the user's browser and could keep serving altered HTML and JavaScript on later visits, harvest API data, or redirect requests to another server. Moving the site to HTTPS afterward wouldn't be an instant fix either: requests are handled first by the installed service worker, which can return a forged response from its cache or change how the network is accessed. The risk of this kind of attack is one of the reasons browsers require HTTPS for service workers. That's where the secure context requirement comes from. A page counts as secure if it was delivered over HTTPS or from an address the browser trusts on its own: localhost and its subdomains, the 127.0.0.0/8 range and ::1. Home or office network addresses like 192.168.x.x and 10.x.x.x aren't on that list. To the browser, that's plain HTTP, however much you trust your router. The sneakiest part is the way the browser says no. In the specification, navigator.serviceWorker is marked as available only in secure contexts, and on an HTTP page it simply isn't there. The check 'serviceWorker' in navigator, which almost every registration example starts with, quietly returns false, the code skips registration, and not a single error shows up in the console. That explains the story from the start of this article: on the phone, the worker didn't fail with an error. It never even tried to start. Let’s Encrypt and Free HTTPS Certificates for PWAs About ten years ago, many teams heard "use HTTPS" as "spend money and deal with hassle." You bought a certificate from a certificate authority and renewed it by hand, and for an internal tool or side project, that meant an extra line in the budget and an extra date on the calendar that everyone forgot until the morning the site started scaring visitors with a red warning. Let's Encrypt, a nonprofit certificate authority run by ISRG, changed that. On December 3, 2015, it entered public beta: invitations were no longer required, certificates were free, and the announcement stated the goal plainly: "We want HTTPS to be the default." Cost was only half of the problem. The server proves on its own that it controls the domain and obtains a certificate on its own via the ACME protocol, with no human involved and no correspondence with a certificate authority, and in March 2019 the IETF made that protocol official as RFC 8555. Let's Encrypt certificates are valid for 90 days by default, and such a short lifetime is practical only because there's no need to renew them by hand. These days, getting HTTPS for a worker takes about one toggle in a hosting panel, or a Certbot container next to nginx. I think this quietly ended many of the PWA arguments on internal projects: the money argument is gone, and what's left is the laziness argument. Testing a PWA on iPhone With HTTPS, Tunnels and mkcert On the laptop itself everything works, because http://localhost is on the trusted list and browsers on macOS will happily register a worker with no certificate at all. Trouble starts when you pick up the phone. On an iPhone, localhost means the iPhone itself. The laptop can only be reached by its network address, and to Safari that address is ordinary insecure HTTP. There are two workable paths: route traffic through a tunnel with a real certificate, or teach the phone to trust your own. The tunnel is easier. The command cloudflared tunnel --url http://localhost:5173 brings up what Cloudflare calls a quick tunnel. You get a random subdomain on trycloudflare.com, requests to it travel through Cloudflare's network to your local server, and the address has a public certificate, so the iPhone trusts it with no setup. ngrok http 5173 does the same through its own service. Cloudflare states plainly that quick tunnels are intended only for development and testing, and points to a different tunnel type for production. The tunnel has a catch that people only notice a week later. The random address changes on every launch, and a new subdomain means a new origin to the browser (the scheme, host, and port by which a browser tells one site from another). To an iPhone, every such address becomes a separate app: yesterday's icon points to a subdomain that no longer exists, today's starts with empty storage and no worker, and a row of identical icons for your PWA gradually grows on the test phone. If you plan to test an update on top of an old version, set up a permanent address: a named tunnel or your own domain. The second path goes through a local certificate authority, and the easiest way to set one up is with mkcert. The command mkcert -install creates a root certificate on your Mac and adds it to the system trust store, and mkcert 192.168.1.20 localhost issues a certificate for the dev server. The Mac is now happy. The phone isn't yet. You need to move the rootCA.pem file from the folder shown by mkcert -CAROOT to the iPhone via AirDrop, email, or a local web server, and install it as a profile. Then comes a step almost everyone skips: open Settings > General > About > Certificate Trust Settings and turn on your certificate under "Enable Full Trust for Root Certificates." Apple's instructions say explicitly that for profiles downloaded from a website or sent by email, trust for SSL/TLS has to be enabled manually, and without that switch the installed profile changes nothing. This setup has a price, and the mkcert README names it plainly: the rootCA-key.pem file gives full power to intercept secure requests from your machine, and it must never be shared. A root certificate on the phone works in the same direction. As long as it's trusted, whoever holds the key can issue a certificate for any domain, and the iPhone will accept it. I keep such profiles only on test devices and remove them when the test setup is no longer needed. Clicking through a certificate warning doesn't count as trust. A page with a self-signed certificate will open if you push through the warning screen, but registering a worker on it usually ends in an error, and until someone surfaces that error on screen, developers keep fixing the manifest. How to Register a Service Worker and Read Registration Errors Once you have HTTPS, registration itself looks modest, and almost everything interesting in it happens in the catch branch: // app.js if ('serviceWorker' in navigator) { window.addEventListener('load', async () => { try { const reg = await navigator.serviceWorker.register('/sw.js', { scope: '/', updateViaCache: 'none', }); console.info('[sw] scope:', reg.scope); } catch (error) { console.error('[sw] registration failed:', error); } }); } else { console.warn('[sw] navigator.serviceWorker is missing: HTTP or an old browser'); } The else branch handles that silent refusal on HTTP: better one warning line in the console than half an hour of guessing. I hook registration to load so that downloading and installing the worker doesn't compete with the first render for the network, which is noticeable on slow mobile connections. The scope parameter sets the scope, meaning the prefix of addresses whose requests the worker will intercept. By default it equals the directory where the script itself lives. A worker at /assets/sw.js can't take control of / unless the server sends a Service-Worker-Allowed: / header along with the script, so it's simpler to keep sw.js in the root. We'll come back to updateViaCache in the section on caches; it has a surprising wrinkle. Don't leave the catch branch empty or skip it. Its errors usually tell a specific story that you won't see on a laptop. The specification forbids redirects when loading a worker script, so an sw.js that the server redirects to the www version of the domain, to a CDN or to a login page won't register at all. If the server serves the script with the wrong MIME type (the content type from the Content-Type header), registration fails with a SecurityError. This happens most often in SPAs, single-page apps where JavaScript handles all navigation and the server answers any unknown path with index.html: instead of a script, the browser gets HTML. One more small thing helps with debugging. The browser adds a Service-Worker: script header to the request for a worker script, so such requests are easy to find in logs or to single out with a separate rule on a CDN, the network of caching servers in front of your backend. Installing is up to the user. iOS has no beforeinstallprompt event and no way to trigger the system install dialog from code. The person opens the Share menu and picks Add to Home Screen, and in iOS 26 sees a toggle there, "Open as Web App," which is on by default. What happens in the system at that moment, where the icon and splash screen come from and what MDM has to do with it, we'll cover in Part 4 of this series. Web App Manifest Fields Safari Supports on iOS The manifest is hooked up with one line in <head>: <link rel="manifest" href="/manifest.webmanifest">. The file can have any name, and manifest.json works too, because the browser follows the link and doesn't look for a file by name. The right response type is application/manifest+json. It's registered with IANA together with the .webmanifest extension, and the specification recommends serving the manifest with it. Plain application/json is accepted in practice as well, but I'd go with the first so the next person on the team doesn't have to guess what's behind that address. Safari 15.4 brought a small but welcome change: the browser now downloads the manifest along with the page, whereas before it did so only at the moment the user added the site to the Home Screen. The manifest request is now visible in Web Inspector right away, and you don't have to install the app every time just to check. Here's the file for our imaginary app: { "id": "/", "name": "Field Journal", "short_name": "Journal", "start_url": "/?source=home-screen", "scope": "/", "display": "standalone", "theme_color": "#0d1b2a", "background_color": "#f4f7f9", "icons": [ { "src": "/icons/icon-192.png", "sizes": "192x192", "type": "image/png" }, { "src": "/icons/icon-512.png", "sizes": "512x512", "type": "image/png" } ] } Going by MDN's compatibility tables, Safari on iOS reads noticeably less from this file than Chrome does. It has understood name, short_name, start_url, scope and display since iOS 11.3, the very beginning of manifest support. theme_color arrived in iOS 15, icons in 15.4, and id in 16.4. Icons come with a condition described in the Safari 15.4 release notes: Safari takes them from the manifest only if the HTML has no apple-touch-icon and the icon either has no purpose or has any. WebKit added the id field in the same release as Web Push, so that notifications and badges would work for several installs of the same app, for example a work copy and a personal copy with different accounts. According to MDN, the iPhone ignores most of the rest. orientation, description and share_target aren't supported on iOS, and Safari understands shortcuts only on macOS, starting with version 17.4. Of the display values, standalone and browser work on iPhone, while fullscreen and minimal-ui aren't supported. The exception is background_color: Safari uses it, together with name, when creating the web app's launch screen. You don't need to remove the other fields from a shared manifest. They're useful on Android and Chrome, and they don't bother Safari. But don't count on them for anything critical on iPhone. short_name is the label under the icon, and it's meant to survive the cramped Home Screen: "Field Journal" might not fit there, "Journal" will. start_url doesn't have to point to the site root. A parameter like ?source=home-screen is handy for analytics, so the server can see launches from the icon, but don't use it to check authorization, because anyone can type such an address. Relative paths inside the manifest are resolved against the manifest's own address, and a start_url from a different origin must, per the specification, be ignored, with the current page's address used instead. id comes with a trap that's easy to miss. If the field is missing, the app's identity per the specification is start_url, and a harmless tweak of an analytics parameter can look to the browser like the appearance of a different app. I set id explicitly in the first release and never touch it afterward. "id": "/" is resolved against the origin and becomes the site root, a stable name that survives any experiments with the start page. display: "standalone" asks for a separate window without an address bar, and theme_color sets the color of the system elements around the page. Both fields are requests to the browser and have nothing to do with your interface's CSS. The browser will carry them out in its own way. Why Apple Meta Tags Still Matter in 2026 Apple taught the iPhone to open sites without browser chrome long before the W3C took up the manifest, and it did so with its own meta tags. They haven't gone anywhere, and I still keep both sets in <head>: <link rel="manifest" href="/manifest.webmanifest"> <meta name="theme-color" content="#0d1b2a" media="(prefers-color-scheme: dark)"> <meta name="theme-color" content="#f4f7f9" media="(prefers-color-scheme: light)"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> <link rel="apple-touch-icon" href="/icons/apple-touch-icon-180.png"> apple-mobile-web-app-capable historically asked iOS to launch the site from the icon without Safari's interface. In iOS 26 that became the default behavior for any site, but duplicating it is still useful if some of your users are on older iOS versions or on corporate devices that are rarely updated. apple-mobile-web-app-status-bar-style controls the status bar, the one with the clock and battery, and Apple's old reference describes three values for it. With default and black, the page starts below the status bar, while with black-translucent it takes the whole screen and the status bar partly overlaps it. It looks great right up until the first header slides under the clock, so check your insets with env(safe-area-inset-top) as soon as you use that value. The most useful thing in this bunch, I think, is apple-touch-icon. Since Safari 15.4 it explicitly takes priority over manifest icons, and WebKit explains this as a wish to preserve the behavior of sites that have a separate icon drawn for iOS. Use an opaque 180×180 PNG: a transparent background was historically handled badly on iOS, and on older versions the alpha channel turned into a black backing. If the tag isn't there at all, iOS, according to Apple's documentation, looks for files with the apple-touch-icon prefix in the site root on its own. That's where the endless 404s for /apple-touch-icon.png in your server logs come from, the ones nobody can usually explain. Safari won't use an icon whose only purpose in the manifest is "maskable", since that value comes from Android's icon conventions. theme-color arrived in Safari 15 and turned out to be more noticeable than expected. In Safari on iOS 15 it colors the status bar and the area visible when you scroll past the edge of the page, and with a media attribute you can set different colors for light and dark themes, as in the example above. After that the story gets murkier: MDN's tables note that as of Safari 26, the theme-color color is used only in installed web apps. I didn't find that wording in Apple's release notes, so check on whichever iOS version you have in hand, and don't promise your designer a colored header in a Safari tab. Service Worker Lifecycle and skipWaiting on iPhone The most persistent misconception about workers is that a worker runs in the background all the time, like a daemon on a server. The browser starts it for an event (a page request, a push notification, a message from a tab) and may stop it right after the event is handled. A worker has no DOM and no window, and global variables live only until the next stop, so a "sync in progress" flag stored in a variable will be empty after the worker wakes up. Anything that has to survive the worker being stopped should be written to Cache Storage or IndexedDB. A service worker's lifecycle is designed so that a new version doesn't break pages that are already open. After register(), the browser downloads the script and compares it byte for byte with the installed one, and if there's no difference, that's the end of it. A new script gets an install event, in which it usually puts the app shell into the cache, meaning the HTML, styles and scripts without which the interface can't be drawn. Then it waits in the waiting state while the old version serves open tabs, and only when none are left does it get activate, clean up old caches and start receiving fetch events from pages in its scope. On the first run there's one more confusing detail: the page that registered the worker isn't controlled by it until the next navigation, unless the worker calls clients.claim(). // sw.js const VERSION = 'shell-v7'; const SHELL = ['/', '/assets/app.8d91.js', '/assets/app.3c2e.css']; self.addEventListener('install', (event) => { event.waitUntil(caches.open(VERSION).then((cache) => cache.addAll(SHELL))); }); self.addEventListener('activate', (event) => { event.waitUntil( caches.keys().then((keys) => Promise.all(keys.filter((key) => key !== VERSION).map((key) => caches.delete(key))) ) ); }); self.addEventListener('message', (event) => { if (event.data?.type === 'skip-waiting') self.skipWaiting(); }); event.waitUntil() looks like a housekeeping detail, but without it the browser doesn't know the event is still in progress and is free to stop the worker before the cache is written. On the iPhone, where background processes inevitably get put to sleep, that's a very practical risk. cache.addAll() has a quirk of its own: it's atomic, and if even one file in the list returns a 404, the whole installation fails. The new worker then simply doesn't start, users stay on the old version, and your logs are empty, because the error happened on the phone. I don't write the list of app shell files (everything the app can't launch without) by hand. The build tool generates it from the real files with hashes in their names, so it can't contain an address that doesn't exist on the server. Optional images I cache in a separate step, so that one unavailable file can't fail the installation. The waiting state is often mistaken for a bug, though it's protection against shooting yourself in the foot. If a new worker takes over a tab where old JavaScript is already running, you get a mix of the old interface with the new cache or a new API contract. Calling self.skipWaiting() on every install, together with clients.claim() in activate, removes the waiting state entirely, and for a static site or strictly backward-compatible releases that's a fine choice. For an app with an IndexedDB migration or a changing offline data format, I don't do that, and in the code above the worker skips waiting only on a message from the page. The page, noticing registration.waiting, shows a "New version available" banner, sends reg.waiting.postMessage({ type: 'skip-waiting' }) on tap, and reloads in the controllerchange event handler. In my experience, on iOS this scheme noticeably reduces the number of mysterious bug reports from people who come back to the app after a couple of days in the background. Background work on the iPhone is more modest than tutorials promise. Background Sync, Periodic Background Sync and Background Fetch aren't available in Safari or other browsers on iOS. So a photo upload queue that will "send the files by itself ten minutes from now, while the phone sits in your pocket" won't work. A Service Worker wakes up only when the app is open or when an event arrives that iOS is willing to deliver, such as a push. Send what has accumulated when the app returns to the foreground, and show the user that an operation is waiting for a connection. We'll cover updates, push, and IndexedDB storage in detail in Part 5 of this series. For now, remember this: a worker wakes up briefly and remembers nothing between wake-ups. Service Worker Cache Strategies and Opaque Responses In the fetch handler, the worker decides the fate of every request, and the most common mistake here is answering everything from the cache because that's what the tutorial did. For a hashed /assets/app.8d91.js, the Cache First strategy is nearly perfect: a file with that name will never change, and it doesn't need the network. For GET /api/orders, the same strategy will show yesterday's order as today's, and the user won't even know. In practice, there are three strategies, and each negotiates freshness with the user in its own way. The simplest, Cache First, returns the copy and goes to the network only on a miss. It belongs on versioned files, fonts, and icons. For HTML documents and data where freshness matters more than milliseconds, I use Network First: network first, and the saved response only when the network is down. The third strategy, Stale-While-Revalidate, is trickier. The user immediately gets the saved copy while the worker requests a fresh version in the meantime and puts it in the cache for next time. The response arrives quickly but may lag by one visit. For avatars, configs and feeds, that's usually fine. In the fetch handler, the strategy is chosen by what kind of data it is: self.addEventListener('fetch', (event) => { const { request } = event; const url = new URL(request.url); if (request.method !== 'GET' || url.origin !== self.location.origin) return; if (request.mode === 'navigate') { event.respondWith(networkFirst(request, '/')); } else if (url.pathname.startsWith('/assets/')) { event.respondWith(cacheFirst(request)); } else if (url.pathname.startsWith('/api/feed')) { event.respondWith(staleWhileRevalidate(event)); } }); async function cacheFirst(request) { return (await caches.match(request)) ?? fetch(request); } async function networkFirst(request, fallbackUrl) { const cache = await caches.open(VERSION); try { const response = await fetch(request); if (response.ok) cache.put(request, response.clone()); return response; } catch { return (await cache.match(request)) ?? caches.match(fallbackUrl); } } async function staleWhileRevalidate(event) { const cache = await caches.open(VERSION); const cached = await cache.match(event.request); const fresh = fetch(event.request).then((response) => { if (response.ok) cache.put(event.request, response.clone()); return response; }); event.waitUntil(fresh.catch(() => {})); return cached ?? fresh; } Requests for which the worker didn't call respondWith() go to the network as usual, and I consider that the right default: anything you haven't explicitly classified should behave as if there were no worker. For the same reason, the first line filters out other hosts and everything except GET. For a third-party CDN without CORS, the browser gives the worker what's called an opaque response. Its status is 0, the body and headers are unavailable to the script, and response.ok is false. Code that checks ok won't cache it, and images that "seemed to be cached" will vanish when you're offline. You can put such a response in with put(), but then you won't know whether you've just saved a 404 for a month. Caching third-party resources makes sense only if the CDN sends Access-Control-Allow-Origin and the request goes in cors mode. Otherwise it's easier to proxy the files through your own domain or accept that they load only online. In staleWhileRevalidate, note event.waitUntil(). Without it, the cached response goes to the page, the browser decides the work is done, and it may stop the worker before the fresh version lands in the cache. Tutorials often skip that line, though on iPhone it's exactly what separates a real staleWhileRevalidate from an imitation. I handle page navigations separately, by request.mode === 'navigate'. A SPA opens addresses like /orders/42 that aren't in the cache as files, and if the offline fallback covers only /index.html, opening such a link directly without a connection ends in an error. For a SPA it's reasonable to return the cached shell /, as in the code above, and let the app decide which data to show. A server-rendered site can't do this: if you answer every address with the same shell, the worker will break link previews, diagnostics and SEO, and the team will then blame Safari. There, the fallback should be a separate page like /offline.html. credentials is a trap that can go unnoticed for a long time. fetch(event.request) inherits the settings of the original request, while a new Request assembled by hand sends cookies only to its own origin by default. As long as the API lives on the same domain, there's no difference. When the API moves to api.example.com while the app stays on app.example.com, requests rebuilt in the worker suddenly arrive without cookies, the user looks logged out only on the screens that passed through the worker, and reproducing it on desktop doesn't always work the first time. Responses with personal data are better kept out of the Cache API unless you've thought through sign-out. The cache is tied to the origin and knows nothing about the session, so on sign-out, delete the entries holding the user's data, and don't put responses that depend on a cookie or token into a shared cache out of habit. POST can't be put there at all, since the Cache API accepts only GET, and offline writes have to be built as a queue of operations in IndexedDB with an idempotency key and an explicit sync status. While the worker starts after a long idle period, navigation waits for it, and on iPhone that's sometimes visible as an extra empty frame. Since Safari 15.4 there's Navigation Preload: the browser starts the network request for the page in parallel with starting the worker, and in the handler you pick up the ready response from event.preloadResponse. It's enabled by calling self.registration.navigationPreload.enable() in activate, and it makes sense when documents already go through Network First. HTTP Cache vs Cache API and How sw.js Updates There are actually two caches in the browser, and they know nothing about each other. The first, the HTTP cache, is always there. The browser itself stores server responses in it and decides on its own when they've gone stale, going by the Cache-Control, ETag and Vary headers. You control it only indirectly, through those headers. It can even answer a fetch() from a worker without waiting for the network, and it can evict an entry when it needs space. You can't rely on it offline. The second, the Cache API, exists specifically for the worker. It's storage that you manage by hand: what to put in, under which key, when to delete it, and what to return instead of the network. That's why offline is built on it: the files needed for launch sit where you yourself put them. The Cache API doesn't read headers. Even if the server allowed a file to be cached for a year via max-age=31536000, it won't get into the Cache API on its own, because only your code puts things there. And conversely, an entry in the Cache API won't be updated via ETag until you write that check yourself. The classic cache problem looks like this. A bundle is named /app.js with no hash, the worker put it in the Cache API at install, you shipped a fix, and caches.match('/app.js') keeps returning the old version: the key is the same, after all. The fix is simple: give the file a hashed name, such as /assets/app.8f3a.js, plus the header Cache-Control: public, max-age=31536000, immutable. Then you won't have to ask users to "clear the cache." Nobody deletes the old URL; people just stop requesting it. Hashed names have a flip side, and it hits SPAs especially hard. After a deploy, old files disappear from the server, while the user still has the previous version's shell open, and that shell asks for /assets/app.8d91.js. If the server answers any unknown path with index.html and a 200 status, the browser gets HTML instead of a script, fails while parsing it, and the user sees a white screen. Worse, a strategy that caches responses with a 200 status will save that HTML under the name of a JavaScript file. So I keep old assets on the server for at least one release and don't let the SPA fallback into /assets/: a missing file there must return 404, and nothing else. On nginx you can set it up like this: location = /sw.js { add_header Cache-Control "no-cache"; } location = /manifest.webmanifest { types { } default_type application/manifest+json; add_header Cache-Control "no-cache"; } location /assets/ { add_header Cache-Control "public, max-age=31536000, immutable"; try_files $uri =404; } location / { try_files $uri $uri/ /index.html; } sw.js itself isn't part of the hashing scheme. It needs no-cache so that every check reaches the server, including through CDNs and proxies. This is where the promised detail about updateViaCache comes in. Per the specification, the default is 'imports', and with it the worker's main script is requested bypassing the HTTP cache during an update check anyway, while only files pulled in with importScripts() go through the cache. The value 'none' extends the same rule to imports, and 'all' brings the HTTP cache back for everything. According to MDN, Safari has supported this setting since version 11.1. I set 'none' in projects where several teams touch the cache infrastructure: it does no harm, and it insures against a single forgotten importScripts with a long max-age. The specification describes quite strictly when a browser checks for an update. A check is triggered on navigation to a page within the worker's scope, on functional events such as push if more than a day has passed since the last check, and on a manual call to registration.update(). If the last check was more than 86,400 seconds ago, the request for the script bypasses the HTTP cache in any case, even with updateViaCache: 'all'. For a SPA there's a trap hidden here: transitions through history.pushState don't count as navigation, and an app that's been open on an iPhone since morning may never ask the server about a new sw.js until evening. A "Check for updates" button in settings that calls update() and offers a restart when a waiting worker appears helps more than hoping for the daily timer. The browser will dutifully go to the network, but between it and the server there may be a CDN that someone once told to cache everything with a .js extension for a week. Then half of the iPhones sit for a week on a worker that no longer exists on the server, so check the rule for sw.js on the CDN separately, preferably with a request from a phone. Service Worker Scope Mistakes Including the Trailing Slash The specification compares scope as a string prefix, with no understanding of directories. A worker or manifest with a scope of /app will also capture /app-v2 and /application, and a new storefront will suddenly live on the old dashboard's cache. Write /app/ with a trailing slash, and after registering, look at registration.scope in the console. If it isn't the prefix you expected, you can stop debugging cache strategies. One origin can have several registrations with different scopes, and each page is controlled by the one whose scope matches its address and is longer than the others. That sounds convenient until the landing page has registered a worker on /, the dashboard has its own on /app/, and both start answering shared resources in their own ways. I've seen a test environment where, after such a story, the dashboard behaved differently depending on which page the person first landed on, and sorting it out took longer than writing both workers. It's calmer when an origin has one owner: one worker with routing inside fetch, or separate subdomains if the teams really can't share a file. A subdomain means a different origin, different cookies and a different cache, so it's a real boundary, not a free optimization. There's an API mix-up related to this. navigator.serviceWorker.getRegistration() takes a page address (scope has nothing to do with it) and returns the registration that controls that address, and without an argument it uses the current page's address. If you're looking for the dashboard's registration from the home page, pass an address inside the dashboard. Otherwise you'll get the landing page's registration and wonder for a long time why it has no waiting. Then there's the habit of registering sw.js?v=3 to "reset the cache." Jake Archibald, in an article on the worker lifecycle, explains where that ends up: the HTML of the page, which carries the new address, is itself sitting in the old worker's cache, and the user simply won't see the new address. His wording is worth quoting: "You've put yourself in a position where you need to update the service worker in order to update the service worker." Keep one permanent address, /sw.js, and leave freshness to the headers and to the byte-by-byte comparison that the browser does without you. The manifest's scope has its own mix-up, and it's often explained wrongly. Per the specification, if scope isn't specified, it's computed from start_url: the start address is taken without the file name, query and fragment, and the directory where the manifest lies plays no role here. The manifest can sit in the site root, but with "start_url": "/warehouse/index.html" and no scope, it gets a scope of /warehouse/, and a link to /account takes the person beyond the app's boundary. There the page, per the specification, stops being considered part of the app and is shown with browser elements again. If start_url doesn't lie inside an explicitly specified scope, the browser silently discards that scope and falls back to the default, so a typo in one field changes the boundaries of the whole app with no warning. The way out is simple: an explicit "scope": "/" where the account should be part of the app, or a conscious decision that it lives separately. The second, in my observation, almost nobody chooses. One last pitfall: manifest.webmanifest and sw.js must not hide behind authorization. If the server redirects an unauthenticated request to a login page, the worker won't register, because redirects are forbidden. The manifest case is funnier: Safari gets the login page's HTML in its place, and the icon on the Home Screen may well end up named "Sign In." Both files are public and contain no personal data, so serve them as static files, bypassing the app's router. In the Network tab this takes five seconds to check. If a text/html document arrived in place of the manifest, the installed app will take its name from the page's <title>. First Launch From the Home Screen Icon on iPhone The first visit happens entirely in a Safari tab. The page arrives over HTTPS, Safari downloads the manifest along with it, the script registers the worker, and the worker puts the shell in the cache and activates according to your policy. Then the person adds the site to the Home Screen, and here many expect the app to take along everything the worker managed to store. It won't. A web app on the Home Screen has its own storage, separate from Safari's, with its own cookies, localStorage, IndexedDB, and Cache Storage, so someone who has just signed in within a tab will see the login screen again in the app. On first launch from the icon, the worker registers again, now inside the app, and if there's no connection at that moment, there's nothing to show. That’s the second cause of the broken first launch from the story at the start of the article: even with correct HTTPS, the first launch of an installed app has to happen online. On the second launch, the icon opens start_url, and the already-active worker intercepts requests within its scope. If the network is gone, a well-thought-out strategy returns the shell and clearly shows that the data is offline and how fresh it is. If someone put index.html in the cache a year ago under Cache First and forgot about it, the user gets an interface that looks like it works and is hopelessly out of date, and an honest "data is unavailable right now" page would have been better than that surprise. Storage is also subject to a quota on the user's phone. The popular rule "iOS wipes everything for a PWA after seven days of non-use" wrongly stretches to installed apps a restriction that WebKit introduced in 2020 for site storage in Safari. In the same post, WebKit wrote that web apps on the Home Screen have their own days-of-use counter and that, under normal use, their data shouldn't be deleted. The system can still evict data when space runs short, so sync critical data to the server, show queue status, and test a cold start after a long idle period. How to Test a PWA on a Real iPhone Test on a real device, and test the installed app specifically: a Safari tab and the icon on the Home Screen use different storage, so success in one tells you nothing about the other. I run the same scenario every time: install on a clean phone, open from the icon with the network on, turn on airplane mode, fully close the app (swipe it up in the app switcher), and open it again. Then do the same thing on top of an old version to see how the new worker comes up, and separately, sign out, after which no one else's data should remain in the cache. One successful page reload on a laptop proves absolutely nothing. For debugging on an iPhone, there's Web Inspector. On the phone, it's turned on in Safari's settings, under the advanced options, after which the device connects to the Mac with a cable and its pages appear in Safari's Develop menu on the Mac. The console, network, and storage are all there, but debugging workers is, in places, less pleasant than in Chrome DevTools. I don't rely on the inspector alone, and I log what I'll need later to analyze a bug report: the worker version, the result of cleaning up old caches, and the cases when the offline fallback kicked in. A line like console.info('[sw]', VERSION, 'activate') and the shell version number somewhere on a settings screen save hours of back-and-forth with a user who has no idea which version they have. There's one scenario where none of the above helps. A link to your app sent in a messenger will open not in the installed web app, and not necessarily in Safari, but in the messenger's built-in browser. It has its own storage and its own restrictions, so the cache you've built is unavailable there. Why the same address behaves differently on iPhone in Safari, in a built-in browser, and on the Home Screen, and what WebKit has to do with it, we'll cover in Part 3 of this series.

By Konstantin Shkurko

LangChainGo vs Genkit Go: Where Genkit Shines

There are two realistic choices for building AI services in Go today: LangChainGo, the Go port of the Python framework everyone knows, and Genkit Go, Google's open-source Gen AI framework. LangChainGo is still the first result most Go developers find. It has 9,677 stars and the biggest provider catalog in the ecosystem. It is also eight months without a commit, eleven months without a release, and still on 0.1.x after three and a half years. This article compares them the way that actually matters: by building the same production service in both and looking at what each one costs you. Not a feature checklist, a real application with real requirements. If you have already decided and want the migration mechanics, that is the companion article: How to migrate from LangChainGo to Genkit Go. The State of Both Projects LangChainGoGenkit GoLatest releasev0.1.14, October 2025v1.13.1Last commit to mainJanuary 2026activeAge and version3.5 years, still 0.1.xGA since v1.0Open issues242Open pull requests171, oldest from June 2023Stars9,677 LangChainGo is not archived, and "dead" overstates it. But eleven months without a release is long enough for the ecosystem to move underneath a library, and it has. What Is Broken in LangChainGo Today Three concrete things, all of which affect a service in production right now. The quickstart returns a 404. DefaultOptions() hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) fails on the first call. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, whose README states support ended on November 30, 2025. Streaming is broken on every current Gemini model. gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash all fail the same way: Plain Text error in stream mode: invalid character ']' looking for beginning of value The OpenAI provider streams fine, so this is the Google provider on the legacy SDK, not the library everywhere. There is a fourth consequence that follows from the third, and it is the nastiest one. LangChainGo installs a streaming function automatically whenever you attach a callbacks handler, in chains/options.go and again in agents/mrkl.go. Since streaming is broken, turning on observability breaks your Gemini agents: Plain Text WITHOUT callbacks handler: err=<nil> out="The current year is 2026." WITH callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value Feature by Feature CapabilityLangChainGoGenkit GoStructured outputFormat instructions in the prompt, parse the text backgenkit.GenerateData[T], real response schemaToolsCall(ctx, string) (string, error), no schemagenkit.DefineTool[In, Out], typed with JSON schemaAgentsReAct, driven by parsing Action: linesNative function calling, plus an agent primitive with sessions (experimental)CompositionChain interfaces over map[string]anyTyped Go functions (DefineFlow)RetryWrite it yourselfmiddleware.RetryCross-vendor fallbackWrite it yourselfmiddleware.FallbackObservability17 observe-only callback methodsBuilt-in tracing plus a Developer UIOpenTelemetryNoneBuilt inLocal vector storeNone, all 15 stores need a serverlocalvec, file-backedStreaming on GeminiBrokenWorksHTTP endpointnet/http boilerplategenkit.Handler(flow) The single most important row is retry and fallback, and the reason is architectural rather than cosmetic. A Genkit middleware wraps a call and receives a next(), so it can decide to call again, or call something else. LangChainGo's callbacks.Handler returns nothing from every method. You can watch a call, you cannot change it. That missing seam is why every resilience concern ends up in your application code. What We Are Building A support-ticket triage service. Every requirement here is something a real team asks for: Retrieve the relevant policy documents for an incoming ticket.Let the model call a tool to act on it.Return a typed struct, not free text.Retry transient model failures.Survive a vendor outage by falling back to a different provider.Trace every step, so somebody can debug it at 3 am. The Genkit Go Version The whole service, in one file: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{}, )) // RAG store. embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001") store, retriever, err := localvec.DefineRetriever(g, "policy", localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil) localvec.Index(ctx, docs, store) // A tool the model can call. refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.", func(ctx *ai.ToolContext, a struct { OrderID string `json:"order_id"` AmountCents int `json:"amount_cents"` }) (string, error) { return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil }) // The flow. triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) { found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket)) if err != nil { return nil, err } out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)), ai.WithDocs(found.Documents...), ai.WithTools(refund), ai.WithPrompt("Triage this ticket using the policy documents, and act on it: %s", ticket), ai.WithUse( &middleware.Retry{MaxRetries: 2}, &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}, ), ) return out, err }) 73 lines, and it returns exactly what you asked for: Plain Text {Category:billing Severity:1 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued a refund of $25.50 (2550 cents) to resolve the duplicate charge on order A-1029.} The interesting part is which requirements are not code. Retry and fallback are two entries in a list. Tracing does not appear at all, because Genkit instruments flows, generate calls, tool calls, and retrieval itself. The RAG store is three calls because localvec ships in the box. The LangChainGo Version Requirements 1, 2, and 3 port directly. Requirements 4, 5, and 6 do not exist in the library, so they become your code: FileLinesWhy it existsmain.go110the actual applicationmemstore.go68all 15 vector stores need an external service, so the local one is yoursresilience.go58no middleware layer, so retry and fallback are yourstracer.go36observability is a 17-method interface to implementtotal272 None of those three extra files contain interesting code. memstore.go is cosine similarity and a sort. tracer.go is six methods that log and eleven empty ones that exist only to satisfy the interface: Go // Everything below exists only to satisfy the interface. func (t *tracer) HandleText(context.Context, string) {} func (t *tracer) HandleLLMStart(context.Context, []string) {} func (t *tracer) HandleLLMError(context.Context, error) {} func (t *tracer) HandleChainStart(context.Context, map[string]any) {} // ...seven more resilience.go wraps the primary model to add retries and a fallback, and it has a sharp edge worth seeing: Go clean := make([]llms.CallOption, 0, len(opts)) for _, o := range opts { var probe llms.CallOptions o(&probe) // Model names are vendor-specific, and gpt-5-mini rejects 'stop' // outright, which the ReAct agent always sets. Both must go. if probe.Model != "" || len(probe.StopWords) > 0 { continue } clean = append(clean, o) } return r.backup.GenerateContent(ctx, msgs, clean...) CallOption is an opaque function, so to find out what an option does you have to apply it to a probe struct and inspect the result. Miss the stop words and the fallback fails with Unsupported parameter: 'stop' is not supported with this model. Both versions work and produce equivalent results. So far this is a story about volume, not capability. The outage test is where that changes. What Happens When a Model Goes Down Point the primary at a retired model in both versions, which is a real 404 from the live API. Genkit heals and finishes: Plain Text WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" {Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued full refund of $25.50 ...} Two calls fail over, once for the initial generate and once for the turn after the tool call. The typed struct survives a vendor switch. LangChainGo heals at the model layer, then breaks one layer up: Plain Text 2026/09/11 20:01:45 primary failed (googleapi: Error 404 ...), falling back 2026/09/11 20:01:54 PARSE FAILED: input text should start with ```json and end with ``` The hand-written fallback did its job. The output parser then rejected the answer because outputparser.Defined.Parse hard-requires code fences: Go const opening = "```json" const closing = "```" if text[:len(opening)] != opening || text[len(text)-len(closing):] != closing { return target, fmt.Errorf("input text should start with %s and end with %s", opening, closing) } Gemini tends to emit those fences. OpenAI does not. The typed output layer is quietly coupled to one vendor's formatting habit, and the coupling surfaces at the exact moment you switch vendors, which is the worst possible time. That same function has a second problem. When the response is shorter than seven characters, it does not return an error; it panics: Plain Text input "ok" PANIC: runtime error: slice bounds out of range [:7] with length 2 input "" PANIC: runtime error: slice bounds out of range [:7] with length 0 An empty completion or a refusal becomes an uncaught panic in your request handler. Dependency Weight The same typed-extraction program, each in its own clean module: LangChainGoGenkit GoModules in the build graph6241Binary size38.0 MB28.7 MB The LangChainGo-only half of that list is worth reading: the deprecated generative-ai-go, plus cloud.google.com/go/vertexai and aiplatform, plus sprig, gonja (a Jinja2 template engine), go.starlark.net (the calculator tool evaluates expressions in Starlark), logrus, tiktoken-go, pkg/errors and json-iterator. A Python-shaped dependency tree, in Go. Where Genkit Go Shines Pulling the thread through all of it, the difference comes down to four things Genkit has, and LangChainGo does not. Middleware is the first. Because a Genkit middleware wraps a call and can decide to call it again or call something else, resilience becomes library code. Retry, cross-vendor fallback, and tool approval are list entries instead of wrapper types you maintain forever. Types are the second. Typed flows, typed tools and schema-backed output mean the compiler carries the contract. That is not just tidier: it is why the Genkit service survived the failover and the LangChainGo one did not. Nothing in the Genkit path was slicing strings out of a model response to find the data. Plugins are the third. A model is a string reference in a registry, so swapping Gemini for GPT is a configuration change. In LangChainGo, two providers are two concrete structs of two different types, which is why a hand-written fallback has to strip vendor-specific call options and still gets it wrong the first time. The Developer UI is the fourth, and it is the one you feel every day rather than during an incident. LangChainGo has no equivalent to it at all. One command starts it next to your app: Shell genkit start -- go run . It reads the registry your code built, so nothing has to be declared twice. On the triage service, it found the flow, the tool, 39 models, and 3 embedders without any configuration: Plain Text Flows (1) triageTicket Tools (1) issue_refund Models (39) googleai/gemini-2.5-flash, ... Embedders (3) googleai/gemini-embedding-001, ... Open a flow, and the input form is generated from your Go struct. A Ticket with Subject and Body fields gives you this, pre-filled and ready to edit: JSON { "body": "", "subject": "" } Press Run and you get the typed output back as a tree, not a blob of text: JSON { "category": "Billing", "urgency": 4, "action": "Refund issued" } The part that earns its place is the trace underneath. Every run is a span tree with per-span timings, and the tool call is a span of its own: Plain Text triageTicket 4.64s ├─ generate 2.52s │ ├─ googleai/gemini-2.5-flash 1.67s │ ├─ issue_refund 1ms │ └─ generate (2) 846ms │ └─ googleai/gemini-2.5-flash 844ms └─ generate 2.12s └─ googleai/gemini-2.5-flash 2.12s You can read the whole shape of the run off that: the first generate called the tool, the model went back for a second turn with the tool result, and a separate generate did the typed extraction. Nobody wrote a line of logging for it. Failures land in the same place. A run that hit a real Gemini constraint showed the error inline with a full Go stack trace, and marked every span in the tree that was affected: Plain Text Error 400, Message: Function calling with a response mime type: 'application/json' is unsupported, Status: INVALID_ARGUMENT, Details: [] That error is worth knowing on its own, and it is model-specific rather than a Genkit limitation. Running the same tools-plus-typed-output call across five models makes the line clear: Plain Text gemini-2.5-flash FAIL Function calling with a response mime type: 'application/json' is unsupported gemini-3-flash-preview OK gemini-3.5-flash OK gemini-3.6-flash OK gemini-3.8-flash OK Gemini 2.5 refuses function calling combined with a JSON response mime type, so on that model a tool call and a typed extraction have to be two steps. Every Gemini 3 model lifted the restriction, which is why the triage service above runs both in one call on gemini-3.5-flash. The useful part is that the Developer UI told me which span failed and why, instead of leaving me to bisect it. The sidebar also carries Traces, Evaluations and Datasets, so the same UI covers running an action by hand, reading what it did, and building a dataset out of it. In LangChainGo, the equivalent of all of this is a callbacks.Handler you write yourself, which returns nothing from every method and, as shown above, breaks your Gemini agents when you attach it. Underneath all four, someone is shipping. When gemini-2.0-flash was retired, that was a 404 to work around in LangChainGo and a fallback that healed itself in Genkit. What LangChainGo Still Does Well This is not a demolition. There are real reasons it is still in use: Provider catalog. Seventeen model providers, including Bedrock, Cloudflare, Ernie, HuggingFace, Maritaca, Mistral, watsonx, and llamafile. Genkit's plugin set is narrower.Vector store catalog. Fifteen integrations, including Dolt, MariaDB, and OpenSearch, which Genkit has no equivalent for.Conversation memory is less typing. chains.NewConversation(llm, memory.NewConversationBuffer()) beats passing resp.History() around by hand. Genkit's answer is an agent session store, which is more capable but sits behind an experimental flag, so on a plain generate call LangChainGo is still terser.The ReAct agent works. On a sequential two-tool task, it succeeds consistently. The cost is the code you own, not flakiness.Familiarity. If your team already thinks in chains, memories, and retrievers from Python, the vocabulary transfers directly. If you depend on a provider or a vector store only LangChainGo supports, that is a real reason to stay. Weigh it against an unmaintained dependency rather than against line counts. The Scorecard Non-blank, non-comment lines. RequirementGenkit GoLangChainGoFull service: RAG + tool + typed output + retry + fallback + tracing73, one file272, four filesBehavior under a vendor outageheals, typed output intactheals at model layer, output parser failsTool with two typed arguments3155 (hand-written unmarshalling)Retry and cross-vendor fallback2 config lines58 lines, vendor-specific options to stripLocal vector store for RAGships as localvec68 lines, no local option existsObservabilitybuilt-in, zero lines36 lines, and it breaks Gemini agentsStreaming on current Gemini modelsworksbroken on all four testedQuickstart with no optionsworks404, default model retiredModules/binary for the same program41 / 28.7 MB62 / 38.0 MB Conclusion If you are starting a Go AI project today, use Genkit Go. That part is not close. If you already have LangChainGo in production, the decision is about risk rather than elegance. The library is eight months without a commit, its Google provider is built on an SDK that reached end of life in November 2025, its quickstart returns a 404, its streaming is broken on every current Gemini model, and turning on observability breaks the agents it is meant to observe. Those are not problems that resolve themselves. The clearest signal is the outage test. Both versions were supposed to survive a vendor failure, and technically both did at the model layer. Only one came back with the data the caller asked for, because in the other one the layer above was matching on code fences. Further reading: How to migrate from LangChainGo to Genkit Go, the step-by-step guideGenkit Go vs ADK Go 2.0Building Agents in Go: Where Genkit Shines vs ADK 2.0Stop using Python for GenAI, use Genkit GoGenkit documentationLangChainGo repositoryMastering Genkit: Go Edition

By Xavier Portilla Edo

CORE

Genkit Flows as senro Steps: Extremely Powerful AI Pipelines in Go

A Genkit flow is an excellent unit of AI work. It is typed, traced, testable, and genkit.DefineFlow gives you something you can call from anywhere. What a flow is not is a unit of orchestration. The moment you have twelve documents to summarise, a flow whose output feeds two other flows, or a model call that fails on the third of forty items, you end up writing the same scaffolding again: a worker pool, a retry loop, a way to avoid re-running the eleven items that already succeeded, logs you can actually read, and some way to find out what happened at three in the morning. senro is a pipeline engine where a step already has all of that: retries that distinguish infrastructure from workload, an action cache that skips work it has already done, an append-only event stream, and a live attach protocol. I covered it in the introduction and the monorepo article. This article shows how to combine them. Put a Genkit flow inside a senro step, and every model call becomes a graph node with its own log, state, retry policy, and cache entry. Every example below ran against the real Gemini API with googleai/gemini-2.5-flash, on Genkit Go v1.13.1 and senro v1.4.0. The summaries are what the model actually returned. What You Need Shell go get github.com/firebase/genkit/go go get github.com/xavidop/senro The example is a small corpus of release notes in docs/*.md that gets summarised one document at a time, then merged into a digest. Step 1: The Flow Nothing senro-specific here. This is an ordinary Genkit flow: Go // SummarizeInput is what the summarize flow takes. type SummarizeInput struct { Path string `json:"path"` Text string `json:"text"` } // Summary is what it returns. type Summary struct { Path string `json:"path"` Summary string `json:"summary"` } func defineFlows(g *genkit.Genkit) *core.Flow[SummarizeInput, Summary, struct{}] { return genkit.DefineFlow(g, "summarize", func(ctx context.Context, in SummarizeInput) (Summary, error) { text, err := genkit.GenerateText(ctx, g, ai.WithSystem("You summarize release notes for an operator. "+ "Answer in one sentence, no preamble."), ai.WithPrompt(fmt.Sprintf("Summarize these release notes:\n\n%s", in.Text))) if err != nil { return Summary{}, err } return Summary{Path: in.Path, Summary: text}, nil }) } Called on its own, it does what you expect: Plain Text The billing reconciliation job was moved to a queue worker, accelerating invoice settlement from up to 24 hours to within two minutes. Step 2: Wrap the Flow in a Step senro has two kinds of steps: a command and a registered Go function. A Genkit flow is a Go function, so it is the second kind: Go // summarize is the registered flow. A func step runs in the pipeline's own // process, so a package-level handle is all the wiring the two need. var summarize *core.Flow[SummarizeInput, Summary, struct{}] type SummarizeParams struct { Path string `json:"path"` Name string `json:"name"` } func init() { senro.RegisterFunc("ai/summarize", RunSummarize) } // RunSummarize runs the flow and writes the summary into the one workspace // this step mounts. func RunSummarize(ctx senro.Ctx, p SummarizeParams) error { dir, ok := ctx.Workspace("doc-" + p.Name) if !ok { return fmt.Errorf("step %s mounts no doc-%s workspace", ctx.StepID(), p.Name) } text, err := os.ReadFile(filepath.Join(string(dir), "doc.md")) if err != nil { return err } res, err := summarize.Run(ctx, SummarizeInput{Path: p.Path, Text: string(text)}) if err != nil { return err } fmt.Fprintf(ctx.Stdout(), "%s\n", res.Summary) return os.WriteFile(filepath.Join(string(dir), "summary.txt"), []byte(res.Summary+"\n"), 0o644) } Three details make this work smoothly: senro.Ctx embeds context.Context, so it passes straight into summarize.Run with no adapter. Genkit's tracing and cancellation behave exactly as they always did.ctx.Stdout() is the step's real log stream, recorded and redacted like a command's output. Writing to os.Stdout instead reaches your terminal and no log file.ctx.Workspace(name) hands the function the same path a mount gives a command, on every executor. The function body does not change when the step later runs in a container or a pod. RegisterFunc registers the function once, from an init. The name is the function's identity: it is what the plan records and what feeds the step's cache key, so renaming it invalidates the cache exactly as renaming a command would. Parameters must be JSON-serializable, and decoding is strict, so a renamed field fails loudly instead of running with a zero value. Step 3: Build the Graph Because the pipeline is a Go program, fanning out over a corpus is a for loop: Go func main() { ctx := context.Background() g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-2.5-flash")) summarize = defineFlows(g) docs, err := filepath.Glob("docs/*.md") if err != nil { log.Fatal(err) } sort.Strings(docs) p := senro.New("ai") w := p.Workflow("summarize") var names, ids []string var wss []*senro.WorkspaceRef corpus := map[string]string{} for _, doc := range docs { text, err := os.ReadFile(doc) if err != nil { log.Fatal(err) } name := strings.TrimSuffix(filepath.Base(doc), ".md") names = append(names, name) ids = append(ids, "summarize/"+name) corpus[name] = string(text) // One workspace per document, so one document changing does not // invalidate the other summaries' cache keys. wss = append(wss, senro.Workspace("doc-"+name, senro.Scope(senro.ScopeRun))) } seed := w.Step("seed", senro.Func("ai/seed", SeedParams{Docs: corpus})) for i, ws := range wss { seed.Mount(ws.At("/doc-"+names[i], senro.RW)) } for i, name := range names { // One AI call, one node: its own log, state, retry and cache entry. w.Step(ids[i], senro.Func("ai/summarize", SummarizeParams{ Path: docs[i], Name: name, })). Needs("seed"). Mount(wss[i].At("/doc", senro.RW)). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File("doc.md")). Outputs(artifact.File("summary.txt")) } digestStep := w.Step("digest", senro.Func("ai/digest", DigestParams{Names: names})). Needs(ids...) for i, ws := range wss { digestStep.Mount(ws.At("/doc-"+names[i], senro.RO)) } if err := senro.Run(ctx, p); err != nil { log.Print(err) os.Exit(1) } } seed writes each document into a workspace of its own. Then one summarize step per document, running in parallel. Then digest, which waits for all of them and merges the results. That loop produces this graph, rendered straight from the run's own plan.json: Plain Text pipeline: ai │ ├─ wave 1 seed func │ ├─ wave 2 (3 parallel) summarize/alpha func · retry 3/infra │ summarize/beta func · retry 3/infra │ summarize/gamma func · retry 3/infra │ └─ wave 3 digest needs summarize/alpha, summarize/beta, summarize/gamma · func Three documents give three parallel nodes. Thirty would give thirty, from the same for loop, with no change to the code. Step 4: Run It Run the pipeline binary directly, the way any Go program runs: Shell go run . Or hand the package to the senro CLI, which builds it, execs it, and attaches automatically. You get a terminal UI on a TTY and plain streaming lines anywhere else: Shell senro run . Plain Text seed -> succeeded summarize/beta -> succeeded summarize/alpha -> succeeded summarize/gamma -> succeeded digest -> succeeded Plain Text - alpha: The billing reconciliation job now uses a queue worker for invoice settlement within two minutes, replacing the nightly cron. - beta: The search index now rebuilds incrementally in under a second without blocking writes, replacing the old full rebuild and requiring operators to delete the `rebuild-search` cron job. - gamma: Service gamma's public API now features per-tenant rate limiting set at 100 requests per second, which may cause HTTP 429 errors for existing bursting integrations. Three real model calls in parallel, each with its own log file, and a merge step that ran once they were all done. While that is running, a second terminal can watch it, or steer it: Shell senro attach # terminal UI senro ui # browser view, prints a one-time link senro attach --run <id> # reopen a finished run from disk Every run also lands in runs/<id>/, so senro runs lists what happened and each step's stdout is a file you can read afterward. One thing to know when you switch between go run and senro run. A func step's cache key includes the binary digest, so the first run after changing how the binary is built re-runs every func step once, then caches normally from there. That is correct behaviour rather than a bug: the function's body lives in the binary, and senro cannot tell whether it changed without hashing it. Step 5: Stop Paying for the Same Call Twice Each summarize step declares Pure(). Run the pipeline again with nothing changed: Plain Text seed -> succeeded summarize/beta -> cached summarize/alpha -> cached summarize/gamma -> cached digest -> succeeded cached is a real state, not a label. Those three steps did not execute: no model call, no tokens, no latency. Their recorded outputs were restored from the content-addressed store, so digest still read three summaries and produced the same result. Now change one document out of three: Plain Text seed -> succeeded summarize/alpha -> cached summarize/beta -> cached summarize/gamma -> succeeded digest -> succeeded One document changed, one model call made. And senro will tell you which component of the key moved: Plain Text $ senro cache explain --run <id> summarize/gamma MISS summarize/gamma key d3eb7450 (previous f68652ad) ✗ input_digests: docs/gamma.md 6750cb09 → ef0fdb64 ✗ workspace_digests: doc-gamma 16774a4e → d43e13c8 ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity, tool_versions, version unchanged For a few dozen documents this is a convenience. For a nightly re-index over ten thousand where one file changed, it is the difference between a bill you notice and one you do not. Note func_identity in that list. For a func step, the cache key covers the binary digest, the registered name, and the parameter digest. So changing your prompt, your model, or the flow's code all correctly invalidate the cache. It is not keyed on the input file alone, which is the mistake a hand-rolled cache usually makes. Give Each Call Its Own Workspace This is the one structural rule to follow when caching model calls. Inputs feeds the input_digests component of the key, but Mount feeds workspace_digests, which is the digest of the entire mounted workspace. Narrow Inputs do not narrow it. Mount one shared tree across a fan-out and every step's key depends on every document, so changing one document re-runs all of them and the cache buys you nothing. One workspace per unit of work, as in the loop above, keeps each key dependent on its own input. Step 6: Let a Model Explain a Failure senro has a second place a model belongs: explaining a failed step. contrib/genkitanalyzer is a ready-made failure analyzer built on Genkit: Shell go get github.com/xavidop/senro/contrib/genkitanalyzer It is a nested module with its own go.mod, so senro itself never pulls in an AI SDK. If you never install it, your dependency graph never hears about Genkit. Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{})) p := senro.New("analyze") w := p.Workflow("build") w.Step("migrate", exec.Command("sh", "-c", `echo "dial tcp 10.0.3.7:5432: connect: connection refused" >&2; exit 1`)) err := senro.Run(ctx, p, senro.WithAnalyzer( genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash")), senro.AnalyzerName("genkit"), senro.AnalyzeTimeout(20*time.Second))) That is the whole integration. When the step fails, an analysis.proposed event lands in the run's ledger: JSON { "id": "migrate@1", "analyzer": "genkit", "duration_ns": 8344562458, "summary": "The 'migrate' step failed because its command explicitly reported a database connection refusal error and exited.", "detail": "... The error message \"dial tcp 10.0.3.7:5432: connect: connection refused\" typically indicates that a client attempted to establish a TCP connection to a server at 10.0.3.7 on port 5432, but the server actively rejected the connection. This could be due to the database server not running, not listening on the specified address/port, or being blocked by a firewall ..." } Because it is an event, the explanation shows up in the TUI, in events.jsonl, and in anything else folding the run. What the Combination Gives You Genkit gives you the model call: typed, traced, provider-agnostic, testable. senro gives you everything around it. ConcernWhat senro already does for a stepOrderingA dependency edge, not a place in a loopOutcomeTen states, so a call that passed on the second attempt is recovered, not silently a successRetriesretry.OnInfra() retries a dropped connection, never a model answer you did not likeLogsOne log file per step per attempt, redactedCostAn action cache keyed on the prompt, parameters, model, and binaryObservabilityOne append-only event stream you can attach to live or replay laterFailure analysisAn analyzer gated behind a human or an explicit policy You can build every one of those yourself. It is never quite the same twice, and it is rarely tested well. Conclusion The pattern is small enough to summarise in four lines: Define the flow with genkit.DefineFlow as you always would.Register a func step with senro.RegisterFunc that calls flow.Run(ctx, ...).Give each call its own workspace and declare Pure() with Inputs and Outputs.Add senro.WithAnalyzer if you want failures explained. From there the graph is a Go program, so the loop over your corpus is a for loop and the logic that decides what runs is testable with go test. senro on GitHubGenkit, and the Go documentationThe last article in this series: Debugging senro pipelines

By Xavier Portilla Edo

CORE

PWA on iOS, Part 1: PWAs on the iPhone Are Older Than They Look

This is the first of eight articles about how web apps work on iOS, and it has less code than the rest. Here I look at where the oddities come from, the ones everyone who ships a PWA to the iPhone ends up living with. Open the <head> of almost any PWA built in 2026 and next to the manifest link you'll most likely find the apple-mobile-web-app-capable meta tag. Safari's old reference says it plainly: "Available in iOS 2.1 and later." Version 2.1 came out on September 12, 2008, when the App Store had already been running for two months. So full-screen web apps on the iPhone showed up after Apple had turned to the native SDK, even though the story is usually told the other way round. PWA behavior on the iPhone is full of small details like this. Behind the icon on your Home Screen are twenty-five years of attempts to turn a web page into a program, and plenty of the decisions made along the way are still sitting in your code. When Windows Trusted HTML as It Trusted Itself In 1999, together with Internet Explorer 5, Microsoft released HTML Applications, or HTA. For its time, the idea was very clear: HTML, CSS, and scripts can already draw an interface, so why not launch such a page in its own window like an ordinary Windows program, with no address bar and no browser buttons? A file with the .hta extension was opened by a system utility, mshta.exe, which took the IE engine but left its security model outside the door. Microsoft wrote about this in its documentation without a trace of embarrassment. HTAs are called "full-fledged applications" that "are trusted," and the summary reads: "an HTA runs like any executable (.exe) file written in C++ or Visual Basic." The user was asked once, before the download, whether to save the file or run it. After that, the app could do everything an ordinary program can: write to disk, start processes, read the registry. It looked roughly like this: HTML <html> <head> <title>Inventory</title> <hta:application applicationname="Inventory" border="thin" showintaskbar="yes" /> <script language="JScript"> function openLog() { var shell = new ActiveXObject("WScript.Shell"); shell.Run("notepad.exe C:\\inventory\\log.txt"); } </script> </head> <body> <button onclick="openLog()">Open log</button> </body> </html> No permission dialog, no origin, no sandbox: the WScript.Shell object runs anything with the current user's rights. For an internal admin tool in the late nineties that was convenient, sysadmins wrote setup wizards, support dashboards, and utilities in HTA that otherwise would have had to be built in Visual Basic. All of it ran only on Windows and only on the IE engine, so HTA couldn't boast the cross-platform reach that people usually pick the web for. Microsoft even patented the approach: the application "Method and apparatus for writing a Windows application in HTML" was filed on May 20, 1999, and patent US 6,662,341 was granted on December 9, 2003. Why .hta Is Still Stripped from Email Attachments The model of "download some markup, get the rights of a program" aged very badly. In MITRE ATT&CK, mshta.exe has a technique of its own, T1218.005, which describes how attackers run malicious .hta files, JavaScript and VBScript through the utility as early as the initial-access stage. A trusted system binary is handy because it slips past policies that only watch for unfamiliar executables. The LOLBAS project, which catalogs legitimate Windows programs that can be abused in attacks, lists mshta.exe with paths in System32 and SysWOW64 all the way up to Windows 11. The loudest episode happened in April 2017. FireEye described attacks through RTF documents for Microsoft Office: inside sat an OLE object with a link; Word went to the attacker's server on opening, got an HTA from there, and handed it to a handler that executed the file. The vulnerability was assigned CVE-2017-0199, and Microsoft fixed it in the update of April 11, 2017. In November 2021, CISA added it to its catalog of vulnerabilities exploited in real attacks. HTA itself wasn't the bug in that chain. It was just the full trust the attackers were after. Today Outlook blocks .hta attachments by default, on the same list as .exe and .bat, and I can't think of a better epitaph for the technology. What do PWAs have to do with it? Well, they look like apps, open from an icon and work offline, but they stay inside the browser security model, with its origins, HTTPS, sandbox, and permissions the user grants one at a time. When, in the next parts, a Service Worker refuses to register over plain HTTP, or iOS won't let a web app near Safari's data, remember mshta.exe: that's exactly the kind of neighbor browsers don't want to see next door anymore. June 2007: No SDK Needed On January 9, 2007, at Macworld, Steve Jobs showed the iPhone and said nothing about third-party apps. Today that looks odd, but back then the phone was sold as a finished product with mail, a browser, maps and an iPod inside, and the question of a platform simply never came up on stage. Developers got their answer on June 11 at WWDC. According to a transcript that 9to5Mac published later, Jobs said the iPhone has "the full Safari engine" inside, so you can write Web 2.0 and Ajax apps that "look exactly and behave exactly like apps on the iPhone," and then added: "There's no SDK that you need!" Apple's press release from the same day says the same thing in a calmer voice. Web apps would look and behave like built-in ones, and they could make a call, send an email, and drop a pin in Google Maps. They would extend what the iPhone can do "without compromising its reliability or security." And they could be updated by "simply changing the code on the developers' own servers." It sounds almost like a definition of a PWA, but in June 2007 there wasn't much behind those words. The app lived in a Safari tab, you reached it through a link or from your bookmarks, and all the integration with the phone came down to tel: and mailto: links plus addresses that opened in Maps. There was no camera API, no background work, and no storage in today's sense. There was no Home Screen icon at all: the Home Screen itself was fixed, and without a jailbreak you couldn't rearrange anything on it. Developers took to this model anyway, and with real enthusiasm. At iPhoneDevCamp in July 2007, Wired reported, the person everybody was talking about was Joe Hewitt, one of the creators of Firefox and Firebug. He had put together iUI, a set of templates that made a site on the iPhone look and act almost like a built-in app. Many developers had already adopted the templates as a base, and Wired wrote that they were clearly becoming the default look of the iPhone web. That August, Hewitt, by then at Facebook, built a version of the social network specifically for the iPhone, and it was a web app too: the native client didn't come out until the summer of 2008. To me, this model still feels more honest than the way it was later retold. The web didn't pretend to be deeply integrated with the phone. It asked the system to perform a system action, and that habit survived all the way to Web Share, and to the way an installed web app on iOS now asks for permission to send notifications. An Icon in January, an SDK in March The button this whole series exists for arrived with iPhone OS 1.1.3 on January 15, 2008. TidBITS described the novelty in detail at the time: Safari got a plus button, its menu had an Add to Home Screen item, and an icon appeared on the Home Screen that Apple called a Web Clip. The same release let you edit the Home Screen for the first time, icons started to wiggle under your finger, and the number of screens went up to nine. Apple's documentation has said the same thing ever since: "The Web Clip feature is available in iOS 1.1.3 and later." The apple-touch-icon was born along with the icon. A site could drop a PNG in the root or point to one in a <link>, and if it did neither, iOS went looking in the root for a file called apple-touch-icon.png. Later versions also tried variants like apple-touch-icon-57x57-precomposed.png. The -precomposed suffix, which has worked since iOS 2.0, was simply a request: Safari, please don't add your own effects to this icon. The era of glossy highlights is long gone, yet I still see requests for these files in server logs, on servers where nobody ever put them. Still, an icon added this way opened an ordinary Safari tab with all the toolbars. The first Web Clip was a bookmark moved to the Home Screen, and nothing more. Apple, meanwhile, had already changed its mind. Back on October 17, 2007, a short letter from Jobs appeared on the company's site. It began with the words "Let me just say it: We want native third-party applications on the iPhone" and promised an SDK in developers' hands in February. He blamed the delay on security: Apple was trying to give developers an open platform and protect users from viruses and malware at the same time, and he pointed to Nokia, which on some of its new phones allowed only apps carrying a digital signature from a known developer. Four months passed between the June "no SDK needed" and this letter. The SDK missed February. It was shown on March 6, 2008, together with the App Store, and the press release from that day contains a phrase that set the course for the next fifteen years: third-party apps "must be approved by Apple and will be available exclusively through the App Store." The store itself opened on July 10, 2008, through iTunes, and by the time the iPhone 3G went on sale, Apple was promising more than 500 native apps in it. iPhone OS 2.0, with the App Store built in, came out the next day, July 11. Full-Screen Mode Came After the App Store Now the detail I started with. The apple-mobile-web-app-capable meta tag hides Safari's interface and opens a site from its icon in full screen. Its neighbor, apple-mobile-web-app-status-bar-style, controls the strip with the clock. Apple marked both as available since iOS 2.1, and that version came out on September 12, 2008. The same reference describes the navigator.standalone property, which lets a page tell that it was launched from the Home Screen. The apple-touch-startup-image splash picture was added only in iPhone OS 3.0, and icons for different screen resolutions in 4.2. So Apple was still finishing the web app mode when the native platform was already on sale. There is no "first they bet on the web, then dropped it" in this timeline. Web apps got modest, minimal support and lived on it for almost ten years. What iOS kept from that period is a set you can still find in old projects: a separate meta tag for every capability, no general description of the app, and installation only by hand, through the Safari menu. A lot of projects have dragged the launch-mode check along since those days, and it usually looks like this: JavaScript const standalone = window.navigator.standalone === true || window.matchMedia('(display-mode: standalone)').matches; if (standalone) { document.documentElement.classList.add('launched-from-home'); } The first half of the condition is a legacy of iPhone OS 2.1, and the second arrived with the web manifest. On the iPhone, this check usually fails not because of the code but because of expectations: navigator.standalone exists only in WebKit on iOS, while the value of display-mode depends on exactly how the site was added to the Home Screen and what was in the manifest at install time. We'll break down how this works in Part 4 of this series. Ten Years on Meta Tags and AppCache Between 2008 and 2018, web apps on the iPhone stood on two things: those same Apple tags and the Application Cache. The tags were simple and tedious. For the splash screen, you had to draw a separate image for almost every iPhone and iPad screen size, in the right orientation and with the right media query. I've come across repositories where the folder of splash images weighed more than the app itself, with a comment sitting next to it along the lines of "don't delete, white screen on iPad otherwise." Offline was worse, and its roots, by the way, aren't Apple's. On May 31, 2007, less than two weeks before WWDC, Google released Gears, a browser plugin that gave pages a local SQLite database, background workers, and a LocalServer module that could serve saved HTML, scripts, and images without a network. Gears was what powered offline access in Gmail. On February 19, 2010, the Gears team wrote on its blog that its effort was shifting toward getting all of Gears' capabilities into web standards like HTML5, and that the plugin itself would get no new features. For the history of PWAs this is a telling moment: a browser vendor retired its own proprietary technology in favor of a standard, and Service Worker would later travel roughly the same road. The standard heir to LocalServer was AppCache. You declared a text file listing your resources in <html manifest="...">, and the browser promised to open the page without a network. Safari on the iPhone supported it, and for web apps launched from the Home Screen it was for a long time the only standard way to open without a connection instead of showing a "You are not connected to the Internet" error page. A typical manifest looked harmless: HTML CACHE MANIFEST # v12 CACHE: /index.html /app.js /styles.css NETWORK: /api/ FALLBACK: / /offline.html The catch was in the update rules. Apple's documentation warned outright that changing the HTML file does nothing until the manifest itself changes, so everybody kept a comment with a version number in it and remembered to bump it. If the manifest did change, the new cache was built in the background and applied only on the next visit, which meant the user saw the old version first and the new one only after a restart. If a single file from the list came back with an error, the update wasn't applied at all. Some users could get stuck on an old bundle for a long time without even knowing it. In May 2012, Jake Archibald, who was then building Lanyrd's mobile site on AppCache, published an article in A List Apart with a famously rude title: "Application Cache is a Douchebag." He went through all of these traps, and if I put it in my own words, AppCache looked like an offline mode but behaved like a hidden deploy that you don't control. Funnily enough, Archibald later ended up among the editors of the Service Worker specification, along with Alex Russell, who comes up below. Service Worker grew straight out of that grievance. Instead of a declarative list, you got a programmable fetch handler, an explicit lifecycle, and a Cache API where you decide for yourself what to store and when. If a legacy project still has <html manifest="cache.appcache"> sitting in it, don't try to migrate gradually by adding a worker alongside. Two offline mechanisms on one origin fight over the same page, and the bug then looks like "some users have the old version" even though git is perfectly clean. Remove AppCache, ship the worker, and check a cold start on a real device. How Apple Campaigned for HTML5 On April 29, 2010, Apple published Jobs's open letter "Thoughts on Flash," explaining why Flash wouldn't be coming to the iPhone, iPod and iPad. Most of the text is about battery life, stability, and the touch interface, but the most interesting part is how it talks about the web. "Rather than use Flash, Apple has adopted HTML5, CSS and JavaScript," Jobs writes, meaning open standards that its devices support quickly and efficiently. The letter ends with a forecast: "New open standards created in the mobile era, such as HTML5, will win on mobile devices (and PCs too)." There's also a line that's hard to read today with a straight face. Adobe, according to Jobs, claimed that Apple's decision was driven by business and a wish to protect the App Store, "but in reality it is based on technology issues." On Flash, I think, he turned out to be right in the end. But the forecast of HTML5's victory came from a company whose Home Screen web apps couldn't send notifications and relied on that very AppCache for offline. Safari was one of the best mobile browsers at the time, and WebKit inside it really did push standards forward, while the app mode stayed roughly where it had been left in 2008. Developers saw the contradiction. In November 2009, that same Joe Hewitt, by then the author, by TechCrunch's estimate, of the most popular iPhone app ever, handed the native Facebook client over to another engineer. He told the publication that the decision was entirely about Apple's policies and that he was fundamentally opposed to the review process. His explanation ended like this: "The web is still unrestricted and free, and so I am returning to my roots as a web developer." I think this contrast explains a whole decade better than any conspiracy theory. Apple was happy to promote the web as a technology for pages and much more careful about the web as an app platform on its own phone. At least that's how it looks from the outside, since the company doesn't publish its motives for each missing API. The most popular explanation for the pause comes down to the App Store commission. You can't prove that kind of causality from public sources, and I wouldn't build technical decisions on it. The engineering explanation is duller, but it can be checked: background execution, persistent storage, and notifications for websites sit badly with a phone's battery, memory, and permission model, and WebKit on iOS has to keep compatibility for years with everything it has already shipped. The commercial incentives don't go away because of that. They just don't explain why, for example, cache.addAll fails entirely because of one broken URL. Who Else Tried to Turn the Web into a Platform? Apple wasn't the only one in those years to look at HTML as a language for apps, and almost all of these attempts ended badly. They're worth remembering, if only so you don't repeat old arguments in PWA debates. Palm showed the Pre smartphone at CES in January 2009, and on June 6 it went on sale through Sprint. The webOS operating system was daring by design: the interface was built on WebKit, and third-party apps were written in HTML, CSS, and JavaScript with the Mojo framework. The Verge, in a long piece on Palm's history, quoted one of its sources explaining the choice: instead of keeping a person around whose job is writing code to center text, you can "stand on the shoulders of giants and let WebKit do it." According to the same piece, Palm ported the V8 engine for speed and was the first to ship it on a mobile device. The rest of the story is short. In 2010, HP bought Palm; on August 18, 2011, it announced it would stop making webOS devices, and in February 2013 it sold the system to LG, where it still lives on in TVs. Google came at it from a different side. The Chrome Web Store opened on December 7, 2010, and besides extensions, it distributed Chrome Apps, packaged and hosted apps that ran in Chrome almost like desktop programs. On August 19, 2016, the Chromium blog published "From Chrome Apps to the Web": packaged apps on Windows, Mac, and Linux were actively used by about 1% of users. Google announced that it would remove them from those platforms and told developers to move to Progressive Web Apps with Service Workers and Web Push. By mid-December 2017, Chrome Apps had vanished from search and the Chrome Web Store directory, although installed apps on those systems kept launching for a while. The final shutdown dates were pushed back several times after that, but the direction never changed again. Mozilla went further than anyone and built a whole operating system in which an app is a web page. The first commercial Firefox OS phone, the ZTE Open, went on sale through Telefónica in Spain on July 2, 2013, for about 90 dollars, aimed at emerging markets. On December 8, 2015, Mozilla announced it was ending development and sales of Firefox OS smartphones, and gave a fairly honest reason: "we weren't able to offer the best user experience possible." In the same statement, Mozilla wrote, not without pride, that Firefox OS had proved the flexibility of the web on devices ranging from cheap smartphones to HD TVs. In September 2016, commercial development stopped altogether, but the code didn't disappear. A fork grew into KaiOS, which ran feature phones such as India's JioPhone. Microsoft, in Windows 8, which went on sale on October 26, 2012, made HTML5 and JavaScript first-class languages for Windows Store apps, right alongside C# and C++. For that, there was the WinJS library, which was open-sourced under the Apache license in April 2014. The last commit to its GitHub repository is dated May 2020. Of the four, webOS survived most visibly, and even that moved into TVs, while Firefox OS carried on in someone else's fork for feature phones. Put these stories together, and the conclusion is an unpleasant one for people who love grand plans. The web as an app language lost every time it needed its own operating system, its own store, or an ecosystem it had to build from scratch. PWAs survived because they ask for none of that and ride inside a browser the user already has. That is exactly what makes iOS a special case. The browser on the iPhone belongs to the same company as the platform with the store, and every browser there has to run on WebKit (apart from regulated exceptions in the EU), so the pace at which PWAs gain new capabilities is set by Apple. WebKit and the DMA get a full treatment in Part 3 of this series. A Name for Something That Already Existed On June 15, 2015, Alex Russell of the Chrome team published "Progressive Web Apps: Escaping Tabs Without Losing Our Soul." He writes that the evening before, over dinner, he and the designer Frances Berriman had listed the properties of a new class of apps: they adapt to any screen, work offline thanks to Service Worker, are always fresh, are served only over TLS, can be found by search engines, can be put on the home screen, and open through an ordinary link. Berriman called them Progressive Open Web Apps. In the body of the post, they shrink to Progressive Apps, while the headline carries the combination that ended up sticking. My favorite line from it says that such apps aren't packaged or distributed through stores because "they're just websites that took all the right vitamins." The same post has a detail worth reading twice if you care about iOS. Russell describes a scenario using the Chrome Dev Summit site: at first it's an ordinary tab, and on the second, third, or fourth visit, once the browser is confident the site is used regularly, it shows an install prompt on its own, filled in with data from the manifest. Safari still has no such prompt, and that's exactly why so many PWAs for the iPhone have a homemade hint somewhere in the interface, with an arrow pointing at the Share button. The word "progressive" comes from the term progressive enhancement. A basic site works in any browser, and wherever Service Workers, a manifest, and push are available, offline support, an icon, and notifications get layered on top. Installation stays optional. If your app shows a blank screen without JavaScript and counts as broken without a registered Service Worker, you've built the exact opposite of what the authors of the term had in mind. That will show up first on an old corporate iPad sitting behind a temperamental proxy. The first big examples appeared quickly, and nearly all of them were about Android. On November 10, 2015, Flipkart engineer Aditya Punjani described how Flipkart Lite was built, the mobile site of India's largest online store: a single-page React app, HTML shells served from the Service Worker cache, offline browsing of recently viewed products, and notifications through Web Push. On April 6, 2017, Twitter announced Twitter Lite at mobile.twitter.com and promised it would take up less than a megabyte on the device. The same announcement has a caveat that deserves a careful read: notifications and offline worked "in Google Chrome and other modern browsers on Android devices." On the iPhone in 2017, both of them were just a fast website, because Service Worker didn't exist in Safari yet. Cupertino, We Have a Problem! iOS 11.3 came out on March 29, 2018, and Apple's press release about it talks about new ARKit features, Animoji, health records in the Health app, and the battery health screen. Not a word about Service Workers. It turned up in the release notes for Safari 11.1, which shipped with that iOS version: support for Service Workers with debugging in Web Inspector, the Cache API, the Web App Manifest for sites added through Add to Home Screen, and Payment Request for paying with Apple Pay. The WebKit overview separately notes that Service Workers are partitioned by the origin of the top-level document, so they can't be used to track a user across sites, and the implementation details are described in the post "Workers at Your Service." The word "support" has to be read very narrowly here. Maximiliano Firtman, who had already written a piece during the beta under the headline "Cupertino, we have a problem," listed right after the release what had been left out. There was no install prompt, so the user had to open the Share menu themselves and find the Add to Home Screen item. There were no push notifications. iOS ignored the icons from the manifest and took apple-touch-icon, and without one it built an icon from a screenshot of the page. There was no splash screen, most of the colors from the manifest were dropped too, and when you left the app its state wasn't preserved: you stepped out to check an SMS code, came back, and the PWA started from scratch. Even Uber's careful PWA threw the user out into Safari at the login step, after which nothing was left of the app mode but the name. Apple's decision makes engineering sense to me. The company switched on the standards it knew how to support inside its own process and under its own battery limits, and it didn't copy Chrome's behavior on Android. For people who had already shipped a PWA on Android and expected identical behavior on both platforms, it looked like mockery, and I can see why they felt that way. What Came Later After that, the story moved in fits and starts. Push for web apps added to the Home Screen arrived only in iOS and iPadOS 16.4, in March 2023, and WebKit's announcement immediately spelled out the conditions: notifications are available only after installation, and permission is requested in response to a user action. The same release brought the Badging API and the ability to add sites to the Home Screen from third-party browsers. Declarative Web Push, which no longer needs an installed Service Worker to show a notification, was added in iOS 18.4 in March 2025. In iOS 26, a site added to the Home Screen opens as a web app by default, and a toggle called "Open as Web App" controls that. I'll say more about storage, because that's where the most outdated advice has piled up. The figure of 50 MB came from write-ups in 2018 and stopped describing reality long ago. The "seven-day rule" appeared in March 2020, when Intelligent Tracking Prevention in Safari started wiping script-writable storage for sites the user hadn't interacted with in seven days of browser use. In the same post, WebKit clarified that web apps on the Home Screen have their own counter of days of use, and their own data shouldn't be deleted in normal operation. The system can still evict it if the device runs out of space, and Apple doesn't name a single public quota figure that covers every case. I draw one practical conclusion from all this. The server stays the source of truth, local storage is a cache that may vanish one day, and a cold start after such a wipe needs to be tested on a real phone. Five years between Service Worker and push meant one simple thing for a product: if your app needed notifications, until spring 2023 native was your only option on the iPhone, however good the PWA turned out. Why an Engineer Needs All This in 2026 The history is here because it's useful for the job. It helps you avoid mistaking an incompatibility for a bug in your own app, and the typical case goes like this: a team writes a correct manifest, checks it in Chrome DevTools, and on the iPhone sees an old icon or a screenshot in its place. Someone who knows only the manifest spec starts hunting for a mistake in the JSON. Someone who remembers the Web Clip looks first at apple-touch-icon, at the way the app was installed, and at the fact that the icon was saved on the device at the moment it was added and most likely won't update by itself. The minimal <head> that I put even into internal admin tools grows out of that same past: HTML <link rel="manifest" href="/manifest.webmanifest"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="default"> <link rel="apple-touch-icon" href="/icons/apple-touch-icon.png"> On recent iOS versions, the apple-mobile-web-app-capable tag simply repeats display: standalone from the manifest, but on a line of tablets that go years without updates, that duplicate is cheaper than working out why the app opened in a tab. Make the icon an opaque 180×180 PNG in a separate file: the system isn't obliged to use a tab favicon for the Home Screen. Another benefit is that it pays to break the word PWA into parts before you even size up the task. On the iPhone, it hides three separate capabilities, and they arrived in different years. The icon and launching without a browser UI came in 2008 along with the meta tags; solid offline support became possible in 2018 with Service Workers, and notifications showed up in 2023, only for installed apps. An internal CRM often gets by with the first. A warehouse app needs the second, plus a queue of operations: iOS still has no Background Sync, so data will only go to the server the next time the app is opened. A messenger needs the third, and this is exactly where the web on the iPhone faces its tightest limits. This is also where the habit of naming a minimum iOS version comes from. "We support iOS" isn't a requirement: iOS 15, 16.4 and 18.4 are very different from one another. In 15, web apps have no push at all; 16.4 brought standard Web Push, and 18.4 added Declarative Web Push on top. Besides, in corporate environments old iPads stay in service much longer than the product team assumes. So I check not a list of APIs but scenarios on a real device: API names say little about how things actually behave in practice. The app should install with the right name and icon and open without the browser bar. After a full close, it should launch with no network; after sign-out, it shouldn't leave the previous user's data in the cache; and it should ask for notification permission only after installation and on a button tap. Lighthouse can do what it's good at, but it won't walk through the Web Clip install path for you (Apple calls that Home Screen icon a Web Clip). Another source of confusion comes from the 2016-2018 Android articles with their beforeinstallprompt, install banners and Google Play packaging. There is no such event on the iPhone, so a snippet that triggers a "native install dialog" is useless for Safari. I'd keep recipes like that in a wiki marked "Chromium only". Otherwise, sooner or later someone will carry them into the shared bundle and spend a week figuring out why the event doesn't fire on the iPhone, when it isn't supposed to. In requirements, instead of "make a PWA," I write three lines: the minimum iOS version, whether push is needed, and what has to work offline. If "sync in the background" shows up in the last line, it's better to end the "a PWA will do" conversation before anyone estimates in story points. And if everything adds up, the actual engineering begins. In Part 2 of this series, we'll put together a minimal setup of HTTPS, a manifest, and a Service Worker, and look at why updating without App Store review works in both directions: you can ship a broken sw.js in a minute, and in that same minute break the app for all your users at once.

By Konstantin Shkurko

ScyllaDB in Enterprise Java: High-Performance NoSQL With Eclipse JNoSQL

Modern enterprise applications must process increasing data volumes while guaranteeing low latency, high availability, and predictable performance. While relational databases are essential for transactional workloads and complex queries, some applications do not fit the relational model. Systems that manage millions of user engagements, financial events, or IoT data often need a persistence solution built for distributed environments and horizontal scalability. In these cases, ScyllaDB affords a compelling option for enterprise architects. ScyllaDB is a high-performance, distributed NoSQL database based on the wide-column data model. Inspired by Apache Cassandra and compatible with its query language (CQL), ScyllaDB uses a shard-per-core architecture to maximize hardware efficiency and preserve consistent performance under intensive workloads. When should ScyllaDB be chosen over traditional relational databases or other NoSQL options? This article examines its column-family model, compares it with Cassandra, and shows how to integrate ScyllaDB into a Java enterprise application. Modern enterprise applications require more than basic data storage. They need high throughput, low latency, and horizontal scalability to manage complex workloads. ScyllaDB is a distributed, high-performance NoSQL database based on the wide-column data model, designed for applications that require predictable performance at scale. Companies such as Discord, Expedia, and Comcast use ScyllaDB in real-world scenarios, including messaging platforms, real-time applications, and high-volume data processing. ScyllaDB is especially well-suited for event-driven architectures, IoT platforms, financial systems, and applications with demanding read-and-write workloads. ScyllaDB uses a schema-based approach with Cassandra Query Language (CQL), which features SQL-like syntax, including statements such as CREATE TABLE, INSERT, and SELECT. Beyond primitive types, ScyllaDB supports collections like List, Set, and Map, as well as User-Defined Types (UDTs) for modeling complex data. This flexibility involves trade-offs: unlike relational databases, ScyllaDB uses query-driven data modeling, limits queries on arbitrary fields, and requires careful use of secondary indexes. This article will examine ScyllaDB’s column-family architecture, compare it with Apache Cassandra, review its capabilities and limitations, and show how to integrate ScyllaDB into enterprise Java applications. Setting Up ScyllaDB With Eclipse JNoSQL ScyllaDB supports deployment in cloud environments, on-premises data centers, and local development setups. Teams can select either self-managed or managed cloud services based on their infrastructure needs. In this tutorial, we will use a single ScyllaDB instance with Docker to focus on the data model and Java integration, avoiding the complexity of a distributed cluster. To start ScyllaDB locally, execute the following command: Shell docker run -d --name scylladb-instance -p 9042:9042 scylladb/scylla This command launches a ScyllaDB container and exposes port 9042, the default for Cassandra Query Language (CQL) connections. After initialization, the database is ready to accept application requests. Integrating ScyllaDB With Eclipse JNoSQL Next, determine how the Java application will connect to ScyllaDB. Starting with Eclipse JNoSQL 1.1.19, a dedicated driver provides native ScyllaDB support, enabling standard SQL operations alongside ScyllaDB-specific features. In this example, we use a Java application with Eclipse MicroProfile Config, CDI, and Jakarta JSON Processing. These technologies are common in MicroProfile and Jakarta EE environments, which helps with integration. First, add the ScyllaDB dependency to your Maven project: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-scylladb</artifactId> <version>${jnosql.version}</version> </dependency> Set the ${jnosql.version} property to Eclipse JNoSQL 1.1.19 or a compatible newer version. Configuring the Database Schema ScyllaDB uses a schema-based approach similar to Apache Cassandra. Developers define keyspaces, tables, column types, and user-defined types explicitly with CQL. In this example, we create a keyspace named developers, two tables for people and movies, and several user-defined types. Eclipse JNoSQL supports providing initialization queries through MicroProfile Config. Add the following properties to META-INF/microprofile-config.properties: Properties files jnosql.column.database=developers jnosql.scylladb.query.1=CREATE KEYSPACE IF NOT EXISTS developers WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 1}; jnosql.scylladb.query.2=CREATE COLUMNFAMILY IF NOT EXISTS developers.Person (id bigint PRIMARY KEY, name text, phones list<text>); jnosql.scylladb.query.3=CREATE TYPE IF NOT EXISTS developers.director (name text, movies set<text>); jnosql.scylladb.query.4=CREATE COLUMNFAMILY IF NOT EXISTS developers.Movie (name text PRIMARY KEY, age int, director FROZEN<director>); jnosql.scylladb.query.5=CREATE INDEX IF NOT EXISTS ageIndex ON developers.Movie(age); jnosql.scylladb.query.6=CREATE TYPE IF NOT EXISTS developers.money (currency text, amount decimal); jnosql.scylladb.query.7=CREATE TYPE IF NOT EXISTS developers.headquarter (city text, country text); This configuration highlights multiple key points. CQL is similar to SQL, using statements likeCREATE, explicit types, and primary keys. ScyllaDB supports collections such as list and set, allowing you to represent multiple values without extra tables. User-defined types (UDTs) group related attributes into structured objects, such as the director type. The Movie table uses FROZEN<director>. In CQL, FROZEN stores the UDT as a single value, preventing independent modification of individual fields. We also created a secondary index on the age column. Although ScyllaDB supports secondary indexes, their effectiveness depends on the workload, cardinality, and query types. Do not treat them as equivalent to relational database indexes or as a substitute for proper partition-key design. Defining database schemas through application configuration works for tutorials but isn't recommended for production. Manage schema evolution separately using version-controlled migration tools such as Liquibase, which supports ScyllaDB via its Cassandra extension. This approach separates application deployment from schema changes and improves control over database updates. Mapping Entities and User-Defined Types With the database schema defined, we can create the Java models. We start with Movie, representing a ScyllaDB entity, and Director, representing a user-defined type. Java @Entity("Movie") public class Movie { @Id("name") private String name; @Column private Integer age; @Column(udt = "director") private Director director; } public class Director { @Column private String name; @Column private Set<String> movies; } The Movie entity uses the standard Jakarta NoSQL mapping approach with @Entity, @Id, and @Column. The director attribute maps to a ScyllaDB user-defined type via @Column(udt = "director"). During serialization and deserialization, Eclipse JNoSQL uses the ScyllaDB driver to convert between the Java object and its CQL UDT representation. A UDT is similar to a structured JSON object that groups related fields, but it's a strongly typed CQL value defined in the schema. The Set<String> field in Director demonstrates ScyllaDB’s native support for collections, allowing you to model a director’s movie titles without a separate relational table. Next, we will define the Person entity: Java @Entity("Person") public class Person { @Id("id") private long id; @Column private String name; @Column private List<String> phones; } The phones attribute demonstrates another collection type ScyllaDB supports: list<text>. Eclipse JNoSQL maps Java List<String> directly to the corresponding CQL collection without extra converters or custom serialization. These examples show how ScyllaDB combines a schema-based model with support for structured types and collections. Eclipse JNoSQL offers familiar mapping annotations, permitting applications to use ScyllaDB’s native features. However, object mapping convenience does not replace the need for query-driven data modeling. In distributed databases like ScyllaDB, partition keys, access patterns, and data distribution remain essential architectural considerations. Running the Application: Jakarta NoSQL and Jakarta Data Once you configure the database and map entities, the next step is to interact with ScyllaDB from Java. Eclipse JNoSQL provides two complementary approaches: Jakarta NoSQL, a template-based API for NoSQL databases, and Jakarta Data, which uses the Repository pattern. Both integrate with CDI, letting developers inject persistence components into applications running on either Java SE with CDI or Jakarta EE application servers. Jakarta NoSQL: The Template Approach The first approach uses Jakarta NoSQL with the ColumnTemplate interface, offering a consistent API for wide-column databases. Developers can insert, update, delete, and retrieve entities without interacting directly with the database driver. Java @Inject ColumnTemplate template; For example, you can create a Person entity, insert it, and retrieve it using a fluent query API: Java Person person = Person.builder() .withId(1L) .withName("Ada Lovelace") .withPhones(List.of("234", "432")) .build(); template.insert(person); Optional<Person> result = template.select(Person.class) .where("id").eq(1L) .singleResult(); The entity mapping automatically handles the Java List<String> and its corresponding CQL collection, so you don't need extra serialization code to persist phone numbers. The Template approach is useful when developers need explicit control over persistence operations, want to build queries programmatically, or prefer a consistent API across NoSQL implementations. Jakarta Data: The Repository Approach The second approach uses Jakarta Data and applies the Repository pattern to simplify persistence. Developers define an interface, and Eclipse JNoSQL supplies the implementation. Consider our PersonRepository: Java @Repository public interface PersonRepository extends CrudRepository<Person, Long> { } This interface provides standard CRUD operations, such as save, findById, and deleteById. The repository can be injected with CDI and used directly in the application: Java @Inject PersonRepository repository; repository.save(person); Optional<Person> result = repository.findById(1L); Jakarta Data also supports query methods derived from method names, allowing developers to express queries without writing implementations. MovieRepository demonstrates this capability and exposes ScyllaDB-specific functionality through its dedicated interface: Java @Repository public interface MovieRepository extends ScyllaDBRepository<Movie, String> { List<Movie> findByAge(Integer age); @CQL("SELECT * FROM developers.Movie") List<Movie> findAllQuery(); } The findByAge method demonstrates query derivation, while the @CQL annotation lets you execute native Cassandra Query Language statements. In this example, the age column has a secondary index, so you can query movies by age without using the partition key. However, query derivation does not remove ScyllaDB’s querying restrictions. Repository methods must match queries supported by the schema and indexes. Retrieving all records is useful for demonstrations but should be used cautiously in production with large datasets. Beyond Standard CRUD Operations While Jakarta NoSQL and Jakarta Data simplify common persistence operations, ScyllaDB integration extends beyond standard repositories and templates. Eclipse JNoSQL offers ScyllaDBTemplate, a specialized implementation that exposes database-specific features while maintaining the familiar Java programming model. With this integration, developers can: Control consistency levels: Specify Cassandra-compatible consistency levels, such as ConsistencyLevel.ONE, to balance consistency, latency, and availability.Execute native CQL: Run database-specific queries directly when standard abstractions are insufficient, including those using ScyllaDB-specific features.Work with structured data: Map collections such as List and Set, as well as user-defined types, without manually converting Java objects to CQL values.Choose the right abstraction: Use the vendor-neutral ColumnTemplate for general persistence, or ScyllaDBTemplate for additional database control. For instance, the specialized template supports an operation such as: Java scyllaDBTemplate.save(person, ConsistencyLevel.ONE); A key architectural feature of Eclipse JNoSQL is that abstraction does not mean losing access to database-specific capabilities. Developers can use standardized APIs for common operations and still access native functionality when needed. Jakarta NoSQL and Jakarta Data complement each other, offering different ways to organize persistence logic. The Template approach provides explicit, flexible interaction, while the Repository approach emphasizes declarative data access and separation of concerns. Both benefit from CDI integration, allowing you to use the same persistence components in standalone Java applications, microservices, or Jakarta EE environments. The choice depends on the application’s architectural requirements, but the underlying ScyllaDB capabilities and data-modeling constraints remain the same. Conclusion ScyllaDB is a strong choice for enterprise architects, especially for applications that demand high throughput, consistent latency, and horizontal scalability. Its schema-based wide-column model, support for collections and user-defined types, and compatibility with Cassandra Query Language provide a familiar and robust solution for distributed data persistence. However, to make informed architectural decisions, it's important to understand its query limitations, secondary indexes, and data-modeling trade-offs. Source: https://github.com/JNOSQL/demos-se#scylla.

By Otavio Santana

CORE

Member Spotlight: Shai Almong

What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.

By Dominique Pugh

The Self-Hosting Math Changed While We Were Still Arguing About Sovereignty

Global AI spending is forecast to hit $2.59 trillion by the end of the year, up 47% from 2025, with infrastructure the single largest line in that number. Underneath most of it sits one decision, repeated per workload: rent the serving or own it. That was always a portfolio question rather than a doctrinal one, and the portfolio was drawn by economics. Those economics moved this year. Most enterprise architecture is still drawn against where they used to sit. What Gated Self-Hosting Actually Is It was never sovereignty. Sovereignty is legitimate. Data residency rules are real, and weights you control sit inside infrastructure you own. The gate was cost at frontier scale. Qwen3.8-Max is 2.4 trillion parameters. Usable throughput from something that size means a multi-node buildout, GPU procurement, cluster management, ongoing inference tuning, and round-the-clock support for a system that can't go down. Cross that threshold, and you haven't self-hosted anything. You've started a model-serving business. That hasn't changed. Frontier scale just stopped being the price of frontier-class capability. Intelligence Per Dollar Is Falling Off a Cliff DeepSeek-V4-Flash runs 284B total parameters with 13B active and a one-million-token context window, and it serves on a single four-GPU node. Qwen3.8-Flash-Next pairs a 125B model with a 51B N-gram embedding table and activates 6B parameters per token; quantized builds are already running on two 64GB cards. Both are open-weight. Both land near where the closed frontier sat a few months ago on the work most enterprises actually run at volume: extraction, triage, code review, document analysis, routine agentic loops. A year ago, hosting a model with frontier-class capability meant a data center. Today, it means a node. The line separating sensible from delusional moved a long way, and most enterprise architecture decisions haven't caught up to where it landed. Serverless Takes Out the Last Real Objection The honest objection to self-hosting was never only capital cost; it was utilization. Owned GPUs bill whether or not you're using them, so the economics only work past a steady consumption floor. Below that floor, you are paying to keep silicon warm. Serverless GPU platforms like Modal drop that floor dramatically. Per-second billing on your own weights, autoscaling to zero, no fleet you have to keep busy to justify. That covers the awkward middle, where the workload is large enough that per-token API pricing starts to hurt but too spiky to justify reserved capacity. Own the weights, rent the seconds, and revisit the buy decision when consumption stabilizes. Post-Training Is Where This Gets Genuinely Interesting The bigger shift isn't cheaper serving. It's what you can do to a model once you control the weights. Most enterprises pay for firm-specific context the same way every time: stuff it into the prompt, pay for it on the way in, pay again on the way out, on every transaction, forever. Engram's work with Harvey is the clearest picture I've seen of the alternative. They built a synthetic law firm, 266 matters and 9,286 files across 100 million tokens, then trained an agent to study the whole workspace before answering anything, holding what it learned in a mix of text notes and weights. It runs on Qwen3.8-27B and answers at $0.13 per query against Opus 4.8 at $1.32, while scoring higher. Probed closed-book, with no tools and no notes, the model's knowledge of the firm climbed from 4.7% to 72.6% over the course of studying. Their framing is the part worth stealing: studying shifts compute from inference time to training time. Where many queries hit the same corpus, paying once to internalize it beats re-reading it on every call. That's an amortization argument, and it strengthens the longer a workspace lives, which describes nearly all enterprise work. Knowledge is the easy half. Capability is the half worth watching, and it runs on the same machinery the open labs converged on this year. The student generates its own rollouts inside an environment you define, and the teacher model grades what it produces token by token. Engram's recipe uses both halves, distillation on model outputs alongside reinforcement learning on problems the model writes for itself. Pull that thread far enough, and the frontier model stops being a runtime dependency and becomes a training input. You pay frontier prices once, during post-training, rather than on every inference call for the life of the system. Where Governed Hosting Still Wins A governed arrangement through AWS Bedrock or Databricks remains the right call for a real set of situations: workloads that genuinely need the current frontier and nothing below it, organizations with no ML engineering capacity to speak of, and volumes low enough that per-token pricing never becomes the largest line on the bill. Those are real conditions, and plenty of enterprises meet them. Far fewer meet them across their whole estate, which is how most of them are buying. Capability was never a pure compute question. It's how infrastructure decisions get made, what security posture is in place, whether an evaluation framework exists to tell you the model is actually performing, and whether anyone is tuning it as usage evolves. Self-hosting neither creates that burden nor removes it. It makes the burden much cheaper to carry, which means the binding constraint is now your people rather than your GPUs. The Math Worth Running Can we afford to be a model-serving company, and do we actually need to be one? For a growing share of workloads, that answer has quietly turned to yes, and most budgets are still built against the old one. What fraction of my traffic genuinely requires frontier intelligence, and what fraction is running on it out of habit? What is my steady-state consumption, and where does it cross the line where owning beats renting? What would it be worth to bake my firm's context and my firm's tasks into weights I control, instead of re-paying for both on every single call? Access to a frontier model's weights is still not the same thing as access to frontier capability. But the weights got cheap enough to serve and malleable enough to shape, and capability turned into something you can build rather than only something you can buy.

By Sid Vivek

Languages

DZone's Featured Languages Resources

Monthly Top Languages Experts

The Latest Languages Topics