New York has had a thriving data science scene for some time, and while there are a few reasons to be cautious about its continued capacity for expansion, the data world is predicting great things for New York City’s tech ecosystem.

However, in light of some of the hurdles New York can present to tech companies, the tech community is prone to asking: “Why New York?”

Drew Conway has the answer. Founder of Alluvium, Venn-diagram aficionado, and Insight Data Science mentor, Drew has been a mainstay in New York’s data science community for nearly a decade. He’s launched companies, built teams, and contributed to the underlying tech infrastructure that has ensured the ongoing success of NYC’s data community. In this post, we get Drew’s take on why New York has always been and continues to be the best place to be a data scientist, and the best place to grow a data science company and build a data science team.

Why New York City?

“The [2008 financial crisis] created a perfect storm of talent exiting afflicted institutions and investment money fleeing the public sector markets.”

First, some history

Contrary to popular belief, New York has actually always been a major data science hub. The reason New York is often overlooked as a cornerstone of the data science community is because almost all the city’s data scientists were previously locked within enormous institutions like banks, ad agencies, and major media companies. They held vague and cryptic titles like quant, statistician, or business intelligence analyst.

Instead of thinking of themselves as a community that could leverage their expertise to transform how data is used, they maintained focus on building specific analytical tools and products for their own particular domains of expertise. This went on for years.

But then something really tragic happened: the financial crisis of 2008, which delivered a huge blow to the city of New York; to both its denizens and its businesses. However, out of this calamity came an opportunity. The crisis created a perfect storm of talent exiting afflicted institutions and investment money fleeing the public sector markets.

Most significantly, New York’s data scientists started coming together to talk about their work and ideas. This gathering of diverse backgrounds and experiences spawned a unique, cohesive data science community that, quite frankly, doesn’t exist anywhere else in the world.

Diversity of talent and of thought

“As a diverse group comprised of skeptical academics, social sector employees, and public and municipal organizations, data scientists in NYC deal with the question of how data is really serving people on a daily basis.”

One of the reasons New York’s data community stands out from the rest is that it is the best at recognizing the need for both natural sciences and social sciences to come together to do truly great and innovative data work. Part of our job as data scientists is to be good at math and modeling complex systems, while also incorporating a deep understanding of human decision making, the most complex system there is. New York’s data science community is a diverse collection of talented data scientists who are uniquely able to balance and implement both of these components into their work.

New York’s data scientists have not only helped to build many innovative and successful businesses (AppNexus, Bitly, Tumblr, Kickstarter, Jet.com, Vimeo, Oscar Health, Enigma, Greenhouse, MongoDB, ZocDoc, OnDeck, Etsy, Venmo, Blue Apron, Fast Forward Labs, Clarifai, etc. etc.) but the Data for Good movement started here too. There are numerous New York-based organizations that have put using data for social good at the core of what they do, such as DataKind, Crisis Text Line, Murmuration, Mt. Sinai’s Arnhold Institute for Global Health, and Teachers Pay Teachers, as well as the NYC Mayor’s Office for Data & Analytics, founded in 2013 under Mayor Bloomberg.

As a diverse group comprised of skeptical academics, social sector employees, and public and municipal organizations, data scientists in NYC deal with the question of how data is really serving people on a daily basis. They also take the time to step back and ask “Okay, there’s lots of good work happening, but what are the limits?” New York City’s density and diversity support this kind of questioning, by keeping all of us in direct contact with the users and consumers we’re hoping to serve.

Density as a forcing function

“If you work at a software firm where you sit by yourself and imagine what your customer needs or wants are, you’ll never be as successful as you could if you’re able to walk down the street from your office and talk directly to your customers.”

Bustling sidewalks and crowded subways aside, New Yorkers living in close proximity to one another has benefits for folks in enterprise businesses as well. A common challenge for many startups is the constant need to better understand their users. Who is using our product? How are they using it? What issues are they having? If you work at a software firm where you sit by yourself and imagine what your customer needs or wants are, you’ll never be as successful as you could if you’re able to walk down the street from your office and talk directly to your customers. New York City has more Fortune 500 companies than any other city in the US. It’s HQ central for so many industries: media, advertising, financial services, banking, fashion, large-scale retail… the list goes on and on.

In New York, we almost take it for granted that on any given day, whether it’s at Data & Society, at Insight Data Science, or at Civic Hall, there are conversations going on about what it means to be a professional data scientist. Instead of staying buried in a text editor all day, many of us are grappling with the ethical and social challenges that come with data science. And in New York City, there is an appetite to discuss and share the latest trends and topics in data science.

The community’s diversity and willingness to ask the “hard questions” about what we’re capable of are reflected in the ecosystem of companies that have emerged in New York. One of the biggest challenges currently facing all data scientists is a lack of a clear career trajectory. What does it mean to be a “Head of Data Science” or “Chief Data Officer”? At the moment there are now a lot of data scientists but not a lot of data science managers. What does it mean to develop leadership inside this community? What are the things that we need to do to create the next generation of data science leaders from all the people sitting in our big tent?

The good news for the world’s future data scientists and managers is that building an innovative data team in New York is not only possible, but their best bet for opportunity and growth. The community’s diversity will allow them to combine methodologies that work well from fields with more established pathways like software engineering, scientific research careers, or product management, and mesh them with less obvious systems modeled after those in finance, healthcare, design, and media. Our ability to foster conversation and collaboration across these areas is what will eventually allow those of us in NYC to shape the trajectory of data science careers well into the future.

Interested in transitioning to a career in data science? Find out more about the Insight Data Science Fellows Program in Boston, New York, Seattle, and Silicon Valley, apply today, or sign up for program updates.

New York City: Data Science’s Best Bet for Growth and Opportunity was originally published in Insight on Medium, where people are continuing the conversation by highlighting and responding to this story.

When Alluvium’s first day was finally set on the calendar I did not want it to pass without notice. In fact, I wanted the day to be special — both memorable and inspiring. Being fortunate enough to live and work in the greatest city in the world, there were many potential locations. Ultimately, however, the decision was easy. There was only one place that might capture the feeling of that moment, and provide a lasting impression of the grandness of our endeavor: the top of One World Trade Center.

Early on a hot and humid NYC summer morning I met the team on the observation deck. As we circled the — truly — spectacular views of our city, we talked. Mostly, about “why”. Why were we starting this company; why were we the right people to start it; and, why should Alluvium even exist? We spent the most time on this last point, as it was something I had spent a lot of time thinking about leading up to the first day. In those preceding days I had scribbled out pages of notes thinking about ways to answer that question. On one sheet I drew two intersecting circles that read “data” and “humanity”, respectively, with Alluvium written in the intersection (yes, a Venn diagram).

It is easy to fall in love with the technical details of building a complex software system. I find this to be particularly pernicious in the Big Deep Data Science Machine Intelligence industry. I too am often swept away in these circular discussions. What is much harder, however, is falling in love with the entrenched human problems at the core of what these systems are meant to address. This is what made our view from the observation deck of One World Trade Center the perfect place to start. In every direction we could see the scale of human complexity. And, perhaps in the starkest of terms, were forced to think about the challenges that we would face by focusing on building a system that seeks to support human operators, rather than replace them.

Looking down on the city, which at that height is revealed much more as a living organism than a planned landscape, I remember telling the team that if there was ever the possibility that a single technology could empower every single person down there; living and working inside one of the most complex systems in the world, than that is what we are building. That is why we exist, so remember this view.

I am sharing this story now because Alluvium recently celebrated another milestone: our first anniversary. In this year we have learned and built some amazing things. While we have not shared much of that, I look forward to sharing much more about what we are building, and for whom, over the next several weeks. Over this time we have also grown, a bit. As part of that growth I have taken to having all new employees meet me at the observation deck on their first day. We talk about why Alluvium exists, and why they are the right person to continue what we started a year ago. Our first company tradition.

I have now done it several times, and beyond the awesomeness of the spectacle, I love having the opportunity to be re-humbled by the view, and see the reaction of my new colleague. Though it is “touristy,” I recommend visiting One World Trade Center to everyone who will listen — even New Yorkers. But that goes double for entrepreneurs, and those intrepid enough to join them.

But, if you cannot make it to New York, I hope you can find your own humbling view. One that unveils a tapestry of opportunity, complexity, and hope. A view from the start.

We are Alluvium, a team of engineers, data scientists, and executives building deeply integrated tools and services to deliver value for industries facing data challenges in the physical world. We believe that data emitted in the physical world is unruly and unrealized, but holds the keys to today’s largest business challenges.

Our mission is to build products that address these challenges, and to give the people working in these industries enhanced abilities. The journey has begun; but, how did we get here, and where are we going?

How did we get here?

The industry of data, or “big data” if you prefer, is young. But, not so young that it is without perspective. I have spent my whole career — now over a decade — working in data, of all sizes. The “big data” industry started in earnest around the mid-2000's with the development of a few seminal technologies that provided useful abstractions for both distributed data storage and computation. These technologies were developed primarily to support improved web search, and their historical origins had a large influence on how the ecosystem developed.

The technology was developed by and for products generating data in the digital world, and the first generation of the industry focused on building products to solve those problems.

An ecosystem of tools, services, and companies have been built to address these digital problems. This is by no means meant to downplay those contributions. A decade later, and we built some amazing technology and products. These are, for the most part, solved problems. What remains unsolved are data problems in the real, physical, world.

The next decade of the big data industry will be about solving these problems. Borrowing what we know about building highly available, scalable, smart systems, and inventing new systems for analyzing streams of data emitted when analog actions and decisions occur.

This is both a natural progression of the industry, but also a fundamental shift in the kinds of technologies, people, and companies that will constitute the next generation of the data industry.

Where are we going?

The promise of better living through connected devices, the so-called ``Internet of Things,’’ has captured the popular zeitgeist. While the entire consumer electronics industry may be set to instrument the lives of consumers — from wearables to smart homes — much of the focus remains on designing and marketing devices that engage consumers over a long period time.

All of the attention paid to imagining this whimsical future belies the present reality: there are massive industries emitting countless streams of data today that are remarkably underutilized.

The next frontier is not about making comfortable lives better through connected devices. It is about building data-driven products that make the hard, dangerous, and crucial jobs that power the global economy frictionless, safer, and more reliable.

In all data there is humanity. In every bit there are traces of of this humanity: in how a choice is made, or how a system is built. In the physical world the complexities of this humanity are magnified. To manage this complexity requires both deep technical expertise and innovative engineering. It also requires considerable empathy for the human beings behind that data.

Those of us who work with data are fond of describing it as messy, but data from the physical world is more than simply messy. It is knotted up in the perpetually flawed mechanism used to convert analog actions to digital signal, and the humanity that underlies it. The complexity of this humanity, however, is also our greatest strength and opportunity. The expertise, experience, and bias that people imprint on the data provide material for building great products.

This is where our journey begins.

Where are you?

We have started with a team that I am extremely proud to call my colleagues and partners. They have built some of the most used and recognized products in both consumer and enterprise data analytics. The combined expertise already under-the-hood here at Alluvium is intimidating in the best possible way.

But, hard problems require many more smart people to solve them.

We are in the very early days, but if this scale of opportunity and challenge is something that excites you, and makes you want to jump out of your chair and start building, we want to meet you.

Send along a note to us at info@alluvium.io, or fill out this short form to tell us a bit more about yourself.

It is time to solve the next generation of really hard problems.