Cities as Digital Laboratories

As World Crowds In, Cities Become Digital Laboratories

New York City amasses data on habits, health and security of its citizens to cope with spiraling growth

By Robert Lee Hotz

Dec. 11, 2015 11:10 a.m. ET

NEW YORK—Gregory Dobler is an astrophysicist who honed his craft by recording spectral images of quasars and black holes. Now, from a high-rise rooftop in Brooklyn, he is training his lens on the expanding universe of New York City.

Every 10 seconds for two years, Dr. Dobler and his colleagues at New York University’s urban observatory have taken a panorama of Manhattan. Across hundreds of wavelengths of light, they are recording the rhythmic pulse of a living city, just as astronomers capture the activity of a variable star.

“Instead of taking pictures of the sky to see what is going on in the heavens, we are taking pictures of the city from a distance to see if we can figure out how the city is functioning,” says Dr. Dobler, a scientist at NYU’s Center for Urban Science and Progress.

Researchers can analyze energy use, air quality, light pollution, heat, traffic and sleep patterns moment-to-moment, building-by-building, in one of the most densely populated cities on Earth. On a recent day, the camera spotted a plume of gases emerging from a skyscraper in Midtown, likely an improper pollution release.

It is one way that researchers, civic entrepreneurs and city managers are using New York for experiments in the emerging science of cities. Hundreds of aging cities have embraced digital technology, but few are moving as quickly as New York to link municipal computer networks, develop novel applications, make digital data public or install so many thousands of sensors to monitor urban life—from water quality, traffic and power use, to the sound of gunfire.

PLAY VIDEO THEN CLICK & DRAG TO LOOK AROUND 
To run smoothly, tomorrow’s cities will rely on millions of networked sensors to monitor everything from trash collection to water quality. Take a virtual-reality tour of where sensors are already in place in New York City. Photo: Robert Libetti/The Wall Street Journal

They hope to turn data generated every day by people in New York into a sustainable design for living that could become a template for digital cities world-wide.

Such innovations are driven by explosive global urban growth. United Nations experts predict that almost all of the world’s population increase during the next three decades will take place in urban centers—a million more people living in New York and a billion new city dwellers in China alone. Indeed, by 2050, as many as seven out of 10 people on Earth will live in an urban area—a global demographic shift so rapid that many consider it a threshold moment for humankind.

“The entire world is now urbanizing at breakneck speed,” says Luis Bettencourt, who studies city dynamics at the Santa Fe Institute in New Mexico. “We’re building more urban infrastructure in the next few decades than over our entire history.”

By most measures, the New York metropolitan area became the world’s first megacity in 1950, with a population of 10 million people or more. By 2050, U.N. planners predict there will be 40 megacities around the world.

For the interactive version of this map, visit the website here and click to play

More than bright lights draw so many millions of people. By many social and economic measures—wages, research funding, new patents and fields of employment, to name a few—the bigger the city the better. Thirty percent of the world’s economy and most of its innovation are concentrated in just 100 cities, according to the business and economics research arm of McKinsey & Co.

By studying dozens of per capita measures world-wide, Dr. Bettencourt and theoretical physicist Geoffrey West detected a fundamental pattern underlying the growth of all cities, from ancient Mexico to modern China. In studies over the past 12 years, they determined that every time the population of a city doubles, every individual measure of human interaction there also increases by 15% to 20%.

Not so long ago, futurists predicted that the ease of electronic connectivity would make big cities obsolete. Instead, Harvard University economist Edward Glaeser and others now say that improvements in information technology strengthen cities that are centers of innovation by speeding the flow of ideas. Urban density facilitates contact between smart people and fosters innovation, increasing urban incomes as new businesses take hold, they say.

“With cities, we increase the possibility of more interactions among ourselves, to create the buzz of a city, to create more ideas, more wealth. That is the attraction of a city and why they are so successful,” says Dr. West.

For the interactive version of this map, visit the website here and click to play

An urban area typically begins where population density and development scales up to around 1,500 people per square mile. By that standard, the number of people living in cities for the first time surpassed the number living in rural areas in 2009, U.N. population experts say.

Urban populations are now growing faster than cities can accommodate them. To house so many people, cities are expected to build new floor space equal to the area of Austria. As grasslands are transformed into building lots and roads, these growing megacities become their own climate zones, hotter than surrounding countryside, altering rainfall and generating more greenhouse gases.

Growing cities also concentrate risks of natural hazards and disease, say analysts at the reinsurance firm Munich Re AG in Germany. Respiratory ailments are likely to worsen because city fumes make pollen and other airborne allergens more potent, research shows. More disinfectants will be needed to purify water and treat urban sewage, threatening drinking water supplies already stretched to capacity.

“We have to get cities right, or we will go off the cliff,” says urban designer Kent Larsen, director of the MIT Media Laboratory’s Changing Places group.

To cope, cities like New York are turning themselves into test tubes of municipal management. To try to become more efficient, secure and sustainable, many are experimenting with networks of cheap wireless sensors that offer an unprecedented view of the places people live, play and work. By 2020, the number of thermostats, pressure gauges, accelerometers, acoustic microphones, cameras, meters and other micro-electromechanical measuring devices linked to the Internet is predicted to reach 50 billion world-wide.

Watch the Video: Researchers and urban planners are turning New York City into a proving ground for Big Data and the interactive technologies that one day might make a planet of cities sustainable and livable. Here, astrophysicist Gregory Dobler and his colleague Masoud Ghandehari at New York University’s Urban Observatory. Photo: Michael Rubenstein for The Wall Street Journal

They include smart irrigation systems in Barcelona, self-regulating streetlights in Glasgow, crowdsourced flood alerts in Jakarta and sensors in Santander, Spain, that monitor everything from air quality to the availability of parking spaces.

“We are entering an era where everything can have an Internet I.P. address, where everything can be a sensor,” says Minerva Tantoco, New York City’s chief technology officer. “This is creating a mesh of connectedness we have never had before.”

Despite its value, the data from so many interconnected sensor networks worries privacy experts.

“There is great promise in these interconnected systems, but there are genuine security and privacy risks,” says Jennifer Urban, a law professor at the University of California, Berkeley who studies data networks. Advanced electricity meters, for example, typically collect and relay information every hour or so and can reveal detailed intimate information about what goes on in a home or an office setting, she says.

“The more that cities collect these kinds of data streams, the richer their picture of their population gets, and it will reveal more things about ourselves than we expect,” says Scott Peppet, a University of Colorado law professor in Boulder who studies the spread of sensor data. “We have to understand what the cities are deploying, what data they are collecting and how that data is going to be used.”

As a proving ground, New York offers formidable challenges. While cities generally are more efficient than suburbs, older ones such as New York rest on a creaking infrastructure. Earlier this year, an international team of 28 urban analysts reported in the Proceedings of The National Academy of Sciences that the New York metropolitan area is the most wasteful, water-guzzling and energy-intensive megacity in the world. It has about one million buildings, 300,000 utility poles, 2,000 bridges, 6,000 miles of the country’s most congested streets and 13,500 miles of aging water mains and sewers.

Even so, New York has a critical advantage: It is a churn of human activity. “The reason New York is so wonderful as a laboratory for any form of urban experimentation or urban innovation is because it is the place where you feel the friction the most,” says Dan Doctoroff, a former deputy mayor under Michael Bloomberg. Now he is CEO of a Google-funded company formed earlier this year called Sidewalk Labs, which is a major investor in the city’s digital experiments.

Bit by bit, a digital skyline is coming into focus, shaped by networks of sensors and millions of smartphones on the move. New York City Mayor Bill de Blasio has pushed such digital innovations, building on initiatives that started under Mayor Bloomberg.

For the interactive version of this map, visit the website here and click to play

Working with Microsoft Corp. , the New York Police Department has created what city officials call the country’s largest public safety data network. As of November, the department’s “Domain Awareness System” links 10,000 public and privately owned surveillance cameras in Lower and Midtown Manhattan, more than 1,000 automated license plate readers that scan several million license plates every day, as well as 600 or so radiation and chemical monitors. The system also connects to data feeds from arrest records, emergency 911 calls and other police files—all in real time.

“It is no longer enough to view each data source separately,” says Jessica Tisch, NYPD deputy commissioner of information and technology. “The thing that allows you to do data-driven management is to view all the different sorts of data at the same time and provide that data to the officers in new ways.”

As part of its deal with Microsoft, the city receives 30% of revenues from sales of the law enforcement system outside New York. Ms. Tisch has the department’s first profit-sharing check for $375,355.20 framed on her office wall.

Earlier this year, the police started testing 300 noise sensors installed on rooftops across 15 square miles of the Bronx and Brooklyn that are programmed to detect the acoustic signature of gunfire. The $1.5 million system, developed by privately held SST Inc., in Newark, Calif., can pinpoint the location of a possible gunshot to within 25 meters and alert police within a minute whether the sound came from a gun or a harmless source such as a backfiring car or a slamming door.

By March 2017, the system will encompass 60 square miles of the city, police officials say.

To share so much data quickly, the department is spending about $340 million to link all its offices, vehicles and uniformed officers. The department is installing 700 miles of fiber-optic cable to connect stations in a private high-speed broadband network and putting wireless computer tablets in every police vehicle. In the past three months, the department has issued high-security smartphones to 12,000 police officers and expects every one of its 34,500 uniformed officers to be linked through a police-issued smartphone by March.

Other sensor networks are woven into the fabric of city life. Across 270 city blocks in Midtown Manhattan, 300 wireless microwave sensors, video cameras and EZ pass readers monitor traffic congestion. In Staten Island and the Bronx, traffic signals are set to automatically detect the approach of a city bus, through wireless sensors, and extend a green light to speed its passage. And in September, the U.S. Department of Transportation awarded New York $20 million to wirelessly connect 10,000 city-owned vehicles to each other in an experimental collision-avoidance system.

To better manage water use, the city has installed 817,000 wireless water meters that transmit readings and leak reports from homes and businesses as often as four times a day. To better handle the 10,000 tons of garbage that New Yorkers throw out every day, the city has installed 700 solar-powered trash cans on street corners that compact litter and wirelessly alert sanitation workers when they need to be emptied.

Dr. Dobler has taken a panorama image of the city every 10 seconds for two years. Researchers have used the photos to analyze energy use, air quality, light pollution, heat, traffic and sleep patterns. PHOTO: MICHAEL RUBENSTEIN FOR THE WALL STREET JOURNAL

At the same time, New York has made more of its own data public than any other city in the world. As of July, it has posted 1,350 data sets of city records online, covering subjects from school attendance, pothole work orders and fire department safety inspections, to noise complaints and construction permits. City officials expect to release 90 more such data sets by the end of the year.

Over the past three years, data analysts also have systematically rewired the city’s management systems, linking hundreds of older computer networks in 30 city agencies and utilities.

With so much pooled data, they have developed algorithms to speed emergency services, uncover tax fraud, detect landlords illegally harassing tenants and target buildings most at risk of fire. When Mayor de Blasio inaugurated free, full-day prekindergarten classes citywide in 2013, the analysts helped pinpoint thousands of eligible 4-year-old children by cross-matching birth records, anonymous social-service information and commercial marketing data for diaper services.

Early next year, New York will make its next digital leap forward. A consortium of companies led by Mr. Doctoroff is expected to install the first of 10,000 curbside transmitters designed to turn New York’s antiquated pay phone system into the world’s largest and fastest free municipal Wi-Fi network.

The $200 million LinkNYC project sets out a citywide network of distinctive kiosks that will offer free one-gigabit per second broadband service, about 20 times faster than available home Internet connections. It will offer free voice and video phone calls, and direct access to a variety of city services, such as 311 and 911 hotlines.

The Wi-Fi signals will be encrypted and no personal data will be collected, city officials say. If all goes according to plan, the LinkNYC network will be entirely paid for by advertising revenue and the sale of aggregated, anonymous data about how people use it. Mr. Doctoroff says the project is expected to generate $500 million in revenues for the city during its first 12 years of operation.

“When we have this network of units on the street, all connected by fiber, our potential to really look at the city and serve citizens in new ways will be completely unparalleled,” says Mr. Doctoroff. “This is going to profoundly change how we engage on the streets of New York.”