A Brief History of Big Data (in Federal Transportation)
August 3, 2016|Thomas Grogan
Guest op-ed by Thomas Grogan, Senior Economist, HDR, Inc.
Big Data is big.
Big enough to generate 3.5 billion search results each day.
Big enough to process over 25 million transit trips daily, to find patterns in ridership to improve safety, service and reduce costs.
Big enough to potentially eliminate six billion metric tons of GHG pollution by reducing inefficient vehicle operations and travel demand.
Big Data can seem complicated because it means many things to many people. But it is not new. At its core, Big Data is still the collection, analysis, and communication of information. What has changed over time is the size, speed, and variety of data which is collected and communicated. However, the challenge lies in using innovative methods to analyze the data and gather meaningful insights which can be used to inform decisions makers.
The Federal Government has been collecting data in increasing size and scale since the first U.S. decennial census in 1790. The methods were rudimentary by today’s standards: the law required that every household be visited; that completed census schedules be posted in public places; and that “the aggregate amount of each description of persons” for every district be transmitted to the president.
Even at a time of great uncertainty in our nation’s future, the government understood the value of quantitatively capturing a moment in time in the country. The total cost was $44,000, or approximately 0.55% of Federal spending at the time (assuming $8 million in nominal terms, based on various estimates). As the nation grew, population data alone would not be enough to provide decision makers with the tools to solve the problem faced by citizens. The next step would be to collect data for how the population moved.
The first mail-out census in 1960 was also the first time that transportation information was formally collected in the decennial census. Under the “Employment Status and Work Experience” section, the multiple choice question was asked “How did this person get to work last week?” This data began to inform transportation planning products used by the Federal Highway Administration. The concentrated effort by the Federal government to collect transportation data that began as one observational data series 56 years ago has grown exponentially in scope, type, and volume of data, as a result of technological progress. And this collected information influences programmatic spending, determines compliance with laws, and guides private sector investment and operational decisions.
In June 2016 alone, the Department of Transportation updated or made available 9 data sets, ranging from numerical to geographic across different modes and types (employment, freight, traffic, performance, maps). Much of this information can be collected and reported automatically by devices; it does not require mailing paper surveys or interviewing people in person. Unlike the data from the first Census, which was only accessible at a single geographical location in a printed document, these data sets are accessible by anyone in the country with a strong internet connection and the right tools and skills to sort through them.
In many cases, real-time information from transit agencies is made developer-friendly in the form of APIs (application program interfaces). When you go online or to an app to track a bus route or order a car service, you’re accessing Big Data, right at your fingertips. A current challenge for agencies and organizations is processing this data on the back-end to gain the appropriate insights for long run transportation policies.
Today, the Census collects much more than the original question asked, and the government established programs with dedicated resources just for transportation data. DOT inherited decades-old data collection programs on the railroad and aviation sectors when they absorbed the functions of the Interstate Commerce Commission and the Civil Aeronautics Board via deregulation. Some might argue that DOT established its own “Big Data” program with the creation of the Bureau of Transportation Statistics (BTS) under ISTEA in 1991. Soon after, the advances in IT technology helped expand the Department’s data and research programs at a rapidly increasing pace.
With the creation of the Research and Innovative Technology Administration (RITA) through the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU), BTS was moved to this administration in 2005. Currently, the Office of the Assistant Secretary for Research and Technology (OST-R) oversees 8 multi-modal research programs, including BTS.
Other organizational developments that are occurring across all agencies include the growth in Chief Innovation and Chief Data Officers (CDO); US DOT appointed its first CDO in July 2014. Perhaps as important as the number of programs are the strategic initiatives the OST-R is spearheading.
The extremely popular “Smart Cities Challenge” by DOT was primarily championed by the OST-R. Making the knowledge and lessons learned of the “Smart Cities Challenge” available to everyone for replication is an important component of this program. This makes the implications for structured data programs and analysis all the more important as the transportation industry embraces Big Data.
This increase in Big Data isn’t limited to just transportation. The future of urban areas depends on accessing Big Data from a wide range of industries including energy, education, health, employment, environment, and housing, as well as transportation. For these “Digital Cities” to become a sustainable reality, we must learn to analyze these different sources of data in conjunction with each other.
The views and opinions expressed in this article are those of Thomas Grogan and do not express those of HDR Inc. or of the Eno Center for Transportation.
September 29, 2023 | Kirbie Ferrell
September 29, 2023 - On September 26 and 27, the fourth annual MOVE America Conference was held in Austin, Texas.
July 21, 2023
July 21, 2023 - Sometimes it is the oldest of references that provide the most compelling insights into today’s pressing...
March 21, 2023 | Karen Price
March 22, 2023 - When we look around today, we see an increasing array of transportation options available to connect...
July 6, 2022 | Jonathan Hammond
July 7, 2022 - It's time for the public transport industry to begin to reflect on the tangible lessons that...
July 6, 2022 | Jonathan Hammond
July 7, 2022 - While they do not usually operate transit, state departments of transportation (DOTs) certainly “drive the bus,”...
October 29, 2021 | Katie Donahue
October 29, 2021 - In light of the catastrophic Colonial Pipeline ransomware attack in May 2021, the Homeland Security Committee...
May 20, 2021 | Madeline Gorman
May 21, 2021 - Improving the speed and reliability of LA County’s bus network will reduce transit travel times, as...
April 16, 2021 | Paul Lewis
April 15, 2021 - This week, Eno released three evaluation reports that aim to understand how the pilots performed during...
January 29, 2021 | Paul Lewis
January 29, 2021 - The prospect of automating bus fleets has agencies very interested in the technology. But if agencies...
December 11, 2020 | Jeff Davis
December 11, 2020 - The Census Bureau this week released the full dataset for the 2019 American Community Survey, including...
September 25, 2020 | Madeline Gorman
September 25, 2020 - There has never been another year quite like this one. But through it all, core values...
June 5, 2020 | Madeline Gorman
June 5, 2020 - By understanding the motivation behind people’s actions, public agencies can do a far better job of...