The data engineer role
The data science field is incredibly broad, encompassing everything from cleaning data to deploying predictive models. However, it's rare for any single data scientist to be working across the spectrum day to day. Data scientists usually focus on a few areas, and are complemented by a team of other scientists and analysts.
Data engineering is also a broad field, but any individual data engineer doesn't need to know the whole spectrum of skills. In this section, we'll sketch the broad outlines of data engineering, then walk through more specific descriptions that illustrate specific data engineering roles.
A data engineer transforms data into a useful format for analysis. Imagine that you're a data engineer working on a simple competitor to Uber called Rebu. Your users have an app on their device through which they access your service. They request a ride to a destination through your app, which gets routed to a driver, who then picks them up and drops them off. After the ride, they're charged, and have the option to rate their driver.
In order to maintain a service like this, you need:
- A mobile app for users
- A mobile app for drivers
- A server that can pass requests from users to drivers, and handle other details like updating payment information