Big data engineers communicate with business users and data scientists to understand the business objectives and translate those objectives into data-processing workflows.
Big data engineers should have a strong knowledge of statistics, extensive programming experience, ideally in Python or Java, and the ability to design and implement solutions for big data challenges. Knowledge and experience in data mining, processing large amounts of raw data, and designing and maintaining relational databases for storage and data acquisition are desired. Experience with NoSQL is preferred.
Employers often require a bachelor’s degree in a related field and four to six years of experience.
Typical duties of a big data engineer
- Gathering and processing raw data and translating analyses
- Evaluating new data sources for acquisition and integration
- Designing and implementing relational databases for storage and processing
- Working directly with the technology and engineering teams to integrate data processing and business objectives