グローバルでサービスを展開する大手外資系ITサービス企業でのBig Data Engineerの求人

- Lead an engagement in the space of Big Data with 10-15 members.
- Interact directly with end-customer as well as lead and manage technical team.
- Report into Enterprise Architect from technical point-of-view and Delivery Manager from delivery perspective.
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
- Work with QA Test analyst to ensure test coverage (Including Integration & Regression testing)
- Develop new program logic and/or assembles standard logic modules to create new applications

- 10 Years of overall technology experience and 5+ years of experience with Spark, Scala
- Experience with relational databases (Preferably Oracle)
- Programming SPARK in Scala & proficiency in SQL to write complex SQL queries
- Strong data analysis and troubleshooting skills
- Knowledge of shell scripts and other languages including Python, R, Java is plus
- Good understanding of AWS & Open Source big data technologies Airflow, Terraform, AWS Glue, AWS Lambda, Spark SQL
- Ability to implement both batch and streaming data pipelines in the AWS and change data capture (CDC) experience
- Experience with Databricks preferable
- Good knowledge of Linux OS and Shell scripting
- Experience working on complex distributed information systems, version control systems, referable SVN, git.
- Experienc in building Big Data platforms and areas of data governance, data security, data quality and related tools.