グローバルでサービスを展開する大手外資系ITサービス企業でのBig Data Engineerの求人
求人ID:1271759
募集終了
転職求人情報
職種
Big Data Engineer
ポジション
担当者〜
おすすめ年齢
20代
30代
40代
50代以上
年収イメージ
応相談(経験・能力を考慮の上当社規定により決定)
仕事内容
- Lead an engagement in the space of Big Data with 10-15 members.
- Interact directly with end-customer as well as lead and manage technical team.
- Report into Enterprise Architect from technical point-of-view and Delivery Manager from delivery perspective.
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
- Work with QA Test analyst to ensure test coverage (Including Integration & Regression testing)
- Develop new program logic and/or assembles standard logic modules to create new applications
- Interact directly with end-customer as well as lead and manage technical team.
- Report into Enterprise Architect from technical point-of-view and Delivery Manager from delivery perspective.
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
- Work with QA Test analyst to ensure test coverage (Including Integration & Regression testing)
- Develop new program logic and/or assembles standard logic modules to create new applications
必要スキル
- 10 Years of overall technology experience and 5+ years of experience with Spark, Scala
- Experience with relational databases (Preferably Oracle)
- Programming SPARK in Scala & proficiency in SQL to write complex SQL queries
- Strong data analysis and troubleshooting skills
- Knowledge of shell scripts and other languages including Python, R, Java is plus
- Good understanding of AWS & Open Source big data technologies Airflow, Terraform, AWS Glue, AWS Lambda, Spark SQL
- Ability to implement both batch and streaming data pipelines in the AWS and change data capture (CDC) experience
- Experience with Databricks preferable
- Good knowledge of Linux OS and Shell scripting
- Experience working on complex distributed information systems, version control systems, referable SVN, git.
- Experienc in building Big Data platforms and areas of data governance, data security, data quality and related tools.
- Experience with relational databases (Preferably Oracle)
- Programming SPARK in Scala & proficiency in SQL to write complex SQL queries
- Strong data analysis and troubleshooting skills
- Knowledge of shell scripts and other languages including Python, R, Java is plus
- Good understanding of AWS & Open Source big data technologies Airflow, Terraform, AWS Glue, AWS Lambda, Spark SQL
- Ability to implement both batch and streaming data pipelines in the AWS and change data capture (CDC) experience
- Experience with Databricks preferable
- Good knowledge of Linux OS and Shell scripting
- Experience working on complex distributed information systems, version control systems, referable SVN, git.
- Experienc in building Big Data platforms and areas of data governance, data security, data quality and related tools.
就業場所
就業形態
正社員
企業名
グローバルでサービスを展開する大手外資系ITサービス企業
企業概要
グローバルでサービスを展開する大手外資系ITサービス企業
企業PR
数々のグローバル企業に価値をもたらしてきた、業界最高水準のITサービスやソリューションをお客様に提供します。
業務カテゴリ
組織カテゴリ
備考
●勤務地補足
東京勤務とリモートワークのハイブリッド
東京勤務とリモートワークのハイブリッド