欧州最大のコンサルティングファームでのData Engineerの求人

Responsibilities:

・ Collaborate with users and project team on requirements and expectations

・ Analyze business requirements, create proper data solutions that meets functional and non-functional requirements

・ Define the data architecture, structure, storage, security model of data in HDFS, Hive, HBase, Solr, RestAPI

・ Design and implement highly scalable data pipelines in Scala/Spark/Java for processing of extreme large amount of data

・ Create and maintain reusable frameworks for data ingestion, validation, normalization, and transformation

・ Automate Unit testing and release with CI/CD pipeline

・ Create scripts for operations, releases, data migrations, data validations

・ Maintain data lineage information

・ Provide technical support for releases, issues, and incidents

Skills:

・ Solid knowledge and experience of Hadoop ecosystem: HDFS, Hive, HBase, Phoenix, Solr, Oozie, Impala, Knox, Zookeeper, YARN, Spark, Livy, HUE, Zeppelin

・ Strong programming skills in Scala and Spark

・ Strong capabilities in designing Database schemas on data platform

・ Good knowledge of Oracle database, MS-SQL Server, MongoDB, Kerberos, Linux, networking

・ Proficient in SQL, shell scripting, git, unit testing, CI/CD tools

・ Understanding of design principles, coding standards and best practices

・ Good problem-solving skills

・ Good data analytics skills

・ Excellent communication, presentation, and interpersonal skills

Preferable:

・ Understanding of REST API, micro-services, ETL, and Salesforce is preferred

・ Hand-on experience with Cloud services is preferred

・ Understanding of streaming data processing and change data capture is preferred

・ Experience with BI tools is preferred

Language:

・ Japanese: intermediate

・ English: intermediate