Senior Big Data Engineer

Job Description:

We are seeking an Senior Big Data Engineers to join our team immediately who can design, develop, maintain, and implement large-scale data processing systems for the Sherlock platform. In this role, you will work closely with cross-functional teams such as data scientists, software engineers, business analysts, and product managers to ensure the successful implementation and scaling of our data infrastructure.
As an ideal candidate, you must have a deep understanding of big data technologies and best practices and the ability to work with large amounts of data from various sources. One of the key skills required for this role is the ability to understand existing solutions and debug performance issues. This means being able to identify bottlenecks in the system, analyze the root cause of performance issues, and implement solutions to improve system efficiency.

Responsibilities:
• Design, develop, and implement large-scale data processing systems using Big Data technologies such as Spark, Cassandra, and Kafka
• Develop data pipelines for data ingestion, transformation, and storage
• Develop and maintain data models and database schemas for structured and unstructured data
• Identify bottlenecks in the system, analyze the root cause of performance issues, and implement solutions to improve system efficiency
• Implement data security and access control measures to ensure the confidentiality, integrity, and availability of data
• Develop and maintain data quality and validation frameworks to ensure the accuracy and completeness of data
• Develop and maintain monitoring and alerting systems to proactively identify and address data processing issues
• Work closely with data scientists, software engineers, business analysts, and product managers to understand data requirements and design solutions that meet their needs
• Continuously evaluate and implement new Big Data technologies and frameworks to improve the performance, scalability, and reliability of our data infrastructure
• Ensure compliance with data privacy regulations such as GDPR and CCPA

Requirements:
• Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related field
• 5+ years of experience in Big Data engineering or a related field
• Strong programming skills in one or more languages such as Python, Java or Scala
• Strong experience with Big Data technologies such as Spark, Cassandra, and Kafka
• Strong experience with data modeling and database design for structured and unstructured data
• Strong experience with data processing and ETL frameworks such as Apache NiFi, Airbyte, or Apache Beam
• Strong experience with data security and access control measures
• Strong experience with data quality and validation frameworks
• Experience with monitoring and alerting systems for Big Data technologies
• Experience with cloud-based Big Data platforms such as Databricks, AWS EMR or Google Cloud Dataproc
• Excellent communication and collaboration skills with the ability to work effectively in a cross-functional team environment