Brief Job Description:
Key Responsibilities:
•
Develop and maintain the backend infrastructure for the Data Lake, ensuring high availability, security, and scalability.
•
Design and implement data ingestion pipelines to efficiently load data from multiple sources into the Data Lake.
•
Develop and optimize ETL (Extract, Transform, Load) processes for data integration, transformation, and enrichment.
•
Implement data storage strategies and data partitioning for efficient retrieval and analysis.
•
Ensure data quality, consistency, and integrity across all data sources and destinations within the Data Lake environment.
•
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide backend support for BI solutions.
•
Monitor and maintain the performance of the Data Lake backend, troubleshooting and resolving any issues.
•
Stay current with the latest trends and best practices in Data Lake and backend technologies to continuously improve the data architecture.
Qualifications:
•
Bachelor’s degree in Computer Science, Information Technology, or related field.
•
3-5 years of experience in backend development for BI/Data Lake environments.
•
Strong experience with Data Lake technologies such as Azure Data Lake, AWS Lake Formation, or Hadoop.
•
Proficiency in scripting and programming languages like Python, Scala, or Java.
•
Experience with cloud platforms (AWS, Azure, or GCP) and data storage services (e.g., S3, Azure Blob Storage).
•
Expertise in building and optimizing ETL pipelines using tools like Apache Spark, Talend, Informatica, or Glue.
•
Strong SQL skills and experience with data warehousing solutions (e.g., Redshift, Snowflake, BigQuery).
•
Knowledge of data governance, security, and compliance best practices.
Preferred Skills:
•
Experience with big data tools such as Apache Kafka, Apache Hive, and Presto.
•
Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
•
Knowledge of CI/CD tools and practices for automated deployment and testing.
•
Understanding of real-time data processing and streaming technologies.
Key Competencies:
•
Strong analytical and problem-solving skills with attention to detail.
•
Excellent communication skills to work effectively with cross-functional teams.
•
Proactive, self-motivated, and a quick learner with a passion for data engineering.
•
Ability to work in a fast-paced environment and manage multiple priorities.
•
Bachelor’s degree in Business, Marketing, or a related field. MBA is a plus.
