Data Engineer

Description:

• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement solutions that meet business needs
• Develop and maintain ETL processes to extract, transform, and load data into Databricks clusters
• Optimize and tune Databricks jobs for performance, reliability, and efficiency
• Troubleshoot and resolve issues related to Databricks clusters and jobs in a timely manner
• Stay up-to-date with industry trends and advancements in big data technologies, providing recommendations for continuous improvement
• Provide technical guidance and mentorship to junior team members
   

Requirements
 

• Proficiency with Databricks, including data ingest, schema design for Bronze, Silver, and Gold layers, and ability to design and implement end-to-end pipelines
• Knowledge of Apache Spark and Databricks architecture
• Experience with relational SQL and NoSQL databases
• Strong programming skills in languages such as Python and/or PySpark
• Experience with data modeling, ETL processes, and data integration
• Familiarity with cloud platforms, especially Azure
• Broad understanding of data warehousing and Big Data concepts, and ability to apply those concepts to implementation
• Excellent problem-solving and analytical skills
• Effective communication and collaboration skills
• Relevant certifications in Databricks or big data technologies are a plus