Our client seeks a Specialized IT Consultant to design, develop, and optimize ETL processes in Azure Databricks, Delta Lake, and Python/PySpark for large-scale data warehousing, analytics, and data lake solutions. This role will be pivotal in ingesting, transforming, and managing structured and unstructured data using Databricks Workflows and Change Data Capture (CDC) tools like GoldenGate.
...
Advantages
- Hybrid work model (3 days onsite at the office)
- Hands-on role working with Databricks, Delta Lake, PySpark, and Azure cloud technologies
- Opportunity to design scalable ETL solutions following Medallion Architecture
- Work on large-scale enterprise data transformation projects
Responsibilities
Works in partnership with clients, advising them on information technology in order to meet their business objectives or overcome problems, works to improve the structure and efficiency of an organization’s I&IT systems. The I&IT Consultant may be used to provide strategic guidance to organizations with regard to Information Management & IT technology, IT infrastructures and the enablement of major business processes through enhancements to IT. Provides subject matter expertise in their field and highly expert technical assistance.
Additional Responsibilities:
This role will focus on data architecture, data warehousing, data lakes, and analytics. The individual will be designing, developing, maintaining, and optimizing ETL (Extract, Transform, Load) processes in Databricks for data warehousing, data lakes, and analytics. The individual will work closely with data architects and business teams to ensure the efficient transformation and movement of data to meet business needs, including handling Change Data Capture (CDC) and streaming data.
- Review business requirements, familiarize with and understand business rules and transactional data model.
- Define conceptual, logical model and physical model mapping from data source to curated model and data mart.
- Analyze requirements and recommend changes to the physical model.
- Develop scripts for the physical model, create database and/or delta lake file structure.
- Access Oracle DB environments, and set the necessary tools for developing solutions.
- Implement data design methodologies and historical and dimensional models.
- Perform data profiling, assess data accuracy, design and document data quality and master data management rules.
- Functionality Review, Data Load review, Performance Review, and Data Consistency checks.
- Help troubleshoot data mart design issues.
- Review the performance of ETL with developers and suggest improvements.
- Participate in end-to-end integrated testing for Full Load and Incremental Load and advise on issues.
Qualifications
Must Haves:
- 7+ years using ETL tools such as Microsoft SSIS, stored procedures, and T-SQL.
- 2+ Delta Lake, Databricks and Azure Databricks pipelines.
- Strong knowledge of Delta Lake for data management and optimization.
- Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
- 2+ years in Python and PySpark.
- Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
- Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
- SQL Server, Oracle.
Summary
This is an exciting opportunity for a Specialized IT Consultant/Data Engineer with strong expertise in Azure Databricks, Delta Lake, PySpark, and real-time data processing. If you have deep experience in ETL pipelines, data modelling, and performance optimization, apply today!
Also, remember that updating your profile on Randstad.ca helps us find you faster when we have roles that match your skills! So even if this role isn’t for you, please update your profile so we can find you!
We look forward to supporting you in your job search! Good luck!
Randstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada's workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.
Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle. We ask that all job applications please identify any accommodation requirements by sending an email to accessibility@randstad.ca to ensure their ability to fully participate in the interview process.
show more
Our client seeks a Specialized IT Consultant to design, develop, and optimize ETL processes in Azure Databricks, Delta Lake, and Python/PySpark for large-scale data warehousing, analytics, and data lake solutions. This role will be pivotal in ingesting, transforming, and managing structured and unstructured data using Databricks Workflows and Change Data Capture (CDC) tools like GoldenGate.
Advantages
- Hybrid work model (3 days onsite at the office)
- Hands-on role working with Databricks, Delta Lake, PySpark, and Azure cloud technologies
- Opportunity to design scalable ETL solutions following Medallion Architecture
- Work on large-scale enterprise data transformation projects
Responsibilities
Works in partnership with clients, advising them on information technology in order to meet their business objectives or overcome problems, works to improve the structure and efficiency of an organization’s I&IT systems. The I&IT Consultant may be used to provide strategic guidance to organizations with regard to Information Management & IT technology, IT infrastructures and the enablement of major business processes through enhancements to IT. Provides subject matter expertise in their field and highly expert technical assistance.
...
Additional Responsibilities:
This role will focus on data architecture, data warehousing, data lakes, and analytics. The individual will be designing, developing, maintaining, and optimizing ETL (Extract, Transform, Load) processes in Databricks for data warehousing, data lakes, and analytics. The individual will work closely with data architects and business teams to ensure the efficient transformation and movement of data to meet business needs, including handling Change Data Capture (CDC) and streaming data.
- Review business requirements, familiarize with and understand business rules and transactional data model.
- Define conceptual, logical model and physical model mapping from data source to curated model and data mart.
- Analyze requirements and recommend changes to the physical model.
- Develop scripts for the physical model, create database and/or delta lake file structure.
- Access Oracle DB environments, and set the necessary tools for developing solutions.
- Implement data design methodologies and historical and dimensional models.
- Perform data profiling, assess data accuracy, design and document data quality and master data management rules.
- Functionality Review, Data Load review, Performance Review, and Data Consistency checks.
- Help troubleshoot data mart design issues.
- Review the performance of ETL with developers and suggest improvements.
- Participate in end-to-end integrated testing for Full Load and Incremental Load and advise on issues.
Qualifications
Must Haves:
- 7+ years using ETL tools such as Microsoft SSIS, stored procedures, and T-SQL.
- 2+ Delta Lake, Databricks and Azure Databricks pipelines.
- Strong knowledge of Delta Lake for data management and optimization.
- Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
- 2+ years in Python and PySpark.
- Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
- Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
- SQL Server, Oracle.
Summary
This is an exciting opportunity for a Specialized IT Consultant/Data Engineer with strong expertise in Azure Databricks, Delta Lake, PySpark, and real-time data processing. If you have deep experience in ETL pipelines, data modelling, and performance optimization, apply today!
Also, remember that updating your profile on Randstad.ca helps us find you faster when we have roles that match your skills! So even if this role isn’t for you, please update your profile so we can find you!
We look forward to supporting you in your job search! Good luck!
Randstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada's workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.
Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle. We ask that all job applications please identify any accommodation requirements by sending an email to accessibility@randstad.ca to ensure their ability to fully participate in the interview process.
show more
