Return To Search

Sr. Databricks Architect & Developer

Location: Nashville, TN

Job Type: contract

Company: IDR

Category: Development and Architecture

Job Description Apply Now

IDR is seeking a Sr. Databricks Architect & Developer to join one of our top clients for a remote opportunity with occasional travel to IA. This role involves designing and developing scalable data pipelines and managing the Databricks platform within a dynamic data migration project in the government services industry.

Position Overview for the Sr. Databricks Architect & Developer:

• Develop high-performance scalable ETL pipelines using Databricks (PySpark, Python, Delta Lake, Databricks Workflows) for data migration from legacy source databases to PostgreSQL.
• Proficiently configure and manage Databricks Landing and Staging schemas to ensure secure and efficient data flows.
• Apply advanced database knowledge (SQL, Databricks SQL, and Postgres) to optimize load performance and large-volume cutovers.
• Contribute to data mapping, conceptual and technical design, and application testing with Databricks Notebooks.
• Implement data masking, support defect resolution, and actively contribute to successful data migration outcomes.

Requirements for the Sr. Databricks Architect & Developer:

• Experience with Databricks, Data Factory, PySpark, Pandas, SQL, PL/SQL, Spark Structured Streaming, AWS (S3, Glue, Lambda, Redshift, EMR), and Python.
• Skilled in ETL pipeline design, DevOps automation, data validation, and transformation.
• Knowledge of SFTP, DoDSAFE, NIPRGPT, and data visualization tools.
• Proven ability to optimize and monitor clusters, manage costs, and use cloud-native monitoring tools like Azure Monitor and CloudWatch.
• Strong background in designing and building high-performance ETL pipelines for data migration, including experience migrating data from VSAM files to PostgreSQL.

What's in it for you?

• Competitive compensation package
• Full Benefits; Medical, Vision, Dental, and more!
• Opportunity to get in with an industry leading organization.

Why IDR?

• 25+ Years of Proven Industry Experience in 4 major markets
• Employee Stock Ownership Program
• Dedicated Engagement Manager who is committed to you and your success.
• Medical, Dental, Vision, and Life Insurance
• ClearlyRated's Best of Staffing® Client and Talent Award winner 12 years in a row.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. IDR is subject to certain governmental recordkeeping and reporting requirements for the administration of civil rights laws and regulations. In order to comply with these laws, we invite employees to voluntarily self-identify their race or ethnicity, gender, and veteran status (if applicable). Submission of this information is voluntary and refusal to provide it will not subject you to any adverse treatment. The information obtained will be kept confidential and may only be used in accordance with the provisions of applicable laws, executive orders, and regulations, including those that require the information to be summarized and reported to the federal government for civil rights enforcement. You may access this form here.

Apply Now