Senior Software Architect – Data Pipelines and Distributed S en The Muse

The Muse is the leading job marketplace and advice destination that helps individuals connect with jobs, companies, and careers they love. We help millions of jobseekers every month while helping companies build a more robust, informed candidate pipeline and strengthen their employer brand through authentic storytelling and targeted job postings.

To help us in our next phase of growth, we are looking for a passionate Senior Software Architect with deep experience building scalable, high-performance data pipelines, feeds, and distributed systems who thrives in an agile, iterative environment where innovation and a bias for progress are critical components.

• Design and Architecture: Develop and maintain robust architectural blueprints.
• Performance Optimization: Identify and address performance bottlenecks, ensuring optimal efficiency and responsiveness.
• Scale, design, and optimize data pipelines
• Feeds: Troubleshoot and resolve issues in job feed processing for clients or programmatic sources.
• Messaging systems: Architect and improve the existing Kafka-based messaging systems to handle high throughput and low-latency requirements.
• Distributed Systems: Architect, design, and implement distributed systems that can handle large-scale data processing and real-time analytics.
• API integration
• Vendor integration: Integrate and manage additional vendors into our programmatic network, adapting to new company feed setups and debugging existing sources as needed.
• Elastic Search optimization: Drive changes and best practices for our ES/Open Search index as part of our wider platform initiatives.
• New feature development: Work closely with the team to build new features and enhance site architecture to support platform functionality.
• Data ingestion: Implement and consume data feeds in various formats, such as JSON, XML, etc., ensuring compatibility and scalability.
• Docker and infrastructure: Use Docker for containerized services and contribute to the management of PostgreSQL, SQLAlchemy, and other components of our infrastructure.
• Technology Selection: Evaluate and recommend appropriate technologies and frameworks for data processing, storage, and distribution, aligning with business needs and technical constraints, as needed
• Collaboration

Mentorship: Provide technical guidance and mentorship to junior team members, fostering a culture of continuous learning and development.

• Proven experience with AWS and Kafka for building and maintaining, designing and implementing scalable, high-performance data architecture, including messaging-driven systems.
• Knowledge of job feed management, development, including programmatic.
• Strong proficiency in Python and familiarity with asynchronous programming.
• Experience with distributed systems and understanding of large-scale data pipeline architecture.
• Hands-on experience integrating and scaling data technologies such as Kafka, ES, DBT and Databricks.
• Experience with cloud platforms (AWS, Google Cloud) and cloud-native technologies.
• Familiarity with Docker, Postgres, and SQLAlchemy for managing databases and infrastructure.
• 8+ years of experience in software development, with a strong focus on data pipelines and distributed systems.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and as part of a team.

• Deep understanding of asynchronous processing and its use in large-scale distributed systems.
• Experience working with JSON, XML, and other structured data formats.
• Familiarity with job board integrations and related technologies.
• Experience with data warehousing and data lake technologies.
• Experience with real-time data processing and streaming technologies.