Lead Data Engineer, Big Data - Data Platform

  • Palo Alto, CA, USA
  • Full-time

Company Description

Common Purpose, Uncommon Opportunity. Everyone at Visa works with one goal in mind – making sure that Visa is the best way to pay and be paid, for everyone everywhere. This is our global vision and the common purpose that unites the entire Visa team. As a global payment technology company, tech is at the heart of what we do: Our VisaNet network processes over 20,000 transactions per second for people and businesses around the world, enabling them to use digital currency instead of cash and checks. We are also global advocates for financial inclusion, working with partners around the world to help those who lack access to financial services join the global economy. Visa’s sponsorships, including the Olympics and FIFA™ World Cup, celebrate teamwork, diversity, and excellence throughout the world. If you have a passion to make a difference in the lives of people around the world, Visa offers an uncommon opportunity to build a strong, thriving career. Visa is fueled by our team of talented employees who continuously raise the bar on delivering the convenience and security of digital currency to people all over the world. Join our team and find out how Visa is everywhere you want to be.

Job Description

  • Drive Big Data Hadoop platform vision and strategy for an entire team of Data Engineers
  • Build and expand on Visa Data Platform’s Hadoop Open source Big data Clusters within an agile collaboration environment
  • Work on big projects across multiple product teams, and with a wide range of stakeholders
  • Imagine working in a startup in the Apache Hadoop Open source Big data platforms domain tasked with processing the plethora of payment data that comes to Visa on an Hourly basis
  • Observe the thriving open source projects in the descriptive and predictive analytics big data space and bring it into Visa to further enrich customer offerings
  • Must love coding – prepare to spend 80% of the time on hands-on development with development teams
  • Deliver end-to-end applications and frameworks ranging from system programming to micro-services to simple front-end applications
  • Be responsible for automated deployment of the product through CICD-based deployments into Kubernetes, Virtual Machines and ability to do small scale implementation fast implementation for localization requirements
  • Security-first mindset to ensure that any software built will satisfy Visa’s stringent security standards
  • Ability to own complete project or a subject area delivery while leading team members in a scrum setting
  • Mentor/manage team members, develop procedures and best practices standards


Basic Qualifications:

  • 10 years of work experience with a Bachelor’s Degree or at least 8 years of work experience with an Advanced Degree n Computer Science or related field (e.g. Masters/MBA/JD/MD) or at least 3 years of work experience with a PhD

Preferred Qualifications:

  • 12-15 years of work experience with a Bachelor’s Degree in Computer Science or related field, or 8-10 years of experience with a Masters in Computer Science or related field
  • Active development experience with Java, Python, Spark, Hive, Kafka, Kstreams, NoSql Databases, while having core fundamentals experience of Apache Hadoop components
  • Experience in building Apache Open source projects, contributed to open source projects and deploying into containers platform 
  • Hand-on experience in optimizing existing Big Data application written in Spark, Hive. Optimizing 
  • Can navigate a Linux terminal with ease
  • Relational database and SQL development experience required
  • Understanding best practices for Big Data (in Hadoop), data warehousing, consumer analytics, knowledge management and key understanding of streaming and other NoSql databases
  • Hands-on experience in developing and managing development of Data Integration applications for large corporations with experience in both batch and online systems.
  • Have experience delivering a localization big data implementation is a big plus
  • In-depth knowledge of the software development life cycle required
  • Strong communication skills with ability to present complex ideas and document in a clear and concise way
  • Quick learner; self-starter, detailed and thorough
  • Proactive sense of urgency and “can do” attitude

Additional Information

Travel Requirements: This position requires the incumbent to travel for work 5% of the time.

Mental/Physical Requirements: This position will be performed in an office setting.  The position will require the incumbent to sit and stand at a desk, communicate in person and by telephone, frequently operate standard office equipment, such as telephones and computers, and reach with hands and arms. 

Working Hours Incumbent must make themselves available during core business hours.

Visa is an EEO Employer.  Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status.  Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

Privacy Policy