Senior Hadoop / Spark Software Engineer / Platform Engineer

Sunnyvale, CA, US

Job Description / Skills Required


The Mellanox Big Data and Machine Learning R&D team is building cutting-edge open-source integrated solutions for accelerating known platforms such as: Apache Spark, Hadoop, TensorFlow, Caffe and more. Our ground-breaking high-speed end-to-end networking solutions are a perfect match for today’s growing needs from distributed systems. Those solutions have already proved themselves in many fields, such as: High Performance Computing, Cloud storage, Cloud compute clusters and many more. Many Big Data and Machine Learning frameworks do not take advantage of these advanced technologies, and our target is to introduce these powerful capabilities to popular frameworks and contribute the code to the open-source community. Among Mellanox high-end networking capabilities are: 100Gb/s Ethernet and InfiniBand, RDMA (Remote Direct Memory Access), GPUDirect, SHArP (Scalable Hierarchical Aggregation and Reduction Protocol), FPGA accelerations and more.


As part of our team, you’ll design and implement advanced acceleration solutions that aim at introducing new hardware capabilities into popular frameworks.

You will be trusted to:
Develop new solutions for network acceleration
Promote our design approaches in the relevant open source communities
Contribute code to upstream
Collaborate with design partners and customers for driving our solutions into production


You will need to have:
Proven contributions to open-source communities in this field
Expert-level understanding of one or more Big Data or Machine Learning platform internals, such as Apache Spark, Hadoop, TensorFlow, Caffe, etc.
Highly technological out-of-the-box thinking
Solid understanding of Linux operating system a plus
Hands on experience working on Hadoop, Spark or other popular Big Data and Machine Learning framework for 2 years or more
Deep understanding of internals of core Hadoop/Spark components – YARN, HDFS, and MapReduce
Strong experience in designing, creating, packaging and integrating Java based projects.