Past Programs

ML Serving infrastructure is becoming ubiquitous in the emerging ML industry as well as in public cloud offerings. Existing solutions overwhelmingly rely on serving models as containers where one container hosts a single model with all its required dependencies. Salesforce and Einstein Platform has a unique multi-tenancy approach that heavily relies on AutoML and enforces automated feature engineering, training, and serving of separate models per tenant, ultimately resulting in serving a total of up to hundreds of thousands of models. In addition, the model size, initialization time and popularity/volume can vary widely based on the underlying customer base of each tenant, introducing what we call the model balancing problem. We present our approach to scale to very large number of models using multi level routing and load balancing, sharing hundreds of models within each container as well as utilizing sophisticated metric driven mechanisms for model initialization, warmup and model balancing. In addition we’ll present our solution for managing model versions and dependencies in shared container scenarios and finally lessons learned on our journey in this nascent space.

The Perlmutter Supercomputer at the National Energy Research Scientific Computing (NERSC) Facility will be deployed in early 2021 to support the large scale simulation, data analysis and machine learning workloads of the Department of Energy, Office of Science funded researchers. Named after Nobel Laureate Saul Perlmutter, the 100 PF Cray system will include next-generation AMD CPUs and NVIDIA GPU compute nodes, 30PBs of an all-flash Lustre file system and the Cray Slingshot Ethernet compatible interconnect. This talk will discuss the Perlmutter architecture, applications, software, and plans to support a growing and diversifying workload including AI and large scale data analysis.

Diversity and Inclusion: The engineering profession cannot reach its maximum potential without embracing the diversity of ideas and experiences that come from people of different backgrounds and creating an inclusive culture where all people can thrive. This talk discuss why diversity is important in engineering, provides a snapshot of diversity metrics in the profession, and highlights how individuals, universities, companies, and the IEEE are working to improve diversity and inclusion in engineering.

We are at the beginning of a new technological transformation called the AI revolution. This is characterized by speed (driven by exponentially fast computing and communication, e.g. 5G), scale (driven by globalization), big data (driven by unprecedented storage capabilities and distributed compute) and, critically, by having platforms that go from being purely digital to be intimately connected to the physical world (internet of things, robotics, etc…). These ingredients are creating the perfect conditions for fueling a new generation of computing tools based on AI technology that allow generating predictions and guiding humans in making critical decisions. Unlike other technologies, AI is a global phenomenon that has low barrier for entry but with profound transformative effects on entire industries such as transportation, manufacturing, agriculture, construction, retails and health-care, to cite a few. We argue that despite all of the potential glowing advances, however, AI-driven technology is still far from achieving the level of accuracy and reliability that is needed for many critical applications — more research and advancements are still needed to enable machines that can perform on par with humans in many high level tasks. In this talk I will examine recent progress in robotics and machine vision in overcoming some of these limitations and open the door for a new generation of AI systems. I will conclude with a brief overview of benefits and risks of the AI revolution and the opportunity to foster a dialogue on possible recommendations for a responsible leadership on this matter.

A fundamental challenge in running software services is managing the conflict between agility and stability. On the one hand, market forces create unending pressure to add features fast. On the other, the faster you try to run, the more likely you are to break something. In this talk, we contend that the "trade-off curve" between agility and stability is fixed for a given organization. We also describe the forces that cause this curve to degrade over time - including the underappreciated challenges of increasing the size of the team. We conclude by discussing investments that can improve the shape of the curve for your organization.

There are multiple steps involved in transitioning a server from the factory to being fully provisioned for an intended workload. These steps include finding the optimal slot for the hardware and to compose the required resources on the hardware for the intended workload. There are many different factors that influence the placement of server hardware in the datacenter, such as physical limitations to connect to a network be it Ethernet or storage networks, power requirements, temperature/cooling considerations, and physical space, etc. In addition to this, there may be custom requirements driven by workload policies (such as security, data privacy, power redundancy, etc.). Once the server has been placed in the right slot it needs to be configured with the appropriate resources for the intended workload. CIED will provide a ranked list of locations for server placement based on the intended workload, connectivity and physical requirements of the server. Once the server is placed in the suggested slot, the solution automatically discovers the server and composes the required resources (compute, storage and networks) for running the appropriate workload. CIED reduces the overall time taken to move hardware from factory to production and also maximizes the server hardware utilization while minimizing downtime by physically placing the resources optimally. From the case study that was undertaken, the time taken to transition a server from factory to being fully provisioned was proportional to the number of devices in the datacenter. With CIED this time is constant irrespective of the complexity or the number of devices in a datacenter.

Data Analytics started with Hadoop in 2005 when it made possible to mine large data sets to extract intelligence from it. These were batch jobs that could run for hours. Then a natural evolution happened around 2010 when Apache Spark and Kafka enabled stream processing of data at scale. The stream processing movement reduced the time from data-to-insights to tens of minutes. But some industries demanded lowering query latencies along with lowering data latencies. For example, the Facebook Newsfeed and the Linkedin FollowFeed applications required super-low data latency. It is 2018, enter Analytics-on-the-fly!

In this talk, Dhruba walks you through the history and evolution of the original Hadoop architecture that separates compute from storage. He inspects the Lambda Architecture and the reasons why it is popular for Stream Processing systems. Then he goes on to describe the Aggregator-Leaf-Tailer (ALT) architecture that provides Analytics-on-the-fly by allowing fast queries on semi structured data. He peels apart the disaggregated and cloud-friendly nature of ALT that allows one to scale compute, storage, data rates and query volumes independently. He describes how replacing the map-reduce framework in earlier generation Hadoop with an RocksDB-based indexing framework in the ALT architecture reduces query latency. He talks about the CQRS pattern in this architecture that isolates query latencies from bursty steams. He elaborates why these new applications demand higher concurrent queries and how the ATL architecture provides it. The talk concludes with a description of how the ALT architecture is already in production in a set of applications at Facebook, Linkedin and Rockset.

Recent successes of AI in several application domains attest the importance of big data in achieving high performance in domain-specific metrics. In the healthcare domain, however, big data in the million scale is generally not available. Furthermore, safeguarding data privacy is extremely crucial. This talk first enumerates key data issues in applying AI in the healthcare domain including data scarcity, out of distribution, and data privacy. Then, we present remedies including transfer learning for remedying data scarcity, knowledge-guided multimodal learning for out of distribution generalization, and distributed ledgers for preserving data privacy.

Soteria is a user right management system designed to safeguard user-data privacy in a transparent and provable manner in compliance to regulations such as GDPR and CCPA. Soteria represents user data rights as formal executable sharing agreements, which can automatically be translated into a human readable form and enforced as data are queried. To support revocation and to prove compliance, an indelible, audited trail of the hash of data access and sharing agreements are stored on a two-layer distributed ledger. The main chain ensures partition tolerance and availability (PA) properties while side chains ensure consistency and availability (CA), thus providing the three properties of the CAP (consistency, availability, and partition tolerance) theorem. Besides depicting the two-layer architecture of Soteria, this paper evaluates representative consensus protocols and recommends side-chain and inter-chain management strategies for improving latency and throughput.

The evolution of the Telecom network to 5G and Edge Computing requires IT compute, storage and networking infrastructure from multiple vendors to be deployed across potentially thousands of geographically distributed and diverse points of presence, including central offices, cell tower huts, wiring closets as well as traditional data centers. This sharply contrasts with the comparatively few homogeneous hyper-scale data centers of the cloud service providers. Telecom operators are demanding open, scalable infrastructure management for this distributed multi-vendor infrastructure based on widely adopted industry standards. In this paper, we are addressing these unique challenges through an open-source, Distributed Management Task Force (DMTF) Redfish®[1] based Resource Aggregation function, helping create a vibrant open-source infrastructure management community around the Resource Aggregator will allow in the building of next-generation telecommunications and other highly distributed networks.

The edge-to-cloud infrastructure and machine learning (ML) are becoming increasingly coupled in the real world. In the last decade, large-scale voice and image applications have driven phenomenal breakthroughs in ML algorithms that we all use in our daily interactions with internet service providers. More recently, different industry verticals have started to rapidly adopt similar AI approaches, but their needs extend well beyond standard datacenter applications. These new uses of ML involve complex and noisy multi-sensor data, sparsely labeled ground truth, and complex deployment environments that span from the edge to the cloud. This talk covers some of the aspects of the intricate relationship between ML and the edge-to-cloud world. This relationship goes both ways, and the talk presents two illustrative examples. In one direction, we need a new infrastructure for optimized ML. So, the first example discusses “infrastructure for AI/ML”: an architecture blueprint for ML in a Highly Autonomous Driving (HAD) application that spans from instrumented test cars (at the edge) to the training core (in the cloud). In the other direction, ML helps to optimize the infrastructure. The second example discusses “AI/ML for infrastructure”: the use of ML for operational intelligence (AI-Ops) to automate the monitoring and predictive maintenance in a high-performance datacenter AI-Ops takes a holistic view which includes both IT telemetry, and facility sensors at the edge of the datacenter.

Federated Learning enables mobile devices to collaboratively learn a shared prediction model while keeping all the training data on device, decoupling the ability to do machine learning from the need to store the data in the cloud. Topics to be presented include: (1) how federated learning differs from more traditional machine learning paradigms; (2) practical algorithms for federated learning that address the unique challenges of this setting; (3) extensions to federated learning, including differential privacy, secure aggregation, and compression for model updates, and (4) a quick overview of federated learning applications and systems at Google.

Content Co-Owner
Peter Kairouz
Research Scientist
Google

All retailers want to know their target buyers better. However, understanding the past and present of their interactions simply isn’t enough these days and predictive analytics is the next step to better understanding their customers. In this session, topics of discussion will include, yet will not be limited to:

• How ML can enable price optimization, product placement and assortment selection
• Using machine learning algorithms effectively for generating suggestions for substitute and complimentary items
• Using deep learning and reinforcement learning to improve order forecasting
• Utilizing optimization algorithms to reduce store costs by optimizing replenishment cycle and safety stock
• Scaling algorithms to generate recommendation for individual stores and to monitor their performance

The move to cloud to leverage agility and scale initiated a fundamental shift from monolithic to microservices architecture. While this shift improved the agility of application development (Dev) teams, it created significant challenges for Operations (Ops) teams. These challenges result from the variability in application structure and behavior, shifting bottlenecks, and limited visibility and control of cloud infrastructure resources and services. Traditional performance management approaches such as queuing networks or using heuristics that capture static relationships between performance and resources are also no longer applicable with highly dynamic virtual cloud environments.

We have designed and implemented a new model-based control platform for managing microservices application performance. Relying on existing monitoring, events and configuration data, especially those collected from the open CNCF framework, such as Kubernetes and Prometheus, and without instrumenting the application code, we automatically discover and build the application structure. Subsequently, using both machine learning (ML) and a priori knowledge of known services, we build a predictive application behavior model for the complete application. The application model and structure can provide sufficient granularity to detect and predict performance problems, isolate the cause of incidents, and recommend remedial actions to the infrastructure and services for problem resolution.

Datacenters are the backbone of internet commerce and cloud storage, and they are also hugely important to support the computational and transactional needs of today’s corporate world. They currently consume 3% the world’s electrical supply, and the world produces upwards of 50 million metric tons of e-waste annually. Thus, Total Cost to the Environment (TCE) is as important as Total Cost of Ownership (TCO) for datacenters.

Intel has a huge datacenter investment with over 290,000 servers (which include over 2 million Xeon high clock cores), over 384 petabytes of storage, and more than 545,000 network ports within its 86-Megawatt data center capacity. This infrastructure is required to support efforts including complex chip design, and Intel’s overall computing needs have grown in excess of 6000% over the past 13 years.

This talk will discuss aspects of the complexities of building and maintaining these datacenters, as well as how green computing initiatives have delivered substantial power reduction costs while significantly contributing towards reducing greenhouse gas emissions. Being Green is not just running datacenters at the most efficient Power Usage Effectiveness (PUE) levels, but also reducing e-waste.

With a growing library of over 310 million images and 17 million videos, Shutterstock needs to quickly ingest, review and publish over 200,000 assets per day efficiently to enable its customers around the world to deliver impactful stories. Using its own innovative practices, Shutterstock developed robust internal libraries that utilize RabbitMQ message-brokers to support multiple applications. Shutterstock’s Software Development Manager, Anusha Dayananda, will provide her observations and learnings on ensuring stability and reliability for distributed systems. The talk will cover best practices on eliminating central control so that engineering teams can easily adopt and operate.

Key Takeaways:

• Best practices for eliminating risks against cluster failures
• How to reduce cross-team dependency
• Ways to optimize engineering workflows for time savings and efficiency

We’re well into the Big Data era. Most organizations embraced collecting data and analyzing what’s happening inside their products. This is crucial to their success, not only to understand what works but also to optimize their services and increase their value to their customers. Several industries are being disrupted just by using technology to optimize existing processes. Think about cabs, short term rentals, or co-working spaces.

There’s also discussion around what constitutes “big” data. Here, we’re not only talking about large volumes of data produced by the likes of Google, Facebook and others very large companies. We’re also talking about the multitude of various datasources and the many teams using them and producing derived datasets. The concept of central data team that does all the data related work is outdated and the entire organization should become an ecosystem where teams depend on each other. Central data teams now become enablers, coaching and providing a safe and flexible environment to move fast while bringing transparency to the increasing complexity of interdependent system. Data processing and micro services have similar requirements in terms of ownership, monitoring and dependency management.

In this talk we will discuss the principles to follow while building the data platform enabling the entire organization to build data driven products, whether using insights from the data or using data directly to build features (for example recommendations).

Every team can consume and produce data using explicit contracts: What they share or don’t, the level of service they provide and the quality of the data. We need to build visibility to the entire org and help evolve the dependency graph with global lineage and schema evolution.

The platform is self-service and gets out of the way to empower users to do the right thing. It provides a safe environment where mistakes can be easily mitigated and the scope of their impact limited. It is flexible to allow users to pick the best tool for the job while facilitating interdependencies. Streaming and batch processing are complementary and work together. Governance is delegated to the appropriate stewards. Sensitive data is properly annotated and secured and its usage tracked and controlled. Cloud being omnipresent, users expect not to have to worry about where the processes are run or where the data is stored. The platform is expected to scale transparently and be billed by the minute.

We’ll discuss the best tools making the data platform and how to build the missing pieces.

As distributed databases expand in popularity, there is ever-growing research into new database architectures that are designed from the start with built-in self-tuning and selfhealing features. In real world deployments, however, migration to these entirely new systems is impractical and the challenge is to keep massive fleets of existing databases available under constant software and hardware change. Apache Cassandra is one such existing database that helped to popularize “scale-out” distributed databases and it runs some of the largest existing deployments of any open-source distributed database. In this paper, we demonstrate the techniques needed to transform the typical, highly manual, Apache Cassandra deployment into a self-healing system. We start by composing specialized agents together to surface the needed signals for a self-healing deployment and to execute local actions. Then we show how to combine the signals from the agents into the cluster level controlplanes required to safely iterate and evolve existing deployments without compromising database availability. Finally, we show how to create simulated models of the database’s behavior, allowing rapid iteration with minimal risk. With these systems in place, it is possible to create a truly self-healing database system within existing large-scale Apache Cassandra deployments.

Mantis and Mantis Query Language (MQL) is an open source platform which provides Netflix service owners with infrastructure and a DSL for instantly accessing real-time insights across the entire ecosystem. Currently Mantis moves over 2 trillion events per day encompassing several petabytes of data. We'll explore the architecture of Mantis and how it enables low-cost real-time data as well as the design of the query language which provides an expressive abstraction over all of this machinery. Ultimately we'll see how Mantis enables use cases from outlier/anomaly detection through A/B testing and much more for a reliable cloud experience and how you can take advantage of the open source version to observe and monitor your own cloud environment.

During challenging times, leaders must step up to communicate with clarity, transparency and credibility. Today, companies have a unique opportunity to act as a trusted, expert voice to their employees, customers, partners and communities. Join this talk to hear more about lessons learned from the front lines of communications during COVID-19 and how you can put these marketing and communication strategies to work to drive engagement and trust with your team members and colleagues, and ultimately to deliver better business and technical outcomes.

When a file needs to be updated by thousands of tests running on a compute farm, waiting for a chance to update slows down the overall runtime as the number of parallel test runs increases. We implemented a file cache that allows each test to write a small temporary file that either updates the shared file or is stored for later updating. A network file lock controls access to the shared file. If the file lock is not acquired, temporary files are merged to reduce the number of future file reads. Temporary files are removed only if their contents is successfully added to the share file or merged into another temporary file. Each test needs to update the shared file many times during a run, so a version number is used to select the temporary file to read and discards the others. The file cache eliminated the time delay for updating the shared file, savings hours of runtime per test suite and speeding up developer check-in time with no penalty for increasing tests or compute nodes. This paper will describe how the file cache was developed, detail the current approach, and illustrate the performance benefits.

In the past decade Hadoop Distributed FileSystem (HDFS) has become the de facto storage choice for large scale analytics. Growing data storage needs at Twitter has forced us to solve multiple scalability, reliability, hardware layout and performance problems in HDFS. At Twitter we have scaled HDFS storage across multiple clusters by incrementally consuming new features and contributing scalability fixes to the open source community. In this presentation we lay out our architecture for scaling HDFS for Exabyte storage, share our learnings managing such a large scale distributed system and note few changes for future. Participants will get an introduction to the challenges of managing a large scale distributed storage system. They will know about Twitter Data Architecture and journey in scalding an open source system.

Log files consisting of events from different services are a rich source of information for large scale analytics. Events can be as simple as log line or as complex as nested structured objects like thrift or protobuffers. At Twitter every service logs events for a particular category and publishes them to the Event Log Aggregation framework. This framework aggregates events of the same category into log files, usually stored on a distributed file system like the Hadoop Distributed File System (HDFS). Large Scale multi-petabyte analytics use these files across hundreds of projects. In this paper we provide an overview of the Event Aggregation framework used at Twitter, highlight its advantages, and compare it with similar frameworks. We also introduce the concept of category group and aggregator group in our architecture. Services at Twitter generate trillions of events with aggregate size exceeding multiple petabytes of data every day. At present this framework handles over three billion events per minute. The main focus of our efforts has been efficient use of hardware resources, scalability and reliability of the framework.

As ML developments shift from research to real world, we encounter many deployment challenges. Teams may be experimenting with various training frameworks, with deployments targeting multiple platforms and hardware. While training using one framework with one hardware target can easily be managed, it becomes challenging with a matrix of multiple frameworks and deployment targets. This fragmented ecosystem introduces deployment complexities and oftentimes custom code is needed to maximize performance for each scenario, which is time-consuming to maintain when models are updated.

To streamline this, the interoperable ONNX model format and ONNX Runtime inference engine can be utilized to deploy models performantly across a variety of hardware. Models trained from PyTorch, Tensorflow, scikit-learn, CoreML, and more can all be converted to the common ONNX format, and the model can then be inferenced using the cross-platform performance-focused ONNX Runtime inference engine, which supports various hardware options for acceleration across CPU and GPUs.

ONNX Runtime is already used in key Microsoft services, on average realizing 2x performance improvements. In this session, we'll share an overview of ONNX Runtime, success stories and usage examples from high volume product groups at Microsoft, and demonstrate ways to integrate this into your AI workflows for immediate impact.