Caffe2

Caffe2 and PyTorch join forces to create a Research + Production platform PyTorch 1.0

Wed, 02 May 2018 00:00:00 +0000

We’d like to share the plans for future Caffe2 evolution. Publicly open-sourced over a year ago, Caffe2 is a light-weight and modular framework that comes production-ready with ultimate scaling capabilities for training and deployment. Its mobile capabilities (Caffe2go) support all major generations of hardware and power one of the largest deployments of mobile deep learning with more than 1 billion devices. Over the past year, we worked with many industry partners to add Caffe2 support for their platform and guaranteed the best possible performance regardless of the platform you run on.

At Facebook, where Caffe2 originates, we support both PyTorch and Caffe2 for the wide range of AI use cases. The main focus of Caffe2 development has been performance and cross-platform deployment whereas PyTorch has focused on flexibility for rapid prototyping and research.

In practice, any deep learning framework is a stack of multiple libraries and technologies operating at different abstraction layers (from data reading and visualization to high-performant compute kernels). Over the past year we saw more components of Caffe2 and PyTorch being shared (e.g. gloo, NNPACK, etc). Also, with our investment into interoperability, we built deep integration between frameworks using the shared ONNX model format.

We realized that in order to deliver the best user experience, it makes sense to combine the beneficial traits of Caffe2 and PyTorch into a single package and enable a smooth transition from fast prototyping to fast execution. It’d also improve our developer efficiency by more easily utilizing a shared set of tools.

Caffe2 and PyTorch projects are merging. Over the next few months, we’re planning to deeply integrate components of the frameworks and effectively unite them as a single package. It will combine the flexible user experience of the PyTorch frontend with scaling, deployment and embedding capabilities of the Caffe2 backend. Following is the high-level outline of the plan.

What’s going to happen?

We’ve already merged code repositories a month ago to streamline some of the developer infrastructure tooling. The following is an outline of our plans of the next months.

Builds and packages

In one of the next releases, python packages (in pip and conda) for Caffe2 and PyTorch will be merged into a single package providing union of the functionalities.
We will continue to provide native library and python extensions as separate install options (which is the case for both Caffe2 and PyTorch today)
All cross-compilation build modes and support for platforms of Caffe2 (iOS, Android, Raspbian, Tegra, etc) will remain intact and we will continue to expand various platforms support.

Model authoring (frontend)

Caffe2’s graph construction APIs (brew, core.Net) will continue to work and we’d provide backward compatibility for existing serialized model NetDefs for the changing functionalities during the refactoring.
Going forward, nn.Module-like abstractions would be preferred for constructing networks. We’re augmenting PyTorch frontend abstractions with a so-called “hybrid frontend” that utilizes tracing and compilation capabilities to extract fully serialized model in graph format (compatible with Caffe2’s NetDef and ONNX) that can be used for efficient deployment. Also, all of Caffe2 current and upcoming backend bindings and functionalities are going to be exposed through the unified hybrid frontend soon. Check out the corresponding PyTorch blog for more details on how hybrid frontend is going to look.
The set of operator implementations of Caffe2 and PyTorch will be merged over time thus expanding functionality of both.
ONNX model format is natively supported for both export and import in Caffe2 and PyTorch today. As we unify the codebases we’re using ONNX as a common model representation and the means to express dynamic model nature suitable for optimization. Thus PyTorch 1.0 will be able to support ONNX natively and interface with other framework or accelerated libraries both for ingesting and emitting models.

Scaling and deployment

Caffe2’s highly scalable execution engine (various accelerated backends and libraries integration and scalable graph executor) remain mostly intact and will gain ability to interoperate smoothly with Python program segments for rapid prototyping.
Caffe2’s existing predictor support will be the primary means of scalable native-only model deployment with accelerated hardware support both in datacenter and on mobile.

Hardware integrations and accelerated libraries support

We’re going to make Caffe2’s diverse devices support and runtime integrations available directly from the prototyping environment of the unified PyTorch 1.0 package.
Integration interface for the libraries targeting accelerated hardware or complete graph runtimes would stay mostly similar to the current Caffe2 code and existing integrations will continue to work. Furthermore, for graph runtimes, we’re working on formalizing the interaction interface with community initiatives like ONNXIFI.

Timeline and further information

We’re hoping to get to a production-ready first release this summer. However, as always, all of the development happens openly on GitHub so you’re welcome to follow along and contribute.

Given the ongoing merge of the projects, going forward we’re going to unify announcements on pytorch.org and cross-reference them on caffe2.ai.

Check out more details on the hybrid frontend and what benefits unification brings to the pytorch community in the corresponding blog.

Caffe2 Now Optimized for ARM Mobile GPUs

Fri, 23 Feb 2018 00:00:00 +0000

Developers are looking to apply AI to an ever expanding range of use cases. As we look to broaden how people can use AI, we’re thrilled to share our recent collaboration between ARM and Facebook to integrate and optimize Caffe2 for ARM’s Mali Graphics Processing Unit (GPU) hardware.

The results of this technical collaboration are now available via the Caffe2 and ARM Compute Library’s open source repositories, allowing Caffe2 developers to benefit from performance improvements on mobile devices using ARM GPUs. The ARM Compute Library can be accessed at https://developer.arm.com/technologies/compute-library. The collaboration will extend into the future for upcoming ARM ML processors as the Compute Library adds support for these new hardware elements.

For additional information about this exciting collaboration please visit https://community.arm.com/graphics/b/blog/posts/mwc18-a-smartphone-camera-with-a-brain.

Reinforcement learning with Caffe2

Thu, 14 Sep 2017 00:00:00 +0000

Reinforcement learning (RL) is an area of machine learning focused on teaching agents a complex relationship between its action and behavior, and maximizing a reward after a duration in an environment. The agent can be a game avatar, recommender system, notification bot, or variety of other systems that make decisions. The reward could be points in a game, or more engagement on a website. Facebook uses RL in different ways, with one example being when to let page owners know how their pages are performing.

Today, we are pleased to announce RL_Caffe2 (https://github.com/caffe2/reinforcement-learning-models), a set of RL libraries built on the Caffe2 platform. Sharing an open-source fork of our Caffe2 RL framework allows us to give back to the community and also collaborate with other institutions as RL finds more applications in industry.

This project, called RL_Caffe2, contains several RL implementations built on Caffe2 and integrated with OpenAI Gym:

DQN: An implementation of the Deep Q Learning network as described in https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf.
SARSA: This is a simplification of DQN that assumes the input data is on-policy: the policy generating the data is updating in real-time. The advantage of SARSA is that, during training, we do not need to know what actions are possible. We only need to know the actions taken.
Actor-Critic: An implementation of the Actor Critic model as described in https://arxiv.org/pdf/1509.02971.pdf

Github (RL_Caffe2): https://github.com/caffe2/reinforcement-learning-models

Caffe2 adds RNN support.

Thu, 03 Aug 2017 00:00:00 +0000

We are excited to share our recent work on supporting a recurrent neural network (RNN).

We did not support RNN models at our open source launch in April. So, over the last several months, we have developed state-of-the-art RNN building blocks to support RNN use cases (machine translation and speech recognition, for example).

Using Caffe2, we significantly improved the efficiency and quality of machine translation systems at Facebook. We got an efficiency boost of 2.5x, which allows us to deploy neural machine translation models into production. As a result, all machine translation models at Facebook have been transitioned from phrase-based systems to neural models for all languages. In addition, several product teams at Facebook, including speech recognition and ads ranking, have started using Caffe2 to train RNN models.

We invite machine learning engineers and researchers to experience Caffe2’s RNN capability. More details about what we implemented and open-sourced for RNN support are outlined below.

Unique Features in Caffe2 RNNs

Caffe2 provides a generic RNN library where the RNN engine is an almost zero-overhead virtual box for executing arbitrary RNN cells. Under the hood, a cell is just a smaller Caffe2 network and benefits from all typical Caffe2 performance advantages. We also have a rich set of APIs that let people use existing RNNCells and implement new ones using Python. MultiRNNCell allows for easy composition of existing cells into more complex ones. For example, you could combine several layers of LSTMCells and then put an AttentionCell on top.

The underlying RNN engine is incredibly flexible. It allows you to select which outputs have gradients and need to be propagated through time, and to define how cells are connected to each other and how they connect to outside world. Each input receives the correct gradient propagated back through time.

The zero-overhead engine is not the only performance win. Another key advantage comes in memory, which allows us to use a large batch size per GPU. The RNN engine supports the recycling of intermediate results across time steps and gives you the power to decide what to recycle. See here for a more detailed analysis of trading memory for compute.

In the diagram above, intermediate results of the backward pass could be reused across time steps. The forward ones, if reused, cause a need to be recomputed on the backward pass. Caffe2 allows you to specify which forward blobs should be dropped to trade compute for memory.

Static RNN

Caffe2 also implements a so-called static RNN. It can be used when back-propagating through time when the sequence length is known in advance. Then, the recurrent net becomes a part of the containing graph, so the global neural network executor, DAGNet, will find the most optimal parallel execution way of running the RNN within the context of the whole model. The static RNN engine supports all existing RNNCells and can be plugged in with almost no changes to the code. For multi-layered LSTM models, we saw a 25 percent speedup over the dynamic RNN implementation.

RNN engine for Beam Search

We follow the practice, common in machine translation, of using beam search at decoding time to improve our estimate of the highest-likelihood output sentence according to the model. We exploited the generality of the RNN abstraction in Caffe2 to implement beam search directly as a single forward network computation, which gives us fast and efficient inference. Beam search decoding as a recurrent network is used regardless of the architecture of the underlying model (RNN, CNN, etc.). We have also open-sourced this method of beam search inference as part of the Caffe2 library.

Homepage: https://caffe2.ai/
Github: https://github.com/pytorch/pytorch
Facebook: https://facebook.com/Caffe2AI

Caffe2 adds 16 bit floating point training support on the NVIDIA Volta platform

Wed, 10 May 2017 00:00:00 +0000

After open sourcing Caffe2 at F8 last month, today we are are excited to share our recent work on low precision 16 bit floating point (FP16) training in collaboration with NVIDIA.

Deep learning workloads often have diverse compute requirements. We have long known that high precision computation like FP64 is not always necessary, and often use FP32 for training. However, many deep neural networks can be trained to a high degree of accuracy using even lower precision such as FP16.

We are working closely with NVIDIA to optimize Caffe2 for the features in NVIDIA’s upcoming Tesla V100, based on their next generation Volta architecture. Caffe2 is excited to be one of the first frameworks that is designed from the ground up to take full advantage of Volta by integrating the latest NVIDIA Deep Learning SDK libraries - NCCL and cuDNN.

Caffe2 with FP16 support will allow machine learning developers using NVIDIA Tesla V100 GPUs to maximize the performance of their deep learning workloads.

NVIDIA using the Tesla V100 and Caffe2 has initially seen 2.5x faster training with FP16 compared to Tesla P100, and up to 5x faster than Tesla K80 GPUs.

FP 16 training performance with NVIDIA Tesla GPUs (ResNet-50, Batch size: 64)

Configuration: Tesla K80 + cuDNN 6 (FP32), Tesla P100 + cuDNN 6 (FP32), Tesla V100 + cuDNN 7 (FP16)

Benchmark performance numbers are provided by NVIDIA using DGX-1 with Tesla P100 (Pascal) and DGX-1V with Tesla V100 (Volta) GPUs.

Caffe2 is open source and available at GitHub for download. Please check out the Caffe2 documentation & tutorials at Caffe2.ai. If you’re thinking about using Caffe2, we’re interested in hearing about your needs. Please participate in our survey. We will send you information about new releases and special developer events/webinars.

We will have our first Caffe2 meetup in San Francisco bay area at the end of May. Please join our Caffe2 meetup group.

Homepage: https://caffe2.ai/
Github: https://github.com/pytorch/pytorch
Survey: https://www.surveymonkey.com/r/caffe2
Meetup: https://www.meetup.com/Caffe2-Bay-Area

Caffe2 Open Source Brings Cross Platform Machine Learning Tools to Developers

Tue, 18 Apr 2017 00:00:00 +0000

Training and deploying AI models is often associated with massive data centers or super computers, with good reason. The ability to continually process, create, and improve models from all kinds of information: images, video, text, and voice, at massive scale, is no small computing feat. Deploying these models on mobile devices so they’re fast and lightweight can be equally daunting. Overcoming these challenges requires a robust, flexible, and portable deep learning framework.

Facebook has been working with others in the open source community to build such a framework. Today, we’re open-sourcing the first production-ready release of Caffe2 - a lightweight and modular deep learning framework emphasizing portability while maintaining scalability and performance.

We’re committed to providing the community with high-performance machine learning tools so that everyone can create intelligent apps and services. Caffe2 is shipping with tutorials and examples that demonstrate learning at massive scale which can leverage multiple GPUs in one machine or many machines with one or more GPUs. Learn to train and deploy models for iOS, Android, and Raspberry Pi. Pre-trained models from the Caffe2 Model Zoo can be run with just a few lines of code.

Caffe2 is deployed at Facebook to help developers and researchers train large machine learning models and deliver AI-powered experiences in our mobile apps. Now, developers will have access to many of the same tools, allowing them to run large-scale distributed training scenarios and build machine learning applications for mobile.

We’ve worked closely with NVIDIA, Qualcomm, Intel, Amazon, and Microsoft to optimize Caffe2 for both cloud and mobile environments. These collaborations will allow the machine learning community to rapidly experiment using more complex models and deploy the next generation of AI-enhanced apps and services. to optimize Caffe2 for both cloud and mobile environments. These collaborations will allow the machine learning community to rapidly experiment using more complex models and deploy the next generation of AI-enhanced apps and services.

Check out the Caffe2 documentation & tutorials at caffe2.ai and see the source code on GitHub! If you’re thinking about using Caffe2, we’re interested in hearing about your needs. Please participate in our survey. We will send you information about new releases and special developer events/webinars.

Homepage: http://caffe2.ai
Github: https://github.com/pytorch/pytorch
Survey: https://www.surveymonkey.com/r/caffe2