This project was completed jointly by Nidhin Pattaniyil and Reshama Shaikh.

As mobile phones have become more accessible, consequently, mobile use has been increasing. Users are utilizing mobile devices over desktop more frequently, and apps on mobile are in high demand. These internet-connected devices provide an opportunity to bring the inference models closer to the user.

Outline

• About the Data: Food-101
• PART 1: Training an Image Classifier Using TensorFlow
• PART 2: Converting the Model
• PART 3: Considerations for Inference: Running on Server vs. Client
• PART 4: Deploying the Web App
• PART 5: Deploying the Mobile App

About the Data: Food-101

The Food-101 data is used for this project, which includes 101 food categories for a total of 101,000 images. Thus, each class has 1,000 images, of which 250 are manually reviewed test images, and 750 are training images. The categories of the ETHZ Food-101 are the 101 most popular categories from the food picture sharing website foodspotting.com.

Data citation

Bossard, Lukas and Guillaumin, Matthieu and Van Gool, Luc, Food 101 Mining Discriminative Components with Random Forests, European Conference on Computer Vision, 2014.

PART 1: Training an Image Classifier using TensorFlow

The image classifier was trained using TensorFlow 2.3.4. The code is available for reference in this notebook on GitHub. There are a number of pre-trained models available in TensorFlow 2 / Keras. We used the ResNet50 and MobileNetV2 architectures. The MobileNet model family is mobile-friendly, and it will be used for the mobile app.

Table 1: Compare Model Architectures

We present below a table comparing models with respect to accuracy, number of parameters, and model size.

MobileNetV2 is a mobile-friendly architecture with fewer parameters. The final model that was used in the mobile app deployment is a MobileNetV2 fine-tuned model which had about 66% accuracy.

PART 2: Converting the Model (Quantization & Optimization)

We downloaded the model.h5 and classes.json (this is the list of food class names) files to our local computer into a folder named model_tf and did our model conversions within a virtual environment.

The model we created, model.h5, is in the format “TensorFlow Keras”. We needed to convert it to the “TensorFlow.js” format because that is needed for client-side inference. We are converting the default tensorflow.js model for a few reasons:

• Model shards: Our model file is large, and the default tensorflow.js breaks the model down into 5 MB shards. For deploying the mobile app, we need one file, so we are specifying weight_shard_size_bytes of 50,000,000 bytes to get that file.
• Inference-speed optimization using GraphModel conversion

How does GraphModel conversion boost TensorFlow.js models’ inference speed? It’s achieved by leveraging TensorFlow (Python)’s ahead-of-time analysis of the model’s computation graph at a fine granularity. The computation-graph analysis is followed by modifications to the graph that reduce the amount of computation while preserving the numeric correctness of the graph’s output result.

• Inference-speed optimization using Quantization: Quantization is a post-training technique to reduce the size of models. Here, we use quantization to decrease the default 32-bit precision to 16-bit precision which will reduce the model file size by half.

We combined all 3 conversion steps into this:

We saved our converted model file and uploaded it to GitHub releases area of our mobile repo: deploying-mobile-app/release

These model and classes files will be used as input when creating the web app.

Table 2: Model Size for MobileNetV2

Note that the quantized model size has been reduced by 50%, from 14MB to 6.9MB. Inference times are similar for both model sizes. For most cases, the literature shows that 16-bit quantization does not impact accuracy significantly.

Web App:  Compare Model Size Before & After Optimization and Quantization

╔═════════════╦════════════════════╗
║ Model       ║ Model Size         ║ 
║             ║                    ║        
╠═════════════╬════════════════════╣
║ MobileNetV2 ║ 14MB               ║
║             ║                    ║                   
╠═════════════╬════════════════════╣
║ MobileNetV2 ║  6.9MB             ║             
║  (16-bit    ║                    ║                   
║   quantized)║                    ║                   
╚═════════════╩════════════════════╝

PART 3: Considerations for Inference: Running on Server vs. Client

Latency/Network connection of user

In server-side inference, once the image/text is received, the inference time is consistent. However, the total time is determined by the user’s network upload speed. In client-side inference, the inference time is dependent on the hardware that the user is running.

Privacy

For sensitive data, users might be uncomfortable with sending the data to a server. Client-side inference allows users to run their workload securely.

Future updates to models

A benefit of server-side inference is the ability to deploy new, state-of-the-art models and consistently update them. In client-side inference, deployment of new models is restricted by the user’s expressed update frequency.

State of the art models vs. mobile optimized

Due to client hardware and storage restrictions, models that are small and optimized for inference are ideal. Most websites on the web are less than 2 MB of JavaScript code and CSS. The simplest model we created was about 20 MB, which is still not ideal for serving on the web. Therefore, most models currently are served on a server.

PART 4: Deploying the Web App

We used this template repository to deploy the web. The template repository is one web app that does both server-side and browser-based inference:
deploying-web-app

We copied the model to our forked repo.

This is the original file from Colab: model.h5
This is the converted TensorFlow.js file: model_tfjs

The structure of the repository with the model files is as below. We placed the original and converted model in the directory backend/assets.

├── backend
│   ├── app.py
│   ├── assets
│   │   ├── classes.json
│   │   ├── model_tf
|   |   |──├──model.h5
│   │   └── model_tfjs
│   │       ├── group1-shard1of1.bin
│   │       └── model.json

Serving the Web App Locally

There are various options for serving web apps in Python, including Flask, Django, and FastAPI.

We served out browser app with FastAPI and used React for the frontend framework.

First, we ran the app locally using Docker.

The repo provides a Docker file to run the app. Run these commands to launch the app. The first time you run using Docker, it could take up to 10 minutes. This command needs to be run at the base level of your repo where the Dockerfile file is located.

Running the above two commands starts a web server running locally on the machine.

The server can be accessed locally at http://localhost:8000.

Voilà! We have a web app running locally! It looks like the following demo.

Demo of Web App

This app is in production at: manning-deploy-imagenet.herokuapp.com

Serving the Web App to a Cloud Platform (Heroku)

Heroku is a nice, free option for deploying the app.

Once the Heroku command-line tools are installed, it can be run using the below commands.

Replace APP_NAME with something unique. The below steps could take some time (5 to 10 minutes) depending on your internet upload speed. For our project, we set APP_NAME="manning-deploy-imagenet".

The app can be tried out here: manning-deploy-imagenet.herokuapp.com

Inference times

We measured the latency for several sample images. We experimented with images of different sizes and visited the site on desktop and mobile.

Table 3: Inference Times for Web App

Web App:  Compare Inference Times

╔═════════════╦════════════════════╦═══════════════════╦════════╗
║ Inference   ║ Desktop Browser    ║ Mobile Browser    ║ Model  ║
║ Source      ║ duration           ║ duration          ║ Size   ║
║             ║ (Inference in ms)  ║ (Inference in ms) ║        ║        
╠═════════════╬════════════════════╬═══════════════════╣════════╣
║ Server      ║ 559                ║ 594               ║ 29MB   ║
║ (Heroku)    ║ (202)              ║ (180)             ║        ║ 
╠═════════════╬════════════════════╬═══════════════════╣════════╣
║ Browser     ║ 50                 ║ 177               ║ 6.9MB  ║
║             ║                    ║                   ║        ║ 
╚═════════════╩════════════════════╩═══════════════════╩════════╝

NOTES
The above results show the latency cost associated with sending an image to the server for inference.

* For the Server (Heroku) model, it is the h5 file, the keras model.
* For the Browser model, it is the post-training quantized and optimized model. (converted using TensorFlow.js)

PART 5: Deploying the Mobile App

The benefit of having inference running natively on a mobile app is that it is functional without an internet connection. Because the model is running on the user’s device, the latency is lower. Inference running locally on the device also protect’s the user’s privacy.

We used React Native and TensorFlowJS React Native adaptor since it allowed us to build a cross-platform mobile app.

We used this template repository to deploy the mobile web; it works for both iOS and Android: deploying-mobile-app

We uploaded the tfjs converted model file to the releases section.

The inference is happening in the file ModelService.tsx. To initialize the TensorFlow.js model, ModelService.tsx calls the below code in the create method.

Below is the core code used for getting a prediction from an image used in classifyImage.

Expo is a free and open-source toolchain built around React Native to help build cross-platform native iOS and Android projects using JavaScript and React. The JavaScript environment that is commonly used for building apps is Node. To run this app, we use yarn, which is used to install JavaScript packages like Expo. After yarn has been installed, we did:

yarn run start

The next step is to open the Expo app and once you are logged in, you should see the app listed.

Demo of Mobile App

This mobile app is in production at:

Table 4: Comparing Inference Times for Mobile App

Note: We would expect the inference time for the mobile app to be faster. It’s a bit slower here. It could be due to the React Native adaptor not being mature.

Mobile App:  Compare Inference Times
╔═════════════╦════════════════════╦═══════════════════╗
║ Inference   ║ Mobile Browser     ║ Mobile App        ║ 
║ Source      ║ duration           ║ duration          ║ 
║             ║ (Inference in ms)  ║ (Inference in ms) ║        
╠═════════════╬════════════════════╬═══════════════════╣ 
║ Pixel 1     ║ 310                ║ 411               ║ 
║ (2016)      ║                    ║                   ║ 
╠═════════════╬════════════════════╬═══════════════════╣ 
║ iPhone XS   ║ 220                ║ 327               ║       
║ (2019       ║                    ║                   ║        
╚═════════════╩════════════════════╩═══════════════════╝

NOTES
Both the mobile browser and app allow you to run inference in less than 1 second. This performance is acceptable for a mobile app.

Summary

This article provides an outline for how to run a deep learning classifier using TensorFlow, and how to serve the model on both web and mobile. For more step-by-step instructions, check out our Manning liveProject: Deploying a Deep Learning Model on Web and Mobile Apps (using TensorFlow and React). The first milestone of this project, which focuses on training the model, is available publicly.

We focus our presentation using TensorFlow-serving because the TensorFlow ecosystem for deploying mobile apps is more mature than the PyTorch one. PyTorch Mobile was released approximately October 2019, and TensorFlow Lite was released in May 2017.

Video

We presented on this topic at PyData Global 2021. There is a 30-minute video with Q&A discussion at the end.

References

• Deploying a Deep Learning Model on Web and Mobile Apps (TensorFlow, React) [Manning liveProject]
• GitHub repo: deploying-web-app (TensorFlow, React)
• GitHub repo: deploying-mobile-app (TensorFlow, React Native)
• Deploying Deep Learning Models on Web and Mobile (fastai, PyTorch, React)
• Deep Learning with JavaScript: Neural networks in TensorFlow.js by Shanqing Cai, Stanley Bileschi, Eric D. Nielsenn and Francois Chollet; Chapter 12. Testing, optimizing, and deploying models

Deploying A Deep Learning Model on Mobile Using TensorFlow and React was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the advances in programming and computing, data scientists can now do state-of-the-art visualizations without requiring in-depth knowledge of D3 or Javascript. We can accomplish this using the python library plotly!

For these examples, I am using Python version 3.9 and plotly version 5.1.0.

A video version of annotations in plotly is available on YouTube.

STEP 1: Read in sample data for visualization

import pandas as pd

event_dates = ['2020-01-01', '2020-02-01', '2020-03-01',
               '2020-04-01', '2020-05-01', '2020-06-01', 
               '2020-07-01', '2020-08-01', '2020-09-01', 
               '2020-10-01', '2020-01-01', '2020-02-01', 
               '2020-03-01', '2020-04-01', '2020-05-01', 
               '2020-06-01', '2020-07-01', '2020-08-01', 
               '2020-09-01', '2020-10-01']

groupid = ['A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A',
           'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B']

outcome = [92,  93, 107,  91, 113,  83,  87,  99, 101, 107, 
           103, 145, 131, 116, 131, 109, 108, 105,  96, 127]

data = {'event_date': event_dates,
        'groupid': groupid,
        'outcome': outcome}

# create the dataframe from the arrays
df = pd.DataFrame(data)

Look at data frame, by sampling a few rows: df.sample(5)

STEP 2: Create a basic, 2-group, line plot

import plotly.express as px

fig = px.line(df
              , x='event_date'
              , y='outcome'
              , color='groupid' 
              , title = "Event Data Over Time for Two Groups")

fig.show()

STEP 3: Extend the x-axis range a bit

Let’s extend the x-axis on the left and right a little bit, so it is cleaner and easier to read and the graph doesn’t go to the edges.

This parameter gives options for the x-axis range: range_x=[..., ...]

# extend the x-axis range

fig = px.line(df
    , x='event_date'
    , y='outcome'
    , color='groupid'
    , title = "Event Data Over Time for Two Groups”
    , range_x=['2019–12–01', '2020–11–01'],)

fig.show()

STEP 4: Add an author and data source footnote

This allows us to add a footnote annotation: fig.add_annotation(...).

fig = px.line(df
    , x='event_date'
    , y='outcome'
    , color='groupid'
    , title = "Event Data Over Time for Two Groups"
    , range_x=['2019-12-01', '2020-11-01'],)

fig.update_layout(autosize=False, width=800, height=600,)

# add today's date
from datetime import date
today = date.today()

# HOW TO ADD A FOOTNOTE TO BOTTOM LEFT OF PAGE
fig.add_annotation(
    text = (f"@reshamas / {today}<br>Source: JHU CSSE")
    , showarrow=False
    , x = 0
    , y = -0.15
    , xref='paper'
    , yref='paper' 
    , xanchor='left'
    , yanchor='bottom'
    , xshift=-1
    , yshift=-5
    , font=dict(size=10, color="grey")
    , align="left"
    ,)

fig.show()

Tip: you may want to adjust the above values for x and y as well as for xshift and yshift, depending on the size of your graph.

Tip: the br in the footnote text represents a line break.

STEP 5: Add an annotation with a box & opacity

fig = px.line(
    df
    , x='event_date'
    , y='outcome'
    , title = "Event Data Over Time for Two Groups"
    , color="groupid"
    , range_x=['2019-12-01', '2020-11-01'],
)
fig.update_layout(
    autosize=False,
    width=800,
    height=600,)

# add today's date
from datetime import date
today = date.today()

# HOW TO ADD A FOOTNOTE TO BOTTOM LEFT OF PAGE
fig.add_annotation(
    text = (f"@reshamas / {today}<br>Source: JHU CSSE")
    , showarrow=False
    , x = 0
    , y = -0.15
    , xref='paper'
    , yref='paper' 
    , xanchor='left'
    , yanchor='bottom'
    , xshift=-1
    , yshift=-5
    , font=dict(size=10, color="grey")
    , align="left"
    ,
    )

# add annotation text with an arrow
fig.add_annotation(
    x='2020-02-01'
    , y=145+1
    , text=f'145 is the<br>the maximum value'
    , yanchor='bottom'
    , showarrow=True
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=12, color="purple", family="Courier New, monospace")
    , align="left"
    ,)

# add annotation with box, color-filled with opacity option
fig.add_annotation(
    x='2020-05-01'
    , y=135
    , text=f'May<br>Day'
    , yanchor='bottom'
    , showarrow=True
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=20, color="green", family="Courier New, monospace")
    , align="left"
    , bordercolor='green'
    , borderwidth=2
    , bgcolor="#CFECEC"
    , opacity=0.8
    ,)

fig.show()

Tip: there are many options for how text in annotations can be presented. There are options for font size, type and color. There are options for adding boxes around text with various formatting options. Arrows can be shown or hidden, using the showarrow=True option.

Tip: the location of the annotation “145 is the maximum value” can be placed using the coordinates, in this case, x='2020-02-01' and y=145. The location of the text can also be shifted using ax=-20 and ay=-30, as an example.

Tip: multiple annotations can be layered on the graph by adding fig.add_annotation object.

STEP 6: Add an annotation with vertical lines

fig = px.line(
    df
    , x='event_date'
    , y='outcome'
    , title = "Event Data Over Time for Two Groups"
    , color="groupid"
    , range_x=['2019-12-01', '2020-11-01'],
)
fig.update_layout(
    autosize=False,
    width=800,
    height=600,)

fig.add_annotation(
    x='2020-03-21'
    , y=145+1
    , text=f'Mar 21<br>First day of spring'
    , yanchor='bottom'
    , showarrow=False
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=12, color="orange", family="Sans Serif")
    , align="left"
    ,)

fig.add_annotation(
    x='2020-06-21'
    , y=145+1
    , text=f'Jun 21<br>First day of summer'
    , yanchor='bottom'
    , showarrow=False
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=12, color="purple", family="Sans Serif")
    , align="left"
    ,)

# add vertical lines
fig.update_layout(shapes=
                  [dict(type= 'line',
                        yref= 'paper', y0= 0, y1= 1,
                        xref= 'x', x0='2020-03-21', x1='2020-03-21',
                        line=dict(color="MediumPurple",
                                  width=3,
                                  dash="dot")
                        ),
                  dict(type= 'line',
                        yref= 'paper', y0= 0, y1= 1,
                        xref= 'x', x0='2020-06-21', x1='2020-06-21',
                        line=dict(color="MediumPurple",
                                  width=3,
                                  dash="solid")
                       )
                  ])
       
fig.show()

Tip: the line type may be adjusted by using these options:

dash="dot" 
dash="solid"

Tip: multiple reference lines can be added using fig.update_layout(shapes=... and the lines can be added as a dictionary.

STEP 7: Add vertical & horizontal rectangle highlight sections

fig = px.line(
    df
    , x='event_date'
    , y='outcome'
    , title = "Event Data Over Time for Two Groups"
    , color="groupid"
    , range_x=['2019-12-01', '2020-11-01'],
)
fig.update_layout(
    autosize=False,
    width=800,
    height=600,)

fig.add_annotation(
    x='2020-03-21'
    , y=145+1
    , text=f'Mar 21<br>First day of spring'
    , yanchor='bottom'
    , showarrow=False
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=12, color="orange", family="Sans Serif")
    , align="left"
    ,)

fig.add_annotation(
    x='2020-06-07'
    , y=145+1
    , text=f'Jun 21<br>First day<br>of summer'
    , yanchor='bottom'
    , showarrow=False
    , arrowhead=1
    , arrowsize=1
    , arrowwidth=2
    , arrowcolor="#636363"
    , ax=-20
    , ay=-30
    , font=dict(size=12, color="purple", family="Sans Serif")
    , align="left"
    ,)

# add vertical lines
fig.update_layout(shapes=
                  [dict(type= 'line',
                        yref= 'paper', y0= 0, y1= 1,
                        xref= 'x', x0='2020-03-21', x1='2020-03-21',
                        line=dict(color="MediumPurple",
                                  width=3,
                                  dash="dot")
                        ),
                  dict(type= 'line',
                        yref= 'paper', y0= 0, y1= 1,
                        xref= 'x', x0='2020-06-21', x1='2020-06-21',
                        line=dict(color="MediumPurple",
                                  width=3,
                                  dash="solid")
                       )
                  ])

# In United States: 'unofficial' summer is from Memorial Day to Labor Day

# Make a vertical highlight section
fig.add_vrect(x0="2020-05-25", x1="2020-09-07", 
              annotation_text="Unofficial<br>Summertime<br>in USA<br>(Memorial Day to<br>Labor Day)", annotation_position="top right",  
              annotation_font_size=11,
              annotation_font_color="Green",
              fillcolor="yellow", opacity=0.25, line_width=0)

# Make a horizontal highlight section
fig.add_hrect(y0=90, y1=100, 
              annotation_text="Observed data<br>of interest", annotation_position="top right",  
              annotation_font_size=11,
              annotation_font_color="Black",
              fillcolor="red", opacity=0.25, line_width=0)

fig.show()

Tip: the horizontal and vertical highlight blocks can be created by adding an annotation section fig.add_hrect or fig.add_vrect. For the horizontal section, specify at which coordinates of y to place the blocks. The same can be done for the vertical highlight block, where x coordinates can be specified. The color of the highlight rectangle sections can be specified using the fillcolor option.

Resources

• Plotly Python Documentation
• Plotly Community Forum

Enriching Data Visualizations with Annotations in Plotly using Python was originally published in Nerd For Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

About

These are the some of the most commonly used commands I use in pandas, shared here for easy reference. These example uses python version 3.9 and pandas version 1.3.0.

1) Download a `csv` file using `wget`

# import libraries
from pathlib import Path
import pandas as pd

# create directories, if they don't already exist
Path("data_raw").mkdir(parents=True, exist_ok=True)
Path("data_derived").mkdir(parents=True, exist_ok=True)

# assign variable names to the directories
path_datain = "data_raw/"
path_dataout = "data_derived/"

data_url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series"
file_name = "time_series_covid19_confirmed_global.csv"

file_url = data_url + "/" + file_name

# get the file
!wget -N {file_url} -P {path_datain}

df = pd.read_csv(path_datain + file_name)

Look at dataframe

df.head()

2) Read in a `.csv` file and create a dataframe

df = pd.read_csv(path_datain + file_name)

3) Make a copy of a dataframe

df2 = df.copy()

4) Explore a dataframe

Read in data

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6, None],
                   "country":["India", "Italy", "Kenya", "Unknown"],
                   "C":[10, 9, 8, 7],
                   "D":[1, 3, None, 5],
                   "E":['2020-01-01', '2020-01-02','2020-01-03', 'NaN']})

Get dimensions of dataframe (# rows, # columns)

df.shape()

(5,5)

Get info

df.info()

>output

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 5 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   A        4 non-null      float64
 1   country  5 non-null      object 
 2   C        5 non-null      int64  
 3   D        4 non-null      float64
 4   E        5 non-null      object 
dtypes: float64(2), int64(1), object(2)
memory usage: 328.0+ bytes

Get unique occurrences of a variable in a dataframe

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6, None],
                   "country":["India", "Italy", "Kenya", "India"],
                   "C":[10, 9, 8, 7],
                   "D":[1, 3, None, 5],
                   "E":['2020-01-01', '2020-01-02','2020-01-03', 'NaN']})

df['country'].unique()

Get frequency counts for a variable in a dataframe

df['country'].value_counts(normalize=False)

NOTE: Use normalize=True to obtain percents.

5) Keep a subset of variables (columns), using `filter`

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6],
                   "B":[1, 2, 3],
                   "C":[10, 9, 8],
                   "D":[1, 3, None]
})

keep_cols = ['A', 'B']

df_v2 = df.filter(keep_cols)

6) “Drop” one or multiple variables in a dataframe

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6],
                   "B":[1, 2, 3],
                   "C":[10, 9, 8],
                   "D":[1, 3, None]
})

df

drop_cols = ['A', 'B']

df.drop(drop_cols, axis=1, inplace=True)

df

7) Take subset of rows in a dataframe: using `isin`

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6],
                   "B":[1, 2, 3],
                   "C":[10, 9, 8],
                   "D":[1, 3, None]
})

df

filtered_list = [2, 4]

check1 = df[df["A"].isin(filtered_list)]

check1

8) Re-order columns

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, 6],
                   "B":[1, 2, 3],
                   "C":[10, 9, 8],
                   "D":[1, 3, None]
})

df.head(2)

new_order = ['D', 'A', 'C', 'B']

df_new = df.reindex(new_order, axis=1)

df_new.head(2)

9) Rename columns in a dataframe

import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[120, 140, 100],
                   "B":[80, 90, 60],
                   "C":[10, 9, 8],
                   "D":[1, 3, None]
})

df

df = df.rename(columns={'A': 'systolic',
                        'B': 'diastolic'
                       })

df

10) Transpose a dataframe (pivot table); Use `melt`

# importing pandas as pd
import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"country":["India", "USA", "Mongolia"],
                   "year": [2019, 2020, 2021],
                   "A":[10, 9, 8],
                   "B":[1, 3, None],
                   "C":['2020-01-01', '2018-01-01', 'NaN']})

df

df_long = df.melt(id_vars=['country', 'year'])

df_long

11) Write out a dataframe to a `.csv` file

# Takes dataframe "df" and writes it out a csv file named "report.csv"

df.to_csv('data_derived/report.csv')

12) Using `mask` to assign new data values

# importing pandas as pd
import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[2, 4, None],
                   "country":["India", "USA", "Unknown"],
                   "C":[10, 9, 8],
                   "D":[1, 3, None],
                   "event_date":['2020-01-01', '2018-01-01', 'NaN']})

df

Changing an existing column

# IF the event_date='2018-01-01', then assign country (existing column in df) to be "United States"

df['country'].mask(df['event_date'] == '2018-01-01', 'United States', inplace=True)

df

Creating a new column

# IF the event_date='2018-01-01', then assign chapter_status (a new column) to be "unbegun"

# NOTE:  the new column needs to be initialized

df['chapter_status'] = ""

df['chapter_status'].mask(df['event_date'] == '2018-01-01', 'unbegun', inplace=True)

df

13) Convert from character to integer

# importing pandas as pd
import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[12, 4, 5, None],
                   "B":[7, 2, 54, 3],
                   "C":[20, 16, 11, 3],
                   "D":[14.3, 3.5, 5.0, 6.8]})

df.head()

df['D_n']= df['D'].astype(int)

df.head()

14) Replace “NaN” with another value

# importing pandas as pd
import pandas as pd
  
# Creating the dataframe 
df = pd.DataFrame({"A":[12, 4, 5, None, 1],
                   "B":[7, 2, 54, 3, None],
                   "C":[20, 16, 11, 3, 8],
                   "D":[14, 3, None, 2, 6]})

df.head()

# replace the NaN values with 1000

df.mask(df.isna(), 1000)

# NOTE: use inplace=True, so it is permanently replaced in the dataframe. 

df.mask(df.isna(), 1000, inplace=True)

Python pandas: Quick Tips was originally published in Nerd For Tech on Medium, where people are continuing the conversation by highlighting and responding to this story.

JupyerCon Keynote Talk

Tackling Toxic Culture in Tech: Racism and Its Legacy

Tema Okun has spent over 30 years working with and for organizations, schools, and community-based institutions as a trainer, facilitator, and coach focused on issues of racial justice and equity. Dr. Okun currently co-leads the Teaching for Equity Fellows Program at Duke University, which works with faculty seeking to develop stronger skills both teaching about race and racism and across lines of race, class, and gender. She also facilitates, coaches, and consults with leaders and organizations nationwide.

Tema was a member of the Educational Leadership faculty at National Louis University in Chicago and has taught undergraduate, master’s, and doctoral level students in educational leadership and education. She is the author of the award-winning The Emperor Has No Clothes: Teaching About Race and Racism to People Who Don’t Want to Know (2010, IAP) and the widely used article White Supremacy Culture. She publishes regularly on the pedagogy of racial and social justice.

Tema is a participant in the Living School for Action and Contemplation and a member of the Bhumisphara Sangha under the leadership of Lama Rod Owens. She is an artist, a poet, and a writer. She lives in Durham NC where she is fortunate to reside among beloved community. Her current project is deepening her ability to love her neighbor as herself. She is finding the instruction easy and the follow through challenging, given how we live in a culture that is afraid to help us do either or both.

Tema Okun at JupyterCon 2020

Three years ago, I was searching for career advice and discovered the Transform Your Workplace podcast with Brandon Laws and Megan Leatherman. I connected with Megan Leatherman and signed up for her newsletter. In Megan’s June 2020 newsletter, she included a reference to Tema’s podcast which really resonated with me.

The JupyterCon leadership had a vision for this conference to invite us to come together as a community, and think how we can make it better. We have strived to focus our attention on diversity and inclusion in all aspects of our programming. We also decided to accentuate equity and social justice in a dedicated keynote speech, and we are delighted to host Tema Okun in this role.

Book

Podcasts

Love and Humility: Finding Refuge (Sep 2020)

04. Love & Humility - Michelle Johnson

White Supremacy Culture at Work (Jul 2020)

Unleashing Social Change: Tema Okun (Jan 2020)

Unleashing Social Change

Episode 18: Tema Okun, Duke University, "Facing Into Your Own Racism with Courage and Love" https://t.co/lVTXLvb61h

@karaswisher: must have Tema Okun on your podcast:Racial justice/equity is built on idea of including people of color into a white world, w/the idea that "we help them".There is very little investigation of the "white world" & the toxicity of whiteness.

Resources

Dismantling Racism

Tema Okun was originally published in Jupyter Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

“URGs and Allies in Data Science” held our second event on March 7, 2020 at Farfetch in New York City. About 55 people attended.

Speaker

Bruno Goncalves

Website: www.data4sci.com
Newsletter: www.data4sci.com/newsletter

Workshop

Time series are everywhere around us. Understanding them requires taking into account the sequence of values seen in previous steps and even long-term temporal correlations. Bruno Goncalves explains a broad range of traditional machine learning (ML) and deep learning techniques to model and analyze time series datasets with an emphasis on practical applications.

Bruno will be presenting a 2-day workshop at an upcoming event:

• O’Reilly Strata Data and AI Conference (New York, NY) (Sep 2020)

Workshop GitHub Repo

• Timeseries_long

Related Links

Meetup event: Time Series Modeling: ML and Deep Learning Approaches with Python

Job Postings at Farfetch

These are current openings in New York City.

• Senior Software Engineer
• Senior Data Scientist

Acknowledgments

• Noemi Derzsy (UADS organizer)
• Farfetch (our venue host)
• Jazzmin Carroll (Farfetch)
• Alice Sieger (Farfetch)
• Sedine Baldeo (Farfetch)
• Ankur Rajdev (Farfetch)
• Gianluca Rossi (Farfetch)
• Ty Shaikh (UADS photography)

Social Media

URGs and Allies in Data Science on Twitter

We've all wondered at one time or another: How do I handle time series data❓ 👉🏽Join us for a workshop with @bgoncalves to find out. 🗓️ Mar 7 (Saturday) 🕘 9am to 4pm 📍 @farfetch 🌐 New York, NY #python #DataScience Sign up here: https://t.co/gTl6EuQMrg

Gianluca Rossi on Twitter

I can't wait to meet you all at this @urgsADS event about Time Series modeling at our office in New York! #MachineLearning #DataScience #python https://t.co/2YsisBI3bz

“URGs and Allies in Data Science” held our inaugural event on Feburary 19, 2020 at Two Sigma in New York City.

Speaker

Hugo Bowne-Anderson

Workshop

This tutorial was an Introduction to Bayesian data science through the lens of simulation or hacker statistics. The objective was to become familiar with many common probability distributions through i) matching them to real-world stories & ii) simulating them. We worked with joint/conditional probabilities, Bayes Theorem, prior/posterior distributions and likelihoods, while seeing their applications in real-world data analyses. We saw the utility of Bayesian inference in parameter estimation and comparing groups and concluded with a dive into the wonderful world of probabilistic programming.

Hugo will be presenting an extended version of this workshop:

• April 20 to 21, 2020 at Strata London
• April 14, 2020 at ODSC East Boston

GitHub repo

• Code + Binder
• Jupyter Notebooks

Related links

• Meetup event listing

Acknowledgements

Two Sigma

Julia Meinwald (Two Sigma)

Alex Alekseyev (Two Sigma)

Ty Shaikh (photography)

Social Media

Arturo Castellanos on Twitter

Super interesting workshop on Bayesian Data Science by Simulation with @hugobowne NYC URGs meetup @DataCamp @twosigma

URGs and Allies in Data Science on Twitter

Record turnout at our inaugural event ➡️➡️ such #DataScience passion in NYC! ⚡️ @hugobowne talks binomial, poisson, bayes, priors & more. Hosted at @twosigma Notebooks: https://t.co/V2aTsjSqC8 https://t.co/qO1tXH6gYg

Hugo Bowne-Anderson on Twitter

iIm really excited to be running a workshop on Bayesian stats, telling stories w/ simulation, probabilistic programming, & more at @urgsADS in February in NYC. Also really chuffed that @reshamas asked me to do so 😊 thanks for the opportunity! https://t.co/4fu4LgTzHp

A The below resources are the most popular and used widely. An added plus, there are no distracting ads on their sites.

Introductory Blog

• Learning Python: From Zero to Hero (quick read)

Free online tutorials

• Automate the Boring Stuff with Python (online book)
• HackerRank: Python
• Learn Python the Hard Way
• CodeCademy
• PyCharm: Introduction to Python

Recommendations

Sukriti Macker on Twitter: "Top 5 FREE Python Courses on Udemy and Coursera!! 😍(Plus a BONUS resource! 🌟)Python Programming Resources 🧵👇 / Twitter"

Top 5 FREE Python Courses on Udemy and Coursera!! 😍(Plus a BONUS resource! 🌟)Python Programming Resources 🧵👇