Python Notes on Software Development Notes

Create fake data with Faker in Python

Eric Callanan — Sunday, 28 Apr 2024

Fake data is useful in a number of scenarios such as in testing applications or libraries or in demonstrating functionality without any privacy or security concerns. While it is possible to create fake data manually, this can be time consuming and there is a better way. Faker is a library that makes it easy to generate fake data, such as names, phone numbers, emails, and other types of data.

There are many uses of fake data as listed below. In this article, I'll detail the basic use of Faker to create fake data, as well its use to populate a Pandas DataFrame and generate a chart.

Unit testing
Populating databases
Creating sample Reports
Simulating financial transactions
Load testing for APIs
Prototyping and design
Testing search functionality
Educational environments
Anonymizing data

Faker installation and use

Faker can be installed with pip install

1pip install faker

Simply import faker and initialize a faker generator, which can be used to create fake data by calling methods on the fake instance. The following generates a fake name by calling the name method on the instance of faker.

1from faker import Faker
2
3fake = Faker()
4
5# fake name
6fake.name()
7
8# 'Stefanie Brown'

Generate fake name using Faker

Show all the Faker methods

The following code lists all the methods on the instance of faker. This shows that there are 297 methods for creating all kinds of data.

 1methods = [x for x in dir(fake) if not x.startswith('_')]
 2print(f"number of methods = {len(methods)}")
 3# number of methods = 297
 4
 5
 6# print(methods)
 7for index in range(0, len(methods), 4):
 8    print(methods[index].ljust(35, " "), end="")
 9    try:
10        print(methods[index+1].ljust(35, " "), end="")
11        print(methods[index+2].ljust(35, " "), end="")
12        print(methods[index+3].ljust(35, " "))
13    except IndexError:
14        break

Methods on Faker:

Methods	methods (contd.)	methods (contd.)	methods (contd.)
aba	add_provider	address	administrative_unit
am_pm	android_platform_token	ascii_company_email	ascii_email
ascii_free_email	ascii_safe_email	bank_country	basic_phone_number
bban	binary	boolean	bothify
bs	building_number	cache_pattern	catch_phrase
century	chrome	city	city_prefix
city_suffix	color	color_hsl	color_hsv
color_name	color_rgb	color_rgb_float	company
company_email	company_suffix	coordinate	country
country_calling_code	country_code	credit_card_expire	credit_card_full
credit_card_number	credit_card_provider	credit_card_security_code	cryptocurrency
cryptocurrency_code	cryptocurrency_name	csv	currency
currency_code	currency_name	currency_symbol	current_country
current_country_code	date	date_between	date_between_dates
date_object	date_of_birth	date_this_century	date_this_decade
date_this_month	date_this_year	date_time	date_time_ad
date_time_between	date_time_between_dates	date_time_this_century	date_time_this_decade
date_time_this_month	date_time_this_year	day_of_month	day_of_week
del_arguments	dga	domain_name	domain_word
dsv	ean	ean13	ean8
ein	email	emoji	enum
factories	file_extension	file_name	file_path
firefox	first_name	first_name_female	first_name_male
first_name_nonbinary	fixed_width	format	free_email
free_email_domain	future_date	future_datetime	generator_attrs
get_arguments	get_formatter	get_providers	hex_color
hexify	hostname	http_method	iana_id
iban	image	image_url	internet_explorer
invalid_ssn	ios_platform_token	ipv4	ipv4_network_class
ipv4_private	ipv4_public	ipv6	isbn10
isbn13	iso8601	items	itin
job	json	json_bytes	language_code
language_name	last_name	last_name_female	last_name_male
last_name_nonbinary	latitude	latlng	lexify
license_plate	linux_platform_token	linux_processor	local_latlng
locale	locales	localized_ean	localized_ean13
localized_ean8	location_on_land	longitude	mac_address
mac_platform_token	mac_processor	md5	military_apo
military_dpo	military_ship	military_state	mime_type
month	month_name	msisdn	name
name_female	name_male	name_nonbinary	nic_handle
nic_handles	null_boolean	numerify	opera
optional	paragraph	paragraphs	parse
passport_dates	passport_dob	passport_full	passport_gender
passport_number	passport_owner	password	past_date
past_datetime	phone_number	port_number	postalcode
postalcode_in_state	postalcode_plus4	postcode	postcode_in_state
prefix	prefix_female	prefix_male	prefix_nonbinary
pricetag	profile	provider	providers
psv	pybool	pydecimal	pydict
pyfloat	pyint	pyiterable	pylist
pyobject	pyset	pystr	pystr_format
pystruct	pytimezone	pytuple	random
random_choices	random_digit	random_digit_above_two	random_digit_not_null
random_digit_not_null_or_empty	random_digit_or_empty	random_element	random_elements
random_int	random_letter	random_letters	random_lowercase_letter
random_number	random_sample	random_uppercase_letter	randomize_nb_elements
rgb_color	rgb_css_color	ripe_id	safari
safe_color_name	safe_domain_name	safe_email	safe_hex_color
sbn9	secondary_address	seed	seed_instance
seed_locale	sentence	sentences	set_arguments
set_formatter	sha1	sha256	simple_profile
slug	ssn	state	state_abbr
street_address	street_name	street_suffix	suffix
suffix_female	suffix_male	suffix_nonbinary	swift
swift11	swift8	tar	text
texts	time	time_delta	time_object
time_series	timezone	tld	tsv
unique	unix_device	unix_partition	unix_time
upc_a	upc_e	uri	uri_extension
uri_page	uri_path	url	user_agent
user_name	uuid4	vin	weights
windows_platform_token	word	words	xml
year	zip	zipcode	zipcode_in_state
zipcode_plus4

Fake methods for Name

It can be seen from the list of all the methods on Faker that there are a number of methods for generating names. We can use dir to list all the methods for generating fake names. This lists 24 methods that have name in the method name. Not all of these are names for people

1[x for x in dir(fake) if not x.startswith('_') and x.find('name') > -1]

Methods in Faker with name int the method

Fake Names	Fake Names (contd.)	Fake Names (contd.)
color_name	cryptocurrency_name	currency_name
domain_name	file_name	first_name
first_name_female	first_name_male	first_name_nonbinary
hostname	language_name	last_name
last_name_female	last_name_male	last_name_nonbinary
month_name	name	name_female
name_male	name_nonbinary	safe_color_name
safe_domain_name	street_name	user_name

Methods for people names in Faker

Fake Names	Fake Names (contd.)	Fake Names (contd.)
first_name_female	name	last_name
last_name_female	name_nonbinary	last_name_nonbinary
name_male	name_female	user_name
first_name_male	first_name
last_name_male	first_name_nonbinary

Generate sample fake names

It can be helpful to see a sample of data from these methods to help identify the best method for the desired fake data. This code calls the methods with name in them to generate a sample for each of the methods. This is very useful to compare the output from the different names.

1for name_method in [x for x in dir(fake) if not x.startswith('_') and x.find('name') > -1]:
2    print(f"{name_method.ljust(23, ' ')} :   { eval(f'fake.{name_method}()') }")

Fake data generated for methods with name

Method Name	Fake value generated
color_name	Navy
cryptocurrency_name	Zcash
currency_name	Surinamese dollar
domain_name	miller-barber.com
file_name	ok.webm
first_name	Aaron
first_name_female	Patricia
first_name_male	Jackson
first_name_nonbinary	Ricky
hostname	web-01.hampton-jenkins.com
language_name	Bihari languages
last_name	Ray
last_name_female	Barker
last_name_male	Vega
last_name_nonbinary	Morales
month_name	August
name	Charles Hoover
name_female	Kathryn Levy
name_male	Timothy Bruce
name_nonbinary	Samantha Carr
safe_color_name	gray
safe_domain_name	example.com
street_name	Kevin Harbor
user_name	haneydaniel

Generate sample fake names using Faker

Regenerating the same fake data - Seed

It is sometimes necessary to generate the same fake data rather than generating random data every time. seed produces the same result when the same methods with the same version of faker are called. This can be required if fake data is being used in unit tests. This is easily achieved by setting a seed similar to random number generation.

 1Faker.seed(42)
 2for i in range(5):
 3    print(fake.name())
 4# Allison Hill
 5# Noah Rhodes
 6# Angie Henderson
 7# Daniel Wagner
 8# Cristian Santos
 9
10
11Faker.seed(42)
12for i in range(5):
13    print(fake.name())
14# Allison Hill
15# Noah Rhodes
16# Angie Henderson
17# Daniel Wagner
18# Cristian Santos

Localized fake data

A locale value can be passed into the constructor for Faker to create fake data in that locale.

1italian_fake = Faker('it_IT')
2for name_method in [x for x in dir(italian_fake) if not x.startswith('_') and x.find('name') == 0]:
3     print(f"{name_method.ljust(23, ' ')} =   { eval(f'italian_fake.{name_method}()') }")
4
5# name                    =   Lando Opizzi
6# name_female             =   Elvira Bonanno-Beccheria
7# name_male               =   Dott. Jacopo Bottaro
8# name_nonbinary          =   Ciro Parmitano

1japanese_fake = Faker('ja_JP')
2for name_method in [x for x in dir(japanese_fake) if not x.startswith('_') and x.find('name') == 0]:
3    print(f"{name_method.ljust(30, ' ')} =   { eval(f'japanese_fake.{name_method}()') }")
4
5# name                           =   山田 春香
6# name_female                    =   小林 知実
7# name_male                      =   橋本 稔
8# name_nonbinary                 =   斎藤 健一

1chinese_fake = Faker('zh_CN')
2for name_method in [x for x in dir(chinese_fake) if not x.startswith('_') and x.find('name') == 0]:
3    print(f"{name_method.ljust(30, ' ')} =   { eval(f'chinese_fake.{name_method}()') }")
4
5# name                           =   张春梅
6# name_female                    =   权建平
7# name_male                      =   贺华
8# name_nonbinary                 =   阎玉梅

Create localized Faker instance to generate localized fake data

Create a chart with fake data

Any of the methods on the instance of Faker can be used in a list comprehension to generate lists of fake values. These lists can be combined into a list of dictionaries and these dictionaries used to load data into a Pandas DataFrame. This can be a great way to quickly generate test data without having to risk exposing customer sensitive data. Once the data is loaded into a DataFrame it can be manipulated or charts created to develop data pipelines with the data.

1Faker.seed(42)
2DATA_SIZE = 500
3names = [fake.first_name() for i in range(DATA_SIZE)]
4ages = [fake.random_int(min=21, max=80) for x in range(DATA_SIZE)]
5states = [fake.state() for x in range(DATA_SIZE)]
6jobs = [fake.job() for x in range(DATA_SIZE)]
7
8data = {'name': names, 'age': ages, 'state': states, 'job': jobs}
9df = pd.DataFrame.from_dict(data)

Top 10 records from a DataFrame containing fake data

	Name	Age	State	Job
0	Danielle	60	Delaware	Surveyor, planning and development
1	Angel	55	Alaska	Surveyor, planning and development
2	Joshua	22	South Dakota	Landscape architect
3	Jeffrey	79	Rhode Island	Community education officer
4	Jill	46	Kansas	Paediatric nurse
5	Erica	58	Virginia	Manufacturing systems engineer
6	Patricia	57	Hawaii	Pharmacologist
7	Christopher	63	Maryland	Arts development officer
8	Robert	22	Oklahoma	Holiday representative
9	Anthony	26	Alabama	Tourism officer

 1mean_age_df = df[['state', 'age']].groupby('state').mean()
 2mean_age_df.rename_axis(None, axis = 0, inplace = True)
 3top_df = mean_age_df.sort_values(by=['age'], ascending = False).head(10)
 4
 5fig, ax = plt.subplots(figsize = (10,7), facecolor=plt.cm.Blues(.2))
 6ax.set_facecolor(plt.cm.Blues(.2))
 7
 8fig.suptitle('Top 10 States with highest average age',
 9             fontsize = 18,  
10             fontweight = 'bold')
11
12states = list(top_df.index)
13y_pos = np.arange(len(states))
14ages = list(top_df['age'])
15
16ax.barh(y_pos, ages, align='center')
17ax.set_yticks(y_pos)
18ax.invert_yaxis()
19ax.set_xlabel('Average Age', fontsize = 14)
20
21# Display highest on top
22ax.set_yticklabels(states, fontsize = 14)
23
24# Hide the right and top spines
25ax.spines['right'].set_visible(False)
26ax.spines['top'].set_visible(False)
27
28plt.show

Create bar chart from fake data loaded into a Pandas DataFrame

Conclusion

Fake data can easily be created with Faker in Python. I can't help asking - do we really need more fake data? I think we do, as it can be useful in a number of situations, such as unit testing or developing data pipelines without the need for real data. There is wide list of data types that can be generated with Faker as well as providing a custom data provider.

Extract data from JSON in Pandas Dataframe

Eric Callanan — Sunday, 24 Jul 2022

Data stored in a csv file can often contain JSON objects in one of the fields. The csv data can easily be loaded into a Pandas Dataframe for analysis. Fields of interest in the JSON object can also be extracted into their own column easily using Dataframe.Apply.

Convert JSON data to Dict

JSON data is a name/value pair structure used to transfer and store data. It maps easily to the Python Dictionary structure. However JSON data in a string cannot be used directly in the Dictionary constructor. This is easily overcome by using the built-in json module to load the string as a Python Dictionary.

1json_string = '{"color": "Red", "wavelength": "620 to 750 nm"}'
2error_dict = dict(json_string)
3
4>>> # ValueError: dictionary update sequence element #0 has length 1; 2 is required

1json_string = '{"color": "Red", "wavelength": "620 to 750 nm"}'
2json_dict = json.loads(json_string)
3
4print(json.dumps(json_dict, indent=4))
5# {
6#     "color": "Red",
7#     "wavelength": "620 to 750 nm"
8# }

Once it is loaded into a Dictionary, elements can be accessed by the keys.

1json_dict.keys()
2# dict_keys(['color', 'wavelength'])
3
4
5json_dict['color']
6# Red

Load csv data with some JSON into Dataframe

Here is sample data where the second column contains JSON objects.

 1df1 = pd.dataframe(
 2    [
 3        ["a", '{"color": "Red",    "wavelength": "620 to 750 nm"}', "11"],
 4        ["b", '{"color": "Orange", "wavelength": "590 to 620 nm"}', "12"],
 5        ["c", '{"color": "Yellow", "wavelength": "570 to 590 nm"}', "13"],
 6        ["d", '{"color": "Green",  "wavelength": "495 to 570 nm"}', "14"],
 7        ["e", '{"color": "Blue",   "wavelength": "450 to 495 nm"}', "15"],
 8        ["f", '{"color": "Indigo", "wavelength": "425 to 450 nm"}', "16"],
 9        ["g", '{"color": "Violet", "wavelength": "380 to 450 nm"}', "17"],
10    ],
11    index=["row 1", "row 2", "row 3", "row 4", "row 5", "row 6", "row 7"],
12    columns=["one", "two", "three"],
13)

Dataframe containing JSON objects in a column

	one	two	three
row 1	a	{ "color": "Red", "wavelength": "620 to 750 nm" }	11
row 2	b	{ "color": "Orange", "wavelength": "590 to 620 nm" }	12
row 3	c	{ "color": "Yellow", "wavelength": "570 to 590 nm" }	13
row 4	d	{ "color": "Green", "wavelength": "495 to 570 nm" }	14
row 5	e	{ "color": "Blue", "wavelength": "450 to 495 nm" }	15
row 6	f	{ "color": "Indigo", "wavelength": "425 to 450 nm" }	16
row 7	g	{ "color": "Violet", "wavelength": "380 to 450 nm" }	17

Extract an element from the JSON object to a new column

The Dataframe apply method is used to apply a function to all data along an axis of the Dataframe. Apply is used to load the json object in each cell as a dictionary and extract the color property, which is appended to the end of the Dataframe as a new column. The apply method can be amazingly fast even for large datasets.

1df1['color'] = df1.apply(lambda x: json.loads(x['two'])['color'], axis = 1)

Dataframe with Color extracted from the JSON data

	one	two	three	color
row 1	a	{ "color": "Red", "wavelength": "620 to 750 nm" }	11	Red
row 2	b	{ "color": "Orange", "wavelength": "590 to 620 nm" }	12	Orange
row 3	c	{ "color": "Yellow", "wavelength": "570 to 590 nm" }	13	Yellow
row 4	d	{ "color": "Green", "wavelength": "495 to 570 nm" }	14	Green
row 5	e	{ "color": "Blue", "wavelength": "450 to 495 nm" }	15	Blue
row 6	f	{ "color": "Indigo", "wavelength": "425 to 450 nm" }	16	Indigo
row 7	g	{ "color": "Violet", "wavelength": "380 to 450 nm" }	17	Violet

Extract all elements from JSON at once

Here are a number of ways to extract all the elements from json objects at once and append the data as columns to the Dataframe. The first loads the JSON data twice once for values and once for keys, this could be improved by defining a function to load the json and return a pandas series. This is not necessary as the Apply method has a result_type parameter that can be set to expand, which will expand list-like results to columns in the Dataframe.

1df1.apply(lambda x: pd.Series(
2    json.loads(x['two']).values(),
3    index=json.loads(x['two']).keys()), 
4    axis=1)

1df1.apply(lambda x: pd.Series(json.loads(x['two'])), axis=1, result_type='expand')

1df1.two.apply(lambda x: pd.Series(json.loads(x)))

All JSON fields extracted to columns

	color	wavelength
row 1	Red	620 to 750 nm
row 2	Orange	590 to 620 nm
row 3	Yellow	570 to 590 nm
row 4	Green	495 to 570 nm
row 5	Blue	450 to 495 nm
row 6	Indigo	425 to 450 nm
row 7	Violet	380 to 450 nm

1df1.join(
2    df1['two'].apply(lambda x: pd.Series(json.loads(x)))
3)

All JSON fields extracted to columns and joined to original Dataframe

	one	two	three	color	wavelength
row 1	a	{ "color": "Red", "wavelength": "620 to 750 nm" }	11	Red	620 to 750 nm
row 2	b	{ "color": "Orange", "wavelength": "590 to 620 nm" }	12	Orange	590 to 620 nm
row 3	c	{ "color": "Yellow", "wavelength": "570 to 590 nm" }	13	Yellow	570 to 590 nm
row 4	d	{ "color": "Green", "wavelength": "495 to 570 nm" }	14	Green	495 to 570 nm
row 5	e	{ "color": "Blue", "wavelength": "450 to 495 nm" }	15	Blue	450 to 495 nm
row 6	f	{ "color": "Indigo", "wavelength": "425 to 450 nm" }	16	Indigo	425 to 450 nm
row 7	g	{ "color": "Violet", "wavelength": "380 to 450 nm" }	17	Violet	380 to 450 nm

Extract elements from different JSON objects

One of the advantages of using JSON to store and transfer data is the flexibility of the JSON storage. Each JSON object is an independent collection of name/value pairs and so a collection of JSON objects may contain different elements. The above code works for all the elements in the series of JSON objects can be expanded into separate columns, elements that are not present in a JSON object will automatically be filled in with NaN.

 1df1 = pd.dataframe(
 2    [
 3        ["a", '{"color": "Red",    "wavelength": "620 to 750 nm", "RGB": "(255,0,0)", "CMYK": "(0.00, 1.00, 1.00, 0.00)"}', "11"],
 4        ["b", '{"color": "Orange", "wavelength": "590 to 620 nm", "CMYK": "(0.00, 0.35, 1.00, 0.00)"}', "12"],
 5        ["c", '{"color": "Yellow", "wavelength": "570 to 590 nm", "RGB": "(255,255,0)", "HSL": "(0.17, 1.00, 0.50)"}', "13"],
 6        ["d", '{"color": "Green",  "wavelength": "495 to 570 nm", "RGB": "(0,250,0)", "HSL": "(0.33, 1.00, 0.50)"}', "14"],
 7        ["e", '{"color": "Blue",   "wavelength": "450 to 495 nm", "RGB": "(0,0,255)", "CMYK": "(1.00, 1.00, 0.00, 0.00)"}', "15"],
 8        ["f", '{"color": "Indigo", "wavelength": "425 to 450 nm", "RGB": "(75,0,130)", "CMYK": "(0.42, 1.00, 0.00, 0.49)"}', "16"],
 9        ["g", '{"color": "Violet", "wavelength": "380 to 450 nm", "RGB": "(238,130,238)", "CMYK": "(0.00, 0.45, 0.00, 0.07)", "HSL": "(0.83, 0.76, 0.72)"}', "17"],
10    ],
11    index=["row 1", "row 2", "row 3", "row 4", "row 5", "row 6", "row 7"],
12    columns=["one", "two", "three"],
13)

Dataframe with inconsistent JSON objects in a column

	one	two	three
row 1	a	{ "color": "Red", "wavelength": "620 to 750 nm", "RGB": "(255,0,0)", "CMYK": "(0.00, 1.00, 1.00, 0.00)" }	11
row 2	b	{ "color": "Orange", "wavelength": "590 to 620 nm", "CMYK": "(0.00, 0.35, 1.00, 0.00)" }	12
row 3	c	{ "color": "Yellow", "wavelength": "570 to 590 nm", "RGB": "(255,255,0)", "HSL": "(0.17, 1.00, 0.50)" }	13
row 4	d	{ "color": "Green", "wavelength": "495 to 570 nm", "RGB": "(0,250,0)", "HSL": "(0.33, 1.00, 0.50)" }	14
row 5	e	{ "color": "Blue", "wavelength": "450 to 495 nm", "RGB": "(0,0,255)", "CMYK": "(1.00, 1.00, 0.00, 0.00)" }	15
row 6	f	{ "color": "Indigo", "wavelength": "425 to 450 nm", "RGB": "(75,0,130)", "CMYK": "(0.42, 1.00, 0.00, 0.49)" }	16
row 7	g	{ "color": "Violet", "wavelength": "380 to 450 nm", "RGB": "(238,130,238)", "CMYK": "(0.00, 0.45, 0.00, 0.07)", "HSL": "(0.83, 0.76, 0.72)" }	17

1df1.join(
2    df1['two'].apply(lambda x: pd.Series(json.loads(x)))
3)

All JSON fields extracted to columns and joined to original Dataframe

	one	two	three	color	wavelength	RGB	CMYK	HSL
row 1	a	{ "color": "Red", "wavelength": "620 to 750 nm", "RGB": "(255,0,0)", "CMYK": "(0.00, 1.00, 1.00, 0.00)" }	11	Red	620 to 750 nm	(255,0,0)	(0.00, 1.00, 1.00, 0.00)	nan
row 2	b	{ "color": "Orange", "wavelength": "590 to 620 nm", "CMYK": "(0.00, 0.35, 1.00, 0.00)" }	12	Orange	590 to 620 nm	nan	(0.00, 0.35, 1.00, 0.00)	nan
row 3	c	{ "color": "Yellow", "wavelength": "570 to 590 nm", "RGB": "(255,255,0)", "HSL": "(0.17, 1.00, 0.50)" }	13	Yellow	570 to 590 nm	(255,255,0)	nan	(0.17, 1.00, 0.50)
row 4	d	{ "color": "Green", "wavelength": "495 to 570 nm", "RGB": "(0,250,0)", "HSL": "(0.33, 1.00, 0.50)" }	14	Green	495 to 570 nm	(0,250,0)	nan	(0.33, 1.00, 0.50)
row 5	e	{ "color": "Blue", "wavelength": "450 to 495 nm", "RGB": "(0,0,255)", "CMYK": "(1.00, 1.00, 0.00, 0.00)" }	15	Blue	450 to 495 nm	(0,0,255)	(1.00, 1.00, 0.00, 0.00)	nan
row 6	f	{ "color": "Indigo", "wavelength": "425 to 450 nm", "RGB": "(75,0,130)", "CMYK": "(0.42, 1.00, 0.00, 0.49)" }	16	Indigo	425 to 450 nm	(75,0,130)	(0.42, 1.00, 0.00, 0.49)	nan
row 7	g	{ "color": "Violet", "wavelength": "380 to 450 nm", "RGB": "(238,130,238)", "CMYK": "(0.00, 0.45, 0.00, 0.07)", "HSL": "(0.83, 0.76, 0.72)" }	17	Violet	380 to 450 nm	(238,130,238)	(0.00, 0.45, 0.00, 0.07)	(0.83, 0.76, 0.72)

1df1.join(
2    df1['two'].apply(lambda x: pd.Series(json.loads(x)))
3).drop(['two'], axis=1)

All JSON fields extracted to columns with original JSON column removed

	one	three	color	wavelength	RGB	CMYK	HSL
row 1	a	11	Red	620 to 750 nm	(255,0,0)	(0.00, 1.00, 1.00, 0.00)	nan
row 2	b	12	Orange	590 to 620 nm	nan	(0.00, 0.35, 1.00, 0.00)	nan
row 3	c	13	Yellow	570 to 590 nm	(255,255,0)	nan	(0.17, 1.00, 0.50)
row 4	d	14	Green	495 to 570 nm	(0,250,0)	nan	(0.33, 1.00, 0.50)
row 5	e	15	Blue	450 to 495 nm	(0,0,255)	(1.00, 1.00, 0.00, 0.00)	nan
row 6	f	16	Indigo	425 to 450 nm	(75,0,130)	(0.42, 1.00, 0.00, 0.49)	nan
row 7	g	17	Violet	380 to 450 nm	(238,130,238)	(0.00, 0.45, 0.00, 0.07)	(0.83, 0.76, 0.72)

Countries with highest deaths from covid

Eric Callanan — Saturday, 18 Dec 2021

A lot of time was spent in 2020 looking at daily covid numbers, which was getting overwhelming. Having taken a break for a while, this article combines covid data and vaccination data with population data to present a view of the countries with the highest deaths from Covid-19 and the vaccination status in the country.

The Jupyter notebook is available on Github Countries-with-highest-deaths-from-covid-19-COVID-19 and it can be launched in a browser using Binder with this link Binder - Countries with highest deaths from COVID-19.

Load Covid data

As shown previously in Countries with highest confirmed cases of Covid-19 with Plotly the data is available from Johns Hopkins University who have made the data available on GitHub. More information about COVID-19 and the coronavirus is available from Coronavirus disease (COVID-19) advice for the public.

The latest data on deaths from Covid-19 is loaded directly from github.

 1# Number of Death Cases - Global: time series data for deaths
 2deaths_path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv'
 3
 4deaths_17_df = pd.read_csv(deaths_path, usecols=['Country/Region', '12/17/21'])
 5deaths_17_df = deaths_17_df.groupby("Country/Region").sum()
 6deaths_17_df.rename_axis(None, axis=0, inplace=True)
 7deaths_17_df
 8
 9"""
10                     12/17/21
11Afghanistan              7332
12Albania                  3158
13Algeria                  6175
14Andorra                   134
15Angola                   1738
16Antigua and Barbuda       117
17Argentina              116892
18Armenia                  7885
19Australia                2142
20Austria                 13438
21"""

 1Top_10 = deaths_17_df.sort_values(by = deaths_17_df.columns[-1], ascending = False).head(10)
 2Top_10
 3
 4"""
 5                12/17/21
 6US                805823
 7Brazil            617395
 8India             476869
 9Mexico            297356
10Russia            289292
11Peru              202076
12United Kingdom    147509
13Indonesia         143986
14Italy             135421
15Iran              130992
16"""

 1bg_color = 'rgba(208,225,242,1.0)'
 2latest_date = pd.to_datetime(deaths_17_df.columns[0])
 3
 4fig = go.Figure(
 5    data = [go.Bar(x = list(Top_10.iloc[:, 0]),
 6                   y = list(Top_10.index),
 7                   hovertemplate = "%{y}: <br>Deaths: %{x:,}<extra></extra>",
 8                   orientation = "h")]
 9)
10
11fig.update_layout(
12    # Set default font
13        font = dict(
14        family = "Droid Sans",
15        size = 14
16    ),
17    # Set figure title
18    title = dict(
19        text = f"<b>Countries with highest deaths from Covid [{latest_date.strftime('%b %d, %Y')}]</b>",
20        xref = 'container',
21        yref = 'container',
22        x = 0.5,
23        y = 0.87,
24        xanchor = 'center',
25        yanchor = 'middle',
26        font = dict(family = 'Droid Sans', size = 24)
27    ),
28    # set the plot bacground color
29    plot_bgcolor = bg_color,
30    # set the hover background color
31    hoverlabel = dict(
32        bgcolor = 'rgba(75,152,201,0.2)',
33        font_size = 16
34    ),
35    paper_bgcolor = bg_color,
36)
37
38fig.update_traces(marker_color = 'rgb(75,152,201)')
39fig['layout']['yaxis']['autorange'] = "reversed"
40fig.show()

Countries with highest deaths from COVID-19

Load Vaccination data

Covid-19 vaccination data is also available publicly on Github at COVID-19 data.

 1# Number of Vaccines - Global: time series data for deaths
 2vaccines_path = "https://raw.githubusercontent.com/govex/COVID-19/master/data_tables/vaccine_data/global_data/time_series_covid19_vaccine_global.csv"
 3vaccines_df = pd.read_csv(vaccines_path)
 4
 5print(f"Shape of vaccines = {vaccines_df.shape}")
 6"""
 7Shape of vaccines = (129128, 8)
 8"""
 9
10print(vaccines_df.iloc[[0,1,2,-3,-2,-1], :])
11"""
12       Country_Region        Date  Doses_admin  People_partially_vaccinated   People_fully_vaccinated Report_Date_String    UID Province_State 
130              Canada  2020-12-14          5.0                          0.0                       0.0         2020-12-14  124.0            NaN 
141               World  2020-12-14          5.0                          0.0                       0.0         2020-12-14    NaN            NaN 
152              Canada  2020-12-15        723.0                          0.0                       0.0         2020-12-15  124.0            NaN 
16129125          Yemen  2021-12-17     786027.0                     556652.0                  366587.0         2021-12-18  887.0            NaN 
17129126         Zambia  2021-12-17    1342989.0                     806611.0                  904201.0         2021-12-18  894.0            NaN 
18129127       Zimbabwe  2021-12-17    7058474.0                    4023289.0                 3035185.0         2021-12-18  716.0            NaN 
19"""

Look at vaccinations for a single Country in the most recent dates.

 1vaccines_df[vaccines_df['Country_Region'] == 'Mexico'].tail(10)
 2"""
 3
 4       Country_Region        Date  Doses_admin  People_partially_vaccinated  People_fully_vaccinated Report_Date_String    UID Province_State 
 5123215         Mexico  2021-12-08  134370326.0                   78816099.0               65543077.0         2021-12-09  484.0            NaN 
 6123849         Mexico  2021-12-09  135022230.0                   78816099.0               65543077.0         2021-12-10  484.0            NaN 
 7124483         Mexico  2021-12-10  135481459.0                   78816099.0               65543077.0         2021-12-11  484.0            NaN 
 8125117         Mexico  2021-12-11  136829147.0                   80191983.0               66003384.0         2021-12-12  484.0            NaN 
 9125751         Mexico  2021-12-12  137169511.0                   80388496.0               66150375.0         2021-12-13  484.0            NaN 
10126385         Mexico  2021-12-13  137357032.0                   80503005.0               66225140.0         2021-12-14  484.0            NaN 
11127019         Mexico  2021-12-14  137357032.0                   80503005.0               66225140.0         2021-12-15  484.0            NaN 
12127651         Mexico  2021-12-15  137357032.0                   81770394.0               66459570.0         2021-12-16  484.0            NaN 
13128283         Mexico  2021-12-16  139820373.0                   81770394.0               66459570.0         2021-12-17  484.0            NaN 
14128915         Mexico  2021-12-17  139820373.0                   81785946.0               66586509.0         2021-12-18  484.0            NaN 
15"""

Look at vaccinations for a single date.

 1vac_17_df = vaccines_df[vaccines_df['Date'] == '2021-12-17']
 2print(vac_17_df.iloc[[0,1,2,4,-4,-3,-2,-1], :])
 3
 4"""
 5       Country_Region        Date   Doses_admin  People_partially_vaccinated  People_fully_vaccinated Report_Date_String    UID Province_State 
 6128496    Afghanistan  2021-12-17  5.228706e+06                 4.397449e+06             3.566192e+06         2021-12-18    4.0            NaN 
 7128497        Albania  2021-12-17  2.226267e+06                 1.111494e+06             1.001189e+06         2021-12-18    8.0            NaN 
 8128498        Algeria  2021-12-17  1.229306e+07                 6.875003e+06             5.391232e+06         2021-12-18   12.0            NaN 
 9128500         Angola  2021-12-17  1.080528e+07                 7.246240e+06             3.559041e+06         2021-12-18   24.0            NaN 
10129124          World  2021-12-17  8.647128e+09                 4.402818e+09             3.622553e+09         2021-12-18    NaN            NaN 
11129125          Yemen  2021-12-17  7.860270e+05                 5.566520e+05             3.665870e+05         2021-12-18  887.0            NaN 
12129126         Zambia  2021-12-17  1.342989e+06                 8.066110e+05             9.042010e+05         2021-12-18  894.0            NaN 
13129127       Zimbabwe  2021-12-17  7.058474e+06                 4.023289e+06             3.035185e+06         2021-12-18  716.0            NaN 
14"""

 1fully_vac_17_df = vac_17_df.groupby("Country_Region").sum()[['People_fully_vaccinated']]
 2fully_vac_17_df.rename_axis(None, axis=0, inplace=True)
 3fully_vac_17_df.drop(['World', 'US (Aggregate)'], axis=0, inplace=True)
 4
 5top_10_vac = fully_vac_17_df.sort_values(by = fully_vac_17_df.columns[-1], ascending = False).head(10)
 6top_10_vac
 7
 8"""
 9           People_fully_vaccinated
10China                 1.162488e+09
11India                 5.330402e+08
12US                    2.034792e+08
13Brazil                1.411751e+08
14Indonesia             1.052381e+08
15Japan                 9.822253e+07
16Mexico                6.658651e+07
17Russia                6.207541e+07
18Pakistan              5.936147e+07
19Germany               5.817144e+07
20"""

Countries with highest vaccinations against COVID-19

Load population data

As detailed in Display COVID-19 numbers relative to population the population data is available from the World Bank data. This is available to download in csv, xml or Excel format. The Excel file contains three worksheets with the population data in the first sheet Data.

Access the excel file to determine the sheet names in the file.

1pop_excel_link = "https://api.worldbank.org/v2/en/indicator/SP.POP.TOTL?downloadformat=excel"
2
3pop_xl = pd.ExcelFile(pop_excel_link, engine = "xlrd")
4print(f"Worksheets = {pop_xl.sheet_names}")  # see all sheet names
5
6"""
7Worksheets = ['Data', 'Metadata - Countries', 'Metadata - Indicators']
8"""

The first sheet *Data* contains the population information.

 1# Load the excel worksheet into a dataframe
 2pop_df = pd.read_excel(
 3    pop_excel_link,
 4    engine = "xlrd",
 5    sheet_name = 'Data',
 6    header = 3)
 7
 8print(pop_df.shape)
 9"""
10(266, 65)
11"""
12
13print(pop_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]])
14"""
15                    Country Name Country Code     Indicator Name         2018         2019         2020 
160                          Aruba          ABW  Population, total     105846.0     106310.0     106766.0 
171    Africa Eastern and Southern          AFE  Population, total  643090131.0  660046272.0  677243299.0 
182                    Afghanistan          AFG  Population, total   37171922.0   38041757.0   38928341.0 
19263                 South Africa          ZAF  Population, total   57792520.0   58558267.0   59308690.0 
20264                       Zambia          ZMB  Population, total   17351714.0   17861034.0   18383956.0 
21265                     Zimbabwe          ZWE  Population, total   14438812.0   14645473.0   14862927.0 
22"""

This can be simplified to just read in the last column as we are interested in the latest population.

 1# Load only the desired columns from the excel worksheet into a dataframe
 2pop_2020_df = pd.read_excel(
 3    pop_excel_link,
 4    engine = "xlrd",
 5    sheet_name = 'Data',
 6    usecols = ['Country Name', 'Country Code', '2019'],
 7    header = 3)
 8
 9pop_2020_df.set_index('Country Name', inplace=True)
10pop_2020_df.rename_axis(None, axis=0, inplace=True)
11print(pop_2020_df.shape)
12"""
13(266, 2)
14"""
15
16print(pop_2020_df.iloc[[0,1,2,3,4,5,-5,-4,-3,-2,-1], :])
17"""
18                            Country Code         2019
19Aruba                                ABW     106310.0
20Africa Eastern and Southern          AFE  660046272.0
21Afghanistan                          AFG   38041757.0
22Africa Western and Central           AFW  446911598.0
23Angola                               AGO   31825299.0
24Albania                              ALB    2854191.0
25Kosovo                               XKX    1788878.0
26Yemen, Rep.                          YEM   29161922.0
27South Africa                         ZAF   58558267.0
28Zambia                               ZMB   17861034.0
29Zimbabwe                             ZWE   14645473.0
30"""

There are a number of regions that are aggregate of other countries. These are removed using the data in the second sheet in the excel file (Metadata - Countries).

 1# Need to remove regions that are not countries
 2metadata_df = pd.read_excel(
 3    pop_excel_link,
 4    engine = "xlrd",
 5    sheet_name = 'Metadata - Countries',
 6    header = 0)
 7
 8# metadata_df.isnan('Region')
 9non_country_regions = list(metadata_df[metadata_df['Region'].isna()]['Country Code'])
10print(non_country_regions)
11
12"""
13['AFE', 'AFW', 'ARB', 'CEB', 'CSS', 'EAP', 'EAR', 'EAS', 'ECA', 'ECS', 'EMU', 'EUU', 'FCS', 'HIC', 'HPC', 'IBD', 'IBT', 'IDA', 'IDB', 'IDX', 'LAC', 'LCN', 'LDC', 'LIC', 'LMC', 'LMY', 'LTE', 'MEA', 'MIC', 'MNA', 'NAC', 'OED', 'OSS', 'PRE', 'PSS', 'PST', 'SAS', 'SSA', 'SSF', 'SST', 'TEA', 'TEC', 'TLA', 'TMN', 'TSA', 'TSS', 'UMC', 'WLD']
14"""

These are the top country populations in the world in 2020.

 1# Remove non-countries from the dataframe
 2pop_2020_df = pop_2020_df[~pop_2020_df['Country Code'].isin(non_country_regions)].copy()
 3top_10_pop = pop_2020_df.sort_values(by = pop_2020_df.columns[-1], ascending = False).head(10)
 4
 5print(top_10_pop)
 6"""
 7                   Country Code          2020
 8China                       CHN  1.410929e+09
 9India                       IND  1.380004e+09
10United States               USA  3.294841e+08
11Indonesia                   IDN  2.735236e+08
12Pakistan                    PAK  2.208923e+08
13Brazil                      BRA  2.125594e+08
14Nigeria                     NGA  2.061396e+08
15Bangladesh                  BGD  1.646894e+08
16Russian Federation          RUS  1.441041e+08
17Mexico                      MEX  1.289328e+08
18"""

Top 10 countries by population

Compare countries in the datasets

There are some discrepancies in the countries between the datasets.

 1def extra_countries(df1, df2):
 2    unmatched_covid_countries = sorted(
 3        list(
 4            df1[
 5                ~df1.index
 6                .isin(list(df2.index))
 7            ].index))
 8    return unmatched_covid_countries
 9
10
11def print_list(data, columns=3):
12    # Pretty display for the list of countries
13    for i, x in enumerate(data):
14        print(f"{x.ljust(38, ' ')}", end='')
15        if i%columns==(columns-1):
16            print()

 1extra = extra_countries(deaths_17_df, pop_2020_df)
 2print_list(extra)
 3
 4"""
 5Bahamas                               Brunei                                Burma                                 
 6Congo (Brazzaville)                   Congo (Kinshasa)                      Czechia                               
 7Diamond Princess                      Egypt                                 Gambia                                
 8Holy See                              Iran                                  Korea, South                          
 9Kyrgyzstan                            Laos                                  MS Zaandam                            
10Micronesia                            Russia                                Saint Kitts and Nevis                 
11Saint Lucia                           Saint Vincent and the Grenadines      Slovakia                              
12Summer Olympics 2020                  Syria                                 Taiwan*                               
13US                                    Venezuela                             Yemen    
14"""

 1extra = extra_countries(pop_2020_df, deaths_17_df)
 2print_list(extra)
 3
 4"""
 5American Samoa                        Aruba                                 Bahamas, The                          
 6Bermuda                               British Virgin Islands                Brunei Darussalam                     
 7Cayman Islands                        Channel Islands                       Congo, Dem. Rep.                      
 8Congo, Rep.                           Curacao                               Czech Republic                        
 9Egypt, Arab Rep.                      Faroe Islands                         French Polynesia                      
10Gambia, The                           Gibraltar                             Greenland                             
11Guam                                  Hong Kong SAR, China                  Iran, Islamic Rep.                    
12Isle of Man                           Korea, Dem. People's Rep.             Korea, Rep.                           
13Kyrgyz Republic                       Lao PDR                               Macao SAR, China                      
14Micronesia, Fed. Sts.                 Myanmar                               Nauru                                 
15New Caledonia                         Northern Mariana Islands              Not classified                        
16Puerto Rico                           Russian Federation                    Sint Maarten (Dutch part)             
17Slovak Republic                       St. Kitts and Nevis                   St. Lucia                             
18St. Martin (French part)              St. Vincent and the Grenadines        Syrian Arab Republic                  
19Turkmenistan                          Turks and Caicos Islands              Tuvalu                                
20United States                         Venezuela, RB                         Virgin Islands (U.S.)                 
21Yemen, Rep.                           
22"""

1extra = extra_countries(deaths_17_df, fully_vac_17_df)
2print_list(extra)
3
4"""
5Diamond Princess                      Eritrea                               Holy See                              
6MS Zaandam                            Summer Olympics 2020                  Tonga   
7"""

1extra = extra_countries(fully_vac_17_df, deaths_17_df)
2print_list(extra)
3"""
4empty list
5"""

 1extra = extra_countries(fully_vac_17_df, pop_2020_df)
 2print_list(extra)
 3
 4"""
 5Bahamas                               Brunei                                Burma                                 
 6Congo (Brazzaville)                   Congo (Kinshasa)                      Czechia                               
 7Egypt                                 Gambia                                Iran                                  
 8Korea, South                          Kyrgyzstan                            Laos                                  
 9Micronesia                            Russia                                Saint Kitts and Nevis                 
10Saint Lucia                           Saint Vincent and the Grenadines      Slovakia                              
11Syria                                 Taiwan*                               US                                    
12Venezuela                             Yemen     
13"""

 1extra = extra_countries(pop_2020_df, fully_vac_17_df)
 2print_list(extra)
 3
 4"""
 5American Samoa                        Aruba                                 Bahamas, The                          
 6Bermuda                               British Virgin Islands                Brunei Darussalam                     
 7Cayman Islands                        Channel Islands                       Congo, Dem. Rep.                      
 8Congo, Rep.                           Curacao                               Czech Republic                        
 9Egypt, Arab Rep.                      Eritrea                               Faroe Islands                         
10French Polynesia                      Gambia, The                           Gibraltar                             
11Greenland                             Guam                                  Hong Kong SAR, China                  
12Iran, Islamic Rep.                    Isle of Man                           Korea, Dem. People's Rep.             
13Korea, Rep.                           Kyrgyz Republic                       Lao PDR                               
14Macao SAR, China                      Micronesia, Fed. Sts.                 Myanmar                               
15Nauru                                 New Caledonia                         Northern Mariana Islands              
16Not classified                        Puerto Rico                           Russian Federation                    
17Sint Maarten (Dutch part)             Slovak Republic                       St. Kitts and Nevis                   
18St. Lucia                             St. Martin (French part)              St. Vincent and the Grenadines        
19Syrian Arab Republic                  Tonga                                 Turkmenistan                          
20Turks and Caicos Islands              Tuvalu                                United States                         
21Venezuela, RB                         Virgin Islands (U.S.)                 Yemen, Rep. 
22"""

Create a mapping between the country names in the datasets.

 1unmatched_covid_countries = extra_countries(deaths_17_df, pop_2020_df)
 2
 3# remove cruise ships and disputed teritories that don't have any equivalent country in population dataset
 4unmatched_covid_countries.remove('Diamond Princess')
 5unmatched_covid_countries.remove('MS Zaandam')
 6unmatched_covid_countries.remove('Holy See')
 7unmatched_covid_countries.remove('Taiwan*')
 8unmatched_covid_countries.remove('Summer Olympics 2020')
 9
10# Manually map these countries to the country names used in COVID-19 dataset
11map_pop = ['Bahamas, The', 'Brunei Darussalam', 'Myanmar', 'Congo, Rep.', 'Congo, Dem. Rep.',
12           'Czech Republic', 'Egypt, Arab Rep.', 'Gambia, The', 'Iran, Islamic Rep.', 
13           'Korea, Rep.', 'Kyrgyz Republic', 'Lao PDR', 'Micronesia, Fed. Sts.', 
14           'Russian Federation', 'St. Kitts and Nevis', 'St. Lucia', 'St. Vincent and the Grenadines', 
15           'Slovak Republic', 'Syrian Arab Republic', 'United States', 'Venezuela, RB', 'Yemen, Rep.'
16          ]
17
18country_map = {}
19for c,p in zip(unmatched_covid_countries, map_pop):
20    country_map[p] = c

Mapping of mis-matched country names between Covid-data and Population data:

COVID-19 Country Name	Population Country Name
Bahamas	Bahamas, The
Brunei	Brunei Darussalam
Burma	Myanmar
Congo (Brazzaville)	Congo, Rep.
Congo (Kinshasa)	Congo, Dem. Rep.
Czechia	Czech Republic
Egypt	Egypt, Arab Rep.
Gambia	Gambia, The
Iran	Iran, Islamic Rep.
Korea, South	Korea, Rep.
Kyrgyzstan	Kyrgyz Republic
Laos	Lao PDR
Micronesia	Micronesia, Fed. Sts.
Russia	Russian Federation
Saint Kitts and Nevis	St. Kitts and Nevis
Saint Lucia	St. Lucia
Saint Vincent and the Grenadines	St. Vincent and the Grenadines
Slovakia	Slovak Republic
Syria	Syrian Arab Republic
US	United States
Venezuela	Venezuela, RB
Yemen	Yemen, Rep.

Add a field to the population data to contain the country name.

 1# Add the covid country to the population dataset
 2pop_2020_df['Covid_Country'] = pop_2020_df.apply(
 3    lambda x: country_map[x.name]
 4        if x.name in country_map 
 5        else x.name, 
 6    axis = 1)
 7
 8print(pop_2020_df.iloc[[1,2,3,4,-4,-3,-2,-1], :])
 9
10"""
11             Country Code        2020 Covid_Country
12Afghanistan           AFG  38928341.0   Afghanistan
13Angola                AGO  32866268.0        Angola
14Albania               ALB   2837743.0       Albania
15Andorra               AND     77265.0       Andorra
16Yemen, Rep.           YEM  29825968.0         Yemen
17South Africa          ZAF  59308690.0  South Africa
18Zambia                ZMB  18383956.0        Zambia
19Zimbabwe              ZWE  14862927.0      Zimbabwe
20"""

Confirm that there are no unmatched countries in the Covid datasets.

 1print(list(deaths_17_df[
 2    ~deaths_17_df.index
 3    .isin(list(pop_2020_df['Covid_Country']))
 4].index))
 5
 6"""
 7['Diamond Princess', 'Holy See', 'MS Zaandam', 'Summer Olympics 2020', 'Taiwan*']
 8"""
 9
10print(list(fully_vac_17_df[
11    ~fully_vac_17_df.index
12    .isin(list(pop_2020_df['Covid_Country']))
13].index))
14
15"""
16['Taiwan*']
17"""

Merge datasets on country

A new dataframe is created with the data for the deaths from covid-19 and the vaccinations as well as the population for each country.

 1covid_pop_df = deaths_17_df.merge(
 2    fully_vac_17_df,
 3    how='left',
 4    left_index = True,
 5    right_on = fully_vac_17_df.index
 6).merge (
 7    pop_2020_df,
 8    how='left',
 9    left_on = covid_df.index,
10    right_on = 'Covid_Country'
11)
12
13covid_pop_df.set_index('key_0', inplace=True)
14covid_pop_df.rename_axis(None, axis=0, inplace=True)
15
16# # Rename column names for deaths and population
17covid_pop_df = covid_pop_df.rename(columns={'12/17/21': 'Deaths'})
18covid_pop_df = covid_pop_df.rename(columns={'2020': 'Population'})
19
20# # Remove unneeded columns
21covid_pop_df.drop(['Country Code', 'Covid_Country'], axis=1, inplace=True)
22
23print(covid_pop_df.iloc[[1,2,3,4,-4,-3,-2,-1], :])
24
25"""
26                    Deaths  People_fully_vaccinated  Population
27Albania               3158                1001189.0   2837743.0
28Algeria               6175                5391232.0  43851043.0
29Andorra                134                  50551.0     77265.0
30Angola                1738                3559041.0  32866268.0
31West Bank and Gaza    4855                      0.0   4803269.0
32Yemen                 1974                 366587.0  29825968.0
33Zambia                3674                 904201.0  18383956.0
34Zimbabwe              4779                3035185.0  14862927.0
35"""

Overview of Countries with highest deaths from covid per capita

The number of people vaccinated is calculated per 1,000 of the population and the number of people died is calculated per 100,000 of the population. The top countries with the highest deaths per capita and those with the highest vaccinations are shown for countries with populations greater than 1 million.

 1covid_pop_df['deaths_100k'] = covid_pop_df.apply(lambda x: (x.Deaths / x.Population) * 100000, axis = 1)
 2
 3covid_pop_df['vac_100'] = covid_pop_df.apply(lambda x: (x.People_fully_vaccinated / x.Population) * 100, axis = 1)
 4
 5print(covid_pop_df.iloc[[1,2,3,4,-4,-3,-2,-1], :])
 6
 7"""
 8                    Deaths  People_fully_vaccinated  Population  deaths_100k    vac_100  
 9Albania               3158                1001189.0   2837743.0   111.285624  35.281172  
10Algeria               6175                5391232.0  43851043.0    14.081763  12.294421  
11Andorra                134                  50551.0     77265.0   173.429108  65.425484  
12Angola                1738                3559041.0  32866268.0     5.288097  10.828857  
13West Bank and Gaza    4855                      0.0   4803269.0   101.076996   0.000000  
14Yemen                 1974                 366587.0  29825968.0     6.618394   1.229087  
15Zambia                3674                 904201.0  18383956.0    19.984817   4.918425  
16Zimbabwe              4779                3035185.0  14862927.0    32.153828  20.421179  
17"""

 1top_10 = (covid_pop_df[covid_pop_df['Population'] > 1000000]
 2          .sort_values(by = ['deaths_100k'], ascending = False)
 3          .head(20))
 4
 5bg_color = 'rgba(208,225,242,1.0)'
 6
 7fig = go.Figure(
 8    data = [go.Bar(x = list(top_10["deaths_100k"]),
 9                   y = list(top_10.index),
10                   hovertemplate = "%{y}: <br>Deaths per 100K: %{x:.0f}<extra></extra>",
11                   orientation = "h")]
12)
13
14fig.update_layout(
15    # Set default font
16        font = dict(
17        family = "Droid Sans",
18        size = 14
19    ),
20    # Set figure title
21    title = dict(
22        text = f"<b>Countries with highest deaths from Covid-19 per 100K of population [Dec 17, 2020]</b>",
23        xref = 'container',
24        yref = 'container',
25        x = 0.5,
26        y = 0.92,
27        xanchor = 'center',
28        yanchor = 'middle',
29        font = dict(family = 'Droid Sans', size = 24)
30    ),
31    # set the plot bacground color
32    plot_bgcolor = bg_color,
33    # set the hover background color
34    hoverlabel = dict(
35        bgcolor = 'rgba(75,152,201,0.2)',
36        font_size = 16
37    ),
38    paper_bgcolor = bg_color,
39)
40
41fig.update_traces(marker_color = 'rgb(75,152,201)')
42fig['layout']['yaxis']['autorange'] = "reversed"
43fig.show()

Countries with highest deaths from Covid-19 per 100K of population

These are the countries with populations over 1 million with the lowest number of deaths from covid-19 per capita.

Countries with lowest deaths from Covid-19 per 100K of population

These are the countries with populations over 1 million with the highest percent of the population vaccinated against covid-19.

Countries with lowest deaths from Covid-19 per 100K of population

Vaccination status in countries with highest deaths

A bar plot in relative mode is used to show percent vaccinated with number of deaths in countries with the highest deaths from cvid-19.

 1top_20 = (covid_pop_df[covid_pop_df['Population'] > 1000000]
 2          .sort_values(by = ['deaths_100k'], ascending = False)
 3          .head(20))
 4
 5neg_vac = [(-1 * x) for x in top_20.vac_100]
 6
 7bg_color = 'rgba(208,225,242,1.0)'
 8
 9fig = go.Figure()
10fig.add_trace(go.Bar(x = [v*6 for v in neg_vac],
11                     y = list(top_20.index),
12                     marker_color = 'rgb(30,75,80)',
13                     name = 'vaccinations',
14                     customdata = [(v * -1) for v in neg_vac],
15                     hovertemplate = "%{y}: <br>Vaccinated: %{customdata:.0f}%<extra></extra>",
16                     orientation = "h"))
17fig.add_trace(go.Bar(x = list(top_20["deaths_100k"]),
18                     y = list(top_20.index),
19                     marker_color = 'rgb(75,152,201)',
20                     name = 'deaths',
21                     hovertemplate = "%{y}: <br>Deaths per 100K: %{x:.0f}<extra></extra>",
22                     orientation = "h"))
23
24
25fig.update_layout(
26    # Set default font
27    font = dict(
28        family = "Droid Sans",
29        size = 14
30    ),
31    # Set figure title
32    title = dict(
33        text = f"<b>Countries with highest deaths from Covid-19 per 100K of population [Dec 17, 2020]</b>",
34        xref = 'container',
35        yref = 'container',
36        x = 0.5,
37        y = 0.95,
38        xanchor = 'center',
39        yanchor = 'middle',
40        font = dict(family = 'Droid Sans', size = 24)
41    ),
42    # set x-axis
43    xaxis = dict(
44        title = '',
45        showticklabels = False,
46    ),
47    
48    legend=dict(
49        orientation="h",
50        yanchor="bottom",
51        y=1.02,
52        xanchor="center",
53        x=0.41
54    ),
55
56    # set the plot bacground color
57    plot_bgcolor = bg_color,
58    # set the hover background color
59    hoverlabel = dict(
60        bgcolor = 'rgba(75,152,201,0.2)',
61        font_size = 16
62    ),
63    paper_bgcolor = bg_color,
64)
65
66fig.update_layout(barmode='relative')
67fig['layout']['yaxis']['autorange'] = "reversed"
68fig.show()

Countries with highest deaths from Covid-19 per 100K of population with percentage vaccinated

Conclusion

The covid-19 pandemic is far from over and their continues to be a number of Country status for Covid and Vaccinations. There is a trend that those countries with higher percentage of the population vaccinated have lower deaths per capita, but there are exceptions to this.

How to filter a Pandas DataFrame

Eric Callanan — Sunday, 07 Nov 2021

There is a filter method on Pandas DataFrame, but it is limited to only filtering on the labels on the index columns. There are multiple ways to filter a DataFrame to focus on the information required. This article demonstrates a number of ways to filter data in a DataFrame.

Comparing a DataFrame to be equal to a value with double equals ( == ) will return a dataframe of boolean values set to true, where the values in the dataframe equal the compare value. This boolean dataframe can be fed into the original dataframe to return a dataframe with only the data that matched the compare value. This is more useful when the comparison is on a single column.

Summary of methods to filter Pandas DataFrame:

Filtering data in Pandas DataFrame
df[df['Rank'] == 16] df[df['Rank'].eq(16)]	Get rows in df that have a Rank equal to 16
df[df['Rank'] >= 16] df[df['Rank'].ge(16)]	Get rows in df that have a Rank greater than or equal to 16
df[df['Rank'] > 16] df[df['Rank'].gt(16)] df[df['Rank'] != 16] df[df['Rank'].ne(16)] df[df['Rank'] < 16] df[df['Rank'].lt(16)] df[df['Rank'] <= 16] df[df['Rank'].le(16)]	Other numeric comparison include greater than Not equal to less than less than or equal to
df[df['Name'] == 'JASMINE'] df[df['Name'].eq('JASMINE')]	Get rows in df that have a Name equal to JASMINE
df[df['Name'].str.startswith('A')]	Get rows in df that have a Name that starts with A
df[df['Name'].str.endswith('A')]	Get rows in df that have a Name that ends with A
df[df['Name'].str.contains('AS')]	Get rows in df that have a Name that contains the letters 'AS'
df[df['Name'].str.contains('as', case=False)]	Get rows in df that have a Name that contains the letters 'as' ignoring case
df[df['Name'].str.contains('ian$\|ine$', case=False, regex=True)]	Get rows in df that have a Name ending in 'ian' or 'ine' ignoring case
df[df['Name'].isin(['SOPHIA','PAUL'])]	Get rows in df that have a Name in a list of names
df[( (df['Name'].str.contains('A')) & (df['Year'] > 1980) & (df['Gender'] == 'Female') & (df['Count'] > 1500) )]	Get rows in df that have a Name containing the letter 'A' and have Gender Female with a count greater than 1500 and are after 1980
df.filter(items=['Year', 'Name'], axis=1)	Use filter method to filter the dataframe to specified columns
df.filter(like='7', axis=0)	Filter DataFrame to index equal to a value
df.filter(regex='^5\|8', axis=0)	Use filter method to filter the dataframe to indexes beginning with 5 or 8

Load sample data

We will use the most popular baby names from popular baby names. This dataset contains data for the top 25 baby names in California based on information entered on birth certificates.

Load the data directly from the website:

1url = "https://data.chhs.ca.gov/dataset/4a8cb74f-c4fa-458a-8ab1-5f2c0b2e22e3/resource/f3fe42ed-4441-4fd8-bf53-92fb80a246da/download/2021-06-18_topbabynames_1960-2019.csv"
2names_df = pd.read_csv(url)

Review the data:

 1print(names_df.shape)
 2names_df.head()
 3
 4(3003, 5)
 5
 6     Year    Gender    Rank   Name       Count
 70    1960    Female    1      SUSAN      3299
 81    1960    Female    2      MARY       3248
 92    1960    Female    3      KAREN      3156
103    1960    Female    4      CYNTHIA    2982
114    1960    Female    5      LISA       2839

Trim the data to a sample set of 10 rows that will be used to demonstrate filtering dataframes.

1df = names_df.sample(n=10, random_state=1)
2df.reset_index(drop=True, inplace=True)

Sample dataframe of top baby names

Sample dataframe of top baby names

Filter by numeric data in a column

A common field type to filter data on is a numeric field. It is possible to filter where the number is equal to a particular number or less than or greater than a number.

1df.dtypes
2
3"""
4Year       int64
5Gender    object
6Rank       int64
7Name      object
8Count      int64
9"""

Selecting a column in a dataframe such as df['Rank'] returns a Pandas.Series, which can be compared to a single number that will return a Pandas.Series of boolean values.

 1df['Rank'] == 16
 2
 3"""
 40    False
 51    False
 62    False
 73    False
 84    False
 95    False
106    False
117     True
128     True
139    False
14"""

DataFrame of booleans that match the criteria

DataFrame of booleans that match the criteria

The Pandas.Series of boolean values can then be used on the original DataFrame to return only the rows where the boolean values are true. Pandas DataFrame also has functions such as eq to perform element-wise comparison, similar to the logic comparisons such as ==.

All logic comparisons work in a similar way

operator	function	description
==	eq	equal to
!=	ne	not equal to
>	gt	greater than
>=	ge	greater than or equal to
<	lt	less than
<=	le	less than or equal to

1df[df['Rank'] == 16]
2
3"""
4   Year  Gender  Rank       Name  Count
57  1983  Female    16    VANESSA   1517
68  1995    Male    16  CHRISTIAN   2761
7"""

Filter DataFrame on numeric value

Filter DataFrame on numeric value

Numeric data Greater Than

1df[df['Rank'] > 19]
2
3"""
4   Year  Gender  Rank     Name  Count
51  2000  Female    24  DESTINY   1188
63  1991  Female    23  JASMINE   1482
79  1960    Male    21     PAUL   2136
8"""

Filter DataFrame on numeric value - greater than

Filter DataFrame on numeric value - greater than

Numeric data Greater Than or Equal To

1df[df['Rank'] >= 19]
2
3"""
4   Year  Gender  Rank     Name  Count
51  2000  Female    24  DESTINY   1188
63  1991  Female    23  JASMINE   1482
74  1967    Male    19   THOMAS   2149
89  1960    Male    21     PAUL   2136
9"""

Filter DataFrame on numeric value - greater than or equal to

Filter DataFrame on numeric value - greater than or equal to

Filter by text in a column

Text fields can be filtered in a similar manner as the numeric fields.

1df[df['Name'] == 'JASMINE']
2
3"""
4   Year  Gender  Rank     Name  Count
53  1991  Female    23  JASMINE   1482
6"""

Filter DataFrame on Text value

Filter DataFrame on Text value

Filter by partial text match in columns

Text comparison is case-sensitive and the logic operators compare the given text with the text in the entire field for an exact match. The greater-than and less-than operators work on text, but these are of limited use. Pandas.Series.str to the rescue! The Pandas.Series.str has a number of string comparisons to allow filtering dataframes on virtually anything. The use of pandas.Series.str.contains and pandas.Series.str.match allows the use or regular expressions, but these may be slower on larger dataframes. The contains function takes a boolean parameter for regex and when set to False, treats the pattern as a literal string. The contains function defaults to searching in a case-sensitive manner, but this can be changed with the parameter case.

Filter on names starting with a letter

1df[df['Name'].str.startswith('A')]
2
3"""
4   Year  Gender  Rank    Name  Count
52  1992  Female     6  AMANDA   2776
6"""

Filter DataFrame on Text value - startswith

Filter DataFrame on Text value - startswith

Filter on names ending with a letter

1df[df['Name'].str.endswith('A')]
2
3"""
4   Year  Gender  Rank     Name  Count
52  1992  Female     6   AMANDA   2776
65  2013  Female     1   SOPHIA   3447
77  1983  Female    16  VANESSA   1517
8"""

Filter DataFrame on Text value - endswith

Filter DataFrame on Text value - endswith

Filter on names containing text

1df[df['Name'].str.contains('AS')]
2
3"""
4   Year  Gender  Rank     Name  Count
53  1991  Female    23  JASMINE   1482
64  1967    Male    19   THOMAS   2149
7"""

Filter DataFrame on Text value - contains

Filter DataFrame on Text value - contains

Match string ignoring case using the case parameter set to False.

1df[df['Name'].str.contains('as', case=False)]
2
3"""
4   Year  Gender  Rank     Name  Count
53  1991  Female    23  JASMINE   1482
64  1967    Male    19   THOMAS   2149
7"""

Filter DataFrame on Text value - contains, ignore case

Filter DataFrame on Text value - contains, ignore case

The default for pandas.Series.str.contains is to use a regular expression. This code searches for all names that end in either ian or ine. When the regex parameter is set to False this does not match any names.

 1df[df['Name'].str.contains('ian$|ine$', case=False, regex=True)]
 2
 3"""
 4   Year  Gender  Rank       Name  Count
 53  1991  Female    23    JASMINE   1482
 66  1980    Male    14      BRIAN   2737
 78  1995    Male    16  CHRISTIAN   2761
 8"""
 9
10df[df['Name'].str.contains('ian$|ine$', case=False, regex=False)]
11
12"""
13Empty DataFrame
14"""

Filter DataFrame on Text value - regex

Filter DataFrame on Text value - regex

Filter by multiple search criteria

Filtering on a single field is great. Filtering data can be even more powerful when combining different search criteria either on the same column or on multiple columns. This can get confusing as the number of conditions grow and it is worth noting that the conditions are applied to each row individually. As an example, suppose we want all the data for names 'SOPHIA' and 'PAUL'. We cannot look for data where name is 'SOPHIA' and where name is 'PAUL', as no individual row has both of these. Instead we need to look for data where name is 'SOPHIA' OR where name is 'PAUL'.

However, if it is an exact match on multiple values, it is better to use pandas.DataFrame.isin function - and it is easier to read.

1df[((df['Name'] == 'SOPHIA') | (df['Name'] == 'PAUL'))]
2
3df[df['Name'].isin(['SOPHIA','PAUL'])]
4
5"""
6   Year  Gender  Rank    Name  Count
75  2013  Female     1  SOPHIA   3447
89  1960    Male    21    PAUL   2136
9"""

Filter DataFrame on Text value - isin

Filter DataFrame on Text value - isin

Filter on all records where the name starts with "J" and ends in "INE".

1df[((df['Name'].str.startswith('J')) & (df['Name'].str.endswith('INE')))]
2
3"""
4   Year  Gender  Rank     Name  Count
53  1991  Female    23  JASMINE   1482
6"""

Filter DataFrame on Text value - startswith and endswith

Filter DataFrame on Text value - startswith and endswith

Filter by multiple columns

Multiple search criteria can also be applied across multiple columns. This code filters on all rows where the "Name" contains the letter "A" and the "Year" is after 1980 and the "Gender" is "Female" and the "Count" is greater than 1,500. The conditions are enclosed in braces and spread out on multiple lines to make the conditions easier to read. The filtered data will only contain rows where all of these conditions are met.

 1df[(
 2    (df['Name'].str.contains('A'))
 3    & (df['Year'] > 1980)
 4    & (df['Gender'] == 'Female')
 5    & (df['Count'] > 1500)
 6   )]
 7
 8"""
 9   Year  Gender  Rank     Name  Count
102  1992  Female     6   AMANDA   2776
115  2013  Female     1   SOPHIA   3447
127  1983  Female    16  VANESSA   1517
13"""

Filter DataFrame on multiple columns

Filter DataFrame on multiple columns

Filter the full names dataset

Filter on all of the Male names ending in "R" in the top ten names after the year 1990.

 1names_df[(
 2    (names_df['Name'].str.endswith('R'))
 3    & (names_df['Year'] > 1990)
 4    & (names_df['Gender'] == 'Male')
 5    & (names_df['Rank'] < 10)
 6   )]
 7
 8"""
 9      Year Gender  Rank         Name  Count
101577  1991   Male     2  CHRISTOPHER   5672
111628  1992   Male     3  CHRISTOPHER   5102
121679  1993   Male     4  CHRISTOPHER   4600
131729  1994   Male     4  CHRISTOPHER   4172
141779  1995   Male     4  CHRISTOPHER   4013
151830  1996   Male     5  CHRISTOPHER   3896
161880  1997   Male     5  CHRISTOPHER   3680
171934  1998   Male     9  CHRISTOPHER   3386
181984  1999   Male     9  CHRISTOPHER   3237
192033  2000   Male     8  CHRISTOPHER   3311
202334  2006   Male     9  CHRISTOPHER   2869
212382  2007   Male     7  CHRISTOPHER   2978
222431  2008   Male     6    ALEXANDER   2865
232434  2008   Male     9  CHRISTOPHER   2607
242480  2009   Male     5    ALEXANDER   2875
252529  2010   Male     4    ALEXANDER   2593
262581  2011   Male     6    ALEXANDER   2454
272632  2012   Male     7    ALEXANDER   2415
282682  2013   Male     7    ALEXANDER   2374
292731  2014   Male     5    ALEXANDER   2485
302783  2015   Male     6    ALEXANDER   2304
312834  2016   Male     7    ALEXANDER   2218
322886  2017   Male     9    ALEXANDER   1998
332935  2018   Male     8    ALEXANDER   1908
342984  2019   Male     7    ALEXANDER   1838
35"""

Filter using Filter method

The filter method on Pandas DataFrame is limited to only filtering on the index column names. This is of limited use, but it does support filtering on regex. The filter method can take 4 parameters but items, like, or regex are mutually exclusive. Using Filter can be useful when the field of interest is the index or can be set as the index of the DataFrame.

Filter on columns in the list.

 1df.filter(items=['Year', 'Name'], axis=1)
 2
 3"""
 4   Year       Name
 50  1982   MICHELLE
 61  2000    DESTINY
 72  1992     AMANDA
 83  1991    JASMINE
 94  1967     THOMAS
105  2013     SOPHIA
116  1980      BRIAN
127  1983    VANESSA
138  1995  CHRISTIAN
149  1960       PAUL
15"""

Filter columns on DataFrame

Filter columns on DataFrame

Filter where the index is 7.

1df.filter(like='7', axis=0)
2
3"""
4   Year  Gender  Rank     Name  Count
57  1983  Female    16  VANESSA   1517
6"""

Filter DataFrame on Index equal to 7

Filter DataFrame on Index equal to 7

Filter where the index begins with 5 or 8 using regex.

1df.filter(regex='^5|8', axis=0)
2
3"""
4   Year  Gender  Rank       Name  Count
55  2013  Female     1     SOPHIA   3447
68  1995    Male    16  CHRISTIAN   2761
7"""

Filter DataFrame with regex on index

Filter DataFrame with regex on index

Conclusion

It is easy to filter data in a Pandas DataFrame, but the filter method may not be the method you are looking for! If you are familiar with SQL, you may find the where method, but this is not used for filtering in Pandas as discussed in "Pandas - Dataframe Where function".

Filtering data in DataFrames is done by matching criteria on a particular column in the DataFrame and feeding the boolean Data Series into the original DataFrame to filter out where there is not a match. This sounds more complicated than it is and this article showed some basic examples of filtering on numeric data and text data.

Create interactive chart with matplotlib and ipywidgets

Eric Callanan — Sunday, 18 Apr 2021

ipywidgets are interactive HTML elements that can be used in Jupyter Notebooks to interact with outputs such as tables and charts. This article briefly introduces ipywidgets and uses them to change the rolling day period in a chart for the rolling average of deaths from covid-19.

ipywidgets or (jupyter-widgets) are interactive HTML elements used to provide the user with tools to interact with data and visualize the effects of parameter changes.

Slider with Interact

The Interact function is a convenient way to quickly create an interactive widget. It takes a function as it's first parameter and then the function parameters with the possible values. Interact creates user interface controls for exploring code and data interactively.

The function below takes one parameter and returns a string stating the value of that parameter. ipywidgets uses widget abbreviations to generate the appropriate UI control, for example when an integer-valued keyword argument of 10 is passed to interact, it generates an integer-valued slider control with a range of -10 to 30.

1def f(x):
2    return f"The selected value is '{x}'"
3
4w.interact(f, x=10)

Note that if you change x to pass in a boolean value, then a checkbox control is created and a textbox is created if a string is passed in.

1def f(x):
2    return f"The selected value is '{x}'"
3
4_ = w.interact(f, x=100)
5_ = w.interact(f, x=True)
6_ = w.interact(f, x="Apple")
7_ = w.interact(f, x=["Apple", "Banana", "Pear"])

It is necessary to specify the start, stop and increment values in a tuple to explicitly create a slider control.

1_ = w.interact(f, x=(1,7,1))

Creating ipywidgets with interact

Creating ipywidgets with interact

There are many widgets documented in the Widget List that are distributed with ipywidgets. These can be combined to create a more complex user interface composed of numerous widgets. As these are HTML elements, the Layout and Styling of ipywidgets utilizes many of the styling mechanisms of web pages. The layout attribute uses a number of CSS properties such as height, width, visibility, border and margin as well as grid-layout and flexbox features. This code created four ipywidgets controls and lays them out using HBox and VBox.

 1slider = w.IntSlider(value=5, min=0, max=10, step=1, description="Number")
 2button = w.Button(
 3    description="The Button",
 4    style=w.ButtonStyle(button_color="Plum"),
 5)
 6dropdown = w.Dropdown(
 7    value="green",
 8    options=["red", "orange", "yellow", "green", "blue", "indigo", "violet"],
 9    description="Rainbow",
10)
11radiobuttons = w.RadioButtons(
12    value="thu",
13    options=["mon", "tue", "wed", "thu", "fri", "sat", "sun"],
14    description="Which Day",
15)
16
17w.HBox([slider, button, dropdown, radiobuttons])
18
19w.VBox([slider, button, dropdown, radiobuttons])

Horizontal and vertical layout of ipywidgets

Horizontal and vertical layout of ipywidgets

The vertical and horizontal boxes can be nested to build up more complex layouts. This shows a horizontal containing two vertical boxes. Border ans padding styles can be applied to the boxes to make it easier to highlight sections.

1w.HBox(
2    [
3        w.VBox([button, radiobuttons]),
4        w.VBox([slider, dropdown]),
5    ]
6)

1box1 = w.VBox([button, radiobuttons])
2box2 = w.VBox([slider, dropdown])
3
4box1.layout = w.Layout(border="solid 1px red", margin="5px", padding="10px")
5box2.layout = w.Layout(border="solid 1px blue", margin="5px", padding="10px")
6
7w.HBox([box1, box2])

Nested ipywidgets in vertical boxes laid out ia a parent horizontal layout

Nested ipywidgets in vertical boxes laid out ia a parent horizontal layout

Rolling average of Deaths from COVID-19

I have previously had a look at the Daily counts and rolling average of COVID-19 numbers, so decided to recreate a chart like this with the addition of widgets. This code retrieves the latest data from Johns Hopkins University Github page, cleans the data and creates the chart for the top five countries with the highest rolling average of deaths.

1# Number of Death Cases - Global: time series data for deaths
2deaths_path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv'
3deaths_df = pd.read_csv(deaths_path)

 1def clean_covid_data(df):
 2    # 1. Drop unwanted columns
 3    df.drop(['Province/State', 'Lat', 'Long'], axis = 1, inplace = True)
 4
 5    # 2. Group by Country
 6    df = df.groupby('Country/Region').sum()
 7
 8    # 3. Convert column headings to datetime
 9    df.columns = pd.to_datetime(df.columns)
10
11    # 4. Remove the column name
12    df.rename_axis(None, axis = 0, inplace = True)
13
14    return df

 1def plot_axis(ax, df, countries, rolling, days):
 2    ax.set_title(f"{rolling}-day rolling average, for the last {days} days")
 3    ax.set_facecolor(plt.cm.Blues(0.2))
 4    rolling_df = df.rolling(rolling, axis=1).mean()
 5    for i, country in enumerate(countries):
 6        ax.plot(
 7            df.columns[-days:],
 8            rolling_df.loc[country].tail(days),
 9            label=country,
10            color=plt.cm.tab10(i),
11        )
12    ax.legend(bbox_to_anchor=(0.99, 1), loc="upper left", facecolor=plt.cm.Blues(0.1))
13    ax.set_xlabel("Date", fontsize="large", fontweight="bold")
14    ax.set_ylabel("Number of Deaths", fontsize="large", fontweight="bold")
15    ax.spines["right"].set_visible(False)
16    ax.spines["top"].set_visible(False)

1def create_rolling_average_chart(df, countries, rolling, days, title):
2    fig, ax = plt.subplots(figsize=(10, 5), facecolor=plt.cm.Blues(0.2))
3    fig.suptitle(title, fontsize="xx-large", fontweight="bold")
4    plot_axis(ax, df, countries, rolling, days)
5    fig.autofmt_xdate()
6    plt.tight_layout()
7    return fig

 1deaths_df = clean_covid_data(deaths_df)
 2daily_deaths_df = deaths_df.diff(axis=1)
 3top_countries = list(
 4    daily_deaths_df.iloc[:, -1].sort_values(ascending=False).head(10).index
 5)
 6create_rolling_average_chart(
 7    daily_deaths_df,
 8    top_countries[:5],
 9    7,
10    100,
11    "Daily rolling average of deaths from COVID-19",
12)

Seven-day-rolling-average-of-deaths-covid-19

Seven-day rolling average of deaths from COVID-19 for five highest countries

Using ipywidgets to modify days for rolling average

The following class is created to put the widgets and the matplotlib chart together. In this example a slider widget is used to select the rolling period from 1 to 10 inclusive. In addition, a dropdown widget is used to set the number of recent days. One of the key aspects to this interaction is the observe method of the widget to register a callback. observe is used to specify a method to call when the widget value changes rolling_slider.observe(self.update_rollout, "value").

  1class average_deaths(widgets.VBox):
  2    df = None
  3    output = widgets.Output()
  4    rolling_days = 7
  5    recent_days = 100
  6
  7    def __init__(self, df):
  8        super().__init__()
  9        self.df = df
 10
 11        # create the chart
 12        self.create_chart()
 13
 14        # create and layout the widget controls
 15        controls = self.create_control_panel()
 16
 17        # add the widgets to the view
 18        self.children = [controls, self.output]
 19
 20    def create_chart(self):
 21        """create the chart in the widget output control"""
 22        with self.output:
 23            self.fig, self.ax = plt.subplots(
 24                figsize=(9.5, 4.5), facecolor=plt.cm.Blues(0.2)
 25            )
 26            self.fig.suptitle(
 27                "Deaths from COVID-19", fontsize="xx-large", fontweight="bold"
 28            )
 29            self.plot_axis()
 30            self.fig.autofmt_xdate()
 31            plt.tight_layout()
 32            self.fig.canvas.toolbar_position = "bottom"
 33
 34    def create_control_panel(self):
 35        """create the slider and dropdown controls"""
 36        rolling_slider = widgets.IntSlider(
 37            value=self.rolling_days,
 38            min=1,
 39            max=10,
 40            step=1,
 41        )
 42        recent_dropdown = widgets.Dropdown(
 43            options=[50, 100, 200, 300, 1000],
 44            value=self.recent_days,
 45            description="",
 46            disabled=False,
 47        )
 48        controls = widgets.HBox(
 49            [
 50                widgets.VBox(
 51                    [
 52                        widgets.Label("Rolling Average Days: "),
 53                        widgets.Label("Recent Number of Days: "),
 54                    ]
 55                ),
 56                widgets.VBox([rolling_slider, recent_dropdown]),
 57            ]
 58        )
 59        controls.layout = self.control_box_layout()
 60
 61        # hook up the widgets to observe changes
 62        rolling_slider.observe(self.update_rollout, "value")
 63        recent_dropdown.observe(self.update_recent, "value")
 64        return controls
 65
 66    def control_box_layout(self):
 67        """create layout for control panel"""
 68        return widgets.Layout(
 69            border="solid 1px Navy",
 70            margin="0px 10px 10px 0px",
 71            padding="5px 5px 5px 5px",
 72        )
 73
 74    def update_rollout(self, change):
 75        """update the chart with changes in rolling frequency"""
 76        self.rolling_days = change.new
 77        self.plot_axis()
 78        self.fig.autofmt_xdate()
 79
 80    def update_recent(self, change):
 81        """update the chart with changes in recent number of days"""
 82        self.recent_days = change.new
 83        self.plot_axis()
 84        self.fig.autofmt_xdate()
 85
 86    def plot_axis(self):
 87        """clear any previous chart and plot the lines for the top countries"""
 88        self.ax.clear()
 89        self.ax.set_title(
 90            f"{self.rolling_days}-day rolling average, for the last {self.recent_days} days"
 91        )
 92        self.ax.set_facecolor(plt.cm.Blues(0.2))
 93        rolling_df = self.df.rolling(self.rolling_days, axis=1).mean()
 94        top_countries = list(
 95            rolling_df.iloc[:, -1].sort_values(ascending=False).head(10).index
 96        )
 97        for i, country in enumerate(top_countries[:5]):
 98            self.ax.plot(
 99                self.df.columns[-self.recent_days :],
100                rolling_df.loc[country].tail(self.recent_days),
101                label=country,
102                color=plt.cm.tab10(i),
103            )
104        self.ax.legend(
105            bbox_to_anchor=(1, 0.8), loc="upper left", facecolor=plt.cm.Blues(0.1)
106        )
107        self.ax.set_xlabel("Date", fontsize="large", fontweight="bold")
108        self.ax.set_ylabel("Number of Deaths", fontsize="large", fontweight="bold")
109        self.ax.spines["right"].set_visible(False)
110        self.ax.spines["top"].set_visible(False)
111
112
113average_deaths(daily_deaths_df)

Interact with matplotlib chart using ipywidgets

Interact with matplotlib chart using ipywidgets

example of using ipywidgets to interact with chart

Example of using ipywidgets to interact with chart

Conclusion

I'm still not sure about the use of ipywidgets. They provide a mechanism of interacting with other user interface in a Jupyter notebook, so they could be used to create a number of variations for charts without having to create the different charts. However, they require a server to deploy or else an interactive Jupyter notebook and if one has the interactive notebook, then the data could be changed and the various charts created. ipywidgets may be more useful when there are easier ways to deploy and interact with the Jupyter notebooks containing them. It would be great if GitHub hosted the Jupyter notebooks in such a way to allow interaction with ipywidgets - Rolling averages of deaths from COVID-19. A work around is to open these notebooks in Binder - Here is a direct link to Binder - Rolling averages of deaths from COVID-19. ipywidgets are interesting, but I'm not seeing the practical application of them yet.

Three Dimensional Shapes and Shading

Eric Callanan — Sunday, 24 Jan 2021

Three dimensional objects are not easy to represent on a 2D surface such as a page or screen. In this article, inspired by chapter 3 of Math for Programmers book, we look at the cross product of two 3D-vectors and representing the surface of any 3D object as a collection of triangles. The cross product can be used to determine the direction of a triangle and this can be used to determine if the triangle is visible and the level of shading.

These are my notes on the book Math for Programmers by Paul Orland.

2D vectors and vector arithmetic
Two Dimensional shapes
Three Dimensional Vectors and Dot Product
Three Dimensional Shapes and Shading

Cross product

The cross product of two vectors v and u (v × u), is a vector that is perpendicular to both v and u, and therefore normal to the plane containing them. The magnitude of the cross product vector equals the area of a parallelogram with the vectors for sides. The direction of the vector of the cross product obeys the right-hand rule.

The orientation of the three axes in three-dimensional space can have two possible orientations. The right-hand rule is a common mnemonic for understanding orientation of axes in three-dimensional space. One can see this by pointing the right index finger along the positive x-axis and curl the remaining fingers toward the positive y-axis, the thumb tells you the direction of the positive z-axis.

The formula to calculate the cross product seems a little complex at first. It is easy to write a python function to calculate the cross product.

$$ \begin{gathered} vector \ u = (u_x, \ u_y, \ u_z) \\\ vector \ v = (v_x, \ v_y, \ v_z) \end{gathered} $$

$$ u \ {\times} \ v = (u_yv_z - u_zv_y, \ u_zv_x - u_xv_z , \ u_xv_y - u_yv_x) $$

Adding color to the x,y,z coordinates makes it easier to see that the resultant coordinate is made from the difference of the products of the other two coordinates. For example the cross product x coordinate is the difference of the y*z and z*y of the vectors. Note that cross product is noncommutative in that u x v is not equal to v x u.

$$ vector \ u = (u_{\color{aqua}x}, \ u_{\color{fuchsia}y}, \ u_{\color{olive}z}), \\\ vector \ v = (v_{\color{aqua}x}, \ v_{\color{fuchsia}y}, \ v_{\color{olive}z}) $$

$$ u \ {\times} \ v = (u_{\color{fuchsia}y}v_{\color{olive}z} - u_{\color{olive}z}v_{\color{fuchsia}y}, \ u_{\color{olive}z}v_{\color{aqua}x} - u_{\color{aqua}x}v_{\color{olive}z} , \ u_{\color{aqua}x}v_{\color{fuchsia}y} - u_{\color{fuchsia}y}v_{\color{aqua}x}) $$

$$ u \ {\times} \ v \neq v \ {\times} \ u $$

 1def cross(u, v):
 2    ux,uy,uz = u
 3    vx,vy,vz = v
 4    return (uy*vz - uz*vy, uz*vx - ux*vz, ux*vy - uy*vx)
 5
 6print(cross((3, 4, 5), (4, 4, 4)))
 7print(cross((4, 4, 4), (3, 4, 5)))
 8
 9"""
10(-4, 8, -4)
11(4, -8, 4)
12"""

 1def plot_axes(ax):
 2    ax.plot((-4, 4), (0, 0), (0, 0), color="grey")
 3    ax.plot((0, 0), (-4, 4), (0, 0), color="grey")
 4    ax.plot((0, 0), (0, 0), (-4, 4), color="grey")
 5
 6
 7def plot_vector(ax, v, c):
 8    ax.plot((0, v[0]), (0, v[1]), (0, v[2]), color=c)
 9
10
11def update_axis(ax, v1, v2):
12    plot_axes(ax)
13    plot_vector(ax, v1, "blue")
14    plot_vector(ax, v2, "red")
15    vx = cross(v1, v2)
16    plot_vector(ax, vx, "cyan")
17    ax.set_title(f"\n{v1} x {v2} \ncross product = {vx}", fontsize=10)
18
19
20fig, axs = plt.subplots(
21    2, 3, figsize=(10, 8), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
22)
23fig.suptitle(
24    f"Cross product of two 3D-vectors",
25    fontsize=18,
26    fontweight="bold",
27)
28
29for i, ax in enumerate(axs.flatten()):
30    ax.set_facecolor(plt.cm.Blues(0.1))
31    ax.set_xlim(-7, 5)
32    ax.set_ylim(-5, 5)
33    ax.set_zlim(-6, 10)
34    ax.set_xlabel("x-axis")
35    ax.set_ylabel("y-axis")
36    ax.set_zlabel("z-axis")
37
38u = (3, 4, 5)
39v = (4, 4, 4)
40update_axis(axs[0][0], u, v)
41update_axis(axs[0][1], v, u)
42update_axis(axs[1][0], u, v)
43update_axis(axs[1][1], v, u)
44
45u = (2, 2, 0)
46v = (0, 4, 0)
47update_axis(axs[0][2], u, v)
48update_axis(axs[1][2], v, u)
49
50for ax in [axs[1][0], axs[1][1], axs[1][2]]:
51    ax.view_init(azim=-80)
52
53fig.tight_layout(pad=2.0)
54plt.show()

Cross product of two 3D vectors

Cross product of 3D vectors viewed from two angles

Length of Cross Product

The magnitude (length) of the cross product is equal to the area of the parallelogram formed by the two vectors. The cross product of parallel vectors is zero as is the cross product of opposite vectors. The magnitude of the cross product of two vectors is at a maximum when those vectors are at 90⁰ to each other.

 1def plot_axes(ax):
 2    ax.plot((-4, 4), (0, 0), (0, 0), color="grey")
 3    ax.plot((0, 0), (-4, 4), (0, 0), color="grey")
 4    ax.plot((0, 0), (0, 0), (-4, 4), color="grey")
 5
 6
 7def plot_vector(ax, v, c):
 8    ax.plot((0, v[0]), (0, v[1]), (0, v[2]), color=c)
 9
10
11def update_axis(ax, v1, v2):
12    plot_axes(ax)
13    plot_vector(ax, v1, "blue")
14    plot_vector(ax, v2, "red")
15    vx = cross(v1, v2)
16    plot_vector(ax, vx, "cyan")
17    ax.set_title(
18        f"\n{v1} x {v2} \nlength cross product = {vector_length(vx):.1F}", fontsize=10
19    )
20
21
22fig, axs = plt.subplots(
23    2, 3, figsize=(10, 8), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
24)
25fig.suptitle(
26    f"Length of cross product of two 3D-vectors",
27    fontsize=18,
28    fontweight="bold",
29)
30
31for i, ax in enumerate(axs.flatten()):
32    ax.set_facecolor(plt.cm.Blues(0.1))
33    ax.set_xlim(-7, 5)
34    ax.set_ylim(-5, 5)
35    ax.set_zlim(-6, 10)
36    ax.set_xlabel("x-axis")
37    ax.set_ylabel("y-axis")
38    ax.set_zlabel("z-axis")
39
40update_axis(axs[0][0], (3, 0, 0), (3, 0, 0))
41update_axis(axs[0][1], (3, 0, 0), (2, 2, 0))
42update_axis(axs[0][2], (3, 0, 0), (1, 3, 0))
43update_axis(axs[1][0], (3, 0, 0), (0, 3, 0))
44update_axis(axs[1][1], (3, 0, 0), (-2, 2, 0))
45update_axis(axs[1][2], (3, 0, 0), (-3, 0, 0))
46
47plt.show()
48
49publish_png_image(fig, "length-of-cross-product-3d-vector.png")

Length of cross product is equal to parallelogram area

Length of cross product is equal to parallelogram area

Triangle composed of 3 vectors

A triangle is composed of three sides or three vectors. Each vector is composed of two sets of coordinates, a start and an end. To form a triangle the three vectors must be connected so the start/end of one vector must match the start/end of another vector.

 1def add_vectors(*v):
 2    return tuple(sum(x) for x in zip(*v))
 3
 4
 5def subtract_vectors(v1, v2):
 6    return tuple(x[0] - x[1] for x in zip(v1, v2))
 7
 8
 9print(subtract_vectors((2, 3, 4), (0, 0, 0)))
10print(subtract_vectors((2, 3, 4), (4, 4, 4)))
11"""
12(2, 3, 4)
13(-2, -1, 0)
14"""

 1def plot_axes(ax):
 2    ax.plot((-3, 3), (0, 0), (0, 0), color="grey")
 3    ax.plot((0, 0), (-3, 3), (0, 0), color="grey")
 4    ax.plot((0, 0), (0, 0), (-3, 3), color="grey")
 5
 6
 7def plot_vector(ax, v1, v2, c):
 8    ax.plot((v1[0], v2[0]), (v1[1], v2[1]), (v1[2], v2[2]), color=c)
 9
10
11def update_axis(ax, *vectors):
12    for v in vectors:
13        plot_vector(ax, v[0], v[1], "blue")
14
15
16fig, axs = plt.subplots(
17    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
18)
19fig.suptitle(
20    f"Cross Product on a Triangle",
21    fontsize=18,
22    fontweight="bold",
23)
24
25titles = ["Vectors", "Triangle", "Cross Product"]
26for i, ax in enumerate(axs.flatten()):
27    ax.set_facecolor(plt.cm.Blues(0.1))
28    ax.set_xlim(-3, 3)
29    ax.set_ylim(-3, 3)
30    ax.set_zlim(-3, 3)
31    ax.set_xlabel("x-axis")
32    ax.set_ylabel("y-axis")
33    ax.set_zlabel("z-axis")
34    ax.set_title(titles[i])
35    plot_axes(ax)
36
37# triangle
38t = ((2, 0, 0), (0, 2, 0), (0, 0, 2))
39orig = (0, 0, 0)
40cp = add_vectors(
41    t[0], cross(subtract_vectors(t[1], t[0]), subtract_vectors(t[2], t[0]))
42)
43
44update_axis(axs[0], (orig, t[0]), (orig, t[1]), (orig, t[2]))
45update_axis(axs[1], (t[0], t[1]), (t[0], t[2]), (t[1], t[2]))
46update_axis(axs[2], (t[0], t[1]), (t[0], t[2]), (t[1], t[2]))
47plot_vector(ax, t[0], cp, "Red")
48
49plt.show()

Use of Cross Product to determine the outward side of a triangle

Use of Cross Product to determine the outward side of a triangle

Shading

shading refers to the process of altering the color of an object based on the source of light and the viewing perspective as well as the angle of the light and the angle of the object surface to the viewer. Ultimately the 3D objects are displayed on the 2D screen and based on calculations using properties like cross product and projecting the visible surfaces onto a 2D surface, the 3D object is presented for viewing.

The vertices of a shape can be plotted to display the shape or the plot_wireframe function from matplotlib can be used. Similarly, instead of calculating the surface and shading of an object, the plot_surface function is used. The following code plots a triangle using vertices as well as the plot_wireframe and plot_surface functions.

 1def plot_axes(ax):
 2    ax.plot((-2, 2), (0, 0), (0, 0), color="grey")
 3    ax.plot((0, 0), (-2, 2), (0, 0), color="grey")
 4    ax.plot((0, 0), (0, 0), (-2, 2), color="grey")
 5
 6
 7def plot_vector(ax, v1, v2, c):
 8    ax.plot((v1[0], v2[0]), (v1[1], v2[1]), (v1[2], v2[2]), color=c)
 9
10
11def plot_polygon(ax, *v):
12    for i in range(len(v)):
13        j = (i + 1) % len(v)
14        plot_vector(ax, v[i], v[j], "Blue")
15
16
17fig, axs = plt.subplots(
18    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
19)
20fig.suptitle(
21    f"Plotting a triangle in 3D",
22    fontsize=18,
23    fontweight="bold",
24)
25
26for i, ax in enumerate(axs.flatten()):
27    ax.set_facecolor(plt.cm.Blues(0.1))
28    ax.set_xlim(-1, 3)
29    ax.set_ylim(-1, 3)
30    ax.set_zlim(-1, 3)
31    ax.set_xlabel("x-axis")
32    ax.set_ylabel("y-axis")
33    ax.set_zlabel("z-axis")
34    plot_axes(ax)
35
36# triangle vertices
37triangle = [(0, 2, 0), (2, 2, 0), (0, 2, 2)]
38plot_polygon(axs[0], *triangle)
39axs[0].set_title("Vertices")
40
41x = np.array([[0, 2], [0, 0]])
42y = np.array([[2, 2], [2, 2]])
43z = np.array([[0, 0], [0, 2]])
44axs[1].plot_wireframe(x, y, z, color="Blue")
45axs[1].set_title("plot_wireframe")
46
47axs[2].plot_surface(x, y, z, color="Blue")
48axs[2].set_title("plot_surface")
49
50plt.show()

plot triangle using plot_wireframe and plot_surface functions

Plot triangle using plot_wireframe and plot_surface functions

 1def plot_vector(ax, v1, v2, c):
 2    ax.plot((v1[0], v2[0]), (v1[1], v2[1]), (v1[2], v2[2]), color=c)
 3
 4
 5def plot_polygon(ax, *v):
 6    for i in range(len(v)):
 7        j = (i + 1) % len(v)
 8        plot_vector(ax, v[i], v[j], "Blue")
 9
10
11fig, axs = plt.subplots(
12    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
13)
14fig.suptitle(
15    f"Vertices in a side",
16    fontsize=18,
17    fontweight="bold",
18)
19
20for i, ax in enumerate(axs.flatten()):
21    ax.set_facecolor(plt.cm.Blues(0.1))
22    ax.set_xlim(-1, 3)
23    ax.set_ylim(-1, 3)
24    ax.set_zlim(-1, 3)
25    ax.set_xlabel("x-axis")
26    ax.set_ylabel("y-axis")
27    ax.set_zlabel("z-axis")
28    ax.set_title(titles[i])
29
30square = [(2, 2, 2), (0, 2, 2), (0, 2, 0), (2, 2, 0)]
31plot_polygon(axs[0], *square)
32axs[0].set_title("Vertices")
33
34x = np.array([[2, 0], [2, 0]])
35y = np.array([[2, 2], [2, 2]])
36z = np.array([[2, 2], [0, 0]])
37axs[1].plot_wireframe(x, y, z, color="Blue")
38axs[1].set_title("plot_wireframe")
39
40axs[2].plot_surface(x, y, z, color="Blue")
41axs[2].set_title("plot_surface")
42
43plt.show()

plot square using plot_wireframe and plot_surface functions

Plot square using plot_wireframe and plot_surface functions

Cube

Plotting a 3D object using vertices and plotting segments and faces can get complicated without functions like plot_wireframe and plot_surface. A NumPy 2-dimensional array is used for the x,y and z coordinates. plot_surface has a number of parameters that can be used to customise the rendered object such as lightsource as used in the third plot.

 1fig, (ax1, ax2, ax3) = plt.subplots(
 2    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
 3)
 4fig.suptitle(
 5    f"Cube",
 6    fontsize=18,
 7    fontweight="bold",
 8)
 9
10for i, ax in enumerate([ax1, ax2, ax3]):
11    ax.set_facecolor(plt.cm.Blues(0.1))
12    ax.set_xlim(-1, 3)
13    ax.set_ylim(-1, 3)
14    ax.set_zlim(-1, 3)
15    ax.set_xlabel("x-axis")
16    ax.set_ylabel("y-axis")
17    ax.set_zlabel("z-axis")
18
19x = np.array(
20    [
21        [2, 0, 0, 2, 2],
22        [2, 0, 0, 2, 2],
23        [0, 2, 2, 0, 0],
24        [0, 2, 2, 0, 0],
25        [2, 0, 0, 2, 2],
26    ]
27)
28
29y = np.array(
30    [
31        [2, 2, 0, 0, 2],
32        [2, 2, 0, 0, 2],
33        [0, 0, 2, 2, 0],
34        [0, 0, 2, 2, 0],
35        [2, 2, 0, 0, 2],
36    ]
37)
38
39z = np.array(
40    [
41        [2, 2, 2, 2, 2],
42        [0, 0, 0, 0, 0],
43        [0, 0, 0, 0, 0],
44        [2, 2, 2, 2, 2],
45        [2, 2, 2, 2, 2],
46    ]
47)
48
49ax1.set_title("Wireframe")
50ax1.plot_wireframe(x, y, z)
51
52ax2.set_title("Surface")
53ax2.plot_surface(x, y, z)
54
55ax3.set_title("Light Source")
56ls = matplotlib.colors.LightSource(4, -4, 4)
57ax3.plot_surface(x, y, z, lightsource=ls)
58
59plt.show()

plot cube using plot_wireframe and plot_surface functions

Plot cube using plot_wireframe and plot_surface functions

Sphere

Like a circle or a curved line in 2D-space can be represented by a sequence of small straight lines, a curved surface in 3D-space can be represented by a collection of small triangles. The higher the number of segments the more curved the sphere will appear.

 1def create_sphere(segments):
 2    u = np.linspace(0, 2 * np.pi, segments)
 3    v = np.linspace(0, np.pi, segments)
 4    x = 20 * np.outer(np.cos(u), np.sin(v))
 5    y = 20 * np.outer(np.sin(u), np.sin(v))
 6    z = 20 * np.outer(np.ones(np.size(u)), np.cos(v))
 7    return (x, y, z)
 8
 9
10fig, (ax1, ax2, ax3) = plt.subplots(
11    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
12)
13fig.suptitle(
14    f"Sphere Wireframes",
15    fontsize=18,
16    fontweight="bold",
17)
18
19for i, ax in enumerate([ax1, ax2, ax3]):
20    ax.set_facecolor(plt.cm.Blues(0.1))
21    ax.set_xlim(-25, 25)
22    ax.set_ylim(-25, 25)
23    ax.set_zlim(-25, 25)
24    ax.set_xlabel("x-axis")
25    ax.set_ylabel("y-axis")
26    ax.set_zlabel("z-axis")
27    n = 5 * i + 10
28    ax.set_title(f"segments: {n}")
29    (x, y, z) = create_sphere(n)
30    ax.plot_wireframe(x, y, z)
31
32plt.show()

Plot sphere using plot_wireframe with increasing numbers of segments

Plot sphere wireframe with increasing numbers of segments

 1def create_sphere(segments):
 2    u = np.linspace(0, 2 * np.pi, segments)
 3    v = np.linspace(0, np.pi, segments)
 4    x = 20 * np.outer(np.cos(u), np.sin(v))
 5    y = 20 * np.outer(np.sin(u), np.sin(v))
 6    z = 20 * np.outer(np.ones(np.size(u)), np.cos(v))
 7    return (x, y, z)
 8
 9
10fig, axs = plt.subplots(
11    2, 3, figsize=(10, 8), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
12)
13fig.suptitle(
14    f"Sphere composed with increasing number of segments",
15    fontsize=18,
16    fontweight="bold",
17)
18
19for i, ax in enumerate(axs.flatten()):
20    ax.set_facecolor(plt.cm.Blues(0.1))
21    ax.set_xlim(-25, 25)
22    ax.set_ylim(-25, 25)
23    ax.set_zlim(-25, 25)
24    ax.set_xlabel("x-axis")
25    ax.set_ylabel("y-axis")
26    ax.set_zlabel("z-axis")
27    n = (10 * i) + 10
28    ax.set_title(f"segments: {n}")
29    (x, y, z) = create_sphere(n)
30    ax.plot_surface(x, y, z)
31
32plt.show()

Plot sphere using plot_surface with increasing numbers of segments

Plot sphere surface with increasing numbers of segments

Plot sphere with decreasing transparency

Plot sphere with decreasing transparency

Torus

A Torus is a geometric surface generated by revolving a circle in 3-D space about an axis that is coplanar with the circle. A real-world objects that approximate a torus of revolution is an inner tube and an example of a solid torus is a ring doughnut or bagel.

 1def create_torus(segments):
 2    u = np.linspace(0, 2 * np.pi, segments)
 3    v = np.linspace(0, 2 * np.pi, segments)
 4    u, v = np.meshgrid(u, v)
 5    a,b = 5,9
 6    x = (b + a * np.cos(u)) * np.cos(v)
 7    y = (b + a * np.cos(u)) * np.sin(v)
 8    z = a * np.sin(u)
 9    return (x, y, z)
10
11
12fig, (ax1, ax2) = plt.subplots(
13    1, 2, figsize=(10, 5), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
14)
15fig.suptitle(
16    f"Torus",
17    fontsize=18,
18    fontweight="bold",
19)
20
21(x, y, z) = create_torus(50)
22
23for i, ax in enumerate([ax1, ax2]):
24    ax.set_facecolor(plt.cm.Blues(0.1))
25    ax.set_xlim(-14, 14)
26    ax.set_ylim(-14, 14)
27    ax.set_zlim(-14, 14)
28    ax.set_xlabel("x-axis")
29    ax.set_ylabel("y-axis")
30    ax.set_zlabel("z-axis")
31
32ax1.set_title(f"wireframe")
33ax1.plot_wireframe(x, y, z)
34
35ax2.set_title(f"surface")
36ax2.plot_surface(x, y, z)
37
38plt.show()

Plot torus

Plot torus

Conclusion

The cross product of two vectors is a vector that is perpendicular to both vectors. The magnitude of the cross product vector equals the area of a parallelogram with the vectors for sides. The direction of the vector of the cross product obeys the right-hand rule.

The surface of 3D objects can be represented by a collection of triangles and the cross product of the vertices can be used to determine whether the surface is visible and level of shading to apply.

Matplotlib has functions for rendering wireframes and surfaces of 3D objects and examples of cube, sphere and torus were shown.

matplotlib a comprehensive library for creating static, animated, and interactive visualizations in Python.

Math for Programmers by Paul Orland - a great book to brush up on your math skills.

Speed up image file transfer by zipping

Eric Callanan — Sunday, 17 Jan 2021

Compressing multiple files into one archive file allows for faster files transfer. Image files in the form of jpg are already compressed and compressing them into zip files does not significantly reduce the file size further. However, it does reduce the number files to one archive file and this makes file transfer faster, even including the time to decompress the zip file at the destination.

Files can be compressed into a zip file for storage to reduce the amount of disk space required. This article demonstrates that transferring large numbers of image files to Google Colab and decompressing is over 300 times faster than transferring the same data as individual files.

Google Colab (Colaboratory)

Google released Google Colab in 2017 as an online collaborating tool for machine learning with cloud storage. This is an impressive resource that is free to use, it presents a Jupyter-like notebook interface that is easily accessed through a browser using a google account. The Colab Jupyter notebooks are saved automatically to the Google Drive associated with the account. The Jupyter notebooks can be shared, copied, reviewed and opened on other machines.

This is a great free resource, but there are some constraints such as inactive sessions will time out and be shut down and all local variables and loaded data are lost when the session ends. In addition, I found that uploading numerous files is very time consuming, even from mapped google drive. This is where use of zip files significantly improved file transfer times.

Image dataset

This experiment was done with some of the images from the dogs-vs-cats data from Kaggle, but it could be done with any set of images. The images were sampled randomly and split into multiple datasets of the following increasing sizes. These were stored in directories as well as each directory compressed into a zip file. The table shows that zipping the directories does not reduce the size much. These zip files and directories were uploaded to Google Drive for easy access from Google Colab. Note that it can take a couple of hours for the files to upload to Google Drive.

Table: Comparison of directory sizes and zip file size for sample images

number of files	directory size (KB)	zip file size (KB)
10	200	199
20	398	397
30	623	621
40	850	848
50	1097	1094
60	1320	1317
70	1571	1567
80	1802	1797
90	2043	2038
100	2209	2203
200	4478	4462
300	6664	6640
400	8811	8778
500	10929	10888
600	13254	13203
700	15520	15461
800	17719	17650
900	20147	20069
1000	22418	22329
2000	45587	45393
3000	68260	67971
4000	91057	90675
5000	114013	113539

Code to create the sample data from the Cats & Dogs image files.

 1def get_file_names_from_dir(directory, ext=".jpg"):
 2    file_list = []
 3    for root, dirnames, filenames in os.walk(directory):
 4        # Only load image files
 5        files = [f for f in filenames if f.lower().endswith(ext)]
 6        for filename in files:
 7            file_path = os.path.join(root, filename)
 8            if os.path.getsize(file_path) > 0:
 9                file_list.append(file_path)
10            else:
11                print(f"{filename} is zero length, so ignoring.")
12    return file_list

 1def make_dir_and_zip(num):
 2    root_dir = r"/tmp/dogs-vs-cats/split"
 3    dest_dir = f"{root_dir}/images_{num}"
 4    if os.path.exists(dest_dir):
 5        shutil.rmtree(dest_dir)
 6    if not os.path.exists(dest_dir):
 7        os.makedirs(dest_dir)
 8
 9    for a in animals_df.sample(num, random_state=42)["animals"]:
10        shutil.copy(a, dest_dir)
11
12    shutil.make_archive(dest_dir, "zip", dest_dir)

1animals = get_file_names_from_dir(r"/tmp/dogs-vs-cats")
2animals_df = pd.DataFrame({'animals': animals})
3
4for i in range(10, 100, 10):
5    print("- ", end="")
6    make_dir_and_zip(i)
7    make_dir_and_zip(i * 10)
8    make_dir_and_zip(i * 100)

Transfer image files individually

The following function copies all files in a directory using copytree function from shutil. This uses time module to calculate the time taken and returns the time taken in milliseconds as well as the number of files copied.

1def copy_directory_and_files(source_path, dest_path):
2    print('Copying files')
3    start = time.time()
4    # Copy the content of source to destination
5    dest = shutil.copytree(source_path, dest_path)
6    copy_dir_time = (time.time() - start) * 1000
7    print(f'directory copied in {copy_dir_time:.0F} milliseconds')
8    num_copied = len(os.listdir(dest_path))
9    return copy_dir_time, num_copied

 1num = 10
 2source_dir = f"{image_remote_dir}images_{num}"
 3dest_dir = f"{local_image_dir}images_{num}"
 4copy_dir_time, num_copied = copy_directory_and_files(source_dir, dest_dir)
 5
 6"""
 7Copying files
 8directory copied in 4021 milliseconds
 9"""
10
11copy_dir_time
12"""
134021
14"""
15
16num_copied
17"""
1810
19"""

Transfer zip file and unzip image files

A similar function is created to copy the zip files and extract the files from the zip file into a directory.

 1def copy_and_extract_data(zip_filename, source_path, dest_path, extract_path):
 2    zource_zip = source_path + zip_filename
 3    local_zip = dest_path + zip_filename
 4
 5    start = time.time()
 6    # Copy the content of source to destination
 7    dest = shutil.copyfile(zource_zip, local_zip)
 8    copy_time = (time.time() - start) * 1000
 9    print(f"Zip file copied in {copy_time:.0F} milliseconds")
10
11    start = time.time()
12    # Unzip the file
13    zip_ref = zipfile.ZipFile(local_zip, "r")
14    zip_ref.extractall(extract_path)
15    zip_ref.close()
16    unzip_time = (time.time() - start) * 1000
17    print(f"Zip file Extracted in {unzip_time:.0F} milliseconds")
18
19    num_extracted = len(os.listdir(extract_path))
20
21    return copy_time, unzip_time, num_extracted

 1num = 10
 2zip_filename = f"images_{num}.zip"
 3extract_path = f"{local_image_dir_zips}images_{num}"
 4copy_time, unzip_time, num_extracted = copy_and_extract_data(
 5    zip_filename, image_remote, local_image_dir_zips, extract_path
 6)
 7
 8"""
 9Zip file copied in 740 milliseconds
10Zip file Extracted in 5 milliseconds
11"""
12
13copy_time, unzip_time, num_extracted
14"""
15740, 5, 10
16"""

time to copy 10 files

Copying zip file and extracting is 5 times faster than copying files individually

Copying just 10 files from Google Drive to a local directory in Google Colab takes 4021 milliseconds. Copying the zip file with the same images takes just 740 milliseconds and only 5 milliseconds to decompress the zip file. I found it surprising that even with just 10 small images (ranging from 5 to 40 KB) that the copying the zip file and extracting is significantly faster. Using the zip file is over five times faster.

Transfer all files

The following code transfers all the zip files and extracts the images as well as transferring the directories with file images. The resulting times are gathered into a dataframe for analysis. Note this time-consuming to run, so the results are saved on each iteration incase connection to the Google Colab is lost and everything gets reset.

 1def copy_twice(num):
 2    print()
 3    print("*" * 30)
 4    print(f"num = {num}")
 5    zip_filename = f"images_{num}.zip"
 6    extract_path = f"{local_image_dir_zips}images_{num}"
 7    copy_time, unzip_time, num_extracted = copy_and_extract_data(
 8        zip_filename, image_remote, local_image_dir_zips, extract_path
 9    )
10    print()
11
12    source_dir = f"{image_remote}images_{num}"
13    dest_dir = f"{local_image_dir}images_{num}"
14    copy_dir_time, num_copied = copy_directory_and_files(source_dir, dest_dir)
15
16    r_df = pd.DataFrame(
17        [[num, copy_time, unzip_time, num_extracted, copy_dir_time, num_copied]],
18        columns=[
19            "num",
20            "copy_zip_time",
21            "unzip_time",
22            "num_extracted",
23            "copy_files_time",
24            "num_copied",
25        ],
26    )
27    return r_df

 1# Create the results dataframe
 2result_df = pd.DataFrame(
 3    [[0, 0, 0, 0, 0, 0]],
 4    columns=[
 5        "num",
 6        "copy_zip_time",
 7        "unzip_time",
 8        "num_extracted",
 9        "copy_files_time",
10        "num_copied",
11    ],
12)
13
14for num in range(10, 101, 10):
15    result_df = result_df.append(copy_twice(num))
16    result_df.to_csv(f"/content/gdrive/MyDrive/image_tests/timing_results_10_{num}.csv")
17
18for i in range(20, 101, 10):
19    num = i * 10
20    result_df = result_df.append(copy_twice(num))
21    result_df.to_csv(f"/content/gdrive/MyDrive/image_tests/timing_results_10_{num}.csv")
22
23for i in range(20, 51, 10):
24    num = i * 100
25    result_df = result_df.append(copy_twice(num))
26    result_df.to_csv(f"/content/gdrive/MyDrive/image_tests/timing_results_10_{num}.csv")

Compare transfer performance

The results confirm what was seen with the initial sample of 10 files. It can be seen that as the number of files increases, the difference between using a zip file and transferring files individually increases. The Times Faster columns is the sum of the time to copy the zip file and the unzip time divided by the time to copy the files individually. This factor increase as the number of files grows and it can be seen that when there are over 4,000 files the use of a zip file to transfer the files is over 300 times faster.

Table: Transfer times of increasing number of files using zip compared to transferring individually

Number of files	Directory size (KB)	Copy zip (sec)	Unzip (sec)	Copy files (sec)	Times faster
10	200	0.7	0.0	4.0	5.4
20	398	1.1	0.0	7.0	6.4
30	623	0.6	0.0	8.3	13.0
40	850	0.9	0.0	11.6	13.1
50	1097	1.1	0.0	13.5	12.2
60	1320	1.2	0.0	16.5	13.1
70	1571	0.7	0.0	18.0	24.7
80	1802	1.3	0.0	19.7	15.3
90	2043	1.2	0.0	23.1	19.2
100	2209	1.2	0.0	24.9	20.3
200	4478	1.4	0.1	64.5	44.8
300	6664	1.3	0.1	96.8	68.5
400	8811	1.5	0.1	133.1	81.9
500	10929	1.4	0.1	160.4	102.8
600	13254	1.2	0.2	196.2	139.9
700	15520	1.5	0.2	225.8	136.2
800	17719	1.5	0.2	256.2	151.8
900	20147	1.1	0.2	290.5	219.9
1000	22418	1.5	0.3	326.2	176.7
2000	45587	2.3	0.6	678.1	235.8
3000	68260	2.5	0.9	1003.0	296.4
4000	91057	3.1	1.2	1330.0	312.9
5000	114013	3.1	1.4	1654.1	371.5

The following charts shows that the transfer of files individually increases in a linear fashion as the number of files increases. Although, there is an increase in the transfer times of the zip files, it seems almost flat in comparison to the increase in individual file transfer.

 1def plot_copy_time_vs_num_chart(df, title):
 2    fig, ax = plt.subplots(figsize=(8, 5), facecolor=plt.cm.Blues(0.2))
 3    fig.suptitle(title, fontsize="x-large", fontweight="bold")
 4    ax.set_facecolor(plt.cm.Blues(0.2))
 5    ax.plot(df.index, df.copy_files_time / 1000, label="Copy files")
 6    ax.plot(df.index, df.copy_unzip_time / 1000, label="Copy zip file and unzip")
 7    ax.legend(facecolor=plt.cm.Blues(0.1))
 8    ax.set_xlabel("Number of Files", fontsize="large")
 9    ax.set_ylabel("Time to transfer (seconds)", fontsize=14)
10    ax.spines["right"].set_visible(False)
11    ax.spines["top"].set_visible(False)
12    return fig
13
14
15fig = plot_copy_time_vs_num_chart(
16    res_df.loc[range(10, 101, 10), :],
17    "Time taken to transfer files vs number of files\n(10 to 100 files)",
18)
19
20fig = plot_copy_time_vs_num_chart(
21    res_df,
22    "Time taken to transfer files vs number of files\n(10 to 5000 files)",
23)

time to copy 10 to 100 files

Time taken to transfer files as zip files or individually with file groups of 10 to 100

time to copy 10 to 5000 files

Time taken to transfer files as zip files or individually with file groups of 10 to 5000

There is such a difference in the transfer times from transferring files individually to transferring the same files in a zip file, that it looks like the zip file transfer is constant. This is not the case as can be shown by just charting the zip file transfer on its own.

time to copy zip files

Looking at just zip file transfer shows an increase in time transfer as the number of files increases

The use of zip file to transfer files is faster by an increasing factor as the number of files increases.

zip-transfer-times-faster

The factor by which zip file transfer is faster increases as the number of files increases

Conclusion

The purpose of compressing these images is to save time when transferring the files not to save storage space. This article showed that storing files in zip file archives allows for much faster file transfer. It is interesting to see that this is true even for a small number of files, such as 10, where the use of a zip file is over 5 times faster that transferring the files individually. The benefits of using a zip file increase significantly as the number of files grows. Transfer times can be reduced by a factor of over 370 when there are 5,000 files to transfer with the use of a zip file. In the case of 5,000 image files, it took 27 minutes and 34 seconds to copy the 5,000 images directly, but only 3 seconds to copy the zip file and 1.4 seconds to unzip it.

Three Dimensional Vectors and Dot Product

Eric Callanan — Sunday, 10 Jan 2021

The world around is in three dimensions, yet it is not as easy to represent on a 2D surface such as a page or screen. This article, inspired by chapter 3 of Math for Programmers book, looks at presenting 3D vectors with Matplotlib. It also shows how to calculate the length of 3D-vectors and explores the dot product of 3D vectors.

These are my notes on the book Math for Programmers by Paul Orland.

2D vectors and vector arithmetic
Two Dimensional shapes
Three Dimensional Vectors and Dot Product

3D vectors

A 2D vector can be represented as two Cartesian coordinates x and y. These represent the distance from the origin in the horizontal and vertical axes. If we imagine this 2D plane lying flat on the desk and imagine another axis rising above or below the desk, then the shapes could take on a third dimension. This axis is given the label z-axis as a measure of depth. Three Cartesian coordinates (x, y and z) are used to represent a vector in 3D-space.

 1fig = plt.figure(facecolor=plt.cm.Blues(0.2))
 2ax = fig.gca(projection="3d")
 3fig.suptitle(
 4    f"3D-vector (2,3,4)",
 5    fontsize=18,
 6    fontweight="bold",
 7)
 8
 9xs, ys, zs = zip((2, 3, 4), (0, 0, 0))
10ax.plot(xs, ys, zs, color="blue")
11ax.set_facecolor(plt.cm.Blues(0.2))
12ax.set_xlim(0, 4)
13ax.set_ylim(0, 4)
14ax.set_zlim(0, 4)
15ax.set_xlabel("x-axis")
16ax.set_ylabel("y-axis")
17ax.set_zlabel("z-axis")
18
19plt.show()

3D vector with Matplotlib

3D vector with Matplotlib

Viewing 3D vectors

It can be difficult to visualize 3D vectors and objects as these are still being presented on 2D screens and surfaces. This is the same vector plus a vector lying flat on the XY plane shown from different Viewing angles and viewing elevations. Both of these vectors have (3,4) for their (x,y) coordinates, but the blue one has a z-coordinate of 4, while the red one is flat on the XY plane (as specified with a z-coordinate of zero).

 1fig, axs = plt.subplots(
 2    1, 4, figsize=(12, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
 3)
 4fig.suptitle(
 5    f"Different 3D viewing angles",
 6    fontsize=22,
 7    fontweight="bold",
 8)
 9
10for i, ax in enumerate(axs.flatten()):
11    ax.set_facecolor(plt.cm.Blues(0.2))
12    xs, ys, zs = zip((0, 0, 0), (3, 4, 4))
13    ax.plot(xs, ys, zs, color="blue")
14    xs, ys, zs = zip((0, 0, 0), (3, 4, 0))
15    ax.plot(xs, ys, zs, color="red")
16    ax.set_xlim(0, 4)
17    ax.set_ylim(0, 4)
18    ax.set_zlim(0, 4)
19    ax.set_xlabel("x-axis")
20    ax.set_ylabel("y-axis")
21    ax.set_zlabel("z-axis")
22
23    view_angle = i * (30)
24    ax.view_init(elev=10.0, azim=view_angle)
25    ax.set_title(f"View Angle = {view_angle}", fontsize=14, pad=5)
26
27fig.tight_layout(pad=2.0)
28plt.show()
29
30publish_png_image(fig, "3d-vector-different-viewing-angles.png")

3D vector from different viewing angles

3D vector from different viewing angles

 1fig, axs = plt.subplots(
 2    1, 4, figsize=(12, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
 3)
 4fig.suptitle(
 5    f"Increasing viewing elevation",
 6    fontsize=22,
 7    fontweight="bold",
 8)
 9
10for i, ax in enumerate(axs.flatten()):
11    ax.set_facecolor(plt.cm.Blues(0.2))
12    xs, ys, zs = zip((0, 0, 0), (3, 4, 4))
13    ax.plot(xs, ys, zs, color="blue")
14    xs, ys, zs = zip((0, 0, 0), (3, 4, 0))
15    ax.plot(xs, ys, zs, color="red")
16    ax.set_xlim(0, 4)
17    ax.set_ylim(0, 4)
18    ax.set_zlim(0, 4)
19    ax.set_xlabel("x-axis")
20    ax.set_ylabel("y-axis")
21    ax.set_zlabel("z-axis")
22
23    elevation = i * 25.0
24    ax.view_init(elev=elevation)
25    ax.set_title(f"Elevation = {elevation}", fontsize=14, pad=5)
26
27fig.tight_layout(pad=2.0)
28plt.show()

3D vector from different viewing elevation

3D vector from different viewing elevation

Length of 3D vector

The Pythagorean theorem is used to calculate the length of a vector in 2D-space. This can be extended to create a formula to calculate the length of a three-dimensional vector. The central fact of 3D-space is that the three planes X, Y & Z are perpendicular to each other, so any line on one plane is at right-angles to a line on another plane.

Think of the XY plane as lying flat on a table and the Z dimension as rising vertically up. Any 3D-vector (x,y,z) will have a corresponding 2D vector (x,y) on the XY plane vertically below it. The length of (0,0) to (x,y) can be calculated using Pythagorean theorem. This line is one of The edges of a right-angled triangle with z being the second edge - allowing the calculation of the length of the 3D-vector (x,y,z).

$$ hypotenuse \ of \ xy = \sqrt{(x^2 + y^2)} $$

This hypotenuse on the XY plane is perpendicular to the Z plane so

$$ vector \ length = \sqrt{((hypotenuse \ of \ xy)^2 + z^2)} $$

$$ vector \ length = \sqrt{ \left( \sqrt{(x^2 + y^2)} \right) ^2 + z^2} $$

The inner square root and squared cancel each other out

$$ \left( \sqrt{ (x^2 + y^2)} \right) ^2 = (x^2 + y^2) $$

$$ vector \ length = \sqrt{(x^2 + y^2 + z^2)} $$

 1fig, axs = plt.subplots(
 2    1, 3, figsize=(10, 4), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
 3)
 4fig.suptitle(
 5    f"Use of Pythagorean theorem to calculate length 3D-vector",
 6    fontsize=18,
 7    fontweight="bold",
 8)
 9
10for i, ax in enumerate(axs.flatten()):
11    ax.set_facecolor(plt.cm.Blues(0.2))
12    xs, ys, zs = zip((0, 0, 0), (3, 4, 4))
13    ax.plot(xs, ys, zs, color="blue")
14    ax.set_xlim(0, 4)
15    ax.set_ylim(0, 4)
16    ax.set_zlim(0, 4)
17    ax.set_xlabel("x-axis")
18    ax.set_ylabel("y-axis")
19    ax.set_zlabel("z-axis")
20
21
22axs[1].plot((0, 3), (0, 0), (0, 0), color="teal")
23axs[1].plot((3, 3), (0, 4), (0, 0), color="teal")
24axs[1].plot((0, 3), (0, 4), (0, 0), color="teal")
25axs[1].text(1.5, 0.2, 0, "x", "x", color="teal")
26axs[1].text(2.5, 1.6, 0, "y", "y", color="teal")
27axs[1].text(0.7, 1.6, 0, r"$\sqrt{(x^2 + y^2)}$", (3, 4, 0), color="teal")
28
29
30axs[2].plot((0, 3), (0, 4), (0, 0), color="red")
31axs[2].plot((3, 3), (4, 4), (0, 4), color="red")
32axs[2].text(1.7, 0.5, 0, r"$\sqrt{(x^2 + y^2)}$", (3, 4, 0), color="red")
33axs[2].text(3.2, 4.0, 2.0, "z", "z", color="red")
34axs[2].text(1, 1, 2, r"$\sqrt{(x^2 + y^2 + z^2)}$", (3, 4, 4), color="blue")
35
36fig.tight_layout(pad=2.0)
37plt.show()

Use of Pythagorean theorem to calculate length 3D-vector

Use of Pythagorean theorem to calculate length 3D-vector

The length of a 3D-vector can be calculated in Python in a similar manner to the length of a 2D-vector. A list comprehension can be used to create a generic vector_length function that will work for multiple dimensions. This function can be used to calculate the length of vectors with dimensions greater than 3.

 1import math as m
 2
 3def vector_2d_length(v):
 4    return m.sqrt(v[0] ** 2 + v[1] ** 2)
 5
 6def vector_3d_length(v):
 7    return m.sqrt(v[0] ** 2 + v[1] ** 2 + v[2] ** 2)
 8
 9def vector_length(v):
10    return m.sqrt(sum((c ** 2) for c in v))
11
12vector_2d_length((3,4))
13"""
145.0
15"""
16
17vector_3d_length((3,4,4))
18"""
196.4031242374328485
20"""
21
22vector_length((3,4))
23"""
245.0
25"""
26
27vector_length((3,4,4))
28"""
296.4031242374328485
30"""
31
32vector_length((3,4,4,4))
33"""
347.54983443527075
35"""
36
37vector_length((3,4,5,6,7,8))
38"""
3914.106735979665885
40"""

Dot product

There are two important mechanisms of multiplying one 3D vector by another; the dot product and the cross product.

The dot product returns a scaler and works on 2D, 3D or higher number of dimensions. The dot product is the sum of the products of the corresponding entries of the two sequences of numbers. The dot product of 2 vectors is a measure of how aligned the vectors are. When vectors are pointing in the same or similar direction, the dot product is positive. When vectors are pointing in opposite direction, the dot product is negative. When two vectors are perpendicular to each other then their dot product is zero regardless of the magnitude of the vectors.

It is very straight forward to calculate the dot product in python. Again, a list comprehension is used to create a generic function that works for vectors with multiple dimensions.

 1def dot(v1, v2):
 2    return sum([c1 * c2 for c1, c2 in zip(v1, v2)])
 3
 4
 5dot((3, 4), (4, 4))
 6dot((4, 4), (-2, 2))
 7dot((-3, -4), (-4, -3))
 8dot((3, 4), (-3, -4))
 9"""
1028
110
1224
13-25
14"""
15
16
17dot((3, 4, 5), (4, 4, 4))
18dot((3, 0, 9), (-6, 4, 2))
19dot((3, -4, -5), (-4, 4, 4))
20"""
2148
220
23-48
24"""

 1def plot_vector(ax, v, c):
 2    ax.annotate(
 3        "",
 4        (v[0], v[1]),
 5        xytext=(0, 0),
 6        arrowprops=dict(ec=c, fc=c, alpha=0.6),
 7    )
 8
 9
10def update_axis(ax, v1, v2):
11    plot_vector(ax, v1, "blue")
12    plot_vector(ax, v2, "red")
13    ax.set_title(f"dot product = {dot(v1, v2)}", fontsize=14)
14
15
16fig, axs = plt.subplots(2, 3, figsize=(12, 8), facecolor=plt.cm.Blues(0.2))
17fig.suptitle(
18    f"dot product",
19    fontsize=22,
20    fontweight="bold",
21)
22
23for i, ax in enumerate(axs.flatten()):
24    ax.set_facecolor(plt.cm.Blues(0.2))
25    ax.spines["right"].set_visible(False)
26    ax.spines["top"].set_visible(False)
27    ax.spines["left"].set_position("zero")
28    ax.spines["bottom"].set_position("zero")
29    ax.set_xlim(-5, 5)
30    ax.set_ylim(-5, 5)
31    ax.set_aspect("equal")
32    ax.grid(True)
33    ax.set_xticks([x for x in range(-4, 5, 1)])
34    ax.set_yticks([x for x in range(-4, 5, 1)])
35
36update_axis(axs[0][0], (3, 4), (4, 4))
37update_axis(axs[0][1], (3, 4), (2, 2))
38update_axis(axs[0][2], (4, 4), (-2, 2))
39update_axis(axs[1][0], (-3, -4), (-4, -4))
40update_axis(axs[1][1], (3, 4), (-4, -3))
41update_axis(axs[1][2], (3, 4), (-3, -4))
42
43fig.tight_layout(pad=2.0)
44plt.show()

dot product of 2D vectors

dot product of 2D vectors

 1def plot_axes(ax):
 2    ax.plot((-4, 4), (0, 0), (0, 0), color="grey")
 3    ax.plot((0, 0), (-4, 4), (0, 0), color="grey")
 4    ax.plot((0, 0), (0, 0), (-4, 4), color="grey")
 5
 6
 7def plot_vector(ax, v, c):
 8    ax.plot((0, v[0]), (0, v[1]), (0, v[2]), color=c)
 9
10
11def update_axis(ax, v1, v2):
12    plot_axes(ax)
13    plot_vector(ax, v1, "blue")
14    plot_vector(ax, v2, "red")
15    ax.set_title(f"dot product = {dot(v1, v2)}", fontsize=14)
16
17
18fig, axs = plt.subplots(
19    2, 3, figsize=(10, 8), facecolor=plt.cm.Blues(0.2), subplot_kw={"projection": "3d"}
20)
21fig.suptitle(
22    f"dot product on 3D-vector",
23    fontsize=18,
24    fontweight="bold",
25)
26
27for i, ax in enumerate(axs.flatten()):
28    ax.set_facecolor(plt.cm.Blues(0.1))
29    ax.set_xlim(-7, 5)
30    ax.set_ylim(-5, 5)
31    ax.set_zlim(-6, 10)
32    ax.set_xlabel("x-axis")
33    ax.set_ylabel("y-axis")
34    ax.set_zlabel("z-axis")
35
36update_axis(axs[0][0], (3, 4, 5), (4, 4, 4))
37update_axis(axs[0][1], (3, 0, 9), (-6, 4, 2))
38update_axis(axs[0][2], (3, -4, -5), (-4, 4, 4))
39
40update_axis(axs[1][0], (3, 4, 5), (4, 4, 4))
41axs[1][0].view_init(azim=-80)
42update_axis(axs[1][1], (3, 0, 9), (-6, 4, 2))
43axs[1][1].view_init(azim=-80)
44update_axis(axs[1][2], (3, -4, -5), (-4, 4, 4))
45axs[1][2].view_init(azim=-80)
46
47fig.tight_layout(pad=2.0)
48plt.show()

dot product of 3D vectors

dot product of 3D vectors

Use of dot product

Two specific uses of dot product are:

Angle of a vector
Length of a vector

Angle

The dot product of two vectors is easy to calculate and can tell if the angle between two vectors is less than, greater than or equal to 90^o. More specifically, the dot product of two vectors is equal to the product of their lengths by the cosine of the angle between the vectors.

$$ v1 \cdot v2 = |v1| \cdot |v2| \cdot cos θ $$

$$ v1 \cdot v2 = |v1| \ {\times} \ |v2| \ {\times} \ cos θ $$

It is more common to want to find the angle from given coordinates, so the formula can be rearranged.

$$ cos θ = \frac {v1 \cdot v2}{|v1| \ {\times} \ |v2|} $$

$$ θ = arccos \left( \frac {v1 \cdot v2}{|v1| \ {\times} \ |v2|} \right) $$

Example of the angle between (3,4,5) and (4,4,4)

$$ θ = arccos \left( \frac {(3,4,5) \cdot (4,4,4)}{|(3,4,5)| \ {\times} \ |(4,4,4)|} \right) $$

$$ θ = arccos \left( \frac {48}{7.07 \ {\times} \ 6.928} \right) = arccos \left( \frac {48}{48.99} \right) = arccos \left( 0.979 \right) = 0.201 Radians = 11.5^{\circ} $$

This can be written in the following Python function.

 1import math as m
 2
 3
 4def angle_between_vectors(v1, v2):
 5    return m.acos(dot(v1, v2) / (vector_length(v1) * vector_length(v2))) * (180 / m.pi)
 6
 7
 8angle_between_vectors((3, 4, 5), (4, 4, 4))
 9"""
1011.536959032815467
11"""
12
13angle_between_vectors((3, 0, 9), (-6, 4, 2))
14"""
1590.0
16"""
17
18angle_between_vectors((3, -4, -5), (-4, 4, 4))
19"""
20168.46304096718453
21"""

Length

The length of a vector is the square root of the dot product of the vector by itself

$$ vector \ length = |v| = \sqrt{v \cdot v} $$

Vector length of (3,4), (3,4,4) and (3,4,4,4)

$$ vector \ length = \sqrt{v \cdot v} = \sqrt{(3,4) \cdot (3,4))} = \sqrt{25} = 5 $$

$$ vector \ length = \sqrt{v \cdot v} = \sqrt{(3,4,4) \cdot (3,4,4))} = \sqrt{41} = 6.40 $$

$$ vector \ length = \sqrt{v \cdot v} = \sqrt{(3,4,4,4) \cdot (3,4,4,4))} = \sqrt{57} = 7.55 $$

 1import math as m
 2
 3
 4def vector_len_dot(v):
 5    return m.sqrt(dot(v, v))
 6
 7
 8vector_len_dot((3,4))
 9"""
105.0
11"""
12
13vector_len_dot((3,4,5))
14"""
157.0710678118654755
16"""
17
18vector_len_dot((4,4,4))
19"""
206.928203230275509
21"""
22
23vector_len_dot((3,4,4,4))
24"""
257.54983443527075
26"""

Conclusion

This article showed how to display 3D-vectors from different angles and perspectives. Even with this, it can be difficult to visualize 3D vectors and objects as these are still being presented on 2D screens. Two mechanisms were shown of calculating the length of a 3D vector. The dot product was examined and some of its uses such as determining if vectors are perpendicular as well as calculating the exact angle between vectors or calculating the length of a vector using the dot product. Next article will look at the cross product as well as shading and plotting 3D objects.

matplotlib a comprehensive library for creating static, animated, and interactive visualizations in Python.

Math for Programmers by Paul Orland - a great book to brush up on your math skills.

Two Dimensional shapes

Eric Callanan — Sunday, 03 Jan 2021

A vector has magnitude and direction. A single point in 2D space can be considered a vector from the origin (0,0) to that point. Vectors can be described anywhere in 2D space using a point for the start and a point for the end. Multiple vectors can be connected together to create 2D shapes. This article will discuss common 2D shapes as well as rotations and translations.

These are my notes on the book Math for Programmers by Paul Orland.

2D vectors and vector arithmetic
Two Dimensional shapes

Basic shapes

Simple shapes can be drawn by combining many vectors together. A square can be drawn using four points for the corners or four vectors for the sides. For vectors to draw shapes, one vector has to start where another vectors ends. previously, vectors were described as a single point where the origin (0,0) is the starting point. Vectors do not have to start at the origin and can be specified with a start point and end point

 1def plot_axes(ax, l_left, u_right):
 2    ax.set_facecolor(plt.cm.Blues(0.1))
 3
 4    # Hide the right and top spines
 5    ax.spines["right"].set_visible(False)
 6    ax.spines["top"].set_visible(False)
 7
 8    # Set the x and y axis to go through origin (0,0)
 9    ax.spines["left"].set_position("zero")
10    ax.spines["bottom"].set_position("zero")
11
12    ax.set_xlim(l_left[0], u_right[0])
13    ax.set_ylim(l_left[1], u_right[1])
14
15    ax.grid(True)
16
17    ax.tick_params(labelsize="medium")

 1def plot_segment(ax, v1, v2):
 2    ax.plot([v1[0], v2[0]], [v1[1], v2[1]], color="green")
 3
 4
 5fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(9, 6.5), facecolor=plt.cm.Blues(0.2))
 6fig.suptitle(
 7    f"Basic Shapes",
 8    fontsize=22,
 9    fontweight="bold",
10)
11
12plot_axes(ax1, (-1, -1), (6, 6))
13ax1.set_title(f"Square", fontsize=14, pad=20)
14plot_segment(ax1, (1, 2), (4, 2))
15plot_segment(ax1, (1, 2), (1, 5))
16plot_segment(ax1, (4, 2), (4, 5))
17plot_segment(ax1, (1, 5), (4, 5))
18
19plot_axes(ax2, (-1, -1), (6, 6))
20ax2.set_title(f"Triangle", fontsize=14, pad=20)
21plot_segment(ax2, (1, 1), (4, 2))
22plot_segment(ax2, (1, 1), (2, 5))
23plot_segment(ax2, (4, 2), (2, 5))
24
25fig.tight_layout(pad=3.0)
26plt.show()
27
28publish_png_image(fig, "draw-square-triangle.png")

Draw a square and triangle

Draw a square and triangle

Polygon

A Polygon is a two-dimensional figure that is described by a finite number of straight line segments connected to form a closed shape. In the code above, there is a lot of repetition to create each segment of the shape. The following defines a mechanism of plotting a polygon based on a series of points. The last point is joined to the first point to close the shape.

1def plot_polygon(ax, *vertices):
2    # join the last vertex to the first to close the shape
3    vertices = vertices + (vertices[0],)
4    ax.plot([x[0] for x in vertices], [x[1] for x in vertices], color="red")

The same two shapes of square and triangle can be plotted using this function.

 1fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(9, 6.5), facecolor=plt.cm.Blues(0.2))
 2fig.suptitle(
 3    f"Basic Shapes plotted as polygons",
 4    fontsize=22,
 5    fontweight="bold",
 6)
 7
 8plot_axes(ax1, (-1, -1), (6, 6))
 9ax1.set_title(f"Square", fontsize=14, pad=20)
10square = [(1, 2), (4, 2), (4, 5), (1, 5)]
11plot_polygon(ax1, *square)
12
13plot_axes(ax2, (-1, -1), (6, 6))
14ax2.set_title(f"Triangle", fontsize=14, pad=20)
15triangle = [(1, 1), (4, 2), (2, 5)]
16plot_polygon(ax2, *triangle)
17
18fig.tight_layout(pad=3.0)
19plt.show()

Draw a square and triangle with polygons

Draw a square and triangle with polygons

Plot complex polygons

A more complicated 2D shape can be plotted just by specifying a series of points in 2D-space. The following shows the code to plot an elephant.

 1fig, ax1 = plt.subplots(1, 1, figsize=(10, 9), facecolor=plt.cm.Blues(0.2))
 2fig.suptitle(
 3    f"More complex shapes",
 4    fontsize=22,
 5    fontweight="bold",
 6)
 7
 8plot_axes(ax1, (-15, -1), (16, 23))
 9ax1.set_title(f"Elephant", fontsize=14, pad=20)
10elephant = [
11    (2, 2),
12    (6, 2),
13    (6, 9),
14    (8, 12),
15    (9, 11),
16    (10, 7),
17    (12, 5),
18    (14, 6),
19    (15, 8),
20    (13, 8),
21    (12, 7.5),
22    (11, 9),
23    (11, 16),
24    (9, 20),
25    (4, 20),
26    (-6, 21),
27    (-10, 20),
28    (-13, 17),
29    (-13, 6),
30    (-12, 12),
31    (-11.5, 9),
32    (-12, 5),
33    (-11, 2),
34    (-8, 2),
35    (-9, 5),
36    (-8, 7),
37    (-8, 4),
38    (-7, 2),
39    (-4, 2),
40    (-5, 5),
41    (-4, 8),
42    (2, 8),
43]
44plot_polygon(ax1, *elephant)
45plt.show()

Plot an elephant polygons

Plot an elephant polygons

Trigonometry and "sohcahtoa"

Trigonometry is the study of the relationships between side lengths and angles of triangles. In the previous article all vectors were described using x and y coordinates - this is the called the Cartesian coordinate system. A vector can also be described using the Polar coordinate, which consists of specifying the length (or magnitude) of the vector and the angle counter-clockwise from the horizontal.

To convert between Cartesian and Polar coordinates, we need to use the trigonometric rules from right-angled triangles. The rules can be remembered with use of the mnemonic SOHCAHTOA. The sine of an angle is equal to the length of the opposite side divided by the length of the hypotenuse (SOH); cosine equals adjacent over hypotenuse and tangent is equal to opposite over adjacent.

$$ sin θ = \frac {opposite} {hypotenuse} $$

$$ cos θ = \frac {adjacent} {hypotenuse} $$

$$ tan θ = \frac {opposite} {adjacent} $$

 1(x, y) = (3, 4)
 2fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(9, 6.5), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(
 4    f"Trigonometry on a right-angled triangle",
 5    fontsize=22,
 6    fontweight="bold",
 7)
 8
 9plot_axes(ax1, (-1, -1), (4.5, 5.5))
10ax1.set_title(f"Vector at point ({x}, {y})", fontsize=14, pad=20)
11ax1.plot(x, y, color="black", marker="o", markersize=15)
12ax1.annotate(
13    "",
14    (x, y),
15    xytext=(0, 0),
16    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
17)
18
19plot_axes(ax2, (-1, -1), (4.5, 5.5))
20ax2.set_title(f"Right-angled triangle for vector ({x}, {y})", fontsize=14, pad=20)
21ax2.plot(x, y, color="black", marker="o", markersize=15)
22ax2.annotate(
23    "", (x, y), xytext=(0, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
24)
25ax2.annotate(
26    "", (x, 0), xytext=(0, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
27)
28ax2.annotate(
29    "", (x, y), xytext=(x, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
30)
31ax2.annotate(
32    "",
33    (3.0, 0.5),
34    xytext=(2.5, 0),
35    arrowprops=dict(
36        arrowstyle="-",
37        color="blue",
38        connectionstyle="angle,angleA=-90,angleB=180,rad=0",
39    ),
40)
41
42
43ax2.tick_params(labelbottom=False)
44ax2.tick_params(labelleft=False)
45
46ax2.text(1.5, -0.2, "a", size=20, color="blue", ha="left", va="top")
47ax2.text(3.2, 2, "o", size=20, color="blue", ha="left", va="center")
48ax2.text(
49    1.5,
50    2.1,
51    r"h",
52    size=20,
53    color="blue",
54    ha="center",
55    va="bottom",
56    rotation=50,
57)
58ax2.annotate(
59    "",
60    (0.56, 0.8),
61    xytext=(0.85, 0),
62    arrowprops=dict(arrowstyle="-", color="blue", connectionstyle="arc3,rad=0.6"),
63)
64ax2.text(0.5, 0.1, "θ", size=20, color="blue", ha="left", va="bottom")
65
66fig.tight_layout(pad=3.0)
67plt.show()

Relationship between angles and line lengths in right-angled triangle

Relationship between angles and line lengths in a right-angled triangle

Conversion between Degrees and Radians

Angles in mathematics are often measured in Radian rather than in degrees. One radian being the angle formed with an arc the length of one radius. This means that 180 degrees is equal to π (pi) radians. Angles are generally given in degrees such as 15^o, 45^o, 90^o, etc.

$$ Angle \ in \ Radians = Angle \ in \ Degrees \times \frac {\pi} {180} $$

$$ Angle \ in \ Degrees = Angle \ in \ Radians \times \frac {180} {\pi} $$

sin, cos and tan are functions in the math module in python that take the angle in Radians. The inverse of these functions are asin, acos and atan also in the math module. These can be used to determine the angles in the right-angled triangle above with vector (3,4).

$$ sin θ = \frac {opposite} {hypotenuse} = \frac {4} {5} = 0.8 $$

sin θ = 0.8, using asin yields an angle of 0.927 radians.

$$ Angle \ in \ Degrees = Angle \ in \ Radians \times \frac {180} {\pi} = 0.927 \times \frac {180} {3.14} = 53.13^{\circ} $$

Validation of SOHCAHTOA
$$θ = 53.13^{\circ} = 0.927 \ Radians$$	$$sin \ 0.927 = 0.7998$$	$$\frac {opposite} {hypotenuse} = \frac {4} {5} = 0.8$$
$$θ = 53.13^{\circ} = 0.927 \ Radians$$	$$cos \ 0.927 = 0.600$$	$$\frac {adjacent} {hypotenuse} = \frac {3} {5} = 0.6$$
$$θ = 53.13^{\circ} = 0.927 \ Radians$$	$$tan \ 0.927 = 1.332$$	$$\frac {opposite} {adjacent} = \frac {4} {3} = 1.333$$

 1# vector of (3,4)
 2import math as m
 3angle = 4/5
 4"""
 50.8
 6"""
 7
 8m.asin(angle)
 9"""
100.9272952180016123
11"""
12
13m.asin(angle) * (180/m.pi)
14"""
1553.13010235415599
16"""
17
18m.sin(0.927), m.cos(0.927), m.tan(0.927)
19"""
200.7998228343401385 0.6002361482517589 1.3325136059694065
21"""
22
234/5, 3/5, 4/3
24"""
250.8 0.6 1.3333333333333333
26"""

Conversion between Cartesian and Polar coordinates

Cartesian coordinates specify a vector with an x-coordinate and a y-coordinate stating the horizontal and vertical distance from the origin. Polar coordinates specify the length (magnitude) of the vector and the angle from the horizontal axis.

There is a problem with relying on results of the inverse functions asin, acos and atan in that different coordinates may yield the same angle. Consider the vectors (3,4) and (-3,4) both have opposite length of 4 and hypotenuse length of 5. Multiple angles have the same sine, so the inverse sine gives one of these angles. It is possible to use inverse of all three functions to determine the exact angle, but the math module in python contains the function atan2 to get the correct angle based on vector coordinates.

 1help(m.atan2)
 2
 3"""
 4Help on built-in function atan2 in module math:
 5
 6atan2(y, x, /)
 7    Return the arc tangent (measured in radians) of y/x.
 8
 9    Unlike atan(y/x), the signs of both x and y are considered.
10"""

The following function makes use of math.atan2 to convert from Cartesian to Polar coordinates.

 1import math as m
 2
 3
 4def to_polar(v):
 5    angle = m.atan2(v[1], v[0])
 6    length = m.sqrt(v[0] ** 2 + v[1] ** 2)
 7    return (length, angle)
 8
 9
10print(to_polar((3, 4)))
11print(to_polar((-3, 4)))
12"""
13(5.0, 0.9272952180016122)
14(5.0, 2.214297435588181)
15"""

 1def to_cartesian(polar_v):
 2    return (polar_v[0] * m.cos(polar_v[1]), polar_v[0] * m.sin(polar_v[1]))
 3
 4
 5print(to_cartesian((5, 0.927)))
 6print(to_cartesian((5, 2.215)))
 7"""
 8(3.0011807412587945, 3.9991141717006924)
 9(-3.0028095170209874, 3.9978913197444705)
10"""

 1(x, y) = (3, 4)
 2fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(9, 6), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(
 4    f"Cartesian and Polar coordinates",
 5    fontsize=22,
 6    fontweight="bold",
 7)
 8
 9plot_axes(ax1, (-1, -1), (4.5, 5.5))
10ax1.set_title(f"Cartesian coordinates", fontsize=14, pad=20)
11ax1.annotate(
12    "",
13    (x, y),
14    xytext=(0, 0),
15    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
16)
17ax1.text(3.1, 4.3, "(3, 4)", size=20, color="blue", ha="center", va="center")
18
19
20plot_axes(ax2, (-1, -1), (4.5, 5.5))
21ax2.set_title(f"Polar coordinates", fontsize=14, pad=20)
22ax2.annotate(
23    "",
24    (x, y),
25    xytext=(0, 0),
26    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
27)
28ax2.text(
29    1.4,
30    2.1,
31    r"5",
32    size=20,
33    color="blue",
34    ha="center",
35    va="bottom",
36    rotation=50,
37)
38ax2.annotate(
39    "",
40    (1.0, 1.3),
41    xytext=(1.6, 0),
42    arrowprops=dict(arrowstyle="<->", color="blue", connectionstyle="arc3,rad=0.4"),
43)
44ax2.text(0.4, 0.1, "0.927", size=20, color="blue", ha="left", va="bottom")
45
46fig.tight_layout(pad=1.0)
47plt.show()

Same vector described with Cartesian and Polar coordinates

Same vector described with Cartesian and Polar coordinates

Regular Polygons

A Regular polygon is a polygon where all angles are equal in measure and all sides have the same length. Polar coordinates can be used to quickly create a series of regular polygons, but it ie easier in Matplotlib to plot with Cartesian coordinates. These are regular polygons that fit inside a circle of radius 1.5.

 1def regular_polygon(n):
 2    return [to_cartesian((1.5, 2 * m.pi * x / n)) for x in range(0, n)]
 3
 4
 5fig, axs = plt.subplots(2, 4, figsize=(9, 6), facecolor=plt.cm.Blues(0.2))
 6fig.suptitle(
 7    f"Regular polygons",
 8    fontsize=22,
 9    fontweight="bold",
10)
11
12for i, ax in enumerate(axs.flatten()):
13    plot_axes(ax, (-2, -2), (2, 2))
14    ax.tick_params(labelbottom=False)
15    ax.tick_params(labelleft=False)
16    n = i + 3
17    ax.set_title(f"{n} sides", fontsize=14, pad=5)
18    p = regular_polygon(n)
19    plot_polygon(ax, *p)
20
21fig.tight_layout(pad=2.0)
22plt.show()

Regular polygons created with Polar coordinates and plotted with Cartesian coordinates

Regular polygons created with Polar coordinates and plotted with Cartesian coordinates

Rotation, Scaling and translation

Now that we have basic shapes, regular polygons and complex shapes, it is possible to plot the shapes, rotate the shapes, move or translate the shapes and/or change the size of (scale) the shapes. Translation moves every point of a shape by the same distance in a given direction. This can be achieved by adding the same (x,y) values to all coordinates of the shape. Rotate moves the shape around the origin by a specified angle, this is more easily done using polar coordinates. Scale is used to multiply each vector in a shape by a specified factor, a factor larger than 1 will increase the size of a shape and a factor less than 1 will decrease the size of the shape. The following functions are used to scale, transform and rotate and the plot_polygon function is updated to take a color as a parameter.

 1def plot_polygon(ax, color, *vertices):
 2    # join the last vertex to the first to close the shape
 3    vertices = vertices + (vertices[0],)
 4    ax.plot([x[0] for x in vertices], [x[1] for x in vertices], color=color)
 5
 6
 7def scale(n, v):
 8    return (n * v[0], n * v[1])
 9
10
11def translate(tran_v, vectors):
12    return [(tran_v[0] + v[0], tran_v[1] + v[1]) for v in vectors]
13
14
15def rotate(angle, vectors):
16    polar_vectors = [to_polar(v) for v in vectors]
17    return [to_cartesian((l, a + angle)) for l, a in polar_vectors]

 1elephant = [
 2    (2, 2),
 3    (6, 2),
 4    (6, 9),
 5    (8, 12),
 6    (9, 11),
 7    (10, 7),
 8    (12, 5),
 9    (14, 6),
10    (15, 8),
11    (13, 8),
12    (12, 7.5),
13    (11, 9),
14    (11, 16),
15    (9, 20),
16    (4, 20),
17    (-6, 21),
18    (-10, 20),
19    (-13, 17),
20    (-13, 6),
21    (-12, 12),
22    (-11.5, 9),
23    (-12, 5),
24    (-11, 2),
25    (-8, 2),
26    (-9, 5),
27    (-8, 7),
28    (-8, 4),
29    (-7, 2),
30    (-4, 2),
31    (-5, 5),
32    (-4, 8),
33    (2, 8),
34]
35
36fig, axs = plt.subplots(2, 2, figsize=(12, 8), facecolor=plt.cm.Blues(0.2))
37fig.suptitle(
38    f"Translate, Scale and Rotate shapes",
39    fontsize=22,
40    fontweight="bold",
41)
42
43titles = ["Elephant", "Scale", "Translate", "Rotate"]
44for i, ax in enumerate(axs.flatten()):
45    plot_axes(ax, (-20, -1), (30, 30))
46    ax.spines["bottom"].set_visible(False)
47    ax.spines["left"].set_visible(False)
48    n = i + 3
49    ax.set_title(f"{titles[i]}", fontsize=14, pad=5)
50    plot_polygon(ax, "red", *elephant)
51
52small_elephant = [scale(0.5, x) for x in elephant]
53plot_polygon(axs[0][1], "blue", *small_elephant)
54
55tran_elephant = translate((15, 6), elephant)
56plot_polygon(axs[1][0], "blue", *tran_elephant)
57
58rot_elephant = rotate(0.2, elephant)
59plot_polygon(axs[1][1], "blue", *rot_elephant)
60
61fig.tight_layout(pad=2.0)
62plt.show()

Scale, translate and rotate elephant shape

Scale, translate and rotate elephant shape

A variation of translation is reflection where the vectors of a shape are translated proportionally through a vertical line. Reflection through the maximum x coordinate is shown below to give the elephant a friend.

 1def vertical_reflection(vectors):
 2    # find max x value
 3    mir_x = max([x[0] for x in elephant])
 4    return [((-1 * (v[0] - mir_x)) + mir_x, v[1]) for v in vectors]
 5
 6
 7fig, ax = plt.subplots(1, 1, figsize=(10, 5), facecolor=plt.cm.Blues(0.2))
 8fig.suptitle(
 9    f"Reflection of a 2D shape",
10    fontsize=22,
11    fontweight="bold",
12)
13
14plot_axes(ax, (-20, -1), (50, 25))
15ax.spines["bottom"].set_visible(False)
16ax.spines["left"].set_visible(False)
17plot_polygon(ax, "red", *elephant)
18mirror_elephant = vertical_reflection(elephant)
19plot_polygon(ax, "blue", *mirror_elephant)
20
21fig.tight_layout(pad=2.0)
22plt.show()

Reflection of elephant shape

Reflection of elephant shape

Conclusion

This article has completed the sections on 2D-space. Plotting basic and complex shapes using Matplotlib was detailed. A walk-through of trigonometric functions on a right-angled triangle and their use in converting between Polar and Cartesian coordinates. Once the initial shapes are plotted multiple versions of the shape can be generated using translation, rotation and scaling of vectors. Next article will start on 3 dimensional space and building on functions used in 2D-space.

matplotlib a comprehensive library for creating static, animated, and interactive visualizations in Python.

Math for Programmers by Paul Orland - a great book to brush up on your math skills.

2D vectors and vector arithmetic

Eric Callanan — Sunday, 27 Dec 2020

This is the first article on my notes from reading Math for Programmers. It looks at representing 2D vectors graphically with Matplotlib and visualising vector arithmetic.

I recently purchased the book Math for Programmers by Paul Orland. I'm going to write a series or articles to help me work through some of the exercises and to keep notes as I go through the book. This will not be a rewrite of the book and may focus on other areas inspired by the book.

A vector is a quantity that has magnitude and direction and that is commonly represented by a directed line segment.

Two-dimensional space (2D space) is flat like a sheet of paper with only two dimensions height and width.

Points in two-dimensional space

2 dimensional space is flat like a sheet of paper with the width being measured along the horizontal axis (x-axis) and height being measured along the vertical axis (y-axis). Conceptually the two axes extend to positive and negative infinity in both directions, but generally a section is chosen to contain the points of interest. The origin, where x = 0 and y = 0 is the reference point for all other points in the 2D plane. It is possible to describe any point in the 2D plane with two pieces of information - the x and y coordinates. The following code renders a chart showing the point (2, 3) in a 2D plane. This point is 2 units horizontally from the origin and 3 units vertically from the origin.

 1(x, y) = (2, 3)
 2fig, ax = plt.subplots(figsize=(10, 10), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(f"Point ({x}, {y}) in 2D space", fontsize="xx-large", fontweight="bold")
 4ax.set_facecolor(plt.cm.Blues(0.2))
 5ax.plot(x, y, color=plt.cm.Set1(0), marker="X", markersize=15)
 6
 7# Hide the right and top spines
 8ax.spines["right"].set_visible(False)
 9ax.spines["top"].set_visible(False)
10
11# Set the x and y axis to go through origin (0,0)
12ax.spines["left"].set_position("zero")
13ax.spines["bottom"].set_position("zero")
14
15plt.xlim((-4, 4))
16plt.ylim((-4, 4))
17plt.grid(True)
18
19ax.tick_params(labelsize="xx-large")
20
21ax.annotate("", (0, 3.2), xytext=(2, 3.2), arrowprops=dict(arrowstyle="<->", ec="blue"))
22ax.text(1, 3.3, "2", size=20, color="blue", ha="center", va="bottom")
23
24ax.annotate("", (2.2, 0), xytext=(2.2, 3), arrowprops=dict(arrowstyle="<->", ec="blue"))
25ax.text(2.4, 1.5, "3", size=20, color="blue", ha="center", va="center")
26
27plt.show()

2d chart showing point (2,3)

The point (2,3) in two-dimensional plane

Three ways to represent a vector in 2D Space

A vector is a quantity that has magnitude and direction in multi-dimensional space, a 2D-vector is easier to understand and represent by a directed line segment in a 2D plane. A 2D-vector is a point in the 2D plane represented by a line segment from the origin to the point. In the example above, the point (2,3) can be drawn as a directed line from the origin (0, 0) to (2, 3). The third way of representing a 2D-vector is by the length of the directed line and the angle that line is set from the horizontal.

The following code show the three mechanisms of representing the same 2D-vector in two-dimensional space. The point (-3, 4) because the length of the vector is 5.

 1def plot_axes(ax, p):
 2    (x, y) = (p[0], p[1])
 3    ax.set_facecolor(plt.cm.Blues(0.1))
 4    ax.plot(x, y, color="black", marker="o", markersize=15)
 5
 6    # Hide the right and top spines
 7    ax.spines["right"].set_visible(False)
 8    ax.spines["top"].set_visible(False)
 9
10    # Set the x and y axis to go through origin (0,0)
11    ax.spines["left"].set_position("zero")
12    ax.spines["bottom"].set_position("zero")
13
14    ax.set_xlim((min(0, x) - 1.5), (max(0, x) + 1.5))
15    ax.set_ylim((min(0, y) - 1.5), (max(0, y) + 1.5))
16    ax.grid(True)
17    ax.tick_params(labelsize="xx-large")

 1(x, y) = (-3, 4)
 2fig, [ax1, ax2, ax3] = plt.subplots(1, 3, figsize=(20, 8), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(
 4    f"Three ways to represent a vector in 2D space", fontsize=30, fontweight="bold"
 5)
 6plot_axes(ax1, (x, y))
 7ax1.set_title("Point at (x,y)", fontsize=22, pad=20)
 8ax1.annotate("", (x, 0), xytext=(x, y), arrowprops=dict(arrowstyle="<->", ec="blue"))
 9ax1.text(-3.2, 2, "4", size=20, color="blue", ha="center", va="center")
10ax1.annotate("", (0, y), xytext=(x, y), arrowprops=dict(arrowstyle="<->", ec="blue"))
11ax1.text(-1.5, 4.2, "-3", size=20, color="blue", ha="center", va="center")
12
13plot_axes(ax2, (x, y))
14ax2.set_title("Arrow from origin to point (x,y)", fontsize=22, pad=20)
15ax2.plot(0, 0, color="black", marker="o", markersize=10)
16ax2.annotate(
17    "",
18    (x, y),
19    xytext=(0, 0),
20    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=20, headlength=50),
21)
22ax2.tick_params(labelbottom=False)
23ax2.tick_params(labelleft=False)
24ax2.text(-3.5, 4.3, "(-3, 4)", size=20, color="blue", ha="left", va="center")
25ax2.text(0.0, -0.3, "(0, 0)", size=20, color="blue", ha="left", va="center")
26
27plot_axes(ax3, (x, y))
28ax3.set_title("Arrow with distance at angle θ", fontsize=22, pad=20)
29ax3.annotate(
30    "",
31    (x, y),
32    xytext=(0, 0),
33    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=20, headlength=50),
34)
35ax3.text(-1.5, 2.1, "5", size=20, color="blue", ha="left", va="center", rotation=-50)
36ax3.spines["left"].set_visible(False)
37ax3.tick_params(labelbottom=False)
38ax3.tick_params(labelleft=False)
39ax3.annotate(
40    "",
41    (-0.4, 0.7),
42    xytext=(0.6, 0),
43    arrowprops=dict(color="blue", connectionstyle="arc3,rad=0.6"),
44)
45ax3.text(0, 0.1, "θ", size=26, color="blue", ha="left", va="bottom")
46
47fig.tight_layout(pad=2.0)
48plt.show()
49
50publish_png_image(fig, "three-ways-2d-vector.png")

Three ways to represent a 2D-vector

Three ways to represent a 2D-vector

Right angled triangle

Pythagorean theorem

The two dimensions in 2D space are horizontal and vertical, which are perpendicular to each other. This can be used to calculate the length of a vector with the help of the Pythagorean theorem. The Pythagorean theorem states that, in a right-angled triangle, the square on the hypotenuse is equal to the sum of the squares of the other two sides.

In a right-angled triangle with sides of a and b and hypotenuse c then the length of c is the square root of the sum of a-squared and b-squared.

$$ c = \sqrt{a^2 + b^2} $$

In the example the vector (3,4) below: $$ vector \ length = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5 $$

 1(x, y) = (3, 4)
 2fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(12, 8), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(
 4    f"Use Pythagorean theorem to calculate length of vector",
 5    fontsize=22,
 6    fontweight="bold",
 7)
 8
 9plot_axes(ax1, (x, y))
10ax1.set_title("Vector at point (3, 4)", fontsize=18, pad=20)
11ax1.annotate(
12    "",
13    (x, y),
14    xytext=(0, 0),
15    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=20, headlength=50),
16)
17
18plot_axes(ax2, (x, y))
19ax2.set_title("Right-angled triangle for vector (3,3)", fontsize=18, pad=20)
20ax2.annotate(
21    "", (x, y), xytext=(0, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
22)
23ax2.annotate(
24    "", (x, 0), xytext=(0, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
25)
26ax2.annotate(
27    "", (x, y), xytext=(x, 0), arrowprops=dict(arrowstyle="-", ec="green", fc="green")
28)
29ax2.tick_params(labelbottom=False)
30ax2.tick_params(labelleft=False)
31
32ax2.text(1.5, -0.2, "3", size=20, color="blue", ha="left", va="top")
33ax2.text(3.2, 2, "4", size=20, color="blue", ha="left", va="center")
34ax2.text(
35    1.3,
36    1.5,
37    r"$\sqrt{(3^2 + 4^2)}$",
38    size=20,
39    color="blue",
40    ha="center",
41    va="bottom",
42    rotation=50,
43)
44
45fig.tight_layout(pad=3.0)
46plt.show()

use pythagorean theorem to calculate vector length

Use of pythagorean theorem to calculate vector length

The length of a vector can be calculated in python with use of the Math functions module.

 1import math as m
 2
 3
 4def vector_length(v):
 5    return m.sqrt(v[0] ** 2 + v[1] ** 2)
 6
 7
 8for i in range(1, 6):
 9    v = (i, 4)
10    print(f"length of {v} is {vector_length(v):.1F}")
11
12"""
13length of (1, 4) is 4.1
14length of (2, 4) is 4.5
15length of (3, 4) is 5.0
16length of (4, 4) is 5.7
17length of (5, 4) is 6.4
18"""

Adding Vectors

Vectors can be added together by adding their individual constituent parts. In a way, this has already been done when considering a single vector like (3,4). This is composed of an x value of 3 and a y value of 4, which are vectors (3,0) and (0,4) and can be added to yield (3,4).

$$ (x_{1}, y_{1}) + (x_{2}, y_{2}) = (x_{1} + x_{2}, \ y_{1} + y_{2}) $$

The example shows the result of adding vector (3,4) to vector (-2, 1)

$$ (3, 4) + (-2, 1) = (3 + (-2), \ 4 + 1) = (1, 5) $$

 1(x1, y1) = (3, 4)
 2(x2, y2) = (-2, 1)
 3fig, axs = plt.subplots(1, 2, figsize=(12, 6), facecolor=plt.cm.Blues(0.2))
 4fig.suptitle(
 5    f"Adding two vectors",
 6    fontsize=22,
 7    fontweight="bold",
 8)
 9
10for a in axs:
11    a.set_facecolor(plt.cm.Blues(0.1))
12    a.plot(x1, y1, color="black", marker="o", markersize=15)
13    a.plot(x2, y2, color="black", marker="o", markersize=15)
14    a.spines["right"].set_visible(False)
15    a.spines["top"].set_visible(False)
16    a.spines["left"].set_position("zero")
17    a.spines["bottom"].set_position("zero")
18    a.set_xlim(-3.5, 4.5)
19    a.set_ylim(-0.5, 5.5)
20    a.grid(True)
21    a.tick_params(labelsize="large")
22
23axs[0].set_title(f"Vectors at point ({x1}, {y1}) and ({x2}, {y2})", fontsize=18, pad=20)
24axs[0].annotate(
25    "",
26    (x1, y1),
27    xytext=(0, 0),
28    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=20, headlength=50),
29)
30axs[0].annotate(
31    "",
32    (x2, y2),
33    xytext=(0, 0),
34    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=20, headlength=50),
35)
36
37
38axs[1].set_title(f"Sum of vectors = (1, 5)", fontsize=18, pad=20)
39axs[1].plot(x1 + x2, y1 + y2, color="black", marker="*", markersize=15)
40axs[1].annotate(
41    "",
42    (x1 + x2, y1 + y2),
43    xytext=(0, 0),
44    arrowprops=dict(ec=plt.cm.Set1(2), fc=plt.cm.Set1(2), headwidth=20, headlength=50),
45)
46axs[1].annotate(
47    "",
48    (x1, y1),
49    xytext=(0, 0),
50    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(0), fc=plt.cm.Set1(0)),
51)
52axs[1].annotate(
53    "",
54    (x2, y2),
55    xytext=(0, 0),
56    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(0), fc=plt.cm.Set1(0)),
57)
58axs[1].annotate(
59    "",
60    (x1 + x2, y1 + y2),
61    xytext=(x2, y2),
62    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(1), fc=plt.cm.Set1(1)),
63)
64axs[1].annotate(
65    "",
66    (x1 + x2, y1 + y2),
67    xytext=(x1, y1),
68    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(1), fc=plt.cm.Set1(1)),
69)
70
71
72fig.tight_layout(pad=3.0)
73plt.show()

Adding two vectors

Adding two vectors is the sum of the individual component parts

Addition can be extended to sum up any number of vectors by summing up their constituent parts. This can easily be performed in python by taking in a list of vectors and returning the resultant vector of the sum of the vectors.

 1def add_vectors(*v):
 2    return (sum([x[0] for x in v]), sum([x[1] for x in v]))
 3
 4
 5add_vectors((-2, 1), (3,4))
 6"""
 7(1, 5)
 8"""
 9
10add_vectors((1, 2), (0,6), (3,4))
11"""
12(4, 12)
13"""
14
15add_vectors((1, 2), (-5,-3), (3,4))
16"""
17(-1, 3)
18"""

Multiplying vectors by numbers

The addition of two identical vectors is equivalent to multiplying the vector by two. This can be extended to multiplying a vector by any number. Like addition, the resultant vector is got by multiplying the individual components by the number.

$$ (x, y) {\times} n = (x {\times} n, \ y {\times} n) $$

 1(x, y) = (3, 4)
 2fig, axs = plt.subplots(1, 2, figsize=(10, 7), facecolor=plt.cm.Blues(0.2))
 3fig.suptitle(
 4    f"Multiplying vector by a number",
 5    fontsize=22,
 6    fontweight="bold",
 7)
 8
 9for a in axs:
10    a.set_facecolor(plt.cm.Blues(0.1))
11    a.plot(x, y, color="black", marker="o", markersize=15)
12    a.spines["right"].set_visible(False)
13    a.spines["top"].set_visible(False)
14    a.spines["left"].set_position("zero")
15    a.spines["bottom"].set_position("zero")
16    a.set_xlim(-1.5, 13)
17    a.set_ylim(-2, 18)
18    a.set_aspect("equal")
19    a.grid(True)
20    a.set_xticks([x for x in range(0, 13, 2)])
21    a.set_yticks([x for x in range(0, 18, 2)])
22    a.tick_params(labelsize="medium")
23
24axs[0].set_title(f"Vector at point ({x}, {y})", fontsize=18, pad=20)
25axs[0].annotate(
26    "",
27    (x, y),
28    xytext=(0, 0),
29    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
30)
31
32axs[1].set_title(f"Multiplication of vector by 2,3,4", fontsize=18, pad=20)
33for i in range(1, 5):
34    axs[1].plot(x * i, y * i, color="black", marker="*", markersize=15)
35    axs[1].annotate(
36        "",
37        (x * i, y * i),
38        xytext=(0, 0),
39        arrowprops=dict(
40            ec=plt.cm.Set1(0), alpha=0.2, fc=plt.cm.Set1(0), headwidth=25, headlength=30
41        ),
42    )
43
44fig.tight_layout(pad=3.0)
45plt.show()

multiplying vector by a number

Multiplying vector (3,4) by number 2, 3 and 4

A function can be created in python to multiply a vector by a number.

 1def multiply_vector(v, n):
 2    return (v[0] * n, v[1] * n)
 3
 4
 5for i in range(1, 6):
 6    print(f"vector (3,4) * {i} = {multiply_vector((3,4), i)}")
 7
 8"""
 9vector (3,4) * 1 = (3, 4)
10vector (3,4) * 2 = (6, 8)
11vector (3,4) * 3 = (9, 12)
12vector (3,4) * 4 = (12, 16)
13vector (3,4) * 5 = (15, 20)
14"""

Back to Pythagorean principle, it can be seen that scaling the vector (3,4) maintains the principle that the sum on the hypotenuse equals the sum of the squares on the other two sides. the famous triangle of 3,4,5 scales to other whole number combinations such as 6,8,10 or 15, 20, 25.

Table: vector (3,4) multiplied by factors showing the Pythagorean calculations

Vector	factor	x-squared	y-Squared	sum	square root
(3, 4)	1	9	16	25	5.0
(6, 8)	2	36	64	100	10.0
(9, 12)	3	81	144	225	15.0
(12, 16)	4	144	256	400	20.0
(15, 20)	5	225	400	625	25.0

Distance and Displacement

In geometry, Displacement is a vector whose length is the shortest distance from the initial to the final position of a point P undergoing motion. Displacement quantifies both the distance and direction along a straight line, whereas distance just measures the distance between two points without any direction.

The difference between two vectors is a displacement vector, although this includes the distance.

The displacement from (4,2) to (1,4) is (-3, 2)

$$ distance = \sqrt{-3^2 + 2^2} = \sqrt{9 + 4} = \sqrt{13} = 3.6 $$

 1(x1, y1) = (4, 2)
 2(x2, y2) = (1, 4)
 3fig, axs = plt.subplots(1, 2, figsize=(12, 6), facecolor=plt.cm.Blues(0.2))
 4fig.suptitle(
 5    f"Displacement from ({x1}, {y1}) to ({x2}, {y2}) is ({x2-x1}, {y2-y1})",
 6    fontsize=22,
 7    fontweight="bold",
 8)
 9
10for a in axs:
11    a.set_facecolor(plt.cm.Blues(0.1))
12    a.plot(x1, y1, color="black", marker="o", markersize=15)
13    a.plot(x2, y2, color="black", marker="o", markersize=15)
14    a.text(1, 4.3, f"({x2}, {y2})", size=16, color="blue", ha="center", va="bottom")
15    a.text(4, 2.4, f"({x1}, {y1})", size=16, color="blue", ha="center", va="bottom")
16    a.spines["right"].set_visible(False)
17    a.spines["top"].set_visible(False)
18    a.spines["left"].set_position("zero")
19    a.spines["bottom"].set_position("zero")
20    a.set_xlim(-3.5, 4.5)
21    a.set_ylim(-0.5, 5.5)
22    a.grid(True)
23    a.tick_params(labelsize="large")
24
25axs[0].set_title(f"Vectors ({x1}, {y1}) and ({x2}, {y2})", fontsize=18, pad=20)
26axs[0].annotate(
27    "",
28    (x1, y1),
29    xytext=(0, 0),
30    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
31)
32axs[0].annotate(
33    "",
34    (x2, y2),
35    xytext=(0, 0),
36    arrowprops=dict(ec=plt.cm.Set1(0), fc=plt.cm.Set1(0), headwidth=25, headlength=30),
37)
38
39
40axs[1].set_title(f"Sum of vectors = (1, 5)", fontsize=18, pad=20)
41axs[1].plot(x2 - x1, y2 - y1, color="black", marker="*", markersize=15)
42axs[1].annotate(
43    "",
44    (x2 - x1, y2 - y1),
45    xytext=(0, 0),
46    arrowprops=dict(ec=plt.cm.Set1(1), fc=plt.cm.Set1(1), headwidth=25, headlength=30),
47)
48axs[1].annotate(
49    "",
50    (x1, y1),
51    xytext=(0, 0),
52    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(0), fc=plt.cm.Set1(0)),
53)
54axs[1].annotate(
55    "",
56    (x2, y2),
57    xytext=(0, 0),
58    arrowprops=dict(ls="--", arrowstyle="->", ec=plt.cm.Set1(0), fc=plt.cm.Set1(0)),
59)
60axs[1].annotate(
61    "",
62    (x2, y2),
63    xytext=(x1, y1),
64    arrowprops=dict(ec=plt.cm.Set1(1), fc=plt.cm.Set1(1), headwidth=25, headlength=30),
65)
66
67
68axs[1].text(
69    -1.1,
70    0.7,
71    f"({x2-x1}, {y2-y1})",
72    size=16,
73    color=plt.cm.Set1(1),
74    ha="center",
75    va="bottom",
76    rotation=-32,
77)
78axs[1].text(
79    2.9,
80    2.7,
81    f"({x2-x1}, {y2-y1})",
82    size=16,
83    color=plt.cm.Set1(1),
84    ha="center",
85    va="bottom",
86    rotation=-32,
87)
88
89fig.tight_layout(pad=3.0)
90plt.show()

subtracting vectors to get displacement

Subtracting vectors to get displacement

Conclusion

This took me longer than I thought to write this article, but it has helped me learn more on Matplotlib and doing math with 2D vectors. The next article is still on 2 dimensional space with notes on segments, shapes, rotations and transformations.

matplotlib a comprehensive library for creating static, animated, and interactive visualizations in Python.

Math for Programmers by Paul Orland - a great book to brush up on your math skills.

Daily counts and rolling average of COVID-19 numbers

Eric Callanan — Sunday, 20 Dec 2020

The daily counts of confirmed cases or deaths from COVID-19 can fluctuate greatly from day to day. This can be down to time factors such as week day vs weekend or delays in testing or gathering and collating results. A rolling average is a calculated average of different subsets of the full data set. It is used to smooth out fluctuations in the data.

Data sources:
The data used in this article is retrieved from the Johns Hopkins University who have made the data available on GitHub. More information about COVID-19 and the coronavirus is available from Coronavirus disease (COVID-19) advice for the public.

Load COVID-19 data into dataframes

There are two sets of data to load the global confirmed cases and the global deaths.

time_series_covid19_deaths_global.csv
time_series_covid19_confirmed_global.csv

The data can be loaded directly from the GitHub page as in the code below. The data is cleaned to remove unwanted fields and group the data by country with the following function.

 1def load_clean_covid_data(csv_path):
 2    df = pd.read_csv(csv_path)
 3
 4    # 1. Drop unwanted columns
 5    df.drop(['Province/State', 'Lat', 'Long'], axis = 1, inplace = True)
 6
 7    # 2. Group by Country
 8    df = df.groupby('Country/Region').sum()
 9
10    # 3. Convert column headings to datetime
11    df.columns = pd.to_datetime(df.columns)
12
13    # 4. Remove the column name
14    df.rename_axis(None, axis = 0, inplace = True)
15
16    return df

Load the data for Confirmed cases and the data for Deaths from COVID-19 into dataframes.

 1
 2confirmed_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')
 3confirmed_df.shape
 4"""
 5(191, 333)
 6"""
 7
 8confirmed_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
 9"""
10             2020-01-22  2020-01-23  2020-01-24  2020-12-17  2020-12-18  2020-12-19  
11Afghanistan           0           0           0       49378       49621       49681  
12Albania               0           0           0       51424       52004       52542  
13Algeria               0           0           0       93933       94371       94781  
14Yemen                 0           0           0        2087        2087        2087  
15Zambia                0           0           0       18504       18575       18620  
16Zimbabwe              0           0           0       11866       12047       12151  
17"""
18
19
20deaths_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv')
21deaths_df.shape
22"""
23(191, 333)
24"""
25
26deaths_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
27"""
28             2020-01-22  2020-01-23  2020-01-24  2020-12-17  2020-12-18  2020-12-19  
29Afghanistan           0           0           0        2025        2030        2047  
30Albania               0           0           0        1055        1066        1074  
31Algeria               0           0           0        2640        2647        2659  
32Yemen                 0           0           0         606         606         606  
33Zambia                0           0           0         369         373         373  
34Zimbabwe              0           0           0         314         316         318  
35"""

Get the top 10 countries

Get the names of the top ten countries based on the highest deaths for the latest date.

 1top_countries = list(deaths_df
 2                 .iloc[:, -1]
 3                 .sort_values(ascending = False)
 4                 .head(10)
 5                 .index)
 6
 7top_countries
 8"""
 9['US',
10 'Brazil',
11 'India',
12 'Mexico',
13 'Italy',
14 'United Kingdom',
15 'France',
16 'Iran',
17 'Russia',
18 'Spain']
19"""

Display the deaths for a country over time

The data in time_series_covid19_deaths_global.csv is the cumulative number of deaths from COVID-19 recorded in each country every day. This is displayed in a line chart.

 1c = 'US'
 2data = deaths_df.loc[c]
 3date = deaths_df.columns[-1].strftime('%b %d %Y')
 4
 5bg_color = 'rgba(208, 225, 242, 1.0)'
 6line_color = 'rgba(75, 152, 201, 1.0)'
 7grid_color = 'rgba(75, 152, 201, 0.3)'
 8
 9fig = go.Figure()
10fig.add_trace(go.Scatter(
11    x = data.index,
12    y = data,
13    name = 'US',
14    hovertemplate = [f"""<b>{c}</b><br>
15<br>Deaths: {x:,.0F}
16<extra></extra>
17""" for x in data] ))
18
19fig.update_layout(
20    # Set figure title
21    title = dict(
22        text = f'Deaths from COVID-19 in <b>"{c}"</b> <br> {date}',
23        xref = 'container',
24        yref = 'container',
25        x = 0.5,
26        y = 0.9,
27        xanchor = 'center',
28        yanchor = 'middle',
29        font = dict(family = 'Droid Sans', size = 28)
30    ),
31    # set x-axis
32    xaxis = dict(
33        title = 'Date',
34        linecolor = line_color,
35        linewidth = 2,
36        gridcolor = grid_color,
37        showticklabels = True,
38        ticks = 'outside',
39    ),
40    # set y-axis
41    yaxis = dict(
42        title = 'Number of deaths',
43        rangemode = 'tozero',
44        linecolor = line_color,
45        linewidth = 2,
46        gridcolor = grid_color,
47        showticklabels = True,
48        ticks = 'outside',
49    ),
50    # set the plot bacground color
51    plot_bgcolor = bg_color,
52    paper_bgcolor = bg_color,
53)
54
55fig.show()

Total deaths from COVID-19 in US - December 19 2020

Display daily counts for a single country

Reduce the dataset by selecting the data for the top 10 countries. The daily count is easily got using pandas.DataFrame.diff function. This calculates the diference of data elements from another element in the dataframe. The default is to calculate the difference between the element in the previous row, but an axis parameter of 1 or columns can be used to calculate the difference between the element in the previous column.

 1top_df = deaths_df.loc[top_countries]
 2top_daily_df = top_df.diff(axis=1)
 3
 4top_daily_df.iloc[:, -10:]
 5"""
 6                2020-12-10  2020-12-11  2020-12-12  2020-12-13  2020-12-14  2020-12-15  2020-12-16  2020-12-17  2020-12-18  2020-12-19  
 7US                  2920.0      3283.0      2354.0      1389.0      1484.0      2984.0      3668.0      3345.0      2814.0      2571.0  
 8Brazil               770.0       672.0       686.0       279.0       433.0       964.0       936.0      1092.0       823.0       706.0  
 9India                413.0       443.0       391.0       336.0       354.0       387.0       355.0       338.0       347.0       341.0  
10Mexico               671.0       693.0       685.0       249.0       345.0       801.0       670.0       718.0       762.0       627.0  
11Italy                887.0       761.0       649.0       484.0       491.0       846.0       680.0       683.0       674.0       553.0  
12United Kingdom       516.0       424.0       520.0       144.0       233.0       506.0       612.0       532.0       490.0       537.0  
13France               296.0       628.0        -1.0       344.0       376.0       791.0       290.0       261.0       612.0       189.0  
14Iran                 284.0       231.0       222.0       247.0       251.0       223.0       213.0       212.0       178.0       175.0  
15Russia               549.0       601.0       553.0       481.0       442.0       564.0       584.0       574.0       602.0       574.0  
16Spain                325.0       280.0         0.0         0.0       389.0       388.0       195.0       181.0       149.0         0.0  
17"""

Table: Deaths from COVID-19 in previous 7 days in countries with highest deaths on December 19 2020

Country	Dec 13	Dec 14	Dec 15	Dec 16	Dec 17	Dec 18	Dec 19
US	1,389	1,484	2,984	3,668	3,345	2,814	2,571
Brazil	279	433	964	936	1,092	823	706
India	336	354	387	355	338	347	341
Mexico	249	345	801	670	718	762	627
Italy	484	491	846	680	683	674	553
United Kingdom	144	233	506	612	532	490	537
France	344	376	791	290	261	612	189
Iran	247	251	223	213	212	178	175
Russia	481	442	564	584	574	602	574
Spain	0	389	388	195	181	149	0

The daily counts is plotted for a single country.

 1c = 'US'
 2data = top_daily_df.loc[c]
 3date = top_daily_df.columns[-1].strftime('%b %d %Y')
 4
 5bg_color = 'rgba(208, 225, 242, 1.0)'
 6line_color = 'rgba(75, 152, 201, 1.0)'
 7grid_color = 'rgba(75, 152, 201, 0.3)'
 8
 9fig = go.Figure()
10fig.add_trace(go.Scatter(
11    x = data.index,
12    y = data,
13    name = 'US',
14    hovertemplate = [f"""<b>{c}</b><br>
15<br>Deaths: {x:,.0F}
16<extra></extra>
17""" for x in data] ))
18
19fig.update_layout(
20    # Set figure title
21    title = dict(
22        text = f'Daily deaths from COVID-19 in <b>"{c}"</b> <br> {date}',
23        xref = 'container',
24        yref = 'container',
25        x = 0.5,
26        y = 0.9,
27        xanchor = 'center',
28        yanchor = 'middle',
29        font = dict(family = 'Droid Sans', size = 28)
30    ),
31    # set x-axis
32    xaxis = dict(
33        title = 'Date',
34        linecolor = line_color,
35        linewidth = 2,
36        gridcolor = grid_color,
37        showticklabels = True,
38        ticks = 'outside',
39    ),
40    # set y-axis
41    yaxis = dict(
42        title = 'Number of deaths',
43        rangemode = 'tozero',
44        linecolor = line_color,
45        linewidth = 2,
46        gridcolor = grid_color,
47        showticklabels = True,
48        ticks = 'outside',
49    ),
50    # set the plot bacground color
51    plot_bgcolor = bg_color,
52    paper_bgcolor = bg_color,
53)
54
55fig.show()

Daily deaths from COVID-19 in US - December 19 2020

Display rolling average on daily counts

The daily counts fluctuate greatly from day to day and it can be difficult to see the overall trend. A rolling average calculates an average of different subsets of the full data set. It is used to smooth out fluctuations in the data. In this case the pandas.Series.rolling function is used to create a number of rolling average to determine the best value to use.

Here is an example of calculating the rolling average of 3 days for the latest 20 days on the number of deaths from COVID-19 in the US. The chart is used to determine the best number of days to set for the rolling window. Rolling window sizes of 3, 5, 7 and 10 were compared. The different results can be switched on and off in the Plotly chart and areas of the chart can be zoomed in on. These results show that 7 is the best number to select for the rolling window size. This is likely due to daily fluctuations based on weekday that are smoothed out when taking the week on average.

 1c = 'US'
 2data = top_daily_df.loc[c].tail(20)
 3rolling = data.rolling(3).mean()
 4
 5data
 6"""
 7['1153.0', '2588.0', '2775.0', '2843.0', '2594.0', '2257.0', '1122.0',
 8'1419.0', '2520.0', '3064.0', '2920.0', '3283.0', '2354.0', '1389.0',
 9'1484.0', '2984.0', '3668.0', '3345.0', '2814.0', '2571.0']
10"""
11
12rolling
13"""
14['NAN', 'NAN', '2172.0', '2735.3', '2737.3', '2564.7', '1991.0',
15'1599.3', '1687.0', '2334.3', '2834.7', '3089.0', '2852.3', '2342.0',
16'1742.3', '1952.3', '2712.0', '3332.3', '3275.7', '2910.0']
17"""

 1c = 'US'
 2data = top_daily_df.loc[c]
 3date = top_daily_df.columns[-1].strftime('%b %d %Y')
 4r3 = data.rolling(3).mean()
 5r5 = data.rolling(5).mean()
 6r7 = data.rolling(7).mean()
 7r10 = data.rolling(10).mean()
 8
 9bg_color = 'rgba(208, 225, 242, 1.0)'
10line_color = 'rgba(75, 152, 201, 1.0)'
11grid_color = 'rgba(75, 152, 201, 0.3)'
12
13fig = go.Figure()
14for d, n in zip([data, r3, r5, r7, r10], [1, 3, 5, 7, 10]):    
15    fig.add_trace(go.Scatter(
16        x = d.index,
17        y = d,
18        name = f'rolling window = "{n}"',
19        hovertemplate = [f"""<b>{c}</b>
20    <br>Rolling window = {n}<br>
21    <br>Deaths: {x:,.0F}
22    <extra></extra>
23    """ for x in d] ))
24
25fig.update_layout(
26    # Set figure title
27    title = dict(
28        text = f'Daily deaths from COVID-19 in <b>"{c}"</b> <br> {date}',
29        xref = 'container',
30        yref = 'container',
31        x = 0.5,
32        y = 0.9,
33        xanchor = 'center',
34        yanchor = 'middle',
35        font = dict(family = 'Droid Sans', size = 28)
36    ),
37    # set x-axis
38    xaxis = dict(
39        title = 'Date',
40        linecolor = line_color,
41        linewidth = 2,
42        gridcolor = grid_color,
43        showticklabels = True,
44        ticks = 'outside',
45    ),
46    # set y-axis
47    yaxis = dict(
48        title = 'Number of deaths',
49        rangemode = 'tozero',
50        linecolor = line_color,
51        linewidth = 2,
52        gridcolor = grid_color,
53        showticklabels = True,
54        ticks = 'outside',
55    ),
56    # set the plot bacground color
57    plot_bgcolor = bg_color,
58    paper_bgcolor = bg_color,
59)
60
61fig.show()

Daily deaths from COVID-19 in US showing means for different rolling window sizes - December 19 2020

Identify the best rolling average window size

It can be easier to identify the better rolling window size by plotting the different options on separate plots. This confirms that the best rolling window setting is for 7 days. The 7-day rolling average smooths out the daily fluctuations and makes it easier to see trends in the data. This confirmed by creating a number of charts for different countries.

 1c = 'US'
 2data = top_daily_df.loc[c]
 3date = top_daily_df.columns[-1].strftime('%b %d %Y')
 4r3 = data.rolling(3).mean()
 5r5 = data.rolling(5).mean()
 6r7 = data.rolling(7).mean()
 7r10 = data.rolling(10).mean()
 8r20 = data.rolling(10).mean()
 9
10bg_color = 'rgba(208, 225, 242, 1.0)'
11line_color = 'rgba(75, 152, 201, 1.0)'
12grid_color = 'rgba(75, 152, 201, 0.3)'
13
14fig = make_subplots(
15    rows = 3,
16    cols = 2,
17    subplot_titles = (
18        "Daily Data",
19        "Rolling Window 3",
20        "Rolling Window 5",
21        "Rolling Window 7",
22        "Rolling Window 10",
23        "Rolling Window 20"))
24
25fig.add_trace(go.Scatter(x = data.index, y = data), row = 1, col = 1)
26fig.add_trace(go.Scatter(x = r3.index, y = r3), row = 1, col = 2)
27fig.add_trace(go.Scatter(x = r5.index, y = r5), row = 2, col = 1)
28fig.add_trace(go.Scatter(x = r7.index, y = r7), row = 2, col = 2)
29fig.add_trace(go.Scatter(x = r10.index, y = r10), row = 3, col = 1)
30fig.add_trace(go.Scatter(x = r20.index, y = r20), row = 3, col = 2)
31
32fig.update_layout(
33    showlegend=False
34)
35
36fig.show()

Deaths from COVID-19 in US showing different rolling averages in multiple subplots

Deaths from COVID-19 in UK showing different rolling averages in multiple subplots

Display rolling averages for top ten countries

Display the 7-day rolling average of Deaths from COVID-19 for the top ten countries with the highest deaths on December 19 2020.

 1date = top_daily_df.columns[-1].strftime('%b %d %Y')
 2
 3bg_color = 'rgba(208, 225, 242, 1.0)'
 4line_color = 'rgba(75, 152, 201, 1.0)'
 5grid_color = 'rgba(75, 152, 201, 0.3)'
 6
 7fig = go.Figure()
 8for i, row in top_daily_df.iterrows():
 9    r7 = row.rolling(7).mean()
10    fig.add_trace(go.Scatter(
11        x = r7.index,
12        y = r7,
13        name = i,
14        hovertemplate = [f"""<b>{r7.name}</b>
15    <br>Deaths: {x:,.0F}
16    <extra></extra>
17    """ for x in r7]))
18
19fig.update_layout(
20    # Set figure title
21    title = dict(
22        text = f'7-day daily average of deaths from COVID-19 in top ten countries<br> {date}',
23        xref = 'container',
24        yref = 'container',
25        x = 0.5,
26        y = 0.9,
27        xanchor = 'center',
28        yanchor = 'middle',
29        font = dict(family = 'Droid Sans', size = 24)
30    ),
31    # set x-axis
32    xaxis = dict(
33        title = 'Date',
34        linecolor = line_color,
35        linewidth = 2,
36        gridcolor = grid_color,
37        showticklabels = True,
38        ticks = 'outside',
39    ),
40    # set y-axis
41    yaxis = dict(
42        title = '7-day daily average of deaths',
43        rangemode = 'tozero',
44        linecolor = line_color,
45        linewidth = 2,
46        gridcolor = grid_color,
47        showticklabels = True,
48        ticks = 'outside',
49    ),
50    # set the plot bacground color
51    plot_bgcolor = bg_color,
52    paper_bgcolor = bg_color,
53)
54
55fig.show()

7-day average of Deaths per day from COVID-19 in countries with highest deaths

A similar plot can be created for confirmed cases for the same ten countries with highest deaths.

7-day average of Confirmed Cases per day of COVID-19 in countries with highest deaths

Display rolling averages and the daily counts

Display the 7-day rolling average of Deaths from COVID-19 as well as the actual daily counts in the same chart. The daily counts are shown as bars and the rolling average is a line. The results for the countries with the highest deaths is displayed. The plots are separated per country. There is a lot of data collected to date and sections of the charts can be chosen to zoom in on a particular time period. Another approach is to just plot the data for the most recent 30 days to give a picture of what the current trend is.

 1date = top_daily_df.columns[-1].strftime('%b %d %Y')
 2
 3bg_color = 'rgba(208, 225, 242, 1.0)'
 4line_color = 'rgba(75, 152, 201, 1.0)'
 5grid_color = 'rgba(75, 152, 201, 0.3)'
 6
 7fig = make_subplots(
 8    rows = 5,
 9    cols = 2,
10    shared_xaxes = True,
11    subplot_titles = (top_daily_df.index)
12    )
13
14j = 0
15for i, row in top_daily_df.iterrows():
16    r7 = row.rolling(7).mean()
17    fig.add_trace(go.Bar(x = row.index, y = row, name = i), row = j%5+1, col = j%2+1)
18    fig.add_trace(go.Scatter(x = r7.index, y = r7, name = i), row = j%5+1, col = j%2+1)
19
20    j += 1
21
22
23fig.update_xaxes(dict(
24        linecolor = line_color,
25        gridcolor = grid_color,
26    ))
27
28fig.update_yaxes(dict(
29        rangemode = 'tozero',
30        linecolor = line_color,
31        gridcolor = grid_color,
32    ))
33
34fig.update_layout(
35    # Set figure title
36    title = dict(
37        text = f'Daily deaths from COVID-19 in <b>"{c}"</b> <br> {date}',
38        xref = 'container',
39        yref = 'container',
40        x = 0.5,
41        y = 0.95,
42        xanchor = 'center',
43        yanchor = 'middle',
44        font = dict(family = 'Droid Sans', size = 22)
45    ),
46    showlegend=False,
47    # set the plot bacground color
48    plot_bgcolor = bg_color,
49    paper_bgcolor = bg_color,
50)
51
52fig.show()

Daily counts and 7-day average of Deaths from COVID-19 in countries with highest deaths

Daily counts and 7-day average of Deaths from COVID-19 for latest 30 days

Similar charts can be created for the daily counts and rolling average of confirmed cases.

Daily counts and 7-day average of Confirmed Cases of COVID-19 in countries with highest deaths

Daily counts and 7-day average of Confirmed Cases of COVID-19 for latest 30 days

Conclusion

Once again calculations can be applied to a whole dataset easily with Pandas. The diff function is used to easily convert the running totals to daily differences and the rolling function is used to group a subset of data together and calculate a running average. The running average helps smooth the day to day fluctuations in the data and makes it easier to spot a trend.

Plotly is an open-source graphing library for Python that produces interactive charts.

Display COVID-19 numbers relative to population

Eric Callanan — Sunday, 13 Dec 2020

The total number of cases of COVID-19 and deaths from COVID-19 in a country will vary based on a number of factors. It does not seem right to compare absolute numbers in different countries, where the populations of those countries varies greatly. This article will show how to take the absolute numbers and express the confirmed cases and deaths per 100,000 of the population and thus allow comparison between countries.

Load COVID-19 data into dataframes

There are two sets of data to load the global confirmed cases and the global deaths.

time_series_covid19_deaths_global.csv
time_series_covid19_confirmed_global.csv

The data can be loaded directly from the GitHub page as in the code below. The data is cleaned to remove unwanted fields and group the data by country with the following function.

 1def load_clean_covid_data(csv_path):
 2    df = pd.read_csv(csv_path)
 3
 4    # 1. Drop unwanted columns
 5    df.drop(['Province/State', 'Lat', 'Long'], axis = 1, inplace = True)
 6
 7    # 2. Group by Country
 8    df = df.groupby('Country/Region').sum()
 9
10    # 3. Convert column headings to datetime
11    df.columns = pd.to_datetime(df.columns)
12
13    # 4. Remove the column name
14    df.rename_axis(None, axis = 0, inplace = True)
15
16    return df

Load the data for Confirmed cases and the data for Deaths from COVID-19 into dataframes.

 1
 2confirmed_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')
 3confirmed_df.shape
 4"""
 5(191, 326)
 6"""
 7
 8confirmed_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
 9"""
10             2020-01-22  2020-01-23  2020-01-24  2020-12-10  2020-12-11  2020-12-12
11Afghanistan           0           0           0       48053       48116       48229
12Albania               0           0           0       46061       46863       47742
13Algeria               0           0           0       90579       91121       91638
14Yemen                 0           0           0        2081        2082        2083
15Zambia                0           0           0       18091       18161       18217
16Zimbabwe              0           0           0       11081       11162       11219
17"""
18
19
20deaths_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv')
21deaths_df.shape
22"""
23(191, 326)
24"""
25
26deaths_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
27"""
28             2020-01-22  2020-01-23  2020-01-24  2020-12-10  2020-12-11  2020-12-12  
29Afghanistan           0           0           0        1935        1945        1956  
30Albania               0           0           0         965         977         989  
31Algeria               0           0           0        2564        2575        2584  
32Yemen                 0           0           0         606         606         606  
33Zambia                0           0           0         364         365         366  
34Zimbabwe              0           0           0         305         306         307  
35"""

Load global population data into dataframe

World population data is available from the World Bank data. This is available to download in csv, xml or Excel format. The Excel file contains three worksheets with the population data in the first sheet Data. This is loaded into a dataframe.

1pop_xl = pd.ExcelFile("/tmp/API_SP.POP.TOTL_DS2_en_excel_v2_1740169.xls", engine = "xlrd")
2pop_xl.sheet_names
3"""
4['Data', 'Metadata - Countries', 'Metadata - Indicators']
5"""

 1# Load the excel worksheet into a dataframe
 2pop_df = pd.read_excel(
 3    "/tmp/API_SP.POP.TOTL_DS2_en_excel_v2_1740169.xls",
 4    engine = "xlrd",
 5    sheet_name = 'Data',
 6    header = 3)
 7
 8pop_df.shape
 9"""
10(264, 65)
11"""
12
13pop_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
14"""
15     Country Name Country Code     Indicator Name        2018        2019  2020  
160           Aruba          ABW  Population, total    105845.0    106314.0   NaN  
171     Afghanistan          AFG  Population, total  37172386.0  38041754.0   NaN  
182          Angola          AGO  Population, total  30809762.0  31825295.0   NaN  
19261  South Africa          ZAF  Population, total  57779622.0  58558270.0   NaN  
20262        Zambia          ZMB  Population, total  17351822.0  17861030.0   NaN  
21263      Zimbabwe          ZWE  Population, total  14439018.0  14645468.0   NaN  
22"""

Extract population data for 2019 into a dataframe.

 1# Get population for countries for 2019
 2pop_2019_df = pop_df[['Country Name', '2019']].copy()
 3
 4pop_2019_df.shape
 5"""
 6(264, 2)
 7"""
 8
 9pop_2019_df.iloc[[0,1,2,3,4,-5,-4-3,-2,-1], :]
10"""
11    Country Name          2019
120          Aruba  1.063140e+05
131    Afghanistan  3.804175e+07
142         Angola  3.182530e+07
153        Albania  2.854191e+06
164        Andorra  7.714200e+04
17259       Kosovo  1.794248e+06
18257        World  7.673534e+09
19262       Zambia  1.786103e+07
20263     Zimbabwe  1.464547e+07
21"""

Load population data for a single year

After exploring the population dataset, the code above can be simplified to just load data for a single year. The population data for the year 2019 can be loaded directly from the Excel file. The Country Code cannot be used, as it is not present in the COVID-19 data.

 1# Load country and data for 2019 from the excel worksheet into a dataframe
 2pop_2019_df = pd.read_excel(
 3    "/tmp/API_SP.POP.TOTL_DS2_en_excel_v2_1740169.xls",
 4    engine = "xlrd",
 5    sheet_name = 'Data',
 6    usecols = ['Country Name', '2019'],
 7    header = 3)
 8
 9pop_2019_df.shape
10"""
11(264, 2)
12"""
13
14pop_2019_df.iloc[[0,1,2,3,4,-5,-4-3,-2,-1], :]
15"""
16    Country Name          2019
170          Aruba  1.063140e+05
181    Afghanistan  3.804175e+07
192         Angola  3.182530e+07
203        Albania  2.854191e+06
214        Andorra  7.714200e+04
22259       Kosovo  1.794248e+06
23257        World  7.673534e+09
24262       Zambia  1.786103e+07
25263     Zimbabwe  1.464547e+07
26"""

Compare countries in COVID-19 datasets with population dataset

Examine the matching of country names between population data and COVID-19 data as country codes are not used. This shows that there are 25 "Country/Region" that are not present in the population data.

 1# Countries in confirmed_df dataframe that are not in pop_2019_df
 2unmatched_covid_countries = sorted(
 3    list(
 4        confirmed_df[
 5            ~confirmed_df.index
 6            .isin(list(pop_2019_df['Country Name']))
 7        ].index))
 8
 9len(unmatched_covid_countries)
10"""
1125
12"""
13
14unmatched_covid_countries
15"""
16Bahamas                               Brunei                                Burma                                 
17Congo (Brazzaville)                   Congo (Kinshasa)                      Czechia                               
18Diamond Princess                      Egypt                                 Gambia                                
19Holy See                              Iran                                  Korea, South                          
20Kyrgyzstan                            Laos                                  MS Zaandam                            
21Russia                                Saint Kitts and Nevis                 Saint Lucia                           
22Saint Vincent and the Grenadines      Slovakia                              Syria                                 
23Taiwan*                               US                                    Venezuela                             
24Yemen                                 
25"""

Four of these "Country/Region" cannot be mapped to a country in the population data. Two of these are cruise ships, Taiwan* is a disputed territory and the Holy See may be included in smal states. These are removed from the list of unmatched countries.

 1unmatched_covid_countries.remove('Diamond Princess')
 2unmatched_covid_countries.remove('MS Zaandam')
 3unmatched_covid_countries.remove('Holy See')
 4unmatched_covid_countries.remove('Taiwan*')
 5
 6len(unmatched_covid_countries)
 7"""
 821
 9"""
10
11unmatched_covid_countries
12"""
13Bahamas                               Brunei                                Burma                                 
14Congo (Brazzaville)                   Congo (Kinshasa)                      Czechia                               
15Egypt                                 Gambia                                Iran                                  
16Korea, South                          Kyrgyzstan                            Laos                                  
17Russia                                Saint Kitts and Nevis                 Saint Lucia                           
18Saint Vincent and the Grenadines      Slovakia                              Syria                                 
19US                                    Venezuela                             Yemen                                 
20"""

The remaining "Country/Region" are manually mapped to Countries in the population dataset.

COVID-19 Country Name	Population Country Name
Bahamas	Bahamas, The
Brunei	Brunei Darussalam
Burma	Myanmar
Congo (Brazzaville)	Congo, Rep.
Congo (Kinshasa)	Congo, Dem. Rep.
Czechia	Czech Republic
Egypt	Egypt, Arab Rep.
Gambia	Gambia, The
Iran	Iran, Islamic Rep.
Korea, South	Korea, Rep.
Kyrgyzstan	Kyrgyz Republic
Laos	Lao PDR
Russia	Russian Federation
Saint Kitts and Nevis	St. Kitts and Nevis
Saint Lucia	St. Lucia
Saint Vincent and the Grenadines	St. Vincent and the Grenadines
Slovakia	Slovak Republic
Syria	Syrian Arab Republic
US	United States
Venezuela	Venezuela, RB
Yemen	Yemen, Rep.

Update country names in population dataset

A new column is added to the population dataset to contain the country name from the COVID-19 dataset if it is different or the original country if there is no difference. This could update the country name in place, but it is better to have the two names for verification.

 1map_pop = ['Bahamas, The', 'Brunei Darussalam', 'Myanmar', 'Congo, Rep.', 'Congo, Dem. Rep.',
 2           'Czech Republic', 'Egypt, Arab Rep.', 'Gambia, The', 'Iran, Islamic Rep.',
 3           'Korea, Rep.', 'Kyrgyz Republic', 'Lao PDR', 'Russian Federation', 'St. Kitts and Nevis',
 4           'St. Lucia', 'St. Vincent and the Grenadines', 'Slovak Republic', 'Syrian Arab Republic',
 5           'United States', 'Venezuela, RB', 'Yemen, Rep.'
 6          ]
 7
 8country_map = {}
 9for c,p in zip(unmatched_covid_countries, map_pop):
10    country_map[p] = c
11
12pop_2019_df['Covid_Country'] = pop_2019_df.apply(
13    lambda x: country_map[x['Country Name']]
14        if x['Country Name'] in country_map
15        else x['Country Name'],
16    axis = 1)
17
18pop_2019_df.iloc[[1,2,3,21,29,249], :]
19"""
20          Country Name         2019 Covid_Country
211          Afghanistan   38041754.0   Afghanistan
222               Angola   31825295.0        Angola
233              Albania    2854191.0       Albania
2421        Bahamas, The     389482.0       Bahamas
2529   Brunei Darussalam     433285.0        Brunei
26249      United States  328239523.0            US
27"""

Confirm that there are no unexpected countries in covid data that are not in population data.

 1list(confirmed_df[
 2    ~confirmed_df.index
 3    .isin(list(pop_2019_df['Covid_Country']))
 4].index)
 5"""
 6['Diamond Princess', 'Holy See', 'MS Zaandam', 'Taiwan*']
 7"""
 8
 9list(deaths_df[
10    ~deaths_df.index
11    .isin(list(pop_2019_df['Covid_Country']))
12].index)
13"""
14['Diamond Princess', 'Holy See', 'MS Zaandam', 'Taiwan*']
15"""

Calculate cases and deaths per 100,000 of population

Merge the population numbers for 2019 into the COVID datasets. Rename the column name for 2019 to Population. This is done for the Confirmed cases and the Deaths datasets.

 1# Merge Population data
 2confirmed_pop_df = confirmed_df.merge(
 3    pop_2019_df,
 4    left_index = True,
 5    left_on = confirmed_df.index,
 6    right_on = 'Covid_Country',
 7)
 8
 9# Rename '2019' column name to 'population'
10confirmed_pop_df = confirmed_pop_df.rename(columns={'2019': 'Population'})
11
12confirmed_pop_df.shape
13"""(187, 329)
14"""
15
16confirmed_pop_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-4,-3,-2,-1]]
17"""
18(187, 329)
19     2020-01-22 00:00:00  2020-01-23 00:00:00  2020-01-24 00:00:00  2020-12-12 00:00:00 Country Name  Population Covid_Country  
201                      0                    0                    0                48229  Afghanistan  38041754.0   Afghanistan  
213                      0                    0                    0                47742      Albania   2854191.0       Albania  
2258                     0                    0                    0                91638      Algeria  43053054.0       Algeria  
23260                    0                    0                    0                 2083  Yemen, Rep.  29161922.0         Yemen  
24262                    0                    0                    0                18217       Zambia  17861030.0        Zambia  
25263                    0                    0                    0                11219     Zimbabwe  14645468.0      Zimbabwe  
26"""

 1# Merge Population data
 2deaths_pop_df = deaths_df.merge(
 3    pop_2019_df,
 4    left_index = True,
 5    left_on = deaths_df.index,
 6    right_on = 'Covid_Country',
 7)
 8
 9# Rename '2019' column name to 'population'
10deaths_pop_df = deaths_pop_df.rename(columns={'2019': 'Population'})
11
12deaths_pop_df.shape
13"""(187, 329)
14"""
15
16deaths_pop_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-4,-3,-2,-1]]
17"""
18     2020-01-22 00:00:00  2020-01-23 00:00:00  2020-01-24 00:00:00  2020-12-12 00:00:00 Country Name  Population Covid_Country  
191                      0                    0                    0                 1956  Afghanistan  38041754.0   Afghanistan  
203                      0                    0                    0                  989      Albania   2854191.0       Albania  
2158                     0                    0                    0                 2584      Algeria  43053054.0       Algeria  
22260                    0                    0                    0                  606  Yemen, Rep.  29161922.0         Yemen  
23262                    0                    0                    0                  366       Zambia  17861030.0        Zambia  
24263                    0                    0                    0                  307     Zimbabwe  14645468.0      Zimbabwe  
25"""

To adjust the numbers of confirmed cases and deaths to 100,000 of the population, divide the number by the population and multiply by 100,000.

 1# Confirmed cases per 100,000 of population
 2confirmed_per_100k_df = confirmed_pop_df.copy()
 3col_names = list(confirmed_per_100k_df.columns)
 4[(col_names.remove(x)) for x in ['Country Name', 'Population', 'Covid_Country']]
 5confirmed_per_100k_df[col_names] = (
 6    confirmed_per_100k_df[col_names]
 7    .div(confirmed_per_100k_df['Population'], axis = 0)
 8    .mul(100000, axis = 0)
 9)
10
11confirmed_per_100k_df.shape
12"""
13(187, 329)
14"""
15
16confirmed_per_100k_df.iloc[[0,1,2,-3,-2,-1], [0,50,100,150,-4,-3,-2,-1]]
17"""
18     2020-01-22 00:00:00  2020-03-12 00:00:00  2020-05-01 00:00:00  2020-06-20 00:00:00  2020-12-12 00:00:00 Country Name  Population  Covid_Country  
191                    0.0             0.031544             6.022330            74.728416           126.779117  Afghanistan  38041754.0    Afghanistan  
203                    0.0             0.805833            27.398307            66.253450          1672.698148      Albania   2854191.0        Albania  
2158                   0.0             0.055745             9.648561            27.015505           212.849012      Algeria  43053054.0        Algeria  
22260                  0.0             0.000000             0.024004             3.161657             7.142876  Yemen, Rep.  29161922.0          Yemen  
23262                  0.0             0.000000             0.610267             8.006257           101.992998       Zambia  17861030.0         Zambia  
24263                  0.0             0.000000             0.273122             3.270636            76.603902     Zimbabwe  14645468.0       Zimbabwe  
25"""

1# Deaths per 100,000 of population
2deaths_per_100k_df = deaths_pop_df.copy()
3col_names = list(deaths_per_100k_df.columns)
4[(col_names.remove(x)) for x in ['Country Name', 'Population', 'Covid_Country']]
5deaths_per_100k_df[col_names] = (
6    deaths_per_100k_df[col_names]
7    .div(deaths_per_100k_df['Population'], axis = 0)
8    .mul(100000, axis = 0)
9)

Countries with highest confirmed cases per 100,000 of population

Countries with populations of less than one million were removed from the dataset. The data is sorted by the numbers of confirmed cases per 100,000 on the latest date and the data for the top 20 countries displayed in a table and bar chart. The changes over time for the top 10 countries is displayed in a line chart.

 1confirmed_per_100k_df = confirmed_per_100k_df[confirmed_per_100k_df['Population'] > 1000000]
 2
 3top_20_confirmed_df = (confirmed_per_100k_df.sort_values(
 4    by = confirmed_per_100k_df.columns[-4], ascending = False)
 5    .head(20)[['Covid_Country',
 6               'Population',
 7               pd.to_datetime('2020-12-12 00:00:00') ]])
 8
 9# Merge Confirmed data
10top_20_confirmed_df = top_20_confirmed_df.merge(
11    confirmed_df.iloc[:, [-1]],
12    right_index = True,
13    left_on = top_20_confirmed_df['Covid_Country'],
14    right_on = confirmed_df.index,
15)
16
17# Reset the index
18top_20_confirmed_df.reset_index(drop = True, inplace = True)
19
20# Drop the merged key
21top_20_confirmed_df = top_20_confirmed_df.drop(['key_0'], axis = 1)
22
23# Rename the column names
24top_20_confirmed_df = top_20_confirmed_df.rename(
25    columns = {'Covid_Country': 'Country',
26               '2020-12-12 00:00:00_x': 'Cases per 100k',
27               '2020-12-12 00:00:00_y': 'Total cases'})

Table: The top 20 countries with highest cases of COVID-19 on November 12 2020

Rank	Country	Population	Cases per 100k	Total cases
1	Bahrain	1,641,172	5,421	88,965
2	Czechia	10,669,709	5,393	575,422
3	Belgium	11,484,055	5,252	603,159
4	Georgia	3,720,382	5,027	187,006
5	Armenia	2,957,731	4,981	147,312
6	Qatar	2,832,067	4,973	140,827
7	US	328,239,523	4,893	16,062,299
8	Moldova	2,657,637	4,731	125,723
9	Slovenia	2,087,946	4,573	95,481
10	Panama	4,246,439	4,488	190,585
11	Switzerland	8,574,832	4,360	373,831
12	Croatia	4,067,500	4,241	172,523
13	Israel	9,053,300	3,930	355,786
14	Serbia	6,944,975	3,764	261,437
15	Spain	47,076,781	3,676	1,730,575
16	Austria	8,877,067	3,603	319,822
17	France	67,059,887	3,587	2,405,255
18	Netherlands	17,332,850	3,540	613,630
19	North Macedonia	2,083,459	3,505	73,025
20	Kuwait	4,207,083	3,471	146,044

 1top_10 = top_20_confirmed_df.head(10)
 2bg_color = 'rgba(208, 225, 242, 1.0)'
 3
 4fig = go.Figure(
 5    data = [go.Bar(x = list(top_10['Cases per 100k']),
 6                   y = list(top_10['Country']),
 7                   hovertemplate = [f"""<b>{r['Country']}</b>: Confirmed Cases<br>
 8<br>Per 100k: {r['Cases per 100k']:,.0F}
 9<br>Total: {r['Total cases']:,.0f}
10<br>Population: {r['Population']:,.0f}
11<extra></extra>
12""" for i,r in top_10.iterrows()],
13                   orientation = 'h')]
14)
15
16fig.update_layout(
17    # Set default font
18        font = dict(
19        family = "Droid Sans",
20        size = 16
21    ),
22    # Set figure title
23    title = dict(
24        text = f"""<b>Countries with highest confirmed cases of COVID-19 per 100,000
25<BR>{pd.to_datetime('2020-12-12 00:00:00').strftime('%b %d %Y')}</b>""",
26        xref = 'container',
27        yref = 'container',
28        x = 0.5,
29        y = 0.91,
30        xanchor = 'center',
31        yanchor = 'middle',
32        font = dict(family = 'Droid Sans', size = 24)
33    ),
34    # set x-axis
35    xaxis = dict(
36        title = dict(
37            text = 'Number of confirmed cases per 100,000 of population',
38            font = dict(family = 'Droid Sans', size = 18)
39        ),
40        showgrid = False,
41        linecolor = bg_color,
42        linewidth = 2,
43        showticklabels = True,
44    ),
45    # set y-axis
46    yaxis = dict(
47        showgrid = False,
48        linecolor = bg_color,
49        linewidth = 4,
50    ),
51    # set the plot bacground color
52    plot_bgcolor = bg_color,
53    # set the hover background color
54    hoverlabel = dict(
55        bgcolor = 'rgba(75, 152, 201, 0.2)',
56        font_size = 16
57    ),
58    paper_bgcolor = bg_color,
59)
60
61fig.update_traces(marker_color = 'rgb(75, 152, 201)')
62fig['layout']['yaxis']['autorange'] = "reversed"
63
64fig.show()

Highest confirmed cases of COVID-19 per 100,000 of population

 1top_10_confirmed_df = (confirmed_per_100k_df.sort_values(
 2    by = confirmed_per_100k_df.columns[-4], ascending = False)
 3    .head(10))
 4
 5top_10 = (top_10_confirmed_df
 6          .drop(['Country Name', 'Population'], axis = 1)
 7          .set_index('Covid_Country')
 8          .T)
 9
10bg_color = 'rgba(208, 225, 242, 1.0)'
11line_color = 'rgba(75, 152, 201, 1.0)'
12grid_color = 'rgba(75, 152, 201, 0.3)'
13
14fig = go.Figure()
15for c in top_10.columns:
16    fig.add_trace(go.Scatter(
17        x = top_10.index,
18        y = top_10[c],
19        name = c,
20        hovertemplate = [f"""<b>{c}</b><br>
21<br>Cases per 100k:<br>{r[c]:,.0F}
22<extra></extra>
23""" for i,r in top_10.iterrows()]
24    ))
25
26fig.update_traces(
27     hoverinfo = 'text+name',
28    mode = 'lines'
29)
30
31fig.update_layout(
32    # Set figure title
33    title = dict(
34        text = f"""<b>Highest confirmed cases of COVID-19 per 100,000</b>\
35<br>{top_10.index[-1].strftime('%b %d %Y')}""",
36        xref = 'container',
37        yref = 'container',
38        x = 0.5,
39        y = 0.9,
40        xanchor = 'center',
41        yanchor = 'middle',
42        font = dict(family = 'Droid Sans', size = 28)
43    ),
44    # set legend
45    legend = dict(
46        orientation = 'v',
47        traceorder = 'normal',
48        font_size = 12,
49        x = 0.05,
50        y = 1.0,
51        xanchor = "left",
52        yanchor = "top"
53    ),
54    # set x-axis
55    xaxis = dict(
56        title = 'Date',
57        linecolor = line_color,
58        linewidth = 2,
59        gridcolor = grid_color,
60        showticklabels = True,
61        ticks = 'outside',
62    ),
63    # set y-axis
64    yaxis = dict(
65        title = 'Cases of COVID-19 per 100.000 of population',
66        rangemode = 'tozero',
67        linecolor = line_color,
68        linewidth = 2,
69        gridcolor = grid_color,
70        showticklabels = True,
71        ticks = 'outside',
72    ),
73    # set the plot background color
74    plot_bgcolor = bg_color,
75    paper_bgcolor = bg_color,
76)
77
78fig.show()

Changes in countries with highest confirmed cases of COVID-19 per 100,000 of population

Countries with highest deaths per 100,000 of population

Similar table and charts are created for the Deaths from COVID-19 data.

Table: The top 20 countries with highest deaths per 100,000 from COVID-19 on November 12 2020

Rank	Country	Population	Deaths per 100k	Total deaths
1	Belgium	11,484,055	154.9	17,792
2	Peru	32,510,453	112.4	36,544
3	Italy	60,297,396	106.2	64,036
4	Spain	47,076,781	101.2	47,624
5	North Macedonia	2,083,459	100.6	2,096
6	Bosnia and Herzegovina	3,301,000	99.9	3,298
7	Slovenia	2,087,946	97.8	2,041
8	United Kingdom	66,834,405	95.9	64,123
9	Moldova	2,657,637	95.8	2,547
10	US	328,239,523	90.7	297,818
11	Argentina	44,938,712	90.5	40,668
12	Mexico	127,575,529	89.1	113,704
13	Czechia	10,669,709	88.6	9,450
14	France	67,059,887	86.0	57,671
15	Brazil	211,049,527	85.8	181,123
16	Chile	18,952,038	83.6	15,846
17	Armenia	2,957,731	83.2	2,462
18	Bulgaria	6,975,761	80.7	5,626
19	Ecuador	17,373,662	79.9	13,874
20	Panama	4,246,439	78.4	3,331

Highest deaths from COVID-19 per 100,000 of population

Changes in countries with highest deaths from COVID-19 per 100,000 of population

Conclusion

Calculations can be applied to a whole dataset easily with Pandas, such as converting the total numbers of COVID-19 cases to numbers relative to a standard number. The challenge here was getting the population, which is available from the World Data Bank, but the names for countries are inconsistent. Once these are mapped to the countries, the population is merged in for each country. The div and mul dataframe functions are used to adjust the numbers relative to 100,000 of the population. The countries with highest relative numbers are not necessarily the same countries with the highest absolute numbers.

Plotly is an open-source graphing library for Python that produces interactive charts.

Add data with a secondary y axis

Eric Callanan — Sunday, 06 Dec 2020

Display two sets of data on the same chart when the data ranges are different, such as the confirmed cases of COVID-19 and deaths from COVID-19. COVID-19 is the disease caused by a new coronavirus (SARS-CoV-2) that the World Health Organisation (WHO) declared a pandemic in March 2020. This article will show how to create an interactive line chart with plotly, showing the changes in confirmed cases and deaths over time.

The data used in this article is retrieved from the Johns Hopkins University who have made the data available on GitHub. More information about COVID-19 and the coronavirus is available from Coronavirus disease (COVID-19) advice for the public. Plotly is an open-source graphing library for Python that produces interactive charts.

Load the data into a Pandas dataframe

The data is available on John Hopkins GitHub page. There are two sets of data to load the global confirmed cases and the global deaths.

time_series_covid19_deaths_global.csv
time_series_covid19_confirmed_global.csv

The data can be loaded directly from the GitHub page as in the code below. The data is cleaned to remove unwanted fields and group the data by country with the following function.

 1def load_clean_data(csv_path):
 2    df = pd.read_csv(csv_path)
 3
 4    # 1. Drop unwanted columns
 5    df.drop(['Province/State', 'Lat', 'Long'], axis = 1, inplace = True)
 6
 7    # 2. Group by Country
 8    df = df.groupby('Country/Region').sum()
 9
10    # 3. Transpose data to put dates in a single column
11    df = df.T
12    df.index = pd.to_datetime(df.index)
13
14    # 4. Remove the column name
15    df.rename_axis(None, axis = 1, inplace = True)
16
17    return df

Load the data for Confirmed cases and the data for Deaths from COVID-19.

 1
 2confirmed_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')
 3confirmed_df.shape
 4"""
 5(314, 191)
 6"""
 7
 8confirmed_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
 9"""
10            Afghanistan  Albania  Algeria  Yemen  Zambia  Zimbabwe
112020-01-22            0        0        0      0       0         0
122020-01-23            0        0        0      0       0         0
132020-01-24            0        0        0      0       0         0
142020-11-28        45966    36790    81212   2160   17589      9822
152020-11-29        46215    37625    82221   2177   17608      9822
162020-11-30        46498    38182    83199   2191   17647      9950
17"""
18
19
20deaths_df = load_clean_data('https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv')
21deaths_df.shape
22"""
23(314, 191)
24"""
25
26deaths_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
27"""
28            Afghanistan  Albania  Algeria  Yemen  Zambia  Zimbabwe
292020-01-22            0        0        0      0       0         0
302020-01-23            0        0        0      0       0         0
312020-01-24            0        0        0      0       0         0
322020-11-28         1752      787     2393    615     357       275
332020-11-29         1763      798     2410    617     357       275
342020-11-30         1774      810     2431    619     357       276
35"""

Global confirmed cases of COVID-19

Show the total global number of confirmed cases of COVID-19 as the numbers changes over time. Plotly is used to display an interactive line chart. An annotation is added to show the current total number on the latest date.

 1def hov_text(ary_x, ary_y, name):
 2    '''Create the hover text for each data point
 3
 4    Keyword arguments:
 5    ary_x    -- list of all the x data points
 6    ary_y    -- list of all the y data points
 7    name     -- name to display before each value
 8
 9    return: A list of hover texts for the selected data
10            with format to display
11    '''
12    txt = [f'''
13<b>{ary_x[i].strftime('%B %d %Y')}</b><br><br>
14{name}: <b>{ary_y[i]:,.0F}</b><br>
15<extra></extra>
16''' for i in range(len(ary_x))]
17    return txt

 1total_confirmed = confirmed_df.sum(axis = 1)
 2
 3bg_color = 'rgba(208, 225, 242, 1.0)'
 4line_color = 'rgba(75, 152, 201, 1.0)'
 5grid_color = 'rgba(75, 152, 201, 0.3)'
 6
 7fig = go.Figure()
 8fig.add_trace(go.Scatter(x = total_confirmed.index,
 9                         y = total_confirmed,
10                         name = 'Confirmed',
11                         hovertemplate = hov_text(total_confirmed.index,
12                                                  total_confirmed,
13                                                  'Confirmed Cases')))
14
15fig.update_traces(
16    hoverinfo = 'text+name',
17    mode = 'lines'
18)
19
20fig.update_layout(
21    # Set figure title
22    title = dict(
23        text = f'<b>Global confirmed cases from COVID-19</b>',
24        xref = 'container',
25        yref = 'container',
26        x = 0.5,
27        y = 0.9,
28        xanchor = 'center',
29        yanchor = 'middle',
30        font = dict(family = 'Droid Sans', size = 28)
31    ),
32    # set legend
33    legend = dict(
34        orientation = 'h',
35        traceorder = 'normal',
36        font_size = 12,
37        x = 0.0,
38        y = -0.3,
39        xanchor = 'left',
40        yanchor = 'top'
41    ),
42    # set x-axis
43    xaxis = dict(
44        title = 'Date',
45        linecolor = line_color,
46        linewidth = 2,
47        gridcolor = grid_color,
48        showticklabels = True,
49        ticks = 'outside',
50    ),
51    # set y-axis
52    yaxis = dict(
53        title = 'Number of confirmed cases',
54        rangemode = 'tozero',
55        linecolor = line_color,
56        linewidth = 2,
57        gridcolor = grid_color,
58        showticklabels = True,
59        ticks = 'outside',
60    ),
61    showlegend = False,
62    # set the plot bacground color
63    plot_bgcolor = bg_color,
64    paper_bgcolor = bg_color,
65)
66
67# Add text for current date and total
68fig.add_annotation(
69    text = f"{total_confirmed.index[-1].strftime('%B %d')}<BR> {total_confirmed[-1]:,.0F}",
70    xref = 'paper',
71    yref = 'paper',
72    x = 0.5,
73    y = 1.0,
74    showarrow = False,
75    bgcolor = 'rgba(180, 210, 233, 1.0)',
76    borderpad = 10,
77    font = dict(family = 'Droid Sans', size = 28)
78)
79
80fig.show()

Global confirmed cases from COVID-19

Global deaths from COVID-19

Show the total global number of deaths from COVID-19 as the numbers changes over time. The code is similar to the last chart except the data source and titles.

 1total_deaths = deaths_df.sum(axis = 1)
 2
 3bg_color = 'rgba(208, 225, 242, 1.0)'
 4line_color = 'rgba(75, 152, 201, 1.0)'
 5grid_color = 'rgba(75, 152, 201, 0.3)'
 6
 7fig = go.Figure()
 8fig.add_trace(go.Scatter(x = total_deaths.index,
 9                         y = total_deaths,
10                         name = 'Deaths',
11                         hovertemplate = hov_text(total_deaths.index,
12                                                  total_deaths,
13                                                 'Deaths')))
14
15fig.update_traces(
16    hoverinfo = 'text+name',
17    mode = 'lines'
18)
19
20fig.update_layout(
21    # Set figure title
22    title = dict(
23        text = f'<b>Global deaths from COVID-19</b>',
24        xref = 'container',
25        yref = 'container',
26        x = 0.5,
27        y = 0.9,
28        xanchor = 'center',
29        yanchor = 'middle',
30        font = dict(family = 'Droid Sans', size = 28)
31    ),
32    # set legend
33    legend = dict(
34        orientation = 'h',
35        traceorder = 'normal',
36        font_size = 12,
37        x = 0.0,
38        y = -0.3,
39        xanchor = 'left',
40        yanchor = 'top'
41    ),
42    # set x-axis
43    xaxis = dict(
44        title = 'Date',
45        linecolor = line_color,
46        linewidth = 2,
47        gridcolor = grid_color,
48        showticklabels = True,
49        ticks = 'outside',
50    ),
51    # set y-axis
52    yaxis = dict(
53        title = 'Number of deaths',
54        rangemode = 'tozero',
55        linecolor = line_color,
56        linewidth = 2,
57        gridcolor = grid_color,
58        showticklabels = True,
59        ticks = 'outside',
60    ),
61    showlegend = False,
62    # set the plot bacground color
63    plot_bgcolor = bg_color,
64    paper_bgcolor = bg_color,
65)
66
67# Add text for current date and total
68fig.add_annotation(
69    text = f"{total_deaths.index[-1].strftime('%B %d')}<BR> {total_deaths[-1]:,.0F}",
70    xref = 'paper',
71    yref = 'paper',
72    x = 0.5,
73    y = 1.0,
74    showarrow = False,
75    bgcolor = 'rgba(180, 210, 233, 1.0)',
76    borderpad = 10,
77    font = dict(family = 'Droid Sans', size = 28)
78)
79
80fig.show()

Global deaths from COVID-19

Show Confirmed cases and deaths from COVID-19 on a single graph

Just showing the data for confirmed cases and deaths on the same chart does not work well as there is such a difference in scale between the number ranges. The number of deaths looks flat relative to the total number of cases, but these numbers are growing.

Global deaths from COVID-19

One solution is to plot the Deaths data using a second y axis so that each set of data will use the full vertical space available. These y axes are also marked with color associated with the data that uses the axis.

The code to generate this chart is wrapped up in a function that takes the two dataframes and the chart title and returns a plotly graph figure.

 1def plot_confirmed_and_deaths(df1, df2, region):
 2    bg_color = 'rgba(208, 225, 242, 1.0)'
 3    line_color = 'rgba(75, 152, 201, 1.0)'
 4    grid_color = 'rgba(75, 152, 201, 0.3)'
 5
 6    # Linechart with secondary y axis
 7    fig = make_subplots(specs=[[{"secondary_y": True}]])
 8
 9    fig.add_trace(
10        go.Scatter(
11            x = df1.index,
12            y = df1,
13            name = 'Confirmed Cases',
14            hovertemplate = hov_text(df1.index,
15                                     df1,
16                                     'Confirmed Cases')),
17        secondary_y = False)
18
19    fig.add_trace(
20        go.Scatter(
21            x = df2.index,
22            y = df2,
23            name = 'Deaths',
24            line = dict(dash = 'dashdot'),
25            hovertemplate = hov_text(df2.index,
26                                     df2,
27                                     'Deaths')),
28        secondary_y = True)
29
30    fig.update_layout(
31        # Set figure title
32        title = dict(
33            text = f'Confirmed cases and deaths from COVID-19<BR><b>{region}</b>',
34            xref = 'container',
35            yref = 'container',
36            x = 0.5,
37            y = 0.9,
38            xanchor = 'center',
39            yanchor = 'middle',
40            font = dict(family = 'Droid Sans', size = 28)
41        ),
42        # set legend
43        legend = dict(
44            orientation = 'v',
45            traceorder = 'normal',
46            font_size = 12,
47            x = 0.1,
48            y = 0.9,
49            xanchor = "left",
50            yanchor = "top"
51        ),
52        # set x-axis
53        xaxis = dict(
54            title = 'Date',
55            linecolor = line_color,
56            linewidth = 2,
57            gridcolor = grid_color,
58            showticklabels = True,
59            ticks = 'outside',
60        ),
61        # set y-axis
62        yaxis = dict(
63            title = 'Number of Confirmed Cases',
64            color = 'rgba(80, 80, 250, 1.0)',
65            rangemode = 'tozero',
66            linecolor = 'rgba(80, 80, 250, 1.0)',
67            linewidth = 2,
68            gridcolor = 'rgba(80, 80, 250, 0.3)',
69            showticklabels = True,
70            ticks = 'outside'
71        ),
72        # set the plot bacground color
73        plot_bgcolor = bg_color,
74        paper_bgcolor = bg_color,
75    )
76
77    # set the secondary y-axis
78    fig.update_yaxes(
79        title_text = 'Number of Deaths',
80        color = 'rgba(237, 34, 13, 1.0)',
81        range = [0, (df2[-1] * 1.08)],
82        linecolor = 'rgba(237, 34, 13, 1.0)',
83        linewidth = 2,
84        gridcolor = 'rgba(237, 34, 13, 0.3)',
85        showticklabels = True,
86        ticks = 'outside',
87        secondary_y = True
88    )
89
90    return fig

This plot_confirmed_and_deaths function is used to create the line plot showing both confirmed cases and deaths from COVID-19 globally.

1df1 = confirmed_df.sum(axis = 1)
2df2 = deaths_df.sum(axis = 1)
3
4f = plot_confirmed_and_deaths(df1, df2, 'Global')
5f.show()

Global deaths from COVID-19

Display data for selected country

The same function can be used to create a chart for any country.

1country = 'US'
2f = plot_confirmed_and_deaths(
3    confirmed_df[country],
4    deaths_df[country],
5    country)
6f.show()

Global deaths from COVID-19 in United States

Global deaths from COVID-19 in United Kingdom

Global deaths from COVID-19 in Ireland

Global deaths from COVID-19 in Brazil

Conclusion

Pandas dataframe is great for loading data from csv files and filtering the data to an area of interest quickly. Plotly is great to create interactive charts. It is informative to see the rise in the number of deaths from COVID-19 in relation to the rise in the total number of cases detected. This relationship is easier to see when the data can be shown on a single graph. A secondary y-axis is used to display the deaths as the scales are so different. All of the loading, cleaning and creation of the chart can be wrapped up in a couple of functions to create a pipeline to generate these charts for countries of interest. The interactive charts allows the viewer to zoom in on a particular date range and select or deselect data to focus on an area of interest.

Bar chart race showing changes in Covid-19 deaths

Eric Callanan — Sunday, 29 Nov 2020

COVID-19 is the disease caused by a new coronavirus called SARS-CoV-2. World Health Organisation (WHO) first learned of the virus on 31 December 2019. The WHO declared the coronavirus outbreak a pandemic in March 2020. This article will show how to create a bar chart race depicting the countries with the highest number of deaths from coronavirus as they change from day to day.

Retrieve the data and load into dataframe

The data is available on John Hopkins GitHub page. The data for the daily deaths from corona virus is in time_series_covid19_deaths_global.csv file. This file can either be downloaded and loaded into a dataframe or it can be loaded directly from GitHub as in the code below. Load and review the data.

 1# raw csv files from Github
 2deaths_path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv'
 3deaths_df = pd.read_csv(deaths_path)
 4
 5deaths_df.shape
 6"""
 7(312, 171)
 8"""
 9
10deaths_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,4,5,6,-3,-2,-1]]
11"""
12    Province/State  Country/Region        Lat       Long  1/22/20  1/23/20  1/24/20  11/23/20  11/24/20  11/25/20  
130              NaN     Afghanistan  33.939110  67.709953        0        0        0      1695      1712      1725  
141              NaN         Albania  41.153300  20.168300        0        0        0       716       735       743  
152              NaN         Algeria  28.033900   1.659600        0        0        0      2294      2309      2329  
163              NaN         Andorra  42.506300   1.521800        0        0        0        76        76        76  
17267            NaN  Western Sahara  24.215500 -12.885800        0        0        0         1         1         1  
18268            NaN           Yemen  15.552727  48.516388        0        0        0       609       609       611  
19269            NaN          Zambia -13.133897  27.849332        0        0        0       357       357       357  
20270            NaN        Zimbabwe -19.015438  29.154857        0        0        0       273       274       274  
21"""

Group the data by Country/Region

The data in the Country/Region is further broken down by Province/State for some countries such as China or United Kingdom. The data is grouped by Country/Region summing up the number to provide a single total for each country.

 1deaths_df[deaths_df['Country/Region'] == 'China'].head()
 2
 3"""
 4   Province/State Country/Region      Lat      Long  1/22/20  1/23/20  1/24/20  1/25/20  1/26/20  1/27/20  ...  11/16/20  11/17/20  11/18/20  11/19/20  11/20/20  11/21/20  11/22/20  11/23/20  11/24/20  11/25/20  
 558          Anhui          China  31.8257  117.2264        0        0        0        0        0        0  ...         6         6         6         6         6         6         6         6         6         6  
 659        Beijing          China  40.1824  116.4142        0        0        0        0        0        1  ...         9         9         9         9         9         9         9         9         9         9  
 760      Chongqing          China  30.0572  107.8740        0        0        0        0        0        0  ...         6         6         6         6         6         6         6         6         6         6  
 861         Fujian          China  26.0789  117.9874        0        0        0        0        0        0  ...         1         1         1         1         1         1         1         1         1         1  
 962          Gansu          China  35.7518  104.2861        0        0        0        0        0        0  ...         2         2         2         2         2         2         2         2         2         2  
10"""
11
12# Group by Country
13country_deaths_df = deaths_df.groupby(by = ['Country/Region'],
14                                      as_index = False).sum()
15country_deaths_df.shape
16"""
17(191, 312)
18"""

Map countries to specific colors

It is necessary to get a list of all of the top ten counties as they change over time in the dataset.

 1df = country_deaths_df.drop(['Lat', 'Long'], axis = 1)
 2countries = []
 3for c in df.columns[1:]:
 4    df_num = (df[['Country/Region', c]][df[c] > 0]
 5              .sort_values(by = c, ascending = False)
 6              .head(10))
 7    countries.extend(list(df_num['Country/Region']))
 8countries = list(set(countries))
 9
10len(countries)
11"""
1224
13"""

It is necessary to map a set of colors to this set of 24 countries to maintain consistent colors for each country. This is necessary as each bar chart created in the bar chart race is independent of the previous bar chart and the default would reuse the same ten colors. Display the 24 countries with their assigned color.

1cols = list(plt.cm.Dark2.colors + plt.cm.Set3.colors + plt.cm.tab10.colors)
2cols.remove(plt.cm.Set3.colors[8])
3color_dict = {x[0]:x[1] for x in zip(countries, cols)}

Colors for countries with highest deaths from Covid-19

Colors for countries with highest deaths from Covid-19

Create bar chart for top 10 countries for latest day

Extract the data for the country list on the latest date. Sort the data by the number of deaths and create a bar chart of the top ten countries with the highest number of deaths. Add labels to the end of each bar showing the number of deaths for each country and add a label for the date.

 1top_df = (sel_df.iloc[:, [-1]]
 2          .sort_values(by = sel_df.columns[-1], ascending = False)
 3          .head(10))
 4fig, ax = plt.subplots(nrows = 1,
 5                       ncols = 1,
 6                       figsize = (10, 7),
 7                       facecolor = plt.cm.Blues(.2),
 8                       tight_layout = True)
 9bars = ax.barh(y = range(1, len(top_df.index) + 1),
10               tick_label = top_df.index,
11               width = top_df.iloc[: , 0],
12               color = [color_dict[col] for col in top_df.index])
13day = pd.to_datetime(top_df.columns[0]).strftime('%b %d %Y')
14ax.set_title(f'Top ten countries with highest deaths from Covid-19 - {day}',
15             fontsize = 'xx-large',
16             fontweight = 'bold')
17ax.set_ylim(10.8, 0.2)
18ax.set_facecolor(plt.cm.Blues(.2))
19ax.tick_params(labelsize = 'medium')
20ax.grid(True, axis = 'x', color=plt.cm.Blues(0.05))
21[spine.set_visible(False) for spine in ax.spines.values()]
22
23for bar in bars:
24    width = bar.get_width()
25    ax.annotate(f'{width:,.0F}',
26                xy = (width , bar.get_y() + bar.get_height() / 2),
27                xytext = (25, 0),
28                textcoords = "offset points",
29                fontsize = 'x-large',
30                fontweight = 'bold',
31                ha = 'left',
32                va = 'center')
33
34# Add large Date in bottom right on chart
35ax.annotate(pd.to_datetime(top_df.columns[0]).strftime('%b %d'),
36            xy = (1.05, 0.1),
37            xycoords='axes fraction',
38            fontsize = 40,
39            fontweight = 'bold',
40            ha = 'right',
41            va = 'bottom')
42
43plt.show()

Top ten countries with highest deaths from Covid-19 on November 23

Top ten countries with highest deaths from Covid-19 on November 23

Transpose the data to wide format and expand the data

The data is transposed and the date is set as the index so that each row represents the data for a particular date and can be shown in a bar chart.

 1# Transpose the data for the countries of interest
 2wide_df = sel_df.T[countries].copy()
 3
 4# Remove the column name
 5wide_df.rename_axis(None, axis=1, inplace=True)
 6
 7# Set index to datetime
 8wide_df.index = pd.to_datetime([f"{x}" for x in wide_df.index])
 9
10wide_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,-3,-2,-1]]
11"""
12            United Kingdom  Netherlands   Iran  Italy      US  Brazil   Peru
132020-01-22               0            0      0      0       0       0      0
142020-01-23               0            0      0      0       0       0      0
152020-01-24               0            0      0      0       0       0      0
162020-01-25               0            0      0      0       0       0      0
172020-11-21           54721         8946  44327  49261  255946  168989  35549
182020-11-22           55120         8967  44802  49823  256866  169183  35549
192020-11-23           55327         9021  45255  50453  257779  169485  35595
202020-11-24           55935         9111  45738  51306  259925  170115  35641
21"""

Expand the data to plot a smooth transition so the bars on the chart do not jump around, but slide smoothly.

 1expanded_df = wide_df.asfreq('4h')
 2
 3expanded_df.shape
 4"""
 5(1843, 24)
 6"""
 7
 8expanded_df.iloc[-8:, [0,1,2,3,-3,-2,-1]]
 9"""
10                     United Kingdom  Netherlands     Iran    Italy        US    Brazil     Peru  
112020-11-22 20:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
122020-11-23 00:00:00         55327.0       9021.0  45255.0  50453.0  257779.0  169485.0  35595.0  
132020-11-23 04:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
142020-11-23 08:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
152020-11-23 12:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
162020-11-23 16:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
172020-11-23 20:00:00             NaN          NaN      NaN      NaN       NaN       NaN      NaN  
182020-11-24 00:00:00         55935.0       9111.0  45738.0  51306.0  259925.0  170115.0  35641.0  
19"""

Create ranking dataset to rank the countries in each row. The rank order is used to position the bars on the bar chart so that the color remains consistent for each country.

 1rank_df = expanded_df.rank(axis = 1, method = 'first', ascending = False)
 2
 3rank_df.shape
 4"""
 5(1843, 24)
 6"""
 7
 8rank_df.iloc[-8:, [0,1,2,3,-3,-2,-1]]
 9"""
10                     United Kingdom  Netherlands  Iran  Italy   US  Brazil  Peru  
112020-11-22 20:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
122020-11-23 00:00:00             5.0         16.0   8.0    6.0  1.0     2.0  11.0  
132020-11-23 04:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
142020-11-23 08:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
152020-11-23 12:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
162020-11-23 16:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
172020-11-23 20:00:00             NaN          NaN   NaN    NaN  NaN     NaN   NaN  
182020-11-24 00:00:00             5.0         16.0   8.0    6.0  1.0     2.0  11.0  
19"""

Interpolate the results to create a smooth transition from one day to the next. Add incremental values every 4 hours between the given values.

 1expanded_df = expanded_df.interpolate()
 2rank_df = rank_df.interpolate()
 3
 4expanded_df.iloc[-8:, [0,1,2,3,-3,-2,-1]]
 5"""
 6                     United Kingdom  Netherlands     Iran         Italy             US         Brazil          Peru
 72020-11-22 20:00:00    55292.500000       9012.0  45179.5  50348.000000  257626.833333  169434.666667  35587.333333
 82020-11-23 00:00:00    55327.000000       9021.0  45255.0  50453.000000  257779.000000  169485.000000  35595.000000
 92020-11-23 04:00:00    55428.333333       9036.0  45335.5  50595.166667  258136.666667  169590.000000  35602.666667
102020-11-23 08:00:00    55529.666667       9051.0  45416.0  50737.333333  258494.333333  169695.000000  35610.333333
112020-11-23 12:00:00    55631.000000       9066.0  45496.5  50879.500000  258852.000000  169800.000000  35618.000000
122020-11-23 16:00:00    55732.333333       9081.0  45577.0  51021.666667  259209.666667  169905.000000  35625.666667
132020-11-23 20:00:00    55833.666667       9096.0  45657.5  51163.833333  259567.333333  170010.000000  35633.333333
142020-11-24 00:00:00    55935.000000       9111.0  45738.0  51306.000000  259925.000000  170115.000000  35641.000000
15"""

Remove duplicate ranks so that one country does not hide another when they have the exact same rank.

1# Remove any duplicate ranks from the same row
2while ((rank_df.where(~rank_df.apply(pd.Series.duplicated, axis=1), -1)) == -1).any(axis = None):
3    rank_df = rank_df.where(~rank_df.apply(pd.Series.duplicated, axis=1), rank_df*1.01)

Display a sample bar chart from the expanded dataframe using the ranking to order the bars. Add annotations to each bar to display the number of deaths. Add a large display of the current day, which will make the animation more understandable.

 1fig, ax = plt.subplots(nrows = 1,
 2                       ncols = 1,
 3                       figsize = (10, 7),
 4                       facecolor = plt.cm.Blues(.2),
 5                       tight_layout = True)
 6p = expanded_df.columns.map(len).max()
 7bar_num = 10
 8i = 1117
 9sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
10bars = ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
11               tick_label = [x.rjust(p, ' ') for x in sel_df.columns],
12               width = sel_df.iloc[i],
13               color = [color_dict[col] for col in sel_df.columns],
14               alpha = 0.8)
15
16plt.setp(ax.get_xticklabels(), fontsize='small')
17plt.setp(ax.get_yticklabels(), fontsize='medium', fontfamily = 'monospace')
18cur_day = expanded_df.index[i].strftime('%Y-%m')
19ax.set_title(f'Countries with highest deaths from Covid-19 on {cur_day}',
20             fontsize = 'xx-large',
21             fontweight = 'bold')
22ax.set_ylim(10.8, 0.2)
23ax.set_facecolor(plt.cm.Blues(.2))
24ax.grid(True, axis = 'x', color=plt.cm.Blues(.05))
25ax.set_axisbelow(True)
26[spine.set_visible(False) for spine in ax.spines.values()]
27
28for bar in bars:
29    width = bar.get_width()
30    ax.annotate(f'{width:,.0F}',
31                xy = (width , bar.get_y() + bar.get_height() / 2),
32                xytext = (25, 0),
33                textcoords = "offset points",
34                fontsize = 'large',
35                fontweight = 'bold',
36                ha = 'left',
37                va = 'center')
38
39plt.show()

Countries with highest deaths from Covid-19 on sample date

Countries with highest deaths from Covid-19 on sample date

Display a sample of the bar charts for random dates to ensure the bar charts are displaying correctly with the correct ranking and countries maintain their color.

 1sample_num = 5
 2d_df = expanded_df.sample(n = sample_num, random_state = 12).sort_index()
 3r_df = rank_df.loc[d_df.index]
 4
 5fig, axs = plt.subplots(nrows = 1,
 6                        ncols = sample_num,
 7                        figsize = (15, 5),
 8                        facecolor = plt.cm.Blues(.2),
 9                        tight_layout = True)
10
11for i, ax in enumerate(axs.flatten()):
12    sel_df = d_df.iloc[:, list(r_df.iloc[i] <= bar_num)]
13    bars = ax.barh(y = r_df.iloc[:, list(r_df.iloc[i] <= bar_num)].iloc[i],
14                   tick_label = sel_df.columns,
15                   width = sel_df.iloc[i],
16                   color = [color_dict[col] for col in sel_df.columns],
17                   alpha = 0.8)
18
19    cur_day = d_df.index[i].strftime('%Y-%m-%d')
20    ax.set_title(f'{cur_day}',
21                 fontsize = 'large',
22                 fontweight = 'bold')
23    ax.set_ylim(10.8, 0.2)
24    ax.set_facecolor(plt.cm.Blues(.2))
25    ax.tick_params(labelsize = 'xx-small')
26    ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
27    ax.set_axisbelow(True)
28    [spine.set_visible(False) for spine in ax.spines.values()]
29
30    for bar in bars:
31        width = bar.get_width()
32        ax.annotate(f'{width:,.0F}',
33                    xy = (width , bar.get_y() + bar.get_height() / 2),
34                    xytext = (5, 0),
35                    textcoords = "offset points",
36                    fontsize = 'small',
37                    fontweight = 'bold',
38                    ha = 'left',
39                    va = 'center')
40plt.show()

Sample of bar charts for countries with highest deaths from Covid-19

Sample of bar charts for countries with highest deaths from Covid-19

Create the animation

The animation is created by generating a bar chart for each of the rows in the expanded dataframe using the rankings dataframe for bar positions. These images are then combined into sequence using the FuncAnimation function in Matplotlib.

 1def update(i):
 2    ax.clear()
 3
 4    p = expanded_df.columns.map(len).max()
 5    bar_num = 10
 6    sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
 7    bars = ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
 8                   tick_label = [x.rjust(p, ' ') for x in sel_df.columns],
 9                   width = sel_df.iloc[i],
10                   color = [color_dict[col] for col in sel_df.columns],
11                   alpha = 0.8)
12    plt.setp(ax.get_xticklabels(), fontsize='x-small')
13    plt.setp(ax.get_yticklabels(), fontsize='small', fontfamily = 'monospace')
14
15    cur_day = expanded_df.index[i].strftime('%Y-%b-%d')
16    ax.set_title(f'Deaths from Covid-19 - {cur_day}',
17                 fontsize = 'x-large',
18                 fontweight = 'bold',
19                 loc = 'center')
20    ax.set_ylim(10.8, 0.2)
21    ax.set_facecolor(plt.cm.Blues(.2))
22    ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
23    ax.set_axisbelow(True)
24    [spine.set_visible(False) for spine in ax.spines.values()]
25
26    for bar in bars:
27        width = bar.get_width()
28        ax.annotate(f'{width:,.0F}',
29                    xy = (width , bar.get_y() + bar.get_height() / 2),
30                    xytext = (10, 0),
31                    textcoords = "offset points",
32                    fontsize = 'small',
33                    ha = 'left',
34                    va = 'center')
35
36    # Add large Date in bottom right on chart
37    ax.annotate(expanded_df.index[i].strftime('%b %d'),
38                xy = (1.25, 0.1),
39                xycoords='axes fraction',
40                fontsize = 40,
41                fontweight = 'bold',
42                ha = 'right',
43                va = 'bottom')
44
45fig, ax = plt.subplots(figsize = (8, 4),
46                       facecolor = plt.cm.Blues(.2),
47                       dpi = 50,
48                       tight_layout = True)
49
50covid_anim = anim.FuncAnimation(
51    fig = fig,
52    func = update,
53    frames = len(expanded_df),
54    interval = 100)
55
56
57covid_anim.save('COVID_bar_chart_race_2020.gif')

Bar chart race for Countries with highest deaths from Covid-19

Countries with highest deaths from Covid-19

The animation can also be converted to HTML5 video or saved as MP4.

1html = covid_anim.to_html5_video()
2
3covid_anim.save('COVID_bar_chart_race_2020.gif')

MP4 version available here - MP4 bar chart race for Countries with highest deaths from Covid-19

Conclusion

A bar chart race can be an effective way to visualise the increase in deaths from COVID-19 in different contries. There is a bit of work in creating the animated chart, which has been laid out in this article. The concept is straight forward, simply sequence through the data and display a bar chart for each day and the Matplotlib FuncAnimation is great for creating the animation. Some of the things to keep in mind are to implement intermediary data to give a smooth transition and to maintain consistent color for each country.

Display world map with country data

Eric Callanan — Sunday, 22 Nov 2020

Geographical maps can be a great way to present a global view of where data applies to different countries and regions. There are a number of third-party tools and libraries that can be used to help create these maps such as Basemap, Cartopy as well as Geopandas and Geoplot. This article will show how to use Geopandas to display a world map showing countries with higher Under Five Mortality Rates in darker colors.

Load the data

Details on downloading and loading the Under Five Mortality Rate data into a dataframe is described in "Pandas - Load data from Excel file and Display Chart". The following loads the data from a local file and filters on the Median values for each country. The ISO.Code field contains the international standard three-letter code for countries in the world.

 1# Load the excel worksheet into a dataframe
 2u5mr_df = pd.read_excel(
 3    "/tmp/data/Under-five-mortality-rate_2020.xlsx",
 4    engine = "openpyxl",
 5    sheet_name = 'Country estimates (both sexes)',
 6    header = 14,
 7    nrows = 585)
 8
 9# Rename 'Uncertainty.Bounds*' column to 'Uncertainty.Bounds'
10u5mr_df = u5mr_df.rename(columns={'Uncertainty.Bounds*': 'Uncertainty.Bounds'})
11
12# Convert year column names to datetime
13u5mr_df.columns = [x[:-2] if x.endswith('.5') else x for x in u5mr_df.columns]# u5mr_df.columns = [pd.to_datetime(f'{x[:-2]}-12-31') if x.endswith('.5') else x for x in u5mr_df.columns]
14
15# Filter to the Median values
16u5mr_med_df = u5mr_df[u5mr_df['Uncertainty.Bounds'] == 'Median']
17
18# Review the data
19u5mr_med_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,12,-3,-2,-1]]
20
21"""
22(195, 73)
23    ISO.Code Country.Name Uncertainty.Bounds  1950        1959       2017       2018       2019
241        AFG  Afghanistan             Median   NaN         NaN  64.940759  62.541196  60.269399
254        ALB      Albania             Median   NaN         NaN   9.418052   9.525133   9.682407
267        DZA      Algeria             Median   NaN  240.344776  24.319482  23.805926  23.256168
2710       AND      Andorra             Median   NaN         NaN   3.218925   3.085839   2.966929
28574      VNM     Viet Nam             Median   NaN         NaN  20.843125  20.405423  19.935167
29577      YEM        Yemen             Median   NaN         NaN  56.966430  58.460003  58.356138
30580      ZMB       Zambia             Median   NaN  208.929172  64.337901  63.294182  61.663465
31583      ZWE     Zimbabwe             Median   NaN  155.256789  58.234924  55.856832  54.612967
32"""

Load Geopandas

Geopandas and Geoplot are required to create these charts. These can be installed with pip install geopandas and pip install geoplot. Geopandas requires the Geospatial Data Abstraction Library (GDAL) library to be installed - good instructions here on How to install GDAL. Geopandas has a dataset for the contours of all the countries, as well as some information like population estimates, that can be used to plot the world map and colour countries based on population. The following code loads the 'naturalearth_lowres' dataset and plots a map with countries colored based on country population, the darker greens represent higher population. This shows that China and India dominate this map.

 1import geopandas
 2import geoplot
 3import mapclassify
 4
 5world = geopandas.read_file(
 6    geopandas.datasets.get_path('naturalearth_lowres')
 7)
 8fig, ax = plt.subplots(figsize = (10,4), facecolor = plt.cm.Blues(.2))
 9fig.suptitle('Country Populations',
10             fontsize = 'xx-large',  
11             fontweight = 'bold')
12ax.set_facecolor(plt.cm.Blues(.2))
13world.plot(column = 'pop_est',
14           cmap = 'Greens',
15           ax = ax,
16           legend = True)
17
18plt.show()

Map showing country color based on population

Map showing country color based on population

Explore 'naturalearth_lowres' dataset

The 'naturalearth_lowres' dataset is loaded into a Geodataframe, which is a specialised form of a Pandas Dataframe. So all of the functions and properties of Dataframe apply to geodataframe. A geodataframe always contains one geoseries called geometry that holds spatial status.

In the world geodataframe loaded from naturalearth_lowres dataset there are 6 columns and 177 rows.

 1type(world)
 2"""
 3<class 'geopandas.geodataframe.GeoDataFrame'>
 4"""
 5
 6world.shape
 7"""
 8(177, 6)
 9"""
10
11world.columns
12"""
13Index(['pop_est', 'continent', 'name', 'iso_a3', 'gdp_md_est', 'geometry'], dtype='object')
14"""
15
16world.index
17"""
18RangeIndex(start=0, stop=177, step=1)
19"""
20
21world.iloc[[0,1,2,3,-4,-3,-2,-1], :]
22"""
23      pop_est      continent                 name iso_a3 gdp_md_est                                           geometry  
240      920938        Oceania                 Fiji    FJI     8374.0  MULTIPOLYGON (((180.00000 -16.06713, 180.00000...  
251    53950935         Africa             Tanzania    TZA   150600.0  POLYGON ((33.90371 -0.95000, 34.07262 -1.05982...  
262      603253         Africa            W. Sahara    ESH      906.5  POLYGON ((-8.66559 27.65643, -8.66512 27.58948...  
273    35623680  North America               Canada    CAN  1674000.0  MULTIPOLYGON (((-122.84000 49.00000, -122.9742...  
28173    642550         Europe           Montenegro    MNE    10610.0  POLYGON ((20.07070 42.58863, 19.80161 42.50009...  
29174   1895250         Europe               Kosovo    -99    18490.0  POLYGON ((20.59025 41.85541, 20.52295 42.21787...  
30175   1218208  North America  Trinidad and Tobago    TTO    43570.0  POLYGON ((-61.68000 10.76000, -61.10500 10.890...  
31176  13026129         Africa             S. Sudan    SSD    20880.0  POLYGON ((30.83385 3.50917, 29.95350 4.17370, ...  
32"""

Show countries with population greater than 200 million

 1world[(world.pop_est > 200000000)]
 2"""
 3        pop_est      continent                      name iso_a3  gdp_md_est                                           geometry  
 44     326625791  North America  United States of America    USA  18560000.0  MULTIPOLYGON (((-122.84000 49.00000, -120.0000...  
 58     260580739           Asia                 Indonesia    IDN   3028000.0  MULTIPOLYGON (((141.00021 -2.60015, 141.01706 ...  
 629    207353391  South America                    Brazil    BRA   3081000.0  POLYGON ((-53.37366 -33.76838, -53.65054 -33.2...  
 798   1281935911           Asia                     India    IND   8721000.0  POLYGON ((97.32711 28.26158, 97.40256 27.88254...  
 8102   204924861           Asia                  Pakistan    PAK    988200.0  POLYGON ((77.83745 35.49401, 76.87172 34.65354...  
 9139  1379302771           Asia                     China    CHN  21140000.0  MULTIPOLYGON (((109.47521 18.19770, 108.65521 ...  
10"""

Plot the relative population of countries in Africa. This is done by filtering the dataframe on the continent of 'Africa'

 1fig, ax = plt.subplots(figsize = (6,5), facecolor = plt.cm.Blues(.2))
 2fig.suptitle('Africa Populations',
 3             fontsize = 'xx-large',  
 4             fontweight = 'bold')
 5ax.set_facecolor(plt.cm.Blues(.2))
 6ax = world[world.continent == 'Africa'].plot(
 7    column = 'pop_est',
 8    cmap = 'Greens',
 9    ax = ax,
10    legend = True)
11
12plt.show()

Map of Africa showing country color based on population

Map of Africa showing country color based on population

Display country color based on Under Five Mortality Rates in 2019

A global map of the Under Five Mortality Rates can be plotted by merging the Under five mortality rates dataframe with the world geodataframe. The merge is done on the ISO country code. There is a discrepancy in the dataframes in that there are 195 countries in the Under Five Mortality Rates dataframe and only 177 countries in the world geodataframe.

 1# Countries in ufmr dataframe that are not in the world geodataframe
 2u5mr_med_df[~u5mr_med_df['ISO.Code'].isin(list(world['iso_a3']))][['ISO.Code', 'Country.Name']]
 3"""
 4    ISO.Code                      Country.Name
 510       AND                           Andorra
 616       ATG               Antigua and Barbuda
 737       BHR                           Bahrain
 843       BRB                          Barbados
 985       CPV                        Cabo Verde
10112      COM                           Comoros
11118      COK                      Cook Islands
12151      DMA                          Dominica
13187      FRA                            France
14208      GRD                           Grenada
15271      KIR                          Kiribati
16313      MDV                          Maldives
17319      MLT                             Malta
18322      MHL                  Marshall Islands
19328      MUS                         Mauritius
20334      FSM  Micronesia (Federated States of)
21337      MCO                            Monaco
22358      NRU                             Nauru
23379      NIU                              Niue
24382      NOR                            Norway
25391      PLW                             Palau
26436      KNA             Saint Kitts and Nevis
27439      LCA                       Saint Lucia
28442      VCT  Saint Vincent and the Grenadines
29445      WSM                             Samoa
30448      SMR                        San Marino
31451      STP             Sao Tome and Principe
32463      SYC                        Seychelles
33469      SGP                         Singapore
34526      TON                             Tonga
35541      TUV                            Tuvalu
36"""

Show countries that are in the world geodataframe, but not in the Under Five Mortality Rates dataframe.

 1# list all countries in World not in ufmr
 2world[~world['iso_a3'].isin(list(u5mr_med_df['ISO.Code']))][['iso_a3', 'name']]
 3"""
 4    iso_a3                    name
 52      ESH               W. Sahara
 620     FLK            Falkland Is.
 721     -99                  Norway
 822     GRL               Greenland
 923     ATF  Fr. S. Antarctic Lands
1043     -99                  France
1145     PRI             Puerto Rico
12134    NCL           New Caledonia
13140    TWN                  Taiwan
14159    ATA              Antarctica
15160    -99               N. Cyprus
16167    -99              Somaliland
17174    -99                  Kosovo
18"""

There are five countries in the world dataframe with iso_a3 code set to "-99". Three of these are disputed or don't yet have an ISO designation. France and Norway have ISO codes of "FRA" and "NOR" respectively and are updated in the geodataframe with the following.

1# Update ISO codes for France and Norway
2world.loc[world.name == 'France', 'iso_a3'] = 'FRA'
3world.loc[world.name == 'Norway', 'iso_a3'] = 'NOR'

Merge the data from Under Five Mortality Rates with the world geodataframe. The merge is done with an left join on the ISO country code. This reduces the number of countries to 166 that have both mortality information and geometry information. Use of a 'left' join keeps all the countries in the original world geodataframe with the appropriate geometry data. The 11 countries (such as Greenland) that only appear in the world geodataframe will have NaN for all of the under five mortality rate data.

1u5mr_world_df = world.merge(u5mr_med_df,
2                            left_on = 'iso_a3',
3                            right_on = 'ISO.Code',
4                            how = 'left')
5u5mr_world_df.shape
6"""
7(177, 79)
8"""

Create a plot showing the under five mortality rates per country for year 2019. The data in the geopandas.geodataframe can be displayed in a number of ways and it can be confusing to know which parameter to set. This code creates the same plot in four different ways. The first three use the plot function on the geodataframe, which uses matplotlib to generate the plot. The first plot sets a scheme of 'quantiles' for the choropleth classification scheme, which colors the countries based on discrete intervals. When choropleth classification scheme of 'quantiles' is used the legend is of type matplotlib.pyplot.legend so the legend_kwds parameters are different. The default scheme is None, in which case the legend is of type matplotlib.pyplot.colorbar. This is used in chart 2 and 3, with the colorbar being changed from vertical to horizontal. Finally, the fourth plot is rendered using the choropleth function in geoplot module.

 1u5mr_year = u5mr_world_df['2019']
 2fig, axs = plt.subplots(
 3    nrows = 2,
 4    ncols = 2,
 5    figsize = (12,5),
 6    facecolor = plt.cm.Blues(.2))
 7fig.suptitle('National Under Five Mortality Rates in 2019',
 8             fontsize = 'xx-large',  
 9             fontweight = 'bold')
10for ax in axs.flatten():
11    ax.set_facecolor(plt.cm.Blues(.2))
12
13ax1 = axs[0][0]
14u5mr_world_df.plot(
15    ax = ax1,
16    color = 'white',
17    edgecolor = 'black'
18)
19
20u5mr_world_df.plot(
21    column = u5mr_year,
22    scheme = 'quantiles',
23    k = 6,
24    cmap = 'OrRd',
25    ax = ax1,
26    legend = True,
27    legend_kwds = {'title': "UFMR per 1000",
28                   'title_fontsize': 'small',
29                   'frameon': False,
30                   'loc': 'lower center',
31                   'bbox_to_anchor': (-0.2, 0.1, 0.5, 1),
32                   'fontsize': 'xx-small',
33                  },
34)
35[spine.set_visible(False) for spine in ax1.spines.values()]
36ax1.xaxis.set_visible(False)
37ax1.yaxis.set_visible(False)
38
39
40ax2 = axs[0][1]
41u5mr_world_df.plot(
42    ax = ax2,
43    color = 'white',
44    edgecolor = 'black'
45)
46
47u5mr_world_df.plot(
48    column = u5mr_year,
49    cmap = 'OrRd',
50    ax = ax2,
51    legend = True,
52    legend_kwds = {'label': "UFMR per 1000"},
53)
54[spine.set_visible(False) for spine in ax2.spines.values()]
55ax2.xaxis.set_visible(False)
56ax2.yaxis.set_visible(False)
57
58ax3 = axs[1][0]
59u5mr_world_df.plot(
60    ax = ax3,
61    color = 'white',
62    edgecolor = 'black'
63)
64
65u5mr_world_df.plot(
66    column = u5mr_year,
67    cmap = 'OrRd',
68    ax = ax3,
69    legend = True,
70    legend_kwds = {'label': "UFMR per 1000",
71                   'orientation': 'horizontal',
72                   'shrink': 0.7,
73                  },
74)
75[spine.set_visible(False) for spine in ax3.spines.values()]
76ax3.xaxis.set_visible(False)
77ax3.yaxis.set_visible(False)
78
79
80gplt.choropleth(
81    u5mr_world_df,
82    hue = u5mr_year,
83#     scheme = scheme,
84    cmap = 'OrRd',
85    ax = axs[1][1],
86    legend = True
87)
88
89plt.show()

Map showing country Under Five Mortality Rates in 2019

Map showing country Under Five Mortality Rates in 2019

Display country color based on Under Five Mortality Rates in 1985

A function can be created to wrap up the creation of a world map for a particular year based on option 2 above. A color of gray is added to handle missing data using misssing_kwds parameter. This is valuable when dealing with earlier years where there is no data available for many countries and displaying white or no color can be misleading.

 1def create_map_for_year(year, title):
 2    u5mr_year = u5mr_world_df[year]
 3    fig, ax = plt.subplots(
 4        nrows = 1,
 5        ncols = 1,
 6        figsize = (15,6),
 7        facecolor = plt.cm.Blues(.2))
 8    fig.suptitle(title,
 9                 fontsize = 'xx-large',  
10                 fontweight = 'bold')
11    ax.set_facecolor(plt.cm.Blues(.2))
12
13    u5mr_world_df.plot(
14        ax = ax,
15        color = 'white',
16        edgecolor = 'black'
17    )
18
19    u5mr_world_df.plot(
20        column = u5mr_year,
21        cmap = 'OrRd',
22        ax = ax,
23        legend = True,
24        legend_kwds = {'label': "UFMR per 1000",
25                      'shrink': 0.7},
26        missing_kwds = {'facecolor':'Gray'},
27    )
28    [spine.set_visible(False) for spine in ax.spines.values()]
29    ax.xaxis.set_visible(False)
30    ax.yaxis.set_visible(False)
31    return fig

Use the function to create a map for 1985.

1fig = create_map_for_year('1985', 'National Under Five Mortality Rates in 1985')
2plt.show()

Map showing country Under Five Mortality Rates in 1985

Map showing country Under Five Mortality Rates in 1985

Display changes over the decades

The function is modified to plot a map for an axis.

 1def plot_ax_for_year(ax, year):
 2    u5mr_year = u5mr_world_df[year]
 3    ax.set_title(year,
 4                 fontsize = 'xx-large',  
 5                 fontweight = 'bold')
 6    ax.set_facecolor(plt.cm.Blues(.2))
 7
 8    u5mr_world_df.plot(
 9        ax = ax,
10        color = 'white',
11        edgecolor = 'black'
12    )
13
14    u5mr_world_df.plot(
15        column = u5mr_year,
16        cmap = 'OrRd',
17        ax = ax,
18        legend = True,
19        legend_kwds = {'label': "UFMR per 1000",
20                      'shrink': 0.7},
21        missing_kwds = {'facecolor':'Gray'},
22    )
23    [spine.set_visible(False) for spine in ax.spines.values()]
24    ax.xaxis.set_visible(False)
25    ax.yaxis.set_visible(False)
26    return ax

Create a plot with maps through the decades

 1fig, axs = plt.subplots(
 2    nrows = 4,
 3    ncols = 2,
 4    figsize = (15,15),
 5    facecolor = plt.cm.Blues(.2))
 6fig.suptitle('Changes in national Under Five Mortality Rates over time',
 7             fontsize = 'xx-large',  
 8             fontweight = 'bold')
 9years = [f'{x}' for x in range(1950, 2020, 10)] + ['2019']
10for i, ax in enumerate(axs.flatten()):
11    ax = plot_ax_for_year(ax, years[i])
12fig.tight_layout(pad=2)
13plt.show()

Map showing country Under Five Mortality Rates changes from 1950 to 2019

Map showing country Under Five Mortality Rates changes from 1950 to 2019

Conclusion

Geographical maps are a great way to present a global view of where data applies to different countries and regions. Geopandas and Geoplot are used to plot world maps with countries colored based on data of interest. The world maps produced here show the changes over time on Under Five Mortality Rates. This helps visualise particular regions of the world that are consistently doing worse off in addressing child mortality.

Use of Matplotlib creates nice static images, but one drawback of these is precisely that they are static. This data could be more informative if either the maps could be more interactive or displayed as an animation.

Under-five mortality rate:
is the probability of dying between birth and exactly 5 years of age, expressed per 1,000 live births.

Countries with highest confirmed cases of Covid-19 with Plotly

Eric Callanan — Sunday, 15 Nov 2020

COVID-19 is the disease caused by a new coronavirus called SARS-CoV-2. World Health Organisation (WHO) first learned of the virus on 31 December 2019. The WHO declared the coronavirus outbreak a pandemic in March 2020. This article will show how to create an interactive line chart with plotly, showing the changes in confirmed cases over time.

The data used in this article is retrieved from the Johns Hopkins University who have made te data available on GitHub. More information about COVID-19 and the coronavirus is available from Coronavirus disease (COVID-19) advice for the public.

Plotly is an open-source graphing library for Python that produces interactive charts.

Load the data into a Pandas dataframe

 1confirmed_path = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv'
 2confirmed_df = pd.read_csv(confirmed_path)
 3confirmed_df.shape
 4"""
 5(271, 315)
 6"""
 7
 8confirmed_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
 9"""
10    Province/State Country/Region        Lat  11/25/20  11/26/20  11/27/20
110              NaN    Afghanistan  33.939110     45490     45716     45839
121              NaN        Albania  41.153300     34944     35600     36245
132              NaN        Algeria  28.033900     78025     79110     80168
14268            NaN          Yemen  15.552727      2124      2137      2148
15269            NaN         Zambia -13.133897     17535     17553     17569
16270            NaN       Zimbabwe -19.015438      9508      9623      9714
17"""

The data in the Country/Region is further broken down by Province/State for some countries such as China or United Kingdom. A single row is created for each country by grouping the data by Country/Region and summing up the number. the columns 'Province/State', 'Lat', 'Long' are removed from the dataset.

 1# Drop columns 'Province/State', 'Lat', 'Long'
 2confirmed_df.drop(['Province/State', 'Lat', 'Long'], axis = 1, inplace = True)
 3
 4country_df = confirmed_df.groupby('Country/Region').sum()
 5country_df.shape
 6"""
 7(271, 312)
 8"""
 9
10country_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
11"""
12    Country/Region  1/22/20  1/23/20  11/25/20  11/26/20  11/27/20
130      Afghanistan        0        0     45490     45716     45839
141          Albania        0        0     34944     35600     36245
152          Algeria        0        0     78025     79110     80168
16268          Yemen        0        0      2124      2137      2148
17269         Zambia        0        0     17535     17553     17569
18270       Zimbabwe        0        0      9508      9623      9714
19"""

Display confirmed cases in top ten countries on the latest date

Sort the data by the values in the last column in descending order. A bar chart is created using Plotly with the top ten values. This displays an interactive chart that displays the value for each country when the bar is hovered over.

 1Top_10 = (country_df
 2          .sort_values(by=country_df.columns[-1], ascending=False)
 3          [[country_df.columns[-1]]]
 4          .head(10))
 5bg_color = 'rgba(208, 225, 242, 1.0)'
 6
 7fig = go.Figure(
 8    data = [go.Bar(x = list(Top_10.iloc[:, 0]),
 9                   y = list(Top_10.index),
10                   hovertemplate = "%{y}: <br><br>Confirmed: %{x:,.0f}<extra></extra>",
11                   orientation = 'h')]
12)
13
14fig.update_layout(
15    # Set default font
16        font = dict(
17        family = "Droid Sans",
18        size = 16
19    ),
20    # Set figure title
21    title = dict(
22        text = "<b>Top ten countries with highest confirmed cases of covid-19</b>",
23        xref = 'container',
24        yref = 'container',
25        x = 0.5,
26        y = 0.91,
27        xanchor = 'center',
28        yanchor = 'middle',
29        font = dict(family = 'Droid Sans', size = 24)
30    ),
31    # set x-axis
32    xaxis = dict(
33        title = dict(
34            text = 'Number of confirmed cases',
35            font = dict(family = 'Droid Sans', size = 18)
36        ),
37        showgrid = False,
38        linecolor = bg_color,
39        linewidth = 2,
40        showticklabels = True,
41    ),
42    # set y-axis
43    yaxis = dict(
44        showgrid = False,
45        linecolor = bg_color,
46        linewidth = 4,
47    ),
48    # set the plot bacground color
49    plot_bgcolor = bg_color,
50    # set the hover background color
51    hoverlabel = dict(
52        bgcolor = 'rgba(75, 152, 201, 0.2)',
53        font_size = 16
54    ),
55    paper_bgcolor = bg_color,
56)
57
58# Add annotation for the latest date
59fig.add_annotation(
60    xref = "paper",
61    yref = "paper",
62    x = 0.9,
63    y = 0.2,
64    text = f"<b>{pd.to_datetime(Top_10.columns[0]).strftime('%b %d %Y')}</b>",
65    showarrow = False,
66    font = dict(family = 'Droid Sans', size = 40))
67
68fig.update_traces(marker_color = 'rgb(75, 152, 201)')
69fig['layout']['yaxis']['autorange'] = "reversed"
70
71fig.show()

Top ten countries with highest confirmed cases of COVID-19

Display confirmed cases in US

Display a line chart showing the changes in the number of confirmed case over time for a specific country. Start with the US as this has highest number of confirmed cases in the world. The index is converted to datetime format so the axis is displayed better.

 1us_data = country_df.loc['US']
 2# Convert the index to datetime format
 3us_data.index = pd.to_datetime(us_data.index)
 4
 5def hov_text(ary_x, ary_y):
 6    '''Create the hover text for each data point
 7
 8    Keyword arguments:
 9    ary_x    -- list of all the x data points
10    ary_y    -- list of all the y data points
11
12    return: A list of hover texts for the selected country
13            with format to display
14    '''
15    txt = [f'''
16<b>{ary_x[i].strftime('%b %d %Y')}</b><br><br>
17Confirmed Cases: <b>{ary_y[i]:,.0F}</b><br>
18<extra></extra>
19''' for i in range(len(ary_x))]
20    return txt
21
22bg_color = 'rgba(208, 225, 242, 1.0)'
23line_color = 'rgba(75, 152, 201, 1.0)'
24grid_color = 'rgba(75, 152, 201, 0.3)'
25
26fig = go.Figure()
27fig.add_trace(go.Scatter(x = us_data.index,
28                         y = us_data,
29                         name = 'US',
30                         hovertemplate = hov_text(us_data.index, us_data)))
31
32fig.update_traces(
33    hoverinfo = 'text+name',
34    mode = 'lines'
35)
36
37fig.update_layout(
38    # Set figure title
39    title = dict(
40        text = f'<b>Confirmed cases of COVID-19 in US</b>',
41        xref = 'container',
42        yref = 'container',
43        x = 0.5,
44        y = 0.9,
45        xanchor = 'center',
46        yanchor = 'middle',
47        font = dict(family = 'Droid Sans', size = 28)
48    ),
49    # set legend
50    legend = dict(
51        orientation = "h",
52        traceorder = 'normal',
53        font_size = 12,
54        x = 0.0,
55        y = -0.3,
56        xanchor = "left",
57        yanchor = "top"
58    ),
59    # set x-axis
60    xaxis = dict(
61        title = 'Date',
62        linecolor = line_color,
63        linewidth = 2,
64        gridcolor = grid_color,
65        showticklabels = True,
66        ticks = 'outside',
67    ),
68    # set y-axis
69    yaxis = dict(
70        title = 'Number of confirmed cases',
71        linecolor = line_color,
72        linewidth = 2,
73        gridcolor = grid_color,
74        showticklabels = True,
75        ticks = 'outside',
76    ),
77    # set the plot bacground color
78    plot_bgcolor = bg_color,
79    paper_bgcolor = bg_color,
80)
81
82fig.show()

Confirmed cases of COVID-19 in US

Create a function to display confirmed cases for any country

The code above can be wrapped up in a function to create a chart for any country specified.

 1def hov_text(ary_x, ary_y):
 2    '''Create the hover text for each data point
 3
 4    Keyword arguments:
 5    ary_x    -- list of all the x data points
 6    ary_y    -- list of all the y data points
 7
 8    return: A list of hover texts for the selected country
 9            with format to display
10    '''
11    txt = [f'''
12<b>{ary_x[i].strftime('%b %d %Y')}</b><br><br>
13Confirmed Cases: <b>{ary_y[i]:,.0F}</b><br>
14<extra></extra>
15''' for i in range(len(ary_x))]
16    return txt
17
18
19def plot_confirmed_cases_for_country(df, country):
20    sel_data = df.loc[country]
21    # Convert the index to datetime format
22    sel_data.index = pd.to_datetime(sel_data.index)
23
24    bg_color = 'rgba(208, 225, 242, 1.0)'
25    line_color = 'rgba(75, 152, 201, 1.0)'
26    grid_color = 'rgba(75, 152, 201, 0.3)'
27
28    fig = go.Figure()
29    fig.add_trace(go.Scatter(x = sel_data.index,
30                             y = sel_data,
31                             name = 'US',
32                             hovertemplate = hov_text(sel_data.index, sel_data)))
33
34    fig.update_traces(
35        hoverinfo = 'text+name',
36        mode = 'lines'
37    )
38
39    fig.update_layout(
40        # Set figure title
41        title = dict(
42            text = f'<b>Confirmed cases of COVID-19 in {country}</b>',
43            xref = 'container',
44            yref = 'container',
45            x = 0.5,
46            y = 0.9,
47            xanchor = 'center',
48            yanchor = 'middle',
49            font = dict(family = 'Droid Sans', size = 28)
50        ),
51        # set legend
52        legend = dict(
53            orientation = "h",
54            traceorder = 'normal',
55            font_size = 12,
56            x = 0.0,
57            y = -0.3,
58            xanchor = "left",
59            yanchor = "top"
60        ),
61        # set x-axis
62        xaxis = dict(
63            title = 'Date',
64            linecolor = line_color,
65            linewidth = 2,
66            gridcolor = grid_color,
67            showticklabels = True,
68            ticks = 'outside',
69        ),
70        # set y-axis
71        yaxis = dict(
72            title = 'Number of confirmed cases',
73            linecolor = line_color,
74            linewidth = 2,
75            gridcolor = grid_color,
76            showticklabels = True,
77            ticks = 'outside',
78        ),
79        # set the plot bacground color
80        plot_bgcolor = bg_color,
81        paper_bgcolor = bg_color,
82    )
83
84    return fig

Call this function with the dataframe and the country name to create the plot.

1fig = plot_confirmed_cases_for_country(country_df, 'China')
2fig.show()

Confirmed cases of COVID-19 in China

Display confirmed cases for top ten countries

Get the ten country names for the countries with the highest confirmed cases on the latest date.

1top_countries = (country_df
2                 .sort_values(by = country_df.columns[-1], ascending = False)
3                 .head(10)
4                 .index)
5
6"""
7['US',   'India',          'Brazil', 'France',    'Russia',
8'Spain', 'United Kingdom', 'Italy',  'Argentina', 'Colombia']
9"""

Transpose the data so that the date is the index and each row represents all the data for that date. Set the index to be datetime format.

 1wide_df = country_df.T
 2wide_df.index = pd.to_datetime(wide_df.index)
 3
 4wide_df.shape
 5"""
 6(311, 191)
 7"""
 8
 9wide_df.iloc[[0,1,2,-3,-2,-1], [0,1,2,-3,-2,-1]]
10"""
11Country/Region  Afghanistan  Albania  Algeria  Yemen  Zambia  Zimbabwe
122020-01-22                0        0        0      0       0         0
132020-01-23                0        0        0      0       0         0
142020-01-24                0        0        0      0       0         0
152020-11-25            45490    34944    78025   2124   17535      9508
162020-11-26            45716    35600    79110   2137   17553      9623
172020-11-27            45839    36245    80168   2148   17569      9714
18"""

The data is extracted for the top ten countries and displayed on a line chart.

 1def hov_text(ary_x, ary_y, country):
 2    '''Create the hover text for each data point
 3
 4    Keyword arguments:
 5    ary_x    -- list of all the x data points
 6    ary_y    -- list of all the y data points
 7    country  -- name of country
 8
 9    return: A list of hover texts for the selected country
10            with format to display
11    '''
12    txt = [f'''
13<b>{country}<b><br><br>
14Date: <b>{ary_x[i].strftime('%b %d %Y')}</b><br>
15Confirmed Cases: <b>{ary_y[i]:,.0F}</b><br>
16<extra></extra>
17''' for i in range(len(ary_x))]
18    return txt
19
20bg_color = 'rgba(208, 225, 242, 1.0)'
21line_color = 'rgba(75, 152, 201, 1.0)'
22grid_color = 'rgba(75, 152, 201, 0.3)'
23
24df = wide_df[top_countries]
25
26fig = go.Figure()
27for c in top_countries:
28    fig.add_trace(go.Scatter(x = df.index,
29                             y = df[c],
30                             name = c,
31                             hovertemplate = hov_text(df.index, df[c], c)))
32
33fig.update_traces(
34    hoverinfo = 'text+name',
35    mode = 'lines'
36)
37
38fig.update_layout(
39    # Set figure title
40    title = dict(
41        text = f'<b>Confirmed cases of COVID-19 in top ten countries</b>',
42        xref = 'container',
43        yref = 'container',
44        x = 0.5,
45        y = 0.9,
46        xanchor = 'center',
47        yanchor = 'middle',
48        font = dict(family = 'Droid Sans', size = 28)
49    ),
50    # set legend
51    legend = dict(
52        orientation = "h",
53        traceorder = 'normal',
54        font_size = 12,
55        x = 0.0,
56        y = -0.3,
57        xanchor = "left",
58        yanchor = "top"
59    ),
60    # set x-axis
61    xaxis = dict(
62        title = 'Date',
63        linecolor = line_color,
64        linewidth = 2,
65        gridcolor = grid_color,
66        showticklabels = True,
67        ticks = 'outside',
68    ),
69    # set y-axis
70    yaxis = dict(
71        title = 'Number of confirmed cases',
72        linecolor = line_color,
73        linewidth = 2,
74        gridcolor = grid_color,
75        showticklabels = True,
76        ticks = 'outside',
77    ),
78    showlegend = True,
79    # set the plot bacground color
80    plot_bgcolor = bg_color,
81    paper_bgcolor = bg_color,
82)
83
84fig.show()

Confirmed cases of COVID-19 for top ten countries

Conclusion

Pandas and Plotly are a powerful combination to load data, clean data and create interactive charts. Johns Hopkins University has done the hard work of collecting the COVID-19 data and making it available. The data can be loaded directly from the csv file in GitHub. Plotly is used to create bar charts and line charts to display information on countries with the highest confirmed cases of COVID-19.

Display a line chart with a range

Eric Callanan — Sunday, 08 Nov 2020

Under five mortality rate is an estimate for each particular country for that year based on the available information. The accuracy of this value varies from country to country and is dependent on the availability of quality data. The data produced is composed of the median value as well as upper and lower bound limits for each year. An indication of the accuracy of the data can be shown by displaying the upper and lower bounds on a line chart.

Accurate data over time is not available for some countries and Unicef have a process of applying statistical models to estimate the Under Five Mortality Rates based on available data. The Lower and Upper values refer to the lower bound and upper bound of 90% uncertainty intervals.

This article shows how to display the median as well as the lower and upper bounds.

Load the data

Details on downloading and loading the data into a dataframe is described in "Pandas - Load data from Excel file and Display Chart". The following loads the data from a local file and adds an multi-index on Country and Bounds.

 1# Load the excel worksheet into a dataframe
 2u5mr_df = pd.read_excel(
 3    "/tmp/data/Under-five-mortality-rate_2020.xlsx",
 4    engine = "openpyxl",
 5    sheet_name = 'Country estimates (both sexes)',
 6    header = 14)
 7
 8# Drop the last two rows
 9u5mr_df.drop(u5mr_df.tail(2).index, inplace = True)
10
11# Drop ISO.Code column
12u5mr_df.drop(['ISO.Code'], axis = 1, inplace = True)
13
14# Rename 'Uncertainty.Bounds*' column to 'Uncertainty.Bounds'
15u5mr_df = u5mr_df.rename(columns={'Uncertainty.Bounds*': 'Uncertainty.Bounds'})
16
17# Set multiindex to Country and Bounds
18u5mr_df.set_index(['Country.Name', 'Uncertainty.Bounds'], inplace=True)
19
20# Convert year column names to datetime
21u5mr_df.columns = [pd.to_datetime(f'{x[:-2]}-12-31') for x in u5mr_df.columns]
22
23# Review the data
24print(u5mr_df.shape)
25u5mr_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,4,5,6,-4,-3,-2,-1]]
26
27"""
28                                 1960-12-31  1961-12-31  1962-12-31  2016-12-31  2017-12-31  2018-12-31  2019-12-31  
29Country.Name Uncertainty.Bounds                                                                                      
30Afghanistan  Lower                      NaN         NaN  299.425519   57.068054   53.744798   50.640922   47.442070  
31             Median                     NaN         NaN  344.629498   67.572190   64.940759   62.541196   60.269399  
32             Upper                      NaN         NaN  399.287713   78.646207   76.832873   75.405564   74.616041  
33Albania      Lower                      NaN         NaN         NaN    9.075629    9.044506    9.067390    9.068154  
34Zambia       Upper               229.757846  224.353213  219.461322   76.133459   75.733787   77.098449   77.575507  
35Zimbabwe     Lower               129.144390  125.263606  121.661598   50.834193   48.083920   44.136918   41.076802  
36             Median              151.084581  146.858820  142.639204   59.538505   58.234924   55.856832   54.612967  
37             Upper               177.879631  173.725419  168.219351   69.425121   69.987570   70.076887   71.709892  
38"""

Display the changes over time for one country

Nigeria has the highest mediam Under Five Mortality Rate in 2019. This code creates a line chart showing the changes over time.

 1country = 'Nigeria'
 2years = u5mr_df.columns
 3median = u5mr_df.loc[(['Nigeria'], ['Median']), :].iloc[0]
 4
 5title = f'Changes in Under 5 Mortality Rate\n\n {c}'
 6
 7fig, ax = plt.subplots(figsize = (8,5), facecolor = plt.cm.Blues(.2))
 8fig.suptitle(title, fontsize = 'xx-large',  fontweight = 'bold')
 9
10ax.set_facecolor(plt.cm.Blues(.2))
11ax.plot(years, median, label = 'Median')
12ax.set_ylabel('Under-five mortality rate', fontsize = 'large')
13ax.set_xlabel('Year', fontsize=14)
14ax.spines['right'].set_visible(False)
15ax.spines['top'].set_visible(False)
16
17plt.show()

Under five mortality rate over time for Nigeria

Under five mortality rate over time for Nigeria

Display upper, lower and median data for a single country

Extract the data for Nigeria into a separate dataframe. Use the data from this dataframe to plot the lower, upper and median bound values over time. This shows that while the median value is going down, the spread between the lower and upper bounds is widening. The 90% confidence interval for Under Five Mortality in Nigeria in 2019 is from 92 to 152. This is an extraordinarily wide range and shows that there is not much confidence in the median value of 117.

This code adds plot lines for upper, median and lower bounds. This displays each result in a separate color by default.

 1country = 'Nigeria'
 2
 3df_c = u5mr_df.loc[([country]), :]
 4years = df_c.columns
 5lower = df_c.loc[(slice(None), 'Lower'), :].iloc[0]
 6median = df_c.loc[(slice(None), 'Median'), :].iloc[0]
 7upper = df_c.loc[(slice(None), 'Upper'), :].iloc[0]
 8
 9title = f'Changes in Under 5 Mortality Rate\n\n {c}'
10
11fig, ax = plt.subplots(figsize = (8,5), facecolor = plt.cm.Blues(.2))
12fig.suptitle(title, fontsize = 'xx-large',  fontweight = 'bold')
13
14ax.set_facecolor(plt.cm.Blues(.2))
15ax.plot(years, upper, label = 'Upper')
16ax.plot(years, median, label = 'Median')
17ax.plot(years, lower, label = 'Lower')
18ax.legend(bbox_to_anchor = (0.9, 0.9),
19          loc = 'upper right',
20          frameon = False,
21          fontsize = 'medium')
22ax.set_ylabel('Under-five mortality rate', fontsize = 'large')
23ax.set_xlabel('Year', fontsize=14)
24# Hide the right and top spines
25ax.spines['right'].set_visible(False)
26ax.spines['top'].set_visible(False)
27
28plt.show()
29
30
31print(df_c.iloc[:, [13,14,15,-3,-2,-1]])
32"""
33                                 1963-12-31  1964-12-31  1965-12-31  2017-12-31  2018-12-31  2019-12-31  
34Country.Name Uncertainty.Bounds                                                                          
35Nigeria      Lower                      NaN  274.178909  274.741658  101.806684   97.303293   92.166854  
36             Median                     NaN  323.283445  316.175804  122.798947  120.037728  117.202078  
37             Upper                      NaN  380.916122  364.765834  150.074233  151.521302  152.451394  
38"""

Under five mortality rate over time for Nigeria with upper and lower bounds

Under five mortality rate over time for Nigeria with upper and lower bounds

Display shading around median data for a single country

This displays the same information as above, but changes the area between the upper and lower bounds to be shaded in the same color as the median values. The functionality to plot the line on the axis is split out into its own function. The axes fill_between method is used color the area between the lower and upper bounds.

 1def update_axis(ax, df, country):
 2    clr = plt.cm.Purples(0.9)
 3    ax.set_facecolor(plt.cm.Blues(.2))
 4    ax.set_title(country, fontsize = 14, fontweight = 'bold')
 5    x = df.columns
 6    y_l = df.loc[(slice(None), 'Lower'), :].iloc[0]
 7    y_m = df.loc[(slice(None), 'Median'), :].iloc[0]
 8    y_u = df.loc[(slice(None), 'Upper'), :].iloc[0]
 9    ax.plot(x, y_m, label = 'Median', color = clr)
10    ax.fill_between(x, y_l, y_u, alpha=0.3, edgecolor=clr, facecolor=clr)
11    ax.set_ylabel('Under-five mortality rate', fontsize = 'medium')
12    ax.set_xlabel('Year', fontsize = 'medium')
13    ax.tick_params(axis='both', labelsize='small')
14    ax.spines['right'].set_visible(False)
15    ax.spines['top'].set_visible(False)
16
17fig, ax1 = plt.subplots(1, 1, figsize = (8, 5), facecolor = plt.cm.Blues(.2))
18title = f'Changes in Under 5 Mortality Rate'
19fig.suptitle(title, fontsize = 'xx-large',  fontweight = 'bold')
20
21country = 'Nigeria'
22df_c = u5mr_df.loc[([country]), :]
23update_axis(ax1, df_c, country)
24
25plt.show()

Under five mortality rate over time for Nigeria with bounds shaded

Under five mortality rate over time for Nigeria with bounds shaded

Display data for four countries

The update_axis function above can be used to display multiple plots, such as for the four countries with highest under five mortality rates in 2019.

 1fig, axs = plt.subplots(2, 2,
 2                        sharex=True, sharey=True,
 3                        figsize = (10, 7),
 4                        facecolor = plt.cm.Blues(.2))
 5fig.tight_layout(pad = 5.0)
 6title = f'Changes in Under 5 Mortality Rate'
 7fig.suptitle(title, fontsize = 'xx-large',  fontweight = 'bold')
 8
 9countries = (u5mr_df.xs('Median', level='Uncertainty.Bounds')
10             .sort_values(by = u5mr_df.columns[-1], ascending = False)
11             .head(4).index)
12
13for i, ax in enumerate(axs.flatten()):
14    df_c = u5mr_df.loc[([countries[i]]), :]
15    update_axis(ax, df_c, countries[i])
16
17plt.show()

Under five mortality rate over time for four highest countries

Under five mortality rate over time for four highest countries

The same data can be extracted and displayed by changing the sort order to ascending. Countries that are doing better and have much lower under five mortality rates tend to haver much better records, so the 90% confidence interval is also much narrower.

Under five mortality rate over time for four lowest countries

Under five mortality rate over time for four lowest countries

Display data for male and female data

List the worksheets in an excel file

Use Pandas.ExcelFile to find out what sheets are in an Excel file. This shows that the file contains six worksheets with the third and fifth containing the data for Females and Males respectively.

 1u5mr_xl = pd.ExcelFile(u5mr_file)
 2u5mr_xl.sheet_names
 3
 4
 5# output
 6'''
 7['Country estimates (both sexes)',
 8 'Regional & global (both sexes)',
 9 'Country estimates (Female)',
10 'Regional & global (Female)',
11 'Country estimates (Male)',
12 'Regional & global (Male)']
13 '''

The use of the multi-index on the dataframe makes is much easier to extract the data for a particular country and then plot the bounds values. The following put the full process into three functions.

Open the specified spreadsheet into a dataframe

 1def load_dataset(file, sheet):
 2    # Load the excel worksheet into a dataframe
 3    df = pd.read_excel(
 4        file,
 5        engine = "openpyxl",
 6        sheet_name = sheet,
 7        header = 14)
 8
 9    # Drop the last two rows
10    df.drop(df.tail(2).index, inplace = True)
11
12    # Drop ISO.Code column
13    df.drop(['ISO.Code'], axis = 1, inplace = True)
14
15    # Rename 'Uncertainty.Bounds*' column to 'Uncertainty.Bounds'
16    df = df.rename(columns={'Uncertainty.Bounds*': 'Uncertainty.Bounds'})
17
18    # Set multiindex to Country and Bounds
19    df.set_index(['Country.Name', 'Uncertainty.Bounds'], inplace=True)
20
21    # Convert year column names to datetime
22    df.columns = [pd.to_datetime(f'{x[:-2]}-12-31') for x in df.columns]
23
24    print(f'df.shape = {df.shape}')
25    return df

Plot the data on an axis

 1def update_axis(ax, df, country, lbl, clr = plt.cm.Purples(0.9)):
 2    ax.set_facecolor(plt.cm.Blues(.2))
 3    ax.set_title(country, fontsize = 'xx-large', fontweight = 'bold')
 4    x = df.columns
 5    y_l = df.loc[(slice(None), 'Lower'), :].iloc[0]
 6    y_m = df.loc[(slice(None), 'Median'), :].iloc[0]
 7    y_u = df.loc[(slice(None), 'Upper'), :].iloc[0]
 8    ax.plot(x, y_m, label = lbl, color = clr)
 9    ax.fill_between(x, y_l, y_u, alpha=0.3, edgecolor=clr, facecolor=clr)
10    ax.set_ylabel('Under-five mortality rate', fontsize = 'medium')
11    ax.set_xlabel('Year', fontsize = 'medium')
12    ax.tick_params(axis='both', labelsize='small')
13    ax.spines['right'].set_visible(False)
14    ax.spines['top'].set_visible(False)
15
16    ax.legend(bbox_to_anchor = (0.9, 0.9),
17          loc = 'upper right',
18          frameon = False,
19          fontsize = 'medium')

Load the data and display the data from different spreadsheets.

 1f_u5mr_df = load_dataset(
 2    "/tmp/data/Under-five-mortality-rate_2020.xlsx",
 3    "Country estimates (Female)")
 4
 5m_u5mr_df = load_dataset(
 6    "/tmp/data/Under-five-mortality-rate_2020.xlsx",
 7    "Country estimates (Male)")
 8
 9fig, (ax1, ax2, ax3) = plt.subplots(1, 3, sharex=True, sharey=True,
10                               figsize = (12, 5),
11                               facecolor = plt.cm.Blues(.2))
12fig.tight_layout(pad = 5.0)
13title = f'Changes in Under 5 Mortality Rate - females'
14fig.suptitle(title, fontsize = 'xx-large',  fontweight = 'bold')
15
16country = 'Nigeria'
17df_c = f_u5mr_df.loc[([country]), :]
18update_axis(ax1, df_c, country, 'Female')
19update_axis(ax3, df_c, country, 'Female')
20
21df_c = m_u5mr_df.loc[([country]), :]
22update_axis(ax2, df_c, country, 'Male', clr = plt.cm.Greens(0.9))
23update_axis(ax3, df_c, country, 'Male', clr = plt.cm.Greens(0.9))
24
25plt.show()

Under five mortality rate for Males and Females in Nigeria

Under five mortality rate for Males and Females in Nigeria

Conclusion

A single line graph does not give a true picture for some information such as Under Five Mortality Rates as the gathering of the data is challenging in many countries. This data contains upper and lower bound data for 90% confidence interval. The addition of a shaded area above and below the median values provides a lot more information on the data. It can be seen that the poorer countries with higher child mortality rates also have a much greater spread from lower to upper bounds. Countries with lower child mortality rates tend to also have a narrower confidence intervals. Data is probably more accurate due to better record keeping.

Under-five mortality rate:
is the probability of dying between birth and exactly 5 years of age, expressed per 1,000 live births.

Pandas - Dataframe Where function

Eric Callanan — Sunday, 01 Nov 2020

The Where method of Pandas Dataframe is used to replace data in a dataframe where the condition is false. It can return a new dataframe or modify the data in the original dataframe, thereby saving memory and object creation.

The syntax of the Dataframe Where function can be confusing. At a minumum it requires just one parameter - the condition to apply to the dataframe, but it replaces all the data where the condition is false. It can also be used with two parameters where the first parameter is the condition and the second is a value or an operation to be applied when the condition is false.

DataFrame.where(cond, other=nan, inplace=False, axis=None, level=None, errors='raise', try_cast=False)

Where

Pandas dataframe where function is used to replace data in a dataframe where the condition is false. It is not analogous to the Where clause in SQL databases, which is used to filter on data where the condition is true. It is easier to demonstrate with a couple of examples.

Where numbers are even

Create a condition to identify even numbers in the dataframe. The following code applies modulo 2 to each element in the dataframe and shows true where the result is equal to 0.

Create a dataframe.

 1# Create Dataframe
 2np.random.seed(2)
 3df = pd.DataFrame(np.random.randint(0, 10, 21).reshape(-1, 7),
 4                  columns=['A', 'B', 'C', 'D', 'E', 'F', 'G'])
 5
 6"""
 7   A  B  C  D  E  F  G
 80  8  8  6  2  8  7  2
 91  1  5  4  4  5  7  3
102  6  4  3  7  6  1  3
11"""

Create a condition to identify even numbers.

1cond = df % 2 == 0
2df % 2 == 0
3
4       A      B      C      D      E      F      G
50   True   True   True   True   True  False   True
61  False  False   True   True  False  False  False
72   True   True  False  False   True  False  False

Applying this condition in a where clause to the dataframe replaces the data where the condition is false with NaN. In this case where the condition is false is all the odd numbers.

1df.where(cond)
2
3"""
4     A    B    C    D    E   F    G
50  8.0  8.0  6.0  2.0  8.0 NaN  2.0
61  NaN  NaN  4.0  4.0  NaN NaN  NaN
72  6.0  4.0  NaN  NaN  6.0 NaN  NaN
8"""

Subtract 1 from the original dataframe values to round down all the odd numbers to the nearest even number.

1df.where(cond, df-1)
2
3"""
4   A  B  C  D  E  F  G
50  8  8  6  2  8  6  2
61  0  4  4  4  4  6  2
72  6  4  2  6  6  0  2
8"""

Pandas - Use Where to replace odd numbers

Pandas - Use Where to replace odd numbers

Replace negative numbers in place with zero

It is frequently required as part of data cleaning to replace certain values with more valid data. The following replaces negative numbers with zero. The inplace parameter is set to true to replace the values in the original dataframe.

 1# Create a dataframe with some negative numbers
 2np.random.seed(22)
 3df2 = pd.DataFrame(np.random.randint(-20, 100, 21).reshape(-1, 7),
 4                  columns=['A', 'B', 'C', 'D', 'E', 'F', 'G'])
 5
 6df2
 7"""
 8    A   B   C   D   E   F   G
 90  97 -16  24  44  80  82  98
101  87  64 -12  75  80  73  -2
112  -6  25  68  73  19  14  21
12"""

1df2.where(df2>=0, 0, inplace=True)
2
3df2
4"""
5    A   B   C   D   E   F   G
60  97   0  24  44  80  82  98
71  87  64   0  75  80  73   0
82   0  25  68  73  19  14  21
9"""

Pandas - Use Where to replace negative numbers

Pandas - Use Where to replace negative numbers

Replace duplicates in rows

The condition can get more complicated such as looking for duplicate values in a row. The opposite of a condition can then be used by preceding the condition with the tilde character ~ to replace where the condition is true. Technically, the where is still replacing where the condition is false or in this case where the opposite of the condition is false.

 1# Create a new dataframe
 2np.random.seed(22)
 3df3 = pd.DataFrame(np.random.randint(1, 20, 21).reshape(-1, 7),
 4                  columns=['A', 'B', 'C', 'D', 'E', 'F', 'G'])
 5
 6df3
 7"""
 8   A   B   C   D  E   F   G
 90  5  13   1   5  7  12   9
101  5  19  15  14  8   3  10
112  9   9   6  16  3  19  18
12"""

Create a condition to identify duplicates in each row of the dataframe. Use the opposite of this condition to display a True wherever there is a duplicate.

 1# Create a condition to identify duplicates in a row
 2cond = df3.apply(pd.Series.duplicated, axis = 1)
 3df3.where(~cond, True)
 4
 5"""
 6   A     B   C     D  E   F   G
 70  5    13   1  True  7  12   9
 81  5    19  15    14  8   3  10
 92  9  True   6    16  3  19  18
10"""

Modify the duplicates by applying a factor to the original dataframe value.

1df3.where(~cond, df3*1.01)
2
3"""
4   A      B   C      D  E   F   G
50  5  13.00   1   5.05  7  12   9
61  5  19.00  15  14.00  8   3  10
72  9   9.09   6  16.00  3  19  18
8"""

Pandas - Use Where to modify duplicate numbers in each row

Pandas - Use Where to modify duplicate numbers in each row

Conclusion

Pandas Dataframe Where method is used to replace data in the dataframe where the condition is false and is not used for filtering data like the Where clause in SQL. Filtering data based on conditions in rows and columns can be done with loc and iloc. This is more analagous to the Where clause in SQL and more information can be found in Pandas - How to use iloc and loc

Where replaces all values in a dataframe that do not meet the where condition either with NaN or with the specified value(s).

Create a Bar Chart Race with Matplotlib - part 2

Eric Callanan — Sunday, 25 Oct 2020

A Bar Chart Race is an animation where the bars representing the data change in place on the chart to represent change over time. This is the second article to improve on the bar chart race to include changing categories over time while maintaining the color associated with each category.

How to create a basic Bar Chart Race is described in "How to create a Bar Chart race - part 1". In this article the bar chart race will built upon in the following areas:

Handle a larger range of data where the countries change.
Keep the color for each country constant.
Improve the bar chart race by displaying the current value on the bar.
Wrap up in functions to create other bar chart races.

Load the data

 1# Load the excel worksheet into a dataframe
 2u5mr_df = pd.read_excel(
 3    "/tmp/data/Under-five-mortality-rate_2020.xlsx",
 4    sheet_name = 'Country estimates (both sexes)',
 5    header = 14)
 6
 7# Drop the last two rows
 8u5mr_df.drop(u5mr_df.tail(2).index, inplace = True)
 9
10# Rename the columns to Years
11u5mr_df.columns = [x[:-2] if x.endswith('.5') else x for x in u5mr_df.columns]
12
13# Rename 'Uncertainty.Bounds*' column to 'Uncertainty.Bounds'
14u5mr_df = u5mr_df.rename(columns={'Uncertainty.Bounds*': 'Uncertainty.Bounds'})
15
16# Filter to the Median values
17u5mr_med_df = u5mr_df[u5mr_df['Uncertainty.Bounds'] == 'Median']
18
19# Review the data
20u5mr_med_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,4,5,6,-4,-3,-2,-1]]
21"""
22    ISO.Code Country.Name Uncertainty.Bounds  1950  1951  1952        1953       2016       2017       2018       2019
231        AFG  Afghanistan             Median   NaN   NaN   NaN         NaN  67.572190  64.940759  62.541196  60.269399
244        ALB      Albania             Median   NaN   NaN   NaN         NaN   9.419110   9.418052   9.525133   9.682407
257        DZA      Algeria             Median   NaN   NaN   NaN         NaN  24.792098  24.319482  23.805926  23.256168
2610       AND      Andorra             Median   NaN   NaN   NaN         NaN   3.369056   3.218925   3.085839   2.966929
27574      VNM     Viet Nam             Median   NaN   NaN   NaN         NaN  21.220796  20.843125  20.405423  19.935167
28577      YEM        Yemen             Median   NaN   NaN   NaN         NaN  56.823614  56.966430  58.460003  58.356138
29580      ZMB       Zambia             Median   NaN   NaN   NaN  234.418232  66.510929  64.337901  63.294182  61.663465
30583      ZWE     Zimbabwe             Median   NaN   NaN   NaN         NaN  59.538505  58.234924  55.856832  54.612967
31"""

Identify the countries of interest

First get the countries that are in the top 10 for all years in the dataset. Use a lambda expression to apply a sort to each row and combining the results of the top 10 in each row. This consists of the following 41 countries.

 1Afghanistan            Angola                             Bangladesh                         
 2Benin                  Burkina Faso                       Cambodia                           
 3Cameroon               Central African Republic           Chad                               
 4Cote d'Ivoire          Democratic Republic of the Congo   Egypt                              
 5Ethiopia               Gambia                             Ghana                              
 6Guinea                 Guinea-Bissau                      Haiti                              
 7Iraq                   Jordan                             Liberia                            
 8Libya                  Malawi                             Mali                               
 9Mauritania             Mozambique                         Nepal                              
10Niger                  Nigeria                            Oman                               
11Pakistan               Peru                               Republic of Korea                  
12Rwanda                 Senegal                            Sierra Leone                       
13Somalia                South Sudan                        Togo                               
14Turkey                 Yemen

1fields = ['Country.Name'] + [str(x) for x in range(1950, 2020)]
2sel_df = u5mr_med_df[fields].set_index('Country.Name')
3countries = list(sel_df.apply(
4    lambda x: x.sort_values(ascending = False).head(10),
5    axis = 0).index)

Create a color map for these countries.

The original bar chart animation showed changes in the same 5 countries from 2015 to 2019. The bar chart race is an animation of a sequence of bar charts in which the color of the bars is associated with the country for the value on the bar. Each bar chart is generated independently of the previous bar charts, so the color of the countries will change as new countries are added to the chart and other countries are removed. This is confusing in the final animated result.

A color-map is created assigning a color to each of the 41 countries to address this issue. There is no predefined color palette for more than 20 colors. The palette is created by joining colors from multiple palettes together.

1# Colours
2cols = plt.cm.tab20.colors + plt.cm.Dark2.colors + plt.cm.Set3.colors + plt.cm.tab20b.colors
3
4m = zip(countries, cols)
5color_dict = {x[0]:x[1] for x in m}
6color_dict

Unique colors for top countries

Unique colors for top countries

Get the data for these 41 countries of interest

Extract the data for the top 41 countries and transpose the data into wide format. Change the index to the last day of the year for each year and convert to datetime format.

 1# Get the data for the countries of interest
 2data_df = (u5mr_med_df.drop(['ISO.Code', 'Uncertainty.Bounds'], axis=1)
 3           [u5mr_med_df['Country.Name'].isin(countries)]).copy()
 4
 5# Set index to "Country.Name" and Transpose the dataframe
 6wide_df = data_df.set_index('Country.Name').T
 7
 8# Remove the column name
 9wide_df.rename_axis(None, axis=1, inplace=True)
10
11# Set index to datetime for end of year
12wide_df.index = pd.to_datetime([f"{x}-12-31" for x in wide_df.index])
13
14wide_df.iloc[[0,1,2,3,-4,-3,-2,-1], [0,1,2,3,-3,-2,-1]]
15"""
16            Afghanistan     Angola  Bangladesh       Benin        Togo      Turkey      Yemen  
171950-12-31          NaN        NaN  345.207382  349.108392  317.200927         NaN        NaN  
181951-12-31          NaN        NaN  335.272222  345.669180  311.993315         NaN        NaN  
191952-12-31          NaN        NaN  325.617440  342.068496  307.047255         NaN        NaN  
201953-12-31          NaN        NaN  316.000124  338.513583  302.123318  299.282646        NaN  
212016-12-31    67.572190  84.211894   35.680942   97.416395   73.515650   12.148032  56.823614  
222017-12-31    64.940759  80.622302   33.921226   95.133079   71.317235   11.396633  56.966430  
232018-12-31    62.541196  77.672320   32.266390   92.773521   69.115785   10.696923  58.460003  
242019-12-31    60.269399  74.686710   30.753860   90.286429   66.904696   10.046388  58.356138  
25"""

Expand the data set for smooth animation

Expand the dataset with rows every two months and creating a ranking dataframe with rankings for the countries for each year. Fill in the newly created rows in the dataframes using interpolate function. Increment any duplicate rankings for a given row to avoid bars disappearing from the bar chart animation.

 1# Expand the dataset
 2expanded_df = wide_df.asfreq('2M')
 3
 4# Create a ranking dataframe
 5rank_df = expanded_df.rank(axis = 1, method = 'first', ascending = False)
 6
 7expanded_df = expanded_df.interpolate()
 8rank_df = rank_df.interpolate()
 9
10# Remove duplicate ranking
11rank_df = rank_df.where(~rank_df.apply(pd.Series.duplicated, axis=1), rank_df*1.01)

Display bar chart for a single year

Displaying a bar chart for a single year showing the ten countries with the highest Under Five Mortality Rate for that year.

 1fig, ax = plt.subplots(nrows = 1,
 2                       ncols = 1,
 3                       figsize = (10, 7),
 4                       facecolor = plt.cm.Blues(.2),
 5                       tight_layout = True)
 6bar_num = 10
 7i = 138
 8sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
 9ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
10        tick_label = sel_df.columns,
11        width = sel_df.iloc[i],
12        color = [color_dict[col] for col in sel_df.columns],
13        alpha = 0.8)
14cur_year = expanded_df.index[i].strftime('%Y-%m')
15ax.set_title(f'Under Five Mortality Rate - {cur_year}',
16             fontsize = 'xx-large',
17             fontweight = 'bold')
18ax.set_ylim(10.8, 0.2)
19ax.set_facecolor(plt.cm.Blues(.2))
20ax.tick_params(labelsize = 'medium')
21ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
22ax.set_axisbelow(True)
23[spine.set_visible(False) for spine in ax.spines.values()]
24plt.show()

Top ten countries with highest Under Five Mortality Rate in 1973

Top ten countries with highest Under Five Mortality Rate in 1973

Display bar chart with mortality rate on the bars

It can be helpful to see the under five mortality rate on the bar chart as the data is changing. This is done by adding an annotation to each of the bars. The mortality rate is displayed inside of the right edge of the bar. This could be displayed anywhere and works quite well outside to the right of the bar.

 1fig, ax = plt.subplots(nrows = 1,
 2                       ncols = 1,
 3                       figsize = (10, 7),
 4                       facecolor = plt.cm.Blues(.2),
 5                       tight_layout = True)
 6bar_num = 10
 7i = 138
 8sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
 9bars = ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
10               tick_label = sel_df.columns,
11               width = sel_df.iloc[i],
12               color = [color_dict[col] for col in sel_df.columns],
13               alpha = 0.8)
14
15cur_year = expanded_df.index[i].strftime('%Y-%m')
16ax.set_title(f'Under Five Mortality Rate - {cur_year}',
17             fontsize = 'xx-large',
18             fontweight = 'bold')
19ax.set_ylim(10.8, 0.2)
20ax.set_facecolor(plt.cm.Blues(.2))
21ax.tick_params(labelsize = 'medium')
22ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
23ax.set_axisbelow(True)
24[spine.set_visible(False) for spine in ax.spines.values()]
25
26for bar in bars:
27    width = bar.get_width()
28    ax.annotate(f'{width:.0F}',
29                xy = (width , bar.get_y() + bar.get_height() / 2),
30                xytext = (-25, 0),
31                textcoords = "offset points",
32                fontsize = 'xx-large',
33                fontweight = 'bold',
34                ha = 'right',
35                va = 'center')

Top ten countries with highest Under Five Mortality Rates displaying rate on bars

Top ten countries with highest Under Five Mortality Rates displaying rate on bars

Display a random sample of bar charts for different year

The expanded dataframe now has 415 rows, the final animation is created by generating 415 bar charts and creating a sequence of these charts. It is helpful to take a random sample of the dataset to validate that the bar charts are displayed as expected. The random rows are selected using the dataframe sample function.

 1sample_num = 5
 2d_df = expanded_df.sample(n=sample_num, random_state=1)
 3r_df = rank_df.loc[d_df.index]
 4
 5fig, axs = plt.subplots(nrows = 1,
 6                        ncols = sample_num,
 7                        figsize = (15, 5),
 8                        facecolor = plt.cm.Blues(.2),
 9                        tight_layout = True)
10
11for i, ax in enumerate(axs.flatten()):
12    sel_df = d_df.iloc[:, list(r_df.iloc[i] <= bar_num)]
13    bars = ax.barh(y = r_df.iloc[:, list(r_df.iloc[i] <= bar_num)].iloc[i],
14                   tick_label = sel_df.columns,
15                   width = sel_df.iloc[i],
16                   color = [color_dict[col] for col in sel_df.columns],
17                   alpha = 0.8)
18
19    cur_year = d_df.index[i].strftime('%Y-%m')
20    ax.set_title(f'{cur_year}',
21                 fontsize = 'large',
22                 fontweight = 'bold')
23    ax.set_ylim(10.8, 0.2)
24    ax.set_facecolor(plt.cm.Blues(.2))
25    ax.tick_params(labelsize = 'medium')
26    ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
27    ax.set_axisbelow(True)
28    [spine.set_visible(False) for spine in ax.spines.values()]
29
30    for bar in bars:
31        width = bar.get_width()
32        ax.annotate(f'{width:.0F}',
33                    xy = (width , bar.get_y() + bar.get_height() / 2),
34                    xytext = (-5, 0),
35                    textcoords = "offset points",
36                    fontsize = 'medium',
37                    fontweight = 'bold',
38                    ha = 'right',
39                    va = 'center')
40
41plt.show()

Sample bar charts od U5MR from expanded dataset

Sample bar charts od U5MR from expanded dataset

Create bar chart race of highest U5MR over the years

The animation is created by using the FuncAnimation function in Matplotlib. As there are 415 rows in the expladed dataset, this animation can take some time to generate. Note that defining the functions is instantaneous, the time is required when saving the animation to either gif, html or mp4.

 1def update(i):
 2    ax.clear()
 3
 4    p = expanded_df.columns.map(len).max()
 5    bar_num = 10
 6    sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
 7    bars = ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
 8                   tick_label = [x.rjust(p, ' ') for x in sel_df.columns],
 9                   width = sel_df.iloc[i],
10                   color = [color_dict[col] for col in sel_df.columns],
11                   alpha = 0.8)
12    plt.setp(ax.get_xticklabels(), fontsize='x-small')
13    plt.setp(ax.get_yticklabels(), fontsize='small', fontfamily = 'monospace')
14
15    cur_year = expanded_df.index[i].strftime('%Y')
16    ax.set_title(f'Under Five Mortality Rate - {cur_year}',
17                 fontsize = 'larger',
18                 fontweight = 'bold',
19                 loc = 'center')
20    ax.set_ylim(10.8, 0.2)
21    ax.set_facecolor(plt.cm.Blues(.2))
22    ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
23    ax.set_axisbelow(True)
24    [spine.set_visible(False) for spine in ax.spines.values()]
25
26    for bar in bars:
27        width = bar.get_width()
28        ax.annotate(f'{width:.0F}',
29                    xy = (width , bar.get_y() + bar.get_height() / 2),
30                    xytext = (-20, 0),
31                    textcoords = "offset points",
32                    fontsize = 'small',
33                    ha = 'right',
34                    va = 'center')    
35
36fig, ax = plt.subplots(figsize = (8, 5),
37                       facecolor = plt.cm.Blues(.2),
38                       dpi = 150,
39                       tight_layout = True)
40
41u5mr_anim = anim.FuncAnimation(
42    fig = fig,
43    func = update,
44    frames = len(expanded_df),
45    interval = 300)

Generate the gif file.

1u5mr_anim.save('U5MR_bar_chart_race_all_years.gif')

Bar chart race showing top ten countries with highest under five mortality rate from 1950 to 2019

Bar chart race showing top ten countries with highest under five mortality rate from 1950 to 2019"

MP4 file of bar chart race from 1950 to 2019

Wrap up bar chart race creation in functions

There are a number of steps in preparing the data and then expanding, ranking and finally generating the bar chart race. The following wraps this up in a series of functions so these can be used to create a bar chart race from a similar dataset. Three functions are created; one to prepare the data in wide format; one to expand the data; and the final function to create the animation.

prepare_data - Function to get the appropriate data and convert to wide format.

 1def prepare_data(df, highest = True):
 2    # Get all the countries in the top 10 for the years
 3    fields = ['Country.Name'] + [str(x) for x in range(1950, 2020)]
 4    sel_df = df[fields].set_index('Country.Name')
 5    countries = list(sel_df.apply(
 6        lambda x: x.sort_values(ascending = not highest).head(10),
 7        axis = 0).index)
 8
 9    # Create color map for countries
10    cols = plt.cm.tab20.colors + plt.cm.Dark2.colors + plt.cm.Set3.colors + plt.cm.tab20b.colors
11    color_dict = {x[0]:x[1] for x in zip(countries, cols)}
12
13    # Get the data for the countries of interest
14    data_df = (df.drop(['ISO.Code', 'Uncertainty.Bounds'], axis=1)
15               [df['Country.Name'].isin(countries)]).copy()
16
17    # Set index to "Country.Name" and Transpose the dataframe
18    wide_df = data_df.set_index('Country.Name').T
19
20    # Remove the column name
21    wide_df.rename_axis(None, axis=1, inplace=True)
22
23    # Convert index to datetime
24    wide_df.index = pd.to_datetime([f"{x}-12-31" for x in wide_df.index])
25
26    return wide_df, color_dict

expand_data - Function to expand the data and create a ranking dataframe.

 1def expand_data(df, highest = True):
 2    e_df = df.asfreq('2M')
 3
 4    # Create ranking dataset
 5    r_df = e_df.rank(axis = 1, method = 'first', ascending = highest)
 6
 7    # Interpolate
 8    e_df = e_df.interpolate()
 9    r_df = r_df.interpolate()
10
11    # Remove duplicate ranks from the same row
12    r_df = r_df.where(~r_df.apply(pd.Series.duplicated, axis=1), r_df*1.01)
13
14    return e_df, r_df

create_animation - Function to create the animation with the dataframes.

 1def create_animation(expanded_df, rank_df, color_dict, highest = True):
 2    def update2(i):
 3        ax.clear()
 4
 5        p = expanded_df.columns.map(len).max()
 6        bar_num = 10
 7        sel_df = expanded_df.iloc[:, list(rank_df.iloc[i] <= bar_num)]
 8        bars = ax.barh(y = rank_df.iloc[:, list(rank_df.iloc[i] <= bar_num)].iloc[i],
 9                       tick_label = [x.rjust(p, ' ') for x in sel_df.columns],
10                       width = sel_df.iloc[i],
11                       color = [color_dict[col] for col in sel_df.columns],
12                       alpha = 0.8)
13
14        plt.setp(ax.get_xticklabels(), fontsize='small')
15        plt.setp(ax.get_yticklabels(), fontsize='medium', fontfamily = 'monospace')
16
17        cur_year = expanded_df.index[i].strftime('%Y')
18        ax.set_title(f'Under Five Mortality Rate - {cur_year}',
19                     fontsize = 'larger',
20                     fontweight = 'bold',
21                     loc = 'right')
22        ax.set_ylim(10.8, 0.2)
23        if not highest:
24            ax.set_ylim(0.2, 10.8)
25        ax.set_facecolor(plt.cm.Blues(.2))
26        ax.grid(True, axis = 'x', color=plt.cm.Blues(.1))
27        ax.set_axisbelow(True)
28        [spine.set_visible(False) for spine in ax.spines.values()]
29
30
31        h_offset = -20 if highest else 2
32        h_align = 'right' if highest else 'left'
33        for bar in bars:
34            width = bar.get_width()
35            dislpay_value = f'{width:.0F}' if highest else f'{width:.1F}'
36            ax.annotate(dislpay_value,
37                        xy = (width , bar.get_y() + bar.get_height() / 2),
38                        xytext = (h_offset, 0),
39                        textcoords = "offset points",
40                        fontsize = 'small',
41                        ha = h_align,
42                        va = 'center')    
43
44    fig, ax = plt.subplots(figsize = (8, 3),
45                           facecolor = plt.cm.Blues(.2),
46                           dpi = 150,
47                           tight_layout = True)
48
49    data_anim = anim.FuncAnimation(
50        fig = fig,
51        func = update2,
52        frames = len(expanded_df),
53        interval = 200)
54
55    return data_anim

Finally, call these functions to create a bar chart race for countries with either the highest or lowest Under Five Mortality Rates over time. Setting highest = False creates a bar chart race for the countries with the lowest Under Five Mortality Rates from 1950 to 2019.

 1highest = False
 2# 1. Prepare the data
 3df, col_dict = prepare_data(u5mr_med_df, highest)
 4
 5# 2. Expand the data
 6e_df, r_df = expand_data(df)
 7
 8# 3. Create animation
 9data_anim = create_animation(e_df, r_df, col_dict, highest)
10
11# 4. Save animation as gif
12data_anim.save('Bar_chart_race_U5MR_lowest_countries.gif')

Bar chart race showing the countries with lowest under five mortality rate from 1950 to 2019

Bar chart race showing the countries with lowest under five mortality rate from 1950 to 2019

MP4 version available here - MP4 file for bar chart race showing the countries with lowest under five mortality rate from 1950 to 2019

Display U5MR bar chart race for selected countries

The same functions can also be used to compare changes to specific countries over the years.

 1highest = False
 2# 1. Prepare the data
 3df, col_dict = prepare_data(selected_df, highest)
 4
 5# 2. Expand the data
 6e_df, r_df = expand_data(df)
 7
 8# 3. Create animation
 9data_anim = create_animation(e_df, r_df, col_dict, highest)
10
11# 4. Save animation as gif
12data_anim.save('Bar_chart_race_U5MR_selected_countries.gif')

Bar chart race showing under five mortality rate changes for selected countries

Bar chart race showing under five mortality rate changes for selected countries

MP4 version available here - MP4 file for Bar chart race showing under five mortality rate changes for selected countries

Conclusion

The creation of a Bar Chart Race using Matplotlib was improved to handle the countries changing over time. The main challenge is to keep consistent bar colors for each country and this was achieved by creating a color map for all of the countries that appear in the bar chart race. A bar chart race is an animated sequence of bar charts, so the size and number of the charts has an impact on the final size of the gif or mp4 file. The current value for each country is displayed on the bar and this makes it easier to see changes over time.

It was noted that the position of the y-axis changes to accomodate the labels for the countries. This was resolved by right-padding the string for the country name with white space and setting the font to a constant width.

Python Notes on Software Development Notes

Create fake data with Faker in Python

Faker installation and use

Show all the Faker methods

Fake methods for Name

Generate sample fake names

Regenerating the same fake data - Seed

Localized fake data

Create a chart with fake data

Conclusion

Extract data from JSON in Pandas Dataframe

Convert JSON data to Dict

Load csv data with some JSON into Dataframe

Extract an element from the JSON object to a new column

Extract all elements from JSON at once

Extract elements from different JSON objects

Countries with highest deaths from covid

Load Covid data

Load Vaccination data

Load population data

Compare countries in the datasets

Merge datasets on country

Overview of Countries with highest deaths from covid per capita

Vaccination status in countries with highest deaths

Conclusion

How to filter a Pandas DataFrame

Filtering data in Pandas DataFrame

Load sample data

Filter by numeric data in a column

Numeric data Greater Than

Numeric data Greater Than or Equal To

Filter by text in a column

Filter by partial text match in columns

Filter on names starting with a letter

Filter on names ending with a letter

Filter on names containing text

Filter by multiple search criteria

Filter by multiple columns

Filter the full names dataset

Filter using Filter method

Conclusion

Create interactive chart with matplotlib and ipywidgets

Slider with Interact

Widget layout

Rolling average of Deaths from COVID-19

Using ipywidgets to modify days for rolling average

Conclusion

Three Dimensional Shapes and Shading

Cross product

Length of Cross Product

Triangle composed of 3 vectors

Shading

Cube

Sphere

Torus

Conclusion

Speed up image file transfer by zipping

Google Colab (Colaboratory)

Image dataset

Transfer image files individually

Transfer zip file and unzip image files

Transfer all files

Compare transfer performance

Conclusion

Three Dimensional Vectors and Dot Product

3D vectors

Viewing 3D vectors

Length of 3D vector

Dot product

Use of dot product

Angle

Length

Conclusion

Two Dimensional shapes

Basic shapes

Polygon

Plot complex polygons

Trigonometry and "sohcahtoa"

Conversion between Degrees and Radians

Validation of SOHCAHTOA

Top ten countries with highest deaths from Covid-19 on November 23

Top ten countries with highest deaths from Covid-19 on November 23