Data Request and Extract From Star Wars ApI

Description: Using the requests library, download all the people in the Star Wars universe using the Star Wars API (https://swapi.dev/documentation). Show the name of the oldest person (or robot or alien) and list the titles of all the films they appeared in.

The Requests Package

The requests module allows us to send HTTP requests using Python.

The HTTP request returns a Response Object with all the response data (content, encoding, status, etc).

In this blog, we use it to fetch characters from StarWars from the the Star Wars API.

A Quick Look at Our 1st Character – Luke Skywalker

It’s quite easy to featch content from web API using request.

import requests

#Fetch
resp = requests.get('https://swapi.dev/api/people/1')

#Show Request
resp

<Response [200]>

Let’s look at our first character!

#Transform to json
character_json = resp.json()
character_json

{'name': 'Luke Skywalker',
 'height': '172',
 'mass': '77',
 'hair_color': 'blond',
 'skin_color': 'fair',
 'eye_color': 'blue',
 'birth_year': '19BBY',
 'gender': 'male',
 'homeworld': 'http://swapi.dev/api/planets/1/',
 'films': ['http://swapi.dev/api/films/1/',
  'http://swapi.dev/api/films/2/',
  'http://swapi.dev/api/films/3/',
  'http://swapi.dev/api/films/6/'],
 'species': [],
 'vehicles': ['http://swapi.dev/api/vehicles/14/',
  'http://swapi.dev/api/vehicles/30/'],
 'starships': ['http://swapi.dev/api/starships/12/',
  'http://swapi.dev/api/starships/22/'],
 'created': '2014-12-09T13:50:51.644000Z',
 'edited': '2014-12-20T21:17:56.891000Z',
 'url': 'http://swapi.dev/api/people/1/'}

Much clear when transforming JASON into pandas dataframe.

import pandas as pd

#Transform to DataFrame
character_df = pd.json_normalize(data)
character_df

All right, next step, we gonna load all characters from Star Wars.

Welcome All Characters from Star Wars

Before we start, we need to deal with something tricky. The detail about somebody cannot be found in the Star War dataset.

data_full = requests.get('https://swapi.dev/api/people/17').json()
data_full

{'detail': 'Not found'}

So we have to check wheather each person exists or not. How can we do that?

I find that it returns a status code of 200 when requests is succesful.

c1 = requests.get('https://swapi.dev/api/people/1')
c1.status_code

200

However, it returns a status code of 404 when requests fails.

person17 = requests.get('https://swapi.dev/api/people/17')
person17.status_code

404

Also, we can see we have 82 characters in total in the dataset.

count = requests.get('https://swapi.dev/api/people').json()['count']
count

82

Therefore, we loop through until we get 82 successful requests:

df_full = pd.DataFrame()

i = 1

while df_full.shape[0] < count: 
    
    #Make request
    character_request = requests.get('https://swapi.dev/api/people/' + str(i))

    #Check if a character was found:
    status_code = character_request.status_code
    if status_code == 200:

        #Transform to JSON
        character_json = character_request.json()

        #Transform to pandas dataframe
        character_df = pd.json_normalize(character_json)

        #Concatenate to existing dataframe
        df_full = df_full.append(character_df)

    i += 1

Set the column ‘name’ as the index of the dataframe and replace all ‘unknown’ in the column ‘birth_year’ as np.nan.

df_full = df_full.set_index(['name'])
df_full["birth_year"] = df_full["birth_year"].replace('unknown',np.nan)
df_full

82 rows × 15 columns

Get The Oldest Character

birth_year = pd.DataFrame(df_full.birth_year)
birth_year

82 rows × 1 columns

set([each[-3:] for each in df_full.birth_year.to_list() if each is not np.nan])

{'BBY'}

Here we can see that all characters born in XX BBY (except from ‘NaN’).

What does BBY mean?

It means Before the Battle of Yevin. Now it is clear that the number before BBY larger, older the character is.

Who is the oldest character in the Star Wars? ‘Yoda’

birth_year['age'] = (birth_year.birth_year.str.
                   replace('BBY','').
                   astype('float'))
birth_year = birth_year.sort_values(['age'],ascending = False)

birth_year.head(1)

All the Films Yoda Appeared in

films = df_full.loc['Yoda',:]['films']

films_title = []
for each in films:
    films_title.append(requests.get(each).json()['title'])

films_title

['The Empire Strikes Back',
 'Return of the Jedi',
 'The Phantom Menace',
 'Attack of the Clones',
 'Revenge of the Sith']