How to Read CSV File from URL into a Pandas DataFrame
How can we read a CSV file from a URL into a Pandas DataFrame?
Example scenario
Let’s see a real-life example of how we might come across a CSV file to download.
Suppose we want to grab the Chicago Home Price Index data from Fred Economic Data.
There is an option to DOWNLOAD
the CSV (data)
on that page, which will download the CSV data locally.
If we right click CSV (data)
and select Copy link address
, we’ll find the URL that will directly download the CSV data onto our machine.
This URL is quite long, but it can be reduced down to the following URL.
https://fred.stlouisfed.org/graph/fredgraph.csv?id=CHXRSA
Read CSV files using requests
We can use requests
to read a CSV file from a URL.
import requests
import pandas as pd
url = 'https://fred.stlouisfed.org/graph/fredgraph.csv?id=CHXRSA'
r = requests.get(url)
open('temp.csv', 'wb').write(r.content)
df = pd.read_csv('temp.csv')
Read up on the
requests
library in Python.
We can specify the separator using sep
.
df = pd.read_csv('temp.csv', sep=';')
We can also skip the first n
rows or last n
rows.
df = pd.read_csv('temp.csv', skiprows=n, skipfooter=n)
Read more on Panda’s
read_csv()
function. There are lots of parameters we can change.