Getting five- and ten-day weather forecasts
The site that we are currently scraping the weather forecast from (weather.com) also provides the weather forecast for
five and ten days, so in this section, we are going to implement methods to parse these forecast options as well.
The markup of the pages that present data for five and ten days are very similar; they have the same DOM structure and share the same CSS classes, which makes it easier for us to implement just one method that will work for both options. Let's go ahead and add a new method to the wheater_com_parser.py file with the following contents:
def _parse_list_forecast(self, content, args):
criteria = {
'date-time': 'span',
'day-detail': 'span',
'description': 'td',
'temp': 'td',
'wind': 'td',
'humidity': 'td',
}
bs = BeautifulSoup(content, 'html.parser')
forecast_data = bs.find('table', class_='twc-table')
container = forecast_data.tbody
return self._parse(container, criteria)
As I mentioned before, the DOM for the five- and ten-day weather forecasts is very similar, so we create the _parse_list_forecast method, which can be used for both options. First, we define the criteria:
- The date-time is a span element and contains a string representing the day of the week
- The day-detail is a span element and contains a string with the date, for example, SEP 29
- The description is a TD element and contains the weather conditions, for example, Cloudy
- temp is a TD element and contains temperature information such as high and low temperature
- wind is a TD element and contains wind information
- humidity is a TD element and contains humidity information
Now that we have the criteria, we create a BeatufulSoup object, passing the content and the html.parser. All the data that we would like to get is on the table with a CSS class named twc-table. We find the table and define the tbody element as a container.
Finally, we run the _parse method, passing the container and the criteria that we defined. The return of this function will look something like this:
[{'date-time': 'Today',
'day-detail': 'SEP 28',
'description': 'Partly Cloudy',
'humidity': '78%',
'temp': '60°50°',
'wind': 'ESE 10 mph '},
{'date-time': 'Fri',
'day-detail': 'SEP 29',
'description': 'Partly Cloudy',
'humidity': '79%',
'temp': '57°48°',
'wind': 'ESE 10 mph '},
{'date-time': 'Sat',
'day-detail': 'SEP 30',
'description': 'Partly Cloudy',
'humidity': '77%',
'temp': '57°49°',
'wind': 'SE 10 mph '},
{'date-time': 'Sun',
'day-detail': 'OCT 1',
'description': 'Cloudy',
'humidity': '74%',
'temp': '55°51°',
'wind': 'SE 14 mph '},
{'date-time': 'Mon',
'day-detail': 'OCT 2',
'description': 'Rain',
'humidity': '87%',
'temp': '55°48°',
'wind': 'SSE 18 mph '}]
Another method that we need to create is a method that will prepare the data for us, for example, parsing and converting temperature values and creating a Forecast object. Add a new method called _prepare_data with the following content:
def _prepare_data(self, results, args):
forecast_result = []
self._unit_converter.dest_unit = args.unit
for item in results:
match = self._temp_regex.search(item['temp'])
if match is not None:
high_temp, low_temp = match.groups()
try:
dateinfo = item['weather-cell']
date_time, day_detail = dateinfo[:3], dateinfo[3:]
item['date-time'] = date_time
item['day-detail'] = day_detail
except KeyError:
pass
day_forecast = Forecast(
self._unit_converter.convert(item['temp']),
item['humidity'],
item['wind'],
high_temp=self._unit_converter.convert(high_temp),
low_temp=self._unit_converter.convert(low_temp),
description=item['description'].strip(),
forecast_date=f'{item["date-time"]} {item["day-
detail"]}',
forecast_type=self._forecast_type)
forecast_result.append(day_forecast)
return forecast_result
This method is quite simple. First, loop through the results and apply the regex that we created to split the high and low temperatures stored in item['temp']. If there's a match, it will get the groups and assign the value to high_temp and low_temp.
After that, we create a Forecast object and append it to a list that will be returned later on.
Lastly, we add the method that will be invoked when the -5d or -10d flag is used. Create another method called _five_and_ten_days_forecast with the following contents:
def _five_and_ten_days_forecast(self, args):
content = self._request.fetch_data(args.forecast_option.value,
args.area_code)
results = self._parse_list_forecast(content, args)
return self._prepare_data(results)
This method only fetches the contents of the page passing the forecast_option value and the area code, so it will be possible to build the URL to perform the request. When the data is returned, we pass it down to the _parse_list_forecast, which will return a list of Forecast objects (one for each day); finally, we prepare the data to be returned using the _prepare_data method.
Before we run the command, we need to enable this option in the command line tool that we implemented; go over to the __main__.py file, and, just after the definition of the -td flag, add the following code:
argparser.add_argument('-5d', '--fivedays',
dest='forecast_option',
action='store_const',
const=ForecastType.FIVEDAYS,
help='Shows the weather forecast for the next
5 days')
Now, run the application again, but this time using the -5d or --fivedays flag:
$ python -m weatherterm -u Fahrenheit -a SWXX2372:1:SW -p WeatherComParser -5d
It will produce the following output:
>> [Today SEP 28]
High 60° / Low 50° (Partly Cloudy)
Wind: ESE 10 mph / Humidity: 78%
>> [Fri SEP 29]
High 57° / Low 48° (Partly Cloudy)
Wind: ESE 10 mph / Humidity: 79%
>> [Sat SEP 30]
High 57° / Low 49° (Partly Cloudy)
Wind: SE 10 mph / Humidity: 77%
>> [Sun OCT 1]
High 55° / Low 51° (Cloudy)
Wind: SE 14 mph / Humidity: 74%
>> [Mon OCT 2]
High 55° / Low 48° (Rain)
Wind: SSE 18 mph / Humidity: 87%
To wrap this section up, let's include the option to get the weather forecast for the next ten days as well, in the __main__.py file, just below the -5d flag definition. Add the following code:
argparser.add_argument('-10d', '--tendays',
dest='forecast_option',
action='store_const',
const=ForecastType.TENDAYS,
help='Shows the weather forecast for the next
10 days')
If you run the same command as we used to get the five-day forecast but replace the -5d flag with -10d, like so:
$ python -m weatherterm -u Fahrenheit -a SWXX2372:1:SW -p WeatherComParser -10d
You should see the ten-day weather forecast output:
>> [Today SEP 28]
High 60° / Low 50° (Partly Cloudy)
Wind: ESE 10 mph / Humidity: 78%
>> [Fri SEP 29]
High 57° / Low 48° (Partly Cloudy)
Wind: ESE 10 mph / Humidity: 79%
>> [Sat SEP 30]
High 57° / Low 49° (Partly Cloudy)
Wind: SE 10 mph / Humidity: 77%
>> [Sun OCT 1]
High 55° / Low 51° (Cloudy)
Wind: SE 14 mph / Humidity: 74%
>> [Mon OCT 2]
High 55° / Low 48° (Rain)
Wind: SSE 18 mph / Humidity: 87%
>> [Tue OCT 3]
High 56° / Low 46° (AM Clouds/PM Sun)
Wind: S 10 mph / Humidity: 84%
>> [Wed OCT 4]
High 58° / Low 47° (Partly Cloudy)
Wind: SE 9 mph / Humidity: 80%
>> [Thu OCT 5]
High 57° / Low 46° (Showers)
Wind: SSW 8 mph / Humidity: 81%
>> [Fri OCT 6]
High 57° / Low 46° (Partly Cloudy)
Wind: SW 8 mph / Humidity: 76%
>> [Sat OCT 7]
High 56° / Low 44° (Mostly Sunny)
Wind: W 7 mph / Humidity: 80%
>> [Sun OCT 8]
High 56° / Low 44° (Partly Cloudy)
Wind: NNE 7 mph / Humidity: 78%
>> [Mon OCT 9]
High 56° / Low 43° (AM Showers)
Wind: SSW 9 mph / Humidity: 79%
>> [Tue OCT 10]
High 55° / Low 44° (AM Showers)
Wind: W 8 mph / Humidity: 79%
>> [Wed OCT 11]
High 55° / Low 42° (AM Showers)
Wind: SE 7 mph / Humidity: 79%
>> [Thu OCT 12]
High 53° / Low 43° (AM Showers)
Wind: NNW 8 mph / Humidity: 87%
As you can see, the weather was not so great here in Sweden while I was writing this book.