scraping.scrape_programs
@file scrape_programs.py @brief Scrapes the public QCC website for all programs of study with full details.
@details This module scrapes https://www.qcc.edu/programs for the complete list of QCC academic programs and certificates, then visits each program's detail page to extract: - Program description - Total credits required - Area of study (derived from the URL path) - List of required course codes
Like scrape_classes_page.py, this scraper uses Requests and BeautifulSoup since the public QCC website renders content statically. Area of study is derived from the URL path rather than page headings, which proved unreliable. Description extraction scans all content elements for the first substantial paragraph containing program-related keywords, since QCC wraps descriptions in nested div elements rather than placing them as direct siblings of headings.
@date 2026
@par Input https://www.qcc.edu/programs — public QCC programs listing page
@par Output qcc_programs.json — list of program dictionaries with full detail fields
@par Dependencies - requests - beautifulsoup4 - json (stdlib) - re (stdlib) - time (stdlib)
@par Usage python scrape_programs.py
1# ============================================================================= 2# 3# Copyright 2026 4# 5# Author: Noel Mensah 6# GitHub: https://github.com/LSilver17/CSC212---AI-Agent 7# 8# ============================================================================= 9 10""" 11@file scrape_programs.py 12@brief Scrapes the public QCC website for all programs of study with full details. 13 14@details 15This module scrapes https://www.qcc.edu/programs for the complete list of QCC 16academic programs and certificates, then visits each program's detail page to extract: 17 - Program description 18 - Total credits required 19 - Area of study (derived from the URL path) 20 - List of required course codes 21 22Like scrape_classes_page.py, this scraper uses Requests and BeautifulSoup 23since the public QCC website renders content statically. Area of study is 24derived from the URL path rather than page headings, which proved unreliable. 25Description extraction scans all content elements for the first substantial 26paragraph containing program-related keywords, since QCC wraps descriptions 27in nested div elements rather than placing them as direct siblings of headings. 28 29@date 2026 30 31@par Input 32 https://www.qcc.edu/programs — public QCC programs listing page 33 34@par Output 35 qcc_programs.json — list of program dictionaries with full detail fields 36 37@par Dependencies 38 - requests 39 - beautifulsoup4 40 - json (stdlib) 41 - re (stdlib) 42 - time (stdlib) 43 44@par Usage 45 python scrape_programs.py 46""" 47 48import json 49import re 50import time 51import requests 52from bs4 import BeautifulSoup 53from datetime import datetime 54 55## @brief Base URL for constructing absolute links from relative hrefs. 56BASE_URL = "https://www.qcc.edu" 57 58## @brief URL of the QCC public programs listing page. 59PROGRAMS_URL = "https://www.qcc.edu/programs" 60 61## @brief Output path for the scraped JSON file. 62OUTPUT = "qcc_programs.json" 63 64## @brief HTTP headers to send with each request to mimic a browser. 65HEADERS = { 66 "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" 67} 68 69 70def clean_text(text): 71 """ 72 @brief Removes excess whitespace from a string. 73 74 @param text The raw string to clean. 75 @return A normalized single-line string with no extra whitespace. 76 """ 77 return " ".join(text.split()).strip() 78 79 80def area_from_url(url): 81 """ 82 @brief Derives the area of study from the program's URL path segment. 83 84 @details 85 QCC program URLs follow the pattern: 86 https://www.qcc.edu/[area-slug]/[program-slug] 87 88 This function extracts the area slug from the first path segment after 89 the domain and converts it to title case (e.g. "applied-technologies" 90 becomes "Applied Technologies"). 91 92 This approach is more reliable than parsing <h2> tags from the programs 93 listing page, which did not consistently match program entries. 94 95 @param url The full URL of the program detail page. 96 @return The area of study as a title-cased string, or None if the URL 97 does not contain a recognizable path segment. 98 99 @par Example 100 @code 101 area_from_url("https://www.qcc.edu/applied-technologies/some-program") 102 # -> "Applied Technologies" 103 @endcode 104 """ 105 match = re.search(r"qcc\.edu/([^/]+)/", url) 106 if match: 107 slug = match.group(1) 108 return slug.replace("-", " ").title() 109 return None 110 111 112def fetch_program_detail(url): 113 """ 114 @brief Fetches and extracts detail fields from a QCC program detail page. 115 116 @details 117 Sends an HTTP GET request to the program detail URL and parses the response. 118 Extracts three categories of information: 119 120 Description: 121 Scans all <p> and <div> elements in the main content area for the first 122 block exceeding 80 characters that contains program-related keywords 123 (program, students, learn, career, degree, skills, course, prepares, 124 provides, designed, pathway). Blocks matching navigation or boilerplate 125 patterns are skipped. This approach handles QCC's nested div structure 126 where the description is not a direct sibling of the <h1> heading. 127 128 Total Credits: 129 Searches all curriculum tables for a row containing "Total Credits Required" 130 and reads the numeric value from the last cell of that row. 131 132 Required Courses: 133 Collects all course codes matching the pattern "[A-Z]{2,4} [0-9]{3}" from 134 all curriculum table cells, preserving order and deduplicating. 135 136 @param url The full URL of the program detail page. 137 @return A dictionary with keys: 138 - "description" (str or None) 139 - "total_credits" (int or None) 140 - "required_courses" (list of str) 141 142 @note Returns a dictionary with None/empty values if the request fails 143 or the page structure does not match expected patterns. 144 """ 145 result = { 146 "description": None, 147 "total_credits": None, 148 "required_courses": [], 149 } 150 try: 151 resp = requests.get(url, headers=HEADERS, timeout=15) 152 if resp.status_code != 200: 153 return result 154 155 soup = BeautifulSoup(resp.text, "html.parser") 156 main = soup.find("main") or soup.find("body") 157 if not main: 158 return result 159 160 # --- Description --- 161 skip_pattern = re.compile( 162 r"^(Certificate|Associate|This semester|Become an|Apply|Submit|Meet with|" 163 r"Skip to|Contact|Visit|Open Menu|Fulltext|Local|Life-changing|Copyright|" 164 r"In-State|Out-of-State|Some programs|This program may be|High School|" 165 r"Ways to Take|Requirements|Locations|Timeline|Cost|Program Overview|" 166 r"What Will You Learn|Curriculum|Connections|Career|Have more questions)", 167 re.IGNORECASE 168 ) 169 for el in main.find_all(["p", "div"]): 170 text = clean_text(el.get_text()) 171 if len(text) > 80 and not skip_pattern.match(text): 172 if re.search( 173 r"(program|students|learn|career|degree|skills|course|prepares?|provides?|designed|pathway)", 174 text, re.IGNORECASE 175 ): 176 result["description"] = text 177 break 178 179 # --- Total Credits & Required Courses from curriculum table --- 180 courses = [] 181 for table in main.find_all("table"): 182 rows = table.find_all("tr") 183 for row in rows: 184 cells = row.find_all(["td", "th"]) 185 if not cells: 186 continue 187 188 row_text = clean_text(row.get_text()) 189 190 # Total credits row 191 if re.search(r"Total\s+Credits?\s+Required", row_text, re.IGNORECASE): 192 for cell in reversed(cells): 193 num = re.search(r"\b(\d+)\b", clean_text(cell.get_text())) 194 if num: 195 result["total_credits"] = int(num.group(1)) 196 break 197 198 # Course code cells 199 for cell in cells: 200 text = clean_text(cell.get_text()) 201 matches = re.findall(r"\b([A-Z]{2,4}\s+\d{3}[A-Z]?)\b", text) 202 for m in matches: 203 if m not in courses: 204 courses.append(m) 205 206 result["required_courses"] = courses 207 208 except Exception: 209 pass 210 211 return result 212 213 214def scrape_programs(): 215 """ 216 @brief Scrapes the QCC programs listing page and fetches details for each program. 217 218 @details 219 Sends a GET request to PROGRAMS_URL and parses the response to build an 220 initial list of programs with their names, degree types, and URLs. Area of 221 study is derived immediately from the URL via area_from_url(). Then iterates 222 through each program, calls fetch_program_detail() to populate description, 223 total_credits, and required_courses, and applies a 0.3 second polite delay 224 between requests. 225 226 The programs listing page uses <tr> table rows for individual program entries. 227 The program name link is in the first cell and degree type is in the second cell. 228 229 @return A list of program dictionaries with fields: 230 name, area_of_study, degree_type, url, description, 231 total_credits, required_courses, scraped_at. 232 """ 233 print(f"Fetching {PROGRAMS_URL}...") 234 resp = requests.get(PROGRAMS_URL, headers=HEADERS, timeout=30) 235 soup = BeautifulSoup(resp.text, "html.parser") 236 237 programs = [] 238 239 main = soup.find("main") or soup.find("body") 240 241 for el in main.find_all(["h2", "tr"]): 242 if el.name == "tr": 243 cells = el.find_all("td") 244 if len(cells) < 1: 245 continue 246 247 link = cells[0].find("a") 248 if not link: 249 continue 250 251 name = clean_text(link.get_text()) 252 href = link.get("href", "") 253 degree_type = clean_text(cells[1].get_text()) if len(cells) > 1 else None 254 full_url = BASE_URL + href if href.startswith("/") else href 255 256 if not name: 257 continue 258 259 programs.append({ 260 "name": name, 261 "area_of_study": area_from_url(full_url), 262 "degree_type": degree_type, 263 "url": full_url, 264 "description": None, 265 "total_credits": None, 266 "required_courses": [], 267 "scraped_at": datetime.now().isoformat(), 268 }) 269 270 print(f"Found {len(programs)} programs. Fetching detail pages...\n") 271 272 for i, program in enumerate(programs): 273 print(f"[{i+1}/{len(programs)}] {program['name']}") 274 detail = fetch_program_detail(program["url"]) 275 program["description"] = detail["description"] 276 program["total_credits"] = detail["total_credits"] 277 program["required_courses"] = detail["required_courses"] 278 time.sleep(0.3) 279 280 return programs 281 282 283def main(): 284 """ 285 @brief Entry point — runs the programs scraper and saves results to JSON. 286 287 @details 288 Calls scrape_programs() to collect all program data, writes the results 289 to qcc_programs.json, and prints a summary including total programs scraped, 290 descriptions populated, total credits populated, area of study populated, 291 and elapsed time. 292 """ 293 start = datetime.now() 294 print("=" * 50) 295 print(" QCC Programs of Study Scraper") 296 print(f" {start.strftime('%Y-%m-%d %H:%M:%S')}") 297 print("=" * 50) 298 299 programs = scrape_programs() 300 301 with open(OUTPUT, "w", encoding="utf-8") as f: 302 json.dump(programs, f, indent=2) 303 304 elapsed = (datetime.now() - start).seconds 305 filled_desc = sum(1 for p in programs if p["description"] is not None) 306 filled_credits = sum(1 for p in programs if p["total_credits"] is not None) 307 filled_area = sum(1 for p in programs if p["area_of_study"] is not None) 308 print(f"\n✅ Done in {elapsed}s") 309 print(f" Scraped {len(programs)} programs") 310 print(f" Descriptions populated: {filled_desc}/{len(programs)}") 311 print(f" Total credits populated: {filled_credits}/{len(programs)}") 312 print(f" Area of study populated: {filled_area}/{len(programs)}") 313 print(f" Saved to {OUTPUT}") 314 print("=" * 50) 315 316 317if __name__ == "__main__": 318 main()
BASE_URL =
'https://www.qcc.edu'
PROGRAMS_URL =
'https://www.qcc.edu/programs'
OUTPUT =
'qcc_programs.json'
HEADERS =
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}
def
clean_text(text):
71def clean_text(text): 72 """ 73 @brief Removes excess whitespace from a string. 74 75 @param text The raw string to clean. 76 @return A normalized single-line string with no extra whitespace. 77 """ 78 return " ".join(text.split()).strip()
@brief Removes excess whitespace from a string.
@param text The raw string to clean. @return A normalized single-line string with no extra whitespace.
def
area_from_url(url):
81def area_from_url(url): 82 """ 83 @brief Derives the area of study from the program's URL path segment. 84 85 @details 86 QCC program URLs follow the pattern: 87 https://www.qcc.edu/[area-slug]/[program-slug] 88 89 This function extracts the area slug from the first path segment after 90 the domain and converts it to title case (e.g. "applied-technologies" 91 becomes "Applied Technologies"). 92 93 This approach is more reliable than parsing <h2> tags from the programs 94 listing page, which did not consistently match program entries. 95 96 @param url The full URL of the program detail page. 97 @return The area of study as a title-cased string, or None if the URL 98 does not contain a recognizable path segment. 99 100 @par Example 101 @code 102 area_from_url("https://www.qcc.edu/applied-technologies/some-program") 103 # -> "Applied Technologies" 104 @endcode 105 """ 106 match = re.search(r"qcc\.edu/([^/]+)/", url) 107 if match: 108 slug = match.group(1) 109 return slug.replace("-", " ").title() 110 return None
@brief Derives the area of study from the program's URL path segment.
@details QCC program URLs follow the pattern: https://www.qcc.edu/[area-slug]/[program-slug]
This function extracts the area slug from the first path segment after the domain and converts it to title case (e.g. "applied-technologies" becomes "Applied Technologies").
This approach is more reliable than parsing
tags from the programs listing page, which did not consistently match program entries.
@param url The full URL of the program detail page. @return The area of study as a title-cased string, or None if the URL does not contain a recognizable path segment.
@par Example @code area_from_url("https://www.qcc.edu/applied-technologies/some-program")
-> "Applied Technologies"
@endcode
def
fetch_program_detail(url):
113def fetch_program_detail(url): 114 """ 115 @brief Fetches and extracts detail fields from a QCC program detail page. 116 117 @details 118 Sends an HTTP GET request to the program detail URL and parses the response. 119 Extracts three categories of information: 120 121 Description: 122 Scans all <p> and <div> elements in the main content area for the first 123 block exceeding 80 characters that contains program-related keywords 124 (program, students, learn, career, degree, skills, course, prepares, 125 provides, designed, pathway). Blocks matching navigation or boilerplate 126 patterns are skipped. This approach handles QCC's nested div structure 127 where the description is not a direct sibling of the <h1> heading. 128 129 Total Credits: 130 Searches all curriculum tables for a row containing "Total Credits Required" 131 and reads the numeric value from the last cell of that row. 132 133 Required Courses: 134 Collects all course codes matching the pattern "[A-Z]{2,4} [0-9]{3}" from 135 all curriculum table cells, preserving order and deduplicating. 136 137 @param url The full URL of the program detail page. 138 @return A dictionary with keys: 139 - "description" (str or None) 140 - "total_credits" (int or None) 141 - "required_courses" (list of str) 142 143 @note Returns a dictionary with None/empty values if the request fails 144 or the page structure does not match expected patterns. 145 """ 146 result = { 147 "description": None, 148 "total_credits": None, 149 "required_courses": [], 150 } 151 try: 152 resp = requests.get(url, headers=HEADERS, timeout=15) 153 if resp.status_code != 200: 154 return result 155 156 soup = BeautifulSoup(resp.text, "html.parser") 157 main = soup.find("main") or soup.find("body") 158 if not main: 159 return result 160 161 # --- Description --- 162 skip_pattern = re.compile( 163 r"^(Certificate|Associate|This semester|Become an|Apply|Submit|Meet with|" 164 r"Skip to|Contact|Visit|Open Menu|Fulltext|Local|Life-changing|Copyright|" 165 r"In-State|Out-of-State|Some programs|This program may be|High School|" 166 r"Ways to Take|Requirements|Locations|Timeline|Cost|Program Overview|" 167 r"What Will You Learn|Curriculum|Connections|Career|Have more questions)", 168 re.IGNORECASE 169 ) 170 for el in main.find_all(["p", "div"]): 171 text = clean_text(el.get_text()) 172 if len(text) > 80 and not skip_pattern.match(text): 173 if re.search( 174 r"(program|students|learn|career|degree|skills|course|prepares?|provides?|designed|pathway)", 175 text, re.IGNORECASE 176 ): 177 result["description"] = text 178 break 179 180 # --- Total Credits & Required Courses from curriculum table --- 181 courses = [] 182 for table in main.find_all("table"): 183 rows = table.find_all("tr") 184 for row in rows: 185 cells = row.find_all(["td", "th"]) 186 if not cells: 187 continue 188 189 row_text = clean_text(row.get_text()) 190 191 # Total credits row 192 if re.search(r"Total\s+Credits?\s+Required", row_text, re.IGNORECASE): 193 for cell in reversed(cells): 194 num = re.search(r"\b(\d+)\b", clean_text(cell.get_text())) 195 if num: 196 result["total_credits"] = int(num.group(1)) 197 break 198 199 # Course code cells 200 for cell in cells: 201 text = clean_text(cell.get_text()) 202 matches = re.findall(r"\b([A-Z]{2,4}\s+\d{3}[A-Z]?)\b", text) 203 for m in matches: 204 if m not in courses: 205 courses.append(m) 206 207 result["required_courses"] = courses 208 209 except Exception: 210 pass 211 212 return result
@brief Fetches and extracts detail fields from a QCC program detail page.
@details Sends an HTTP GET request to the program detail URL and parses the response. Extracts three categories of information:
Description: Scans all
and
elements in the main content area for the first
block exceeding 80 characters that contains program-related keywords
(program, students, learn, career, degree, skills, course, prepares,
provides, designed, pathway). Blocks matching navigation or boilerplate
patterns are skipped. This approach handles QCC's nested div structure
where the description is not a direct sibling of the
heading.
Total Credits: Searches all curriculum tables for a row containing "Total Credits Required" and reads the numeric value from the last cell of that row.
Required Courses: Collects all course codes matching the pattern "[A-Z]{2,4} [0-9]{3}" from all curriculum table cells, preserving order and deduplicating.
@param url The full URL of the program detail page. @return A dictionary with keys: - "description" (str or None) - "total_credits" (int or None) - "required_courses" (list of str)
@note Returns a dictionary with None/empty values if the request fails or the page structure does not match expected patterns.
def
scrape_programs():
215def scrape_programs(): 216 """ 217 @brief Scrapes the QCC programs listing page and fetches details for each program. 218 219 @details 220 Sends a GET request to PROGRAMS_URL and parses the response to build an 221 initial list of programs with their names, degree types, and URLs. Area of 222 study is derived immediately from the URL via area_from_url(). Then iterates 223 through each program, calls fetch_program_detail() to populate description, 224 total_credits, and required_courses, and applies a 0.3 second polite delay 225 between requests. 226 227 The programs listing page uses <tr> table rows for individual program entries. 228 The program name link is in the first cell and degree type is in the second cell. 229 230 @return A list of program dictionaries with fields: 231 name, area_of_study, degree_type, url, description, 232 total_credits, required_courses, scraped_at. 233 """ 234 print(f"Fetching {PROGRAMS_URL}...") 235 resp = requests.get(PROGRAMS_URL, headers=HEADERS, timeout=30) 236 soup = BeautifulSoup(resp.text, "html.parser") 237 238 programs = [] 239 240 main = soup.find("main") or soup.find("body") 241 242 for el in main.find_all(["h2", "tr"]): 243 if el.name == "tr": 244 cells = el.find_all("td") 245 if len(cells) < 1: 246 continue 247 248 link = cells[0].find("a") 249 if not link: 250 continue 251 252 name = clean_text(link.get_text()) 253 href = link.get("href", "") 254 degree_type = clean_text(cells[1].get_text()) if len(cells) > 1 else None 255 full_url = BASE_URL + href if href.startswith("/") else href 256 257 if not name: 258 continue 259 260 programs.append({ 261 "name": name, 262 "area_of_study": area_from_url(full_url), 263 "degree_type": degree_type, 264 "url": full_url, 265 "description": None, 266 "total_credits": None, 267 "required_courses": [], 268 "scraped_at": datetime.now().isoformat(), 269 }) 270 271 print(f"Found {len(programs)} programs. Fetching detail pages...\n") 272 273 for i, program in enumerate(programs): 274 print(f"[{i+1}/{len(programs)}] {program['name']}") 275 detail = fetch_program_detail(program["url"]) 276 program["description"] = detail["description"] 277 program["total_credits"] = detail["total_credits"] 278 program["required_courses"] = detail["required_courses"] 279 time.sleep(0.3) 280 281 return programs
@brief Scrapes the QCC programs listing page and fetches details for each program.
@details Sends a GET request to PROGRAMS_URL and parses the response to build an initial list of programs with their names, degree types, and URLs. Area of study is derived immediately from the URL via area_from_url(). Then iterates through each program, calls fetch_program_detail() to populate description, total_credits, and required_courses, and applies a 0.3 second polite delay between requests.
The programs listing page uses
@return A list of program dictionaries with fields: name, area_of_study, degree_type, url, description, total_credits, required_courses, scraped_at.
def
main():
284def main(): 285 """ 286 @brief Entry point — runs the programs scraper and saves results to JSON. 287 288 @details 289 Calls scrape_programs() to collect all program data, writes the results 290 to qcc_programs.json, and prints a summary including total programs scraped, 291 descriptions populated, total credits populated, area of study populated, 292 and elapsed time. 293 """ 294 start = datetime.now() 295 print("=" * 50) 296 print(" QCC Programs of Study Scraper") 297 print(f" {start.strftime('%Y-%m-%d %H:%M:%S')}") 298 print("=" * 50) 299 300 programs = scrape_programs() 301 302 with open(OUTPUT, "w", encoding="utf-8") as f: 303 json.dump(programs, f, indent=2) 304 305 elapsed = (datetime.now() - start).seconds 306 filled_desc = sum(1 for p in programs if p["description"] is not None) 307 filled_credits = sum(1 for p in programs if p["total_credits"] is not None) 308 filled_area = sum(1 for p in programs if p["area_of_study"] is not None) 309 print(f"\n✅ Done in {elapsed}s") 310 print(f" Scraped {len(programs)} programs") 311 print(f" Descriptions populated: {filled_desc}/{len(programs)}") 312 print(f" Total credits populated: {filled_credits}/{len(programs)}") 313 print(f" Area of study populated: {filled_area}/{len(programs)}") 314 print(f" Saved to {OUTPUT}") 315 print("=" * 50)
@brief Entry point — runs the programs scraper and saves results to JSON.
@details Calls scrape_programs() to collect all program data, writes the results to qcc_programs.json, and prints a summary including total programs scraped, descriptions populated, total credits populated, area of study populated, and elapsed time.