scraping.scrape_classes_page

@file scrape_classes_page.py @brief Scrapes the public QCC website for the full credit course listing with details.

@details This module scrapes https://www.qcc.edu/classes for the complete list of QCC credit courses, then visits each course's individual detail page to extract: - Description - Credits - Prerequisites - Semesters offered

Unlike scrape_catalog.py which uses Playwright to access The Q portal, this scraper uses the Requests library and BeautifulSoup since the public QCC website renders content statically (no JavaScript required).

Field extraction uses a line-scanning approach rather than CSS selectors, because QCC's public pages present labeled fields as plain text rather than structured HTML elements with identifying CSS classes.

@date 2026

@par Input https://www.qcc.edu/classes — public QCC course listing page

@par Output qcc_classes.json — list of course dictionaries with full detail fields

@par Dependencies - requests - beautifulsoup4 - json (stdlib) - re (stdlib) - time (stdlib)

@par Usage python scrape_classes_page.py

View Source

  1# =============================================================================
  2#
  3# Copyright 2026 
  4#
  5# Author:   Noel Mensah
  6# GitHub:   https://github.com/LSilver17/CSC212---AI-Agent
  7# 
  8# =============================================================================
  9
 10"""
 11@file scrape_classes_page.py
 12@brief Scrapes the public QCC website for the full credit course listing with details.
 13
 14@details
 15This module scrapes https://www.qcc.edu/classes for the complete list of QCC
 16credit courses, then visits each course's individual detail page to extract:
 17    - Description
 18    - Credits
 19    - Prerequisites
 20    - Semesters offered
 21
 22Unlike scrape_catalog.py which uses Playwright to access The Q portal,
 23this scraper uses the Requests library and BeautifulSoup since the public
 24QCC website renders content statically (no JavaScript required).
 25
 26Field extraction uses a line-scanning approach rather than CSS selectors,
 27because QCC's public pages present labeled fields as plain text rather than
 28structured HTML elements with identifying CSS classes.
 29
 30@date 2026
 31
 32@par Input
 33    https://www.qcc.edu/classes — public QCC course listing page
 34
 35@par Output
 36    qcc_classes.json — list of course dictionaries with full detail fields
 37
 38@par Dependencies
 39    - requests
 40    - beautifulsoup4
 41    - json (stdlib)
 42    - re (stdlib)
 43    - time (stdlib)
 44
 45@par Usage
 46    python scrape_classes_page.py
 47"""
 48
 49import json
 50import re
 51import time
 52import requests
 53from bs4 import BeautifulSoup
 54from datetime import datetime
 55
 56## @brief Base URL for constructing absolute links from relative hrefs.
 57BASE_URL    = "https://www.qcc.edu"
 58
 59## @brief URL of the QCC public classes listing page.
 60CLASSES_URL = "https://www.qcc.edu/classes"
 61
 62## @brief Output path for the scraped JSON file.
 63OUTPUT      = "qcc_classes.json"
 64
 65## @brief HTTP headers to send with each request to mimic a browser.
 66HEADERS = {
 67    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
 68}
 69
 70
 71def clean_text(text):
 72    """
 73    @brief Removes excess whitespace from a string.
 74
 75    @param text The raw string to clean.
 76    @return A normalized single-line string with no extra whitespace.
 77    """
 78    return " ".join(text.split()).strip()
 79
 80
 81def fetch_course_detail(url):
 82    """
 83    @brief Fetches and extracts detail fields from a QCC course detail page.
 84
 85    @details
 86    Sends an HTTP GET request to the course detail URL and parses the response
 87    using BeautifulSoup. Since QCC's course pages present fields as labeled
 88    plain text rather than structured HTML elements, this function splits the
 89    page's main content area into clean lines and scans for labeled field patterns.
 90
 91    Extraction rules:
 92    - Credits: finds a line matching "Credits" then reads the numeric value on the next line
 93    - Semesters Offered: finds a line matching "Semester(s) Offered" then reads the next line
 94    - Prerequisites: finds a line matching "Prerequisites?" then reads the next line
 95    - Description: selects the longest text block over 80 characters that does not
 96      match navigation or label patterns
 97
 98    A 0.3 second delay is applied between requests in scrape_classes() to avoid
 99    overloading QCC's server.
100
101    @param url The full URL of the course detail page (e.g. https://www.qcc.edu/courses/financial-accounting-i).
102    @return A dictionary with keys:
103            - "description" (str or None)
104            - "prerequisites" (str or None)
105            - "credits" (str or None)
106            - "semesters_offered" (str or None)
107
108    @note Returns a dictionary with all None values if the request fails or
109          the page structure does not match expected patterns.
110    """
111    result = {
112        "description":       None,
113        "prerequisites":     None,
114        "credits":           None,
115        "semesters_offered": None,
116    }
117    try:
118        resp = requests.get(url, headers=HEADERS, timeout=15)
119        if resp.status_code != 200:
120            return result
121
122        soup = BeautifulSoup(resp.text, "html.parser")
123        main = soup.find("main") or soup.find("body")
124        if not main:
125            return result
126
127        # Split into clean lines
128        raw_lines = [clean_text(line) for line in main.get_text("\n").splitlines() if clean_text(line)]
129
130        # --- Extract Credits ---
131        for i, line in enumerate(raw_lines):
132            if re.match(r"^Credits$", line, re.IGNORECASE):
133                if i + 1 < len(raw_lines):
134                    val = raw_lines[i + 1]
135                    if re.match(r"^\d+(\.\d+)?$", val):
136                        result["credits"] = val
137                        break
138            m = re.match(r"^Credits\s+(\d+(?:\.\d+)?)$", line, re.IGNORECASE)
139            if m:
140                result["credits"] = m.group(1)
141                break
142
143        # --- Extract Semesters Offered ---
144        for i, line in enumerate(raw_lines):
145            if re.match(r"^Semester[s]?\s+Offered$", line, re.IGNORECASE):
146                if i + 1 < len(raw_lines):
147                    result["semesters_offered"] = raw_lines[i + 1]
148                    break
149            m = re.match(r"^Semester[s]?\s+Offered\s{2,}(.+)$", line, re.IGNORECASE)
150            if m:
151                result["semesters_offered"] = clean_text(m.group(1))
152                break
153            m2 = re.match(r"^Semester[s]?\s+Offered[:\s]+(.+)$", line, re.IGNORECASE)
154            if m2:
155                val = clean_text(m2.group(1))
156                if len(val) > 0:
157                    result["semesters_offered"] = val
158                    break
159
160        # --- Extract Prerequisites ---
161        for i, line in enumerate(raw_lines):
162            if re.match(r"^Prerequisites?$", line, re.IGNORECASE):
163                if i + 1 < len(raw_lines):
164                    result["prerequisites"] = raw_lines[i + 1]
165                    break
166            m = re.match(r"^Prerequisites?\s{2,}(.+)$", line, re.IGNORECASE)
167            if m:
168                result["prerequisites"] = clean_text(m.group(1))
169                break
170            m2 = re.match(r"^Prerequisites?[:\s]+(.+)$", line, re.IGNORECASE)
171            if m2:
172                val = clean_text(m2.group(1))
173                if len(val) > 2:
174                    result["prerequisites"] = val
175                    break
176
177        # --- Extract Description ---
178        label_pattern = re.compile(
179            r"^(Area|Course Number|Semester[s]? Offered|Credits|Prerequisites?|"
180            r"Skip to|Primary|Secondary|Contact|Visit|Apply|Copyright|"
181            r"Local|Life-changing|Fulltext|Open Menu|Open Search)",
182            re.IGNORECASE
183        )
184        content = main.find("article") or main
185        text_blocks = []
186        for el in content.find_all(["p", "div", "span"]):
187            t = clean_text(el.get_text())
188            if t:
189                text_blocks.append(t)
190
191        candidates = [t for t in text_blocks if len(t) > 80 and not label_pattern.match(t)]
192        if candidates:
193            result["description"] = max(candidates, key=len)
194
195    except Exception:
196        pass
197
198    return result
199
200
201def scrape_classes():
202    """
203    @brief Scrapes the QCC classes listing page and fetches details for each course.
204
205    @details
206    Sends a GET request to CLASSES_URL and parses the response to build an
207    initial list of courses with their codes, names, and URLs. Then iterates
208    through each course, calls fetch_course_detail() to populate the detail
209    fields, and applies a 0.3 second polite delay between requests.
210
211    The classes listing page uses <h2> tags for department headings and <tr>
212    table rows for individual course entries. The course code is in the first
213    cell and the course name link is in the second cell.
214
215    @return A list of course dictionaries with fields:
216            course_code, department, name, url, description,
217            prerequisites, credits, semesters_offered, scraped_at.
218    """
219    print(f"Fetching {CLASSES_URL}...")
220    resp = requests.get(CLASSES_URL, headers=HEADERS, timeout=30)
221    soup = BeautifulSoup(resp.text, "html.parser")
222
223    courses = []
224    current_dept = None
225
226    main = soup.find("main") or soup.find("body")
227
228    for el in main.find_all(["h2", "tr"]):
229        if el.name == "h2":
230            current_dept = clean_text(el.get_text())
231
232        elif el.name == "tr":
233            cells = el.find_all("td")
234            if len(cells) < 2:
235                continue
236
237            code = clean_text(cells[0].get_text())
238            if not re.match(r"[A-Z]{2,4}\s+\d+", code):
239                continue
240
241            link = cells[1].find("a")
242            if not link:
243                continue
244
245            name = clean_text(link.get_text())
246            href = link.get("href", "")
247            full_url = BASE_URL + href if href.startswith("/") else href
248
249            courses.append({
250                "course_code":       code,
251                "department":        current_dept,
252                "name":              name,
253                "url":               full_url,
254                "description":       None,
255                "prerequisites":     None,
256                "credits":           None,
257                "semesters_offered": None,
258                "scraped_at":        datetime.now().isoformat(),
259            })
260
261    print(f"Found {len(courses)} courses. Now fetching detail pages...\n")
262
263    for i, course in enumerate(courses):
264        print(f"[{i+1}/{len(courses)}] {course['course_code']} — {course['name']}")
265        detail = fetch_course_detail(course["url"])
266        course["description"]       = detail["description"]
267        course["prerequisites"]     = detail["prerequisites"]
268        course["credits"]           = detail["credits"]
269        course["semesters_offered"] = detail["semesters_offered"]
270        time.sleep(0.3)
271
272    return courses
273
274
275def main():
276    """
277    @brief Entry point — runs the classes scraper and saves results to JSON.
278
279    @details
280    Calls scrape_classes() to collect all course data, writes the results
281    to qcc_classes.json, and prints a summary including total courses scraped,
282    credits populated count, semesters offered populated count, and elapsed time.
283    """
284    start = datetime.now()
285    print("=" * 50)
286    print("  QCC Classes Page Scraper")
287    print(f"  {start.strftime('%Y-%m-%d %H:%M:%S')}")
288    print("=" * 50)
289
290    courses = scrape_classes()
291
292    with open(OUTPUT, "w", encoding="utf-8") as f:
293        json.dump(courses, f, indent=2)
294
295    elapsed = (datetime.now() - start).seconds
296    filled_credits   = sum(1 for c in courses if c["credits"] is not None)
297    filled_semesters = sum(1 for c in courses if c["semesters_offered"] is not None)
298    print(f"\n✅ Done in {elapsed}s")
299    print(f"   Scraped {len(courses)} courses")
300    print(f"   Credits populated:           {filled_credits}/{len(courses)}")
301    print(f"   Semesters offered populated: {filled_semesters}/{len(courses)}")
302    print(f"   Saved to {OUTPUT}")
303    print("=" * 50)
304
305
306if __name__ == "__main__":
307    main()

BASE_URL = 'https://www.qcc.edu'

CLASSES_URL = 'https://www.qcc.edu/classes'

OUTPUT = 'qcc_classes.json'

HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}

def clean_text(text): View Source

72def clean_text(text):
73    """
74    @brief Removes excess whitespace from a string.
75
76    @param text The raw string to clean.
77    @return A normalized single-line string with no extra whitespace.
78    """
79    return " ".join(text.split()).strip()

@brief Removes excess whitespace from a string.

@param text The raw string to clean. @return A normalized single-line string with no extra whitespace.

def fetch_course_detail(url): View Source

 82def fetch_course_detail(url):
 83    """
 84    @brief Fetches and extracts detail fields from a QCC course detail page.
 85
 86    @details
 87    Sends an HTTP GET request to the course detail URL and parses the response
 88    using BeautifulSoup. Since QCC's course pages present fields as labeled
 89    plain text rather than structured HTML elements, this function splits the
 90    page's main content area into clean lines and scans for labeled field patterns.
 91
 92    Extraction rules:
 93    - Credits: finds a line matching "Credits" then reads the numeric value on the next line
 94    - Semesters Offered: finds a line matching "Semester(s) Offered" then reads the next line
 95    - Prerequisites: finds a line matching "Prerequisites?" then reads the next line
 96    - Description: selects the longest text block over 80 characters that does not
 97      match navigation or label patterns
 98
 99    A 0.3 second delay is applied between requests in scrape_classes() to avoid
100    overloading QCC's server.
101
102    @param url The full URL of the course detail page (e.g. https://www.qcc.edu/courses/financial-accounting-i).
103    @return A dictionary with keys:
104            - "description" (str or None)
105            - "prerequisites" (str or None)
106            - "credits" (str or None)
107            - "semesters_offered" (str or None)
108
109    @note Returns a dictionary with all None values if the request fails or
110          the page structure does not match expected patterns.
111    """
112    result = {
113        "description":       None,
114        "prerequisites":     None,
115        "credits":           None,
116        "semesters_offered": None,
117    }
118    try:
119        resp = requests.get(url, headers=HEADERS, timeout=15)
120        if resp.status_code != 200:
121            return result
122
123        soup = BeautifulSoup(resp.text, "html.parser")
124        main = soup.find("main") or soup.find("body")
125        if not main:
126            return result
127
128        # Split into clean lines
129        raw_lines = [clean_text(line) for line in main.get_text("\n").splitlines() if clean_text(line)]
130
131        # --- Extract Credits ---
132        for i, line in enumerate(raw_lines):
133            if re.match(r"^Credits$", line, re.IGNORECASE):
134                if i + 1 < len(raw_lines):
135                    val = raw_lines[i + 1]
136                    if re.match(r"^\d+(\.\d+)?$", val):
137                        result["credits"] = val
138                        break
139            m = re.match(r"^Credits\s+(\d+(?:\.\d+)?)$", line, re.IGNORECASE)
140            if m:
141                result["credits"] = m.group(1)
142                break
143
144        # --- Extract Semesters Offered ---
145        for i, line in enumerate(raw_lines):
146            if re.match(r"^Semester[s]?\s+Offered$", line, re.IGNORECASE):
147                if i + 1 < len(raw_lines):
148                    result["semesters_offered"] = raw_lines[i + 1]
149                    break
150            m = re.match(r"^Semester[s]?\s+Offered\s{2,}(.+)$", line, re.IGNORECASE)
151            if m:
152                result["semesters_offered"] = clean_text(m.group(1))
153                break
154            m2 = re.match(r"^Semester[s]?\s+Offered[:\s]+(.+)$", line, re.IGNORECASE)
155            if m2:
156                val = clean_text(m2.group(1))
157                if len(val) > 0:
158                    result["semesters_offered"] = val
159                    break
160
161        # --- Extract Prerequisites ---
162        for i, line in enumerate(raw_lines):
163            if re.match(r"^Prerequisites?$", line, re.IGNORECASE):
164                if i + 1 < len(raw_lines):
165                    result["prerequisites"] = raw_lines[i + 1]
166                    break
167            m = re.match(r"^Prerequisites?\s{2,}(.+)$", line, re.IGNORECASE)
168            if m:
169                result["prerequisites"] = clean_text(m.group(1))
170                break
171            m2 = re.match(r"^Prerequisites?[:\s]+(.+)$", line, re.IGNORECASE)
172            if m2:
173                val = clean_text(m2.group(1))
174                if len(val) > 2:
175                    result["prerequisites"] = val
176                    break
177
178        # --- Extract Description ---
179        label_pattern = re.compile(
180            r"^(Area|Course Number|Semester[s]? Offered|Credits|Prerequisites?|"
181            r"Skip to|Primary|Secondary|Contact|Visit|Apply|Copyright|"
182            r"Local|Life-changing|Fulltext|Open Menu|Open Search)",
183            re.IGNORECASE
184        )
185        content = main.find("article") or main
186        text_blocks = []
187        for el in content.find_all(["p", "div", "span"]):
188            t = clean_text(el.get_text())
189            if t:
190                text_blocks.append(t)
191
192        candidates = [t for t in text_blocks if len(t) > 80 and not label_pattern.match(t)]
193        if candidates:
194            result["description"] = max(candidates, key=len)
195
196    except Exception:
197        pass
198
199    return result

@brief Fetches and extracts detail fields from a QCC course detail page.

@details Sends an HTTP GET request to the course detail URL and parses the response using BeautifulSoup. Since QCC's course pages present fields as labeled plain text rather than structured HTML elements, this function splits the page's main content area into clean lines and scans for labeled field patterns.

Extraction rules:

Credits: finds a line matching "Credits" then reads the numeric value on the next line
Semesters Offered: finds a line matching "Semester(s) Offered" then reads the next line
Prerequisites: finds a line matching "Prerequisites?" then reads the next line
Description: selects the longest text block over 80 characters that does not match navigation or label patterns

A 0.3 second delay is applied between requests in scrape_classes() to avoid overloading QCC's server.

@param url The full URL of the course detail page (e.g. https://www.qcc.edu/courses/financial-accounting-i). @return A dictionary with keys: - "description" (str or None) - "prerequisites" (str or None) - "credits" (str or None) - "semesters_offered" (str or None)

@note Returns a dictionary with all None values if the request fails or the page structure does not match expected patterns.

def scrape_classes(): View Source

202def scrape_classes():
203    """
204    @brief Scrapes the QCC classes listing page and fetches details for each course.
205
206    @details
207    Sends a GET request to CLASSES_URL and parses the response to build an
208    initial list of courses with their codes, names, and URLs. Then iterates
209    through each course, calls fetch_course_detail() to populate the detail
210    fields, and applies a 0.3 second polite delay between requests.
211
212    The classes listing page uses <h2> tags for department headings and <tr>
213    table rows for individual course entries. The course code is in the first
214    cell and the course name link is in the second cell.
215
216    @return A list of course dictionaries with fields:
217            course_code, department, name, url, description,
218            prerequisites, credits, semesters_offered, scraped_at.
219    """
220    print(f"Fetching {CLASSES_URL}...")
221    resp = requests.get(CLASSES_URL, headers=HEADERS, timeout=30)
222    soup = BeautifulSoup(resp.text, "html.parser")
223
224    courses = []
225    current_dept = None
226
227    main = soup.find("main") or soup.find("body")
228
229    for el in main.find_all(["h2", "tr"]):
230        if el.name == "h2":
231            current_dept = clean_text(el.get_text())
232
233        elif el.name == "tr":
234            cells = el.find_all("td")
235            if len(cells) < 2:
236                continue
237
238            code = clean_text(cells[0].get_text())
239            if not re.match(r"[A-Z]{2,4}\s+\d+", code):
240                continue
241
242            link = cells[1].find("a")
243            if not link:
244                continue
245
246            name = clean_text(link.get_text())
247            href = link.get("href", "")
248            full_url = BASE_URL + href if href.startswith("/") else href
249
250            courses.append({
251                "course_code":       code,
252                "department":        current_dept,
253                "name":              name,
254                "url":               full_url,
255                "description":       None,
256                "prerequisites":     None,
257                "credits":           None,
258                "semesters_offered": None,
259                "scraped_at":        datetime.now().isoformat(),
260            })
261
262    print(f"Found {len(courses)} courses. Now fetching detail pages...\n")
263
264    for i, course in enumerate(courses):
265        print(f"[{i+1}/{len(courses)}] {course['course_code']} — {course['name']}")
266        detail = fetch_course_detail(course["url"])
267        course["description"]       = detail["description"]
268        course["prerequisites"]     = detail["prerequisites"]
269        course["credits"]           = detail["credits"]
270        course["semesters_offered"] = detail["semesters_offered"]
271        time.sleep(0.3)
272
273    return courses

@brief Scrapes the QCC classes listing page and fetches details for each course.

@details Sends a GET request to CLASSES_URL and parses the response to build an initial list of courses with their codes, names, and URLs. Then iterates through each course, calls fetch_course_detail() to populate the detail fields, and applies a 0.3 second polite delay between requests.

The classes listing page uses

tags for department headings and table rows for individual course entries. The course code is in the first cell and the course name link is in the second cell.

@return A list of course dictionaries with fields: course_code, department, name, url, description, prerequisites, credits, semesters_offered, scraped_at.

def main(): View Source

276def main():
277    """
278    @brief Entry point — runs the classes scraper and saves results to JSON.
279
280    @details
281    Calls scrape_classes() to collect all course data, writes the results
282    to qcc_classes.json, and prints a summary including total courses scraped,
283    credits populated count, semesters offered populated count, and elapsed time.
284    """
285    start = datetime.now()
286    print("=" * 50)
287    print("  QCC Classes Page Scraper")
288    print(f"  {start.strftime('%Y-%m-%d %H:%M:%S')}")
289    print("=" * 50)
290
291    courses = scrape_classes()
292
293    with open(OUTPUT, "w", encoding="utf-8") as f:
294        json.dump(courses, f, indent=2)
295
296    elapsed = (datetime.now() - start).seconds
297    filled_credits   = sum(1 for c in courses if c["credits"] is not None)
298    filled_semesters = sum(1 for c in courses if c["semesters_offered"] is not None)
299    print(f"\n✅ Done in {elapsed}s")
300    print(f"   Scraped {len(courses)} courses")
301    print(f"   Credits populated:           {filled_credits}/{len(courses)}")
302    print(f"   Semesters offered populated: {filled_semesters}/{len(courses)}")
303    print(f"   Saved to {OUTPUT}")
304    print("=" * 50)

@brief Entry point — runs the classes scraper and saves results to JSON.

@details Calls scrape_classes() to collect all course data, writes the results to qcc_classes.json, and prints a summary including total courses scraped, credits populated count, semesters offered populated count, and elapsed time.