|
ECE405: Introduction to Large-Scale Artificial Intelligence Systems
Meeting Days/Time: Monday, Wednesday & Friday, 10:30 – 11:20 AM (CRN 57997, 3 credits)
Location: Holmes Hall 211
Semester Dates: January 11, 2027 – May 5, 2027
Instructor: Haoyue Tang — Department of Electrical and Computer Engineering, University of Hawaiʻi at Mānoa
Email: haoyue@hawaii.edu
Office Hours: TBA, POST 205E (and by appointment)
[Course Syllabus]
Outline
Language models are the core of modern AI systems. In this course, students build one from scratch. Following Stanford CS336: Language Modeling from Scratch and the previous offering of ECE405 by Prof. Igor Molybog, we go through the full pipeline of a language model: tokenization, Transformer architecture and training, GPU systems and kernels, distributed training, scaling laws, data collection and filtering, inference, evaluation, and post-training (supervised fine-tuning, preference alignment, and reinforcement learning for reasoning). Aim: for students to gain the engineering ability to reproduce results from AI papers. The course is implementation-heavy: four large coding assignments run on GPU clusters.
Prerequisites
Linear algebra, calculus, probability & statistics, and basic machine learning. A solid coding background is required: Python, PyTorch, bash, and git.
Grading
Assignment 1 (Basics): 25%
Assignment 2 (Systems & Scaling): 25%
Assignment 3 (Data): 25%
Assignment 4 (Alignment & Reasoning RL): 25%
No exams. Letter grades follow the UH Mānoa plus/minus scale (see the syllabus).
Course Policies
Late Days: 5 late days for the semester, at most 3 per assignment. Each late day extends a deadline by 24 hours.
Regrades: Within 3 days after grades are released, for objective grading errors only.
Collaboration: Study groups are allowed, but each student writes and submits their own code and write-up. List study-group members on your submission.
AI Tools: LLMs may be used for low-level programming questions and high-level conceptual questions, but not to solve assignment problems directly. Please turn off AI autocomplete in your IDE. Do not consult third-party implementations unless a handout says so.
Academic Integrity: Students are expected to adhere to the UH Mānoa Student Conduct Code.
Accessibility (KOKUA): Students needing accommodations should contact the KOKUA Program.
Assignments
Assignments are adapted from the Stanford CS336 handouts and due Fridays at 11:59 PM HST unless noted. Submit on Lamaku.
| Assignment | Topic | Released | Due |
| A1: Basics | BPE tokenizer, Transformer LM, AdamW; train on TinyStories / OpenWebText | Jan 11 | Feb 12 |
| A2: Systems & Scaling | Profiling, FlashAttention-2 in Triton, distributed data-parallel training, scaling laws | Feb 12 | Mar 12 |
| A3: Data | Common Crawl to pretraining data: extraction, filtering, deduplication | Mar 12 | Apr 9 |
| A4: Alignment & Reasoning RL | SFT, expert iteration, GRPO for math reasoning; optional DPO | Apr 9 | May 5 (Wed)
|
Compute
Assignments need GPUs. Onboarding is in Weeks 1–2. Debug on CPU first and start early, because queues are long right before deadlines.
General References
No required textbook. Lectures draw on publicly available resources:
Schedule
University holidays are marked below. This schedule is a draft and may be adjusted as the semester progresses.
| Week | Dates | Topics | Coursework |
| Week 1 | Jan 11 – 15 | Course overview; language modeling; tokenization (byte-pair encoding) | A1 (Basics) released; Koa onboarding |
| Week 2 | Jan 18 – 22 | PyTorch tensors, einops; resource accounting (FLOPs, memory). Jan 18: MLK Day (No Class) | |
| Week 3 | Jan 25 – 29 | Transformer architecture & hyperparameters | |
| Week 4 | Feb 1 – 5 | Attention alternatives; mixture of experts | |
| Week 5 | Feb 8 – 12 | GPUs & TPUs; memory hierarchy; benchmarking & profiling | A1 due Fri Feb 12; A2 (Systems & Scaling) released |
| Week 6 | Feb 15 – 19 | Kernels & Triton; FlashAttention. Feb 15: Presidents’ Day (No Class) | |
| Week 7 | Feb 22 – 26 | Parallelism I: data parallelism, ZeRO / FSDP | |
| Week 8 | Mar 1 – 5 | Parallelism II: tensor, pipeline, sequence / context parallelism | |
| Week 9 | Mar 8 – 12 | Scaling laws (Chinchilla, compute-optimal training, muP) | A2 due Fri Mar 12; A3 (Data) released |
| — | Mar 15 – 19 | Spring Recess (No Class) | |
| Week 10 | Mar 22 – 26 | Data I: sources, Common Crawl, text extraction. Mar 26: Kūhiō Day / Good Friday (No Class) | |
| Week 11 | Mar 29 – Apr 2 | Data II: quality filtering, deduplication, data mixing, synthetic data | |
| Week 12 | Apr 5 – 9 | Post-training I: supervised fine-tuning; RLHF; preference optimization (PPO, DPO) | A3 due Fri Apr 9; A4 (Alignment & Reasoning RL) released |
| Week 13 | Apr 12 – 16 | Post-training II: RL with verifiable rewards (GRPO); reasoning models | |
| Week 14 | Apr 19 – 23 | Inference: KV caching, quantization, speculative decoding, serving (vLLM) | |
| Week 15 | Apr 26 – 30 | Evaluation: benchmarks & metrics; agents and tool use | |
| Week 16 | May 3 – 5 | Alignment & multimodality; course wrap-up | A4 due Wed May 5
|
|