Why I'm asking: I teach a college data science 101 class with at least 1 section online every semester (and will now be switching to all online for at least the next month - thanks, coronavirus...). At the same time, I'm in the process of redesigning the course, and making new teaching materials and assignments.
I've taken completed several amazing MOOCs in the past (e.g. Princeton's Algorithms I & II, Stanford's Machine Learning), as well as abandoned many more that were simply terrible, so I am no stranger to the online student experience, and I know the things that made certain courses amazing for me personally (e.g. great videos, and well integrated assignments with automated feedback). However, every learner is different, and I would love to hear different perspectives so that I can better redesign my own course.
Was it something about the videos? The assignments? The structure? The instructor? Or something else? Thanks!
If you have control over the courseware platform used for your course, make sure it uses a compatible video player based on html5 video and not some custom implementation (very rare).
In case video speed controls is not available on your courseware, you can pre-process videos to speed them up to 1.5x using this script (save as `fastervid.sh` and run on video lectures before uploading)
#!/bin/bash
if [ -z $1 ]; then
echo "usage $0 input_video.mp4"
exit -1
fi
echo "Converting $1 to 1.5x speed..."
ffmpeg -i "$1" -filter_complex "[0:v]setpts=0.6666666666666*PTS[v];[0:a]atempo=1.5[a]" -map "[v]" -map "[a]" "tmp-$1"
echo "Delaying audio of $1 by 60ms"
ffmpeg -i "tmp-$1" -itsoffset 0.06 -i "tmp-$1" -map "0:0" -map "1:1" -acodec copy -vcodec copy "faster-$1"
# cleanup temp file
rm "tmp-$1"The CmapTools software is very good for this: https://cmap.ihmc.us/cmaptools/
Overall I find concept maps on of the best tools to show an overview of the course (macro maps) or topic (zoomed in maps). As a teacher with extra context about data science you can easily put together something like this for your course that will show learners what there is to learn and allow them to keep track of their progress (e.g. wow when we started all these words were unknown to me, but now I know about 70% of the words on this map already).
I have heard teachers use CmapTools in more active way (asking learners produce concept maps individually or collaboratively), but don't have experience with that myself.
A paper about concept maps: https://cmap.ihmc.us/docs/pdf/TheoryUnderlyingConceptMaps.pd...
And an example of the concept maps I use in my books: https://minireference.com/static/tutorials/conceptmap.pdf
There are so many walls of text out there going over a system's flow that could be expressed in just a couple paragraphs and a companion sequence diagram.
This is also very satisfying:
document.querySelector('video').playbackRate = 100;
but the videos are often served from a different host and I forgot to record the setting that makes it work in that case.
To me, what I consider to be secondary aspects such as interactive quizzes or homework assignments never really made a difference. One thing I noticed that really did not work well for me was asking quiz questions in between that did not require understanding just memorization of what was just said 30 seconds ago. It's like you're checking if they're paying attention v.s. are grasping underlying concepts.
I would recommend you check out CS50 by David Malan (if you haven't already) as it's somewhat of a gold standard for MOOCs in my book.
Good luck with your MOOC! The world needs more of them, make sure you make them as publicly accessible as possible.
Be careful of saying what works well versus what doesn't work well based on perception. I pretty much always use this kind of inline question in lectures, usually gauged to require the student to make a single deductive step from the material just presented. And they all look at me like I'm an idiot for expecting something that simple...except that then some of them get it wrong and have to reintegrate the knowledge, and the number of queries after the fact from misunderstandings of the material went down dramatically when I started doing it.
So I actually tell the students up front that this seems absurd, but play along, because it really does help the integration of knowledge.
Also I've seen that a bit of humor does wonders for learning. Recently did an online course called StatQuest with Josh Starmer(1). I never knew stats / ml could be made this interesting.
Dan Ariely is unusually gifted, so his humor works.
For the vast majority of courses, I prefer when they stick closely to the material, so I can watch it at 1.7x or 2x without breaking stride.
This should substantially change how you teach your class. A normal MOOC has to make learning "easy" so that students keep coming back. Unfortunately, when learning is "easy" it doesn't stick very well.
Effective learning needs to be challenging and force students to dig deep and think critically about what you're teaching them. If you give out hints or shortcuts, they'll probably like you more, but they're not going to build the deep understanding they need to succeed in further courses and in their career. If you're good at making them think critically, the whole learning process will feel hard for them, possibly as if they aren't learning anything, and they might resent you for that. So it's good that they're stuck with you. Do the right thing and make sure they're really understanding the material and not just memorizing formulas and patterns.
Current online courses have really watered down content and zero/terrible ways of evaluating the material.
Things that I like too see in an online course:
- Keep assignments on a regular basis (on par with your campus schedule). Quizzes are great to keep them engaged but do not use them to replace regular assignments.
- Design them to be auto-graded as much as possible. Bonus points if there was some way the students can evaluate their solution locally.
- Something like Piazza/Forum setup to answer student questions and have an appropriate number of TAs.
- Post all the resources used in the videos to be accessible for later reference (slides and links).
If you're recording content for future offerings, ensure that the content is consistently updated to stay relevant.
Although it's a step on the right direction, I'm not a fan of Coursera's "upload a zip file and wait 10 minutes" approach: even that much delay means you've lost your mental "working set" by the time feedback comes. My college professors would give those to us as Make targets in the project scaffolds, which made it closer to "TDD."
In a similar vein, thoughtfully designed interactive environments with good difficulty progression, like Micro-corruption [0]. It's easy to imagine something like this for data science (here's an RStudio workbench and a dataset, go find the needle in the haystack, etc).
1/ Interactive tutorials and quiz format like brilliant.org
2/ A Wikipedia like comprehensive wiki embeded with course website for all kinds of reference.
3/ An online community — could be a piazza/forum.
4/ A github like service integrated with course site specifically for that course where students can submit their projects and be evaluated by other forum members/mentors etc.
The content was engaging. It included guest talks. The course didn't shy from hard work, including dozens of required and recommended papers. The curated papers were my favorite part. The course exposed me to an entire ecosystem that I was unaware of. Too many MOOCs are watered down but this one was just right.
My best advice is to create a course that management will resent open sourcing because it is so valuable. Demand autonomy and use it create a legacy.
The most obvious issue is usually microphone quality. It's baffling that people spend so much effort developing a course and multiply that effort by a shit microphone. If you aren't willing to spend/get money for a good microphone, your philosophy on what is important for online teaching may have some blind spots IMO.
1. Resource curation 2. Guided active retrieval practice (questions, tasks, self assessment and feedback) 3. Collaborative experiences (professional examples, tadks and feedback) 4. Reflection and connection. (Connecting this learning to what has happened before or is likely to happen next
I will get the chapter (With evidence) up on a preprint server shortly -likely the end of the month. If you want the model - dm me on Twitter @maher_au
Good example: write an interpreter for the given (made up and simple) programming language. We had to figure out how to implement certain language features such as clojures.
Bad example: in networking course, we had to recall details about certain protocols, and do calculations. While this might be useful to know how to do in practice, it didn't contribute anything to my understanding, and it just always felt like tedious busy work, and lacked projects that contributed to the course goals.
I don't know data science, but here's a project I've heard of: Somebody's hobby was disc golf. Discs come with ratings on them (I'm assuming things like curve and distance). They created a controlled experiment to take various data points from many discs, and compare them to their stated ratings, and apply a bunch of data science-y things to the data.
Edit: I'd add some type of reference resources like a wiki, and ideally some auto-grader that gives feedback if possible (like what tests failed)
This is Part B, but there's also parts A and C. While I recommend the whole course, you don't need to take it all if you just want the interpreter project.
The networking course was discontinued, but these are the videos: https://www.youtube.com/playlist?list=PLEAYkSg4uSQ2dr0XO_Nwa...
The videos aren't terrible, but the homework was worthless.
- Discrete Optimization (Coursera/Melbourne university, Pascal van Hentenryck): had competitive optimization assignments and course leaderboards. The professor oozed enthousiasm and the nature of the assignments was great for the competitive setup (many different approaches possible, and there were always problems which you would not be able to solve to optimality).
- Microeconomics (EdX/MIT, Jonathan Gruber): very good assignments that made you discover things yourself. Professor presented with a lot of humor and this course was clearly refined over many iterations on campus. Lectures were actually recorded in a classroom setting so not tuned for MOOC delivery at all.
Honorable mentions for Computer Graphics (EdX/UC San Diego, Ravi Ramamoorthi, perfect pacing and balanced, broad set of assignments) and Underactuated Robotics (EdX/MIT, Russ Tedrake, fascinating material and challenging, elegant assignments).
Without naming names, here are two examples:
I once took a MOOC about "bare-metal" microcontroller programming which didn't include any microcontroller coding. Useless.
I once took a MOOC about digital circuit design which used an in-browser circuit simulator for problem sets and labs. Extremely helpful.
I feel sort of cheated when I work through an online course, only to find out that it would have worked just as well if it were a series of YouTube videos.
For college-level data structures, maybe you could ask students to implement their favorite DS from a set of tricky ones? You could write automated tests to verify them so grading is easier for you and development is easier for them.
To take a step back, I think what makes a course great is just its ability to transfer knowledge to students and to spark curiosity (further learning after course finishes). Usually, when you take a course in person, the folks teaching the class have ample opportunity to ensure that transfer is happening, in online courses, the onus is really on the student.
Thus, in an online course, you need to provide clear learning goals, for instance, "dynamic programming", a set of lectures that teaches those concepts, then homework or project based work that directly refers to what has been said in the lectures. Decide what your resources are, say, a book, your videos, and a few homework assignments, then try to have the resources build off each other, and make sure information is available in multiple places. Sometimes with difficult concepts, you need things explained twice, from two different sources!
The next aspect of a great course is the fostering of a learning community. Make sure the students have a place where they can engage (slack, discord, et cetera) and ask questions. Having students make a social connection has been shown to increase their outcomes in online learning, and for me, the early Coursera courses were amazing for this.
Finally, there is the aspect of of running the course. Some students don't do well learning online, and I some of this can be explained by the loss of human to human contact and relationship building and mentoring that can help struggling students during in person office hours. A percentage of students truly rely on this, and there is no online alternative. If you don't grok them pythonz, data science can be a nightmare (R 4 lyfe!!!).
Someone much smarted than me figured out you need 3 things to learn: time invested, a consistent environment, and feedback. Make sure you have enough TAs to actually give student feedback, or figure out an autograder for problems of varying difficulty.
An instructor's understanding of the topic he or she is teaching. It is hard to describe but the vision and the depth are just different. In some occasions, I find that it is my preparation prevents me from fully understanding what the instructors are seeing.
An instructor's passion for the topic, which brings moments for learners to feel "this is fascinating /amazing /cool /interesting. I have never thought about that". It seems that passion cannot be faked or manipulated, which means one cannot be prepared to demonstrate passion when teaching a course.
Teaching and presentation skills, can be trained.
Visualization and other technicals, can always be improved.
Also in online courses it's an important factor that not many e-learning sites fully understand the potential
Courses I've gotten a lot out of: Andrew Ng's ML course and Yaser Abu-Mostafa's learning from data course.
In both courses I really felt like I was learning something meaningful. I very clearly remember how mind blowing it was to me that you could turn complex high level tasks into curve fitting. I excitedly told a colleague of this thing I had learned about called linear regression who turned out to be surprised I had not taken a statistics class. Same thing for generalization bounds & VC dimension in Yaser's class.
I don't know if this can really be generalized to all courses since it might just be matching students to courses that make sense to them, but the worst line in a MOOC is "Well, this is boring, but you're just going to have to grind through it so that we can solve problems later". Shout out to the Convex Optimization MOOC I dropped.
For Data Science 101 this should be easy. Find some problems that students could imagine wanting to work on and use those problems to drive what you talk about.
Also, a pet peeve of mine is courses trying to make me do math by hand. I am 100% going to ask a math package to do an integral for me every time this happens. Sure, somebody has to know all the math deeply, but it's not me.
So I guess the main takeaway here is that you need to understand your audience. What is appropriate for me is probably not appropriate for 1st semester college kids.
A course with missable sections would be good so that if you already know ES6 you don’t need to listen through it on every JS course. Cheat sheets might help with this too.
I don’t care much for a super interesting voice as long as it’s a reasonable voice (not monotonous or unclear) and I can learn the thing.
- the speaker must be standing
- the tone should be light, not necessarily funny, but definitely not always the same
- there should be illustrations
- anecdotes, personal notes, interviews: maybe not at every video but at least one per chapter.
- videos should last less than 15min.
The talk must not aim to be exhaustive. Students will master the material through provided readings, exercises or quizzes. Some may only watch the videos, others will do the full course.
You will probably not have the time to prepare quality videos before the COVID-19 outbreak, so you may limit yourself to an introductory video for every chapter. You may also provide the solution for a couple of exercises in a video on the model of Kahn academy.
You should watch the making-of at the end of "Mindshift" on Coursera, by Barbara Oakley.
Note that "Introduction to logic" on Coursera has no videos but provides great tools and both funny and serious exercises.
Bad videos may spoil the course by making your students bored before they even start learning. So insist on quality, not quantity.
B. Oakley mentioned in my post is filmed full-body and then varies the framing, half-body, head, with digital zoom. That is explained in his making-of.
I really hope somone builds a mooc on top of something like leetcode.com . Start with easy problems see how far you get, get a hint, if you can't then watch a video of concept you are missing and so on.
Here’s my best effort at articulating his teaching style: He walks students directly into the mistakes they’re going to make and then helps guide them out. Most instructors will teach the concept and leave it to students to avoid making mistakes. I learn a lot a better when I know what mistakes to look out for and what to do when I’ve encountered one. But more importantly, I feel like I’ve really mastered the material when I’m confident that I can get myself out of any mess I can get myself into.
One of the keys for me is how I get feedback. Multiple choice quizzes can be well-designed. Most are not. But they scale well.
Personally I like very challenging assignments, but with very lenient grading and partial credit, with lots of detailed feedback from the grader.
If you’re teaching a college course, maybe you have enough TAs to make that happen. You’ll see some of Dr. Joyner’s work is on scaling grading systems and feedback.
To me what's really good about MOOCs is that I can have access to some of the best content and teachers from MIT, etc all from the comfort of my living room.
I think the internet is the best way to distribute information, but for me one of the worst places to learn.
If the learning is solely done via the computer I will fail to even get near to completing the course (self-study, no pressure). But by using pens, paper, books etc, my ability gets to a normal level.
So, encourage them to behave close to the way they do in your classes.
Supporting instructors, and instruction directly has been my main area of focus for the past year, in addition to being directly involved in developing online education platforms for the past 20 years in K-12, Academia and Industry.
I'm not sure if you have a tech background, but making sure you have competent technology advisement at all stages is quickly emerging as a key differentiator. If decision makers in academia haven't had technical literacy to make technical decisions, it shouldn't be a surprise if the experiences are not engaging, because they aren't leveraging digital interactions for what they're capable of, and instead only what a limited non-technical academic background can imagine.
Important:
- Not taking the lecture format classroom of the past 100 years and putting it online in video. MOOCs are handicapped before they even begin, quite often. Sure, great instructors who can explain well are good, but they are the exception, and not the norm. Another key problem remains.. god help you if you have to change the video. The key I focus on is how students are experiencing MOOCs as being the main lens. Academic institutions can have a hard time letting the frame go to be student centric instead of instruction-centric. Online education shouldn't be rooted in the practice of distance education (correspondence, textbooks, phone calls)
- The baseline of the student is that they are far more digitally literate and competent than the institutions they are walking to and most often more than their instructors.
- Much of the Academic online education world is designed and anchored in a past of a web browser on a desktop.. by educators who do not have competency in the possibilities or capabilities of tech.
- When it comes to making digitally engaging learning experiences, first you have to consider if academics rarely learn how to teach, let alone digitally.
- I would seriously pay attention to how students are interacting and studying digitally and focus far less on the existing taxonomies and nomenclature as it does not relate well to self-directed learning.
- The Copernican view of curriculum being the sacred cow of Academia is rooted in Math not changing much in a few hundred years, and generally the rate of change in curriculum being slow. In the real world, Academia is quickly reaching a point where they cannot update fast enough to keep pace with the change in the world, and the gap between students and entering a job is increasing.
- Again, see what students are doing to stay relevant - it's not always formal education.
- Learning how to learn is a course I reference early and often to all sides as the key skill being missed.