On Friday, April 26, the Google Machine Learning pilot program at Mills came to a close. Although the takeoff might have been rocky, the program promises less turbulence in the future.
Last November, the Mills College newsletter announced a partnership with Google: Mills would host the first of Google’s Applied Machine Learning Intensives (AMLI) from February 11 to April 26.
“We can bring great value to an employment market that is looking for this talent that might not otherwise have access to this kind of training,” Kim Roberts, Google’s chief of staff for VP of education and university relations, said.
Naomi Alterman, assistant adjunct professor of computer science at Mills and one of the instructors for the intensive, explained what machine learning is in comparison to statistics.
“Machine learning to me is using statistics to make predictions based on data at a data set scale larger than humans could ever hope to do themselves,” she said. “To me, the difference between ML and statistics is that statistics is limited to what a human is going to do with the data set.”
For example, figuring out how to calculate productivity in relation to amount of coffee consumed, how long a morning run was, and hours of sleep is a simplified way to describe what machine learning can do.
“A less silly case is predicting housing prices, which is both a useful thing that websites like Zillow do, using machine learning, and also a potentially concerning thing especially when you think about historical and socioeconomic issues like redlining,” Alterman said. “These have ways of working their way into data sets which then inherently get encoded into these models.”
The course was 10 weeks long, with 350 hours of time going towards instruction and projects. Housing, dining, and the application was free; the 21 students only had to have 10 weeks free in the spring to attend.
The objective of the program was to gather current college students or recent graduates from diverse backgrounds, locations, and areas of study and teach them a starter course in Machine Learning. Although hosted by Google, the high profile collaboration was not intended as a feeder program for the company; instead the purpose was to assist in bolstering the number of professionals with computer science experience in the workforce.
“There’s a big issue nationally where there aren’t enough computer science instructors to teach computer science courses,” Roberts said. “With the program we have tried to bring underrepresented voices in tech.”
Mills was recommended as the ideal space to host the pilot by Roberts, a Mills alum, who at the time was working as program manager in the applied computing program at Google. Her section was focused on designing computer science initiatives for higher education. Roberts, class of ‘93, said she had a lot of fond and good associations with Mills and considered the fact the college has a computer science program as well as campus proximity to Google’s Mountain View headquarters.
Since it can take a long time to test educational models based on the traditional classroom schedule Roberts said the intensive was to speed up that process. According to Roberts, Google wanted to use this program as a testing place to combine 3 semesters into a shorter amount of time and condense the process.
Karen Gheno, the senior program manager, took over in January after Roberts was promoted. Gheno has been making sure that the pilot launched, and providing the intensive as described to the students and gathering as much information from the pilot to make the next iterations of the program better.
Getting ready for the runway
At the time when Roberts was working on developing the project, she was responsible for the overall design and strategy. She also managed a team of six that included designers and other program managers working on relationship development with the colleges.
“We needed to work with the machine learning curriculum that had been developed internally for Googlers and figure out how we’re going to deliver this content in this program,” Roberts said.
Aside from the more visible aspects of the program, she said there was a lot of work that went on behind the scenes including checking in with students and their satisfaction, consulting with the legal department about handling student data and getting the site launched.
Alterman and McAdams set about editing and restructuring the provided curriculum to better serve their students. It was a tight timeline, and in the beginning they often had to write the lesson minutes before the class started. It was a demanding pace, they said, to rework the class based off of feedback from the students while trying to build the next day’s lesson.
The first four weeks were spent focused on providing the sampler plate of applied machine learning skills, while the last six weeks went towards working on the final project. McAdams and Alterman sent out anonymous daily feedback forms during the lesson heavy time, gauging what worked well and what needed to be modified. A comments and suggestions box was also available for students to input their voice, in addition to the daily feedback form.
“Josh is an amazing instructional designer and has been writing really well paced content throughout this semester which he tests at such an immediate time scale,” Alterman said. “There aren’t many metrics that people care about maximizing here, so there’s a lot more space for us to try a grading policy, try a way of scheduling content throughout the day, seeing how it works and trying something else the next day.”
The access to feedback and freedom to mold the curriculum was a double edged sword. “It’s a high stress environment to work in, but it’s also a really fruitful one in the context of having feet on the ground, ear to the pulse tuning of these sorts of things,” Alterman said.
Hitting turbulence: Curriculum restructuring
Alterman and the other intensive instructor Josh McAdams were brought onto the team a few months before the program began, giving them little time to prepare. While Alterman was from Mills faculty, Josh was brought on by Google. There were also two teaching assistants, Ju de Heer and Hao Yu.
The curriculum that Google provided was informational, but not structured in a way that was compatible with a classroom where each student was at a different level of knowledge with coding, computer science, and machine learning, they both said.
“The content was written by people who have been programming for 20 years,” McAdams said. “They figure they can throw a few hundred lines of code at somebody and they can just figure it out and that’s not a good format for this group of people and we had to figure out how to unwind that and make it more teachable.”
Often, McAdams and Alterman were writing new content as the program progressed.
“Sometimes I was learning things the morning of, before we gave the assignments to students,” Alterman said. “It gave me a lot of confidence in myself to get practice with this, but it’s still extremely stressful to get to work in the morning, be given an assignment that’s about to be given to your students, run through it in 10 minutes when it should take the students an hour and a half, and then try to get through as much of it as you can before they start having questions.”
It seems their hard work paid off.
“They did a really good job trying to get everyone up to speed and I’m really impressed with how fast people caught up,” Mills alum and Computer Science major Regina Wang said. “The instructors created an environment that’s laid back and not competitive at all.”
McAdams and Alterman learned to structure the days starting with new lessons in the morning when the students were able to absorb the information and then placed exercises in the afternoon. Alterman was passionate about incorporating one topic specifically.
“I guess in that same respect that Google is happy to listen to us about creating safe classrooms, Google is also very willing and interested in letting us, meaning the Mills teaching staff, write material about ML ethics and present that within the class,” Alterman said. “Google had stated that they were interested in including that in their course syllabus, but they didn’t actually have any concrete content written for that and they’ve been nothing but supportive of letting us put what we feel is important in the middle of this classroom.”
Including a constant consideration for ethics was something that Alterman wanted to cultivate, and so for each data set, the students were asked to fill out an ethical storyboard. They had to write one paragraph about someone who has been negatively affected by the data set, to give that person a name and explain what happened to them.
Then they had to write another paragraph about a positive thing that really benefited someone and brainstorm ways to mitigate the biases and negative effects that the model could have.
“It’s really heartening to see cohorts of ML students being mostly concerned about the societal implications of these technologies rather than just the technical innovations of them,” Alterman said.
Stabilizing the pilot: incorporating feedback
Throughout the furious pace of the program restructuring as Alterman and McAdams incorporated feedback they were getting from the classroom, they both maintained that Google as a company was flexible with the changes.
Gheno acknowledged that learning curve.
“It’s been a pilot so we’ve learned a lot. The feedback we’ve received from students has been very positive,” Gheno said. “How we were teaching was at the wrong level, but we pivoted. What we learned will make this even better in the future.”
Roberts agreed.
“I think we’re used to pivoting and changing direction when that is what needs to happen. It’s an exciting role as an educator, because schools typically don’t work that way—they’re typically slow to move,” Roberts said. “That’s something I love about working at Google, if something isn’t working then they change direction and try to make it work.”
While the program collaboration was billed as “cutting edge” and “future focused” the reality was not as efficient or smooth as those words might imply. Alterman, McAdams, Roberts, and the students all said they expected the program to come with some problem solving and troubleshooting considering it is the first edition.
“I think it’s very ambitious, especially given the timeline we were trying to create this under, but the feedback we’ve been getting, it sounds the students have been enjoying Mills,” Roberts said. “I feel a lot of appreciation for this first cohort that has gone on this pilot with us. It’s not as smooth and elegant as it will be one day but for a pilot it’s gone really well.”
Jesse Battalino, a master student in the interdisciplinary computer science department at Mills, said he thought there might be some patching going on behind the scenes but it was hard to tell.
“I didn’t ever experience a lapse in instruction. There were always assignments, activities, there were lecture time that was all consistent and there were certain deliverables that we had to give back that we had to get feedback on,” he said. “From the front it all seemed like it was working.”
Wang said she thought the program would be more stressful, but was pleasantly surprised to find that the instructors created an environment that was laid back and not competitive, and the curriculum revisions did not phase her.
“I think all of us came in knowing that it’s the first time they’re doing this, it’s going to be like ‘we’re doing it, we’re going to see what works and what doesn’t and there’s going to be some stuff that doesn’t work’ and it’s something that the instructors have been open about and they’ve taken feedback at every stage. We’ve definitely seen it incorporated even, I think all of us feel very listened to,” Wang said. “Knowing that your input is meaningful is really validating. I think we all feel ok about it because we knew it was a pilot program and I think that we’ve all gotten a lot from it, so it’s all good.”
Another student in the program, Maya Rafalowicz also kept the pilot part in mind.
“I feel like knowing it was a pilot program, I didn’t have that many expectations,” she said. “I really just wanted to find a way to put my foot in the door to the technical field, and my major was molecular biology, I didn’t really have the option to pivot where I wanted to go for a career. I was hoping to gain some skills in order to prepare me for those kinds of jobs.”
McAdams said that the only measure of success was to see how many students get jobs in fields adjacent to machine learning or computer science, which the program provided support with, offering resume and cover letter building advice.
Both Wang and Rafalowicz said that the community and relationships that came out of the experience were the best part.
“The community is one of the best things that has come from it,” Wang said. “That supports the learning because there’s always someone who knows something you don’t and sometimes you know something that other people don’t so I think learning from each other as well as the actual instructors has been really great.”
McAdams agreed.
“This is bound to happen, but you get people and you do spend hundreds of hours together and you make relationships that you know will last beyond whenever we leave,” he said. “I think most of the students seem to be enjoying themselves so I hope that’s true.”
Alterman said that was her highlight as well.
“Regardless of any other methods, if students feel their time has been well spent, that is the most important thing to me and that is what they’ve been saying,” Alterman said.
After she learned that the students had made a WeChat (a Chinese messaging app) group, she felt very happy to see them interacting in that way.
“That made my heart sing; that’s the dream when you create a high intensity environment is that you end up with that environment creating this cohort that trust each other and participate with each others’ lives in a non-transactional way,” she said.
However, the words “cutting edge” and “innovative” do not always indicate a polished product—on the contrary, McAdams says that growth and pushing boundaries is inherent in those terms and ideas.
“I would say that the word “cutting edge” dictates that it will be a little chaotic,” McAdams said. “If it’s cutting edge, then we don’t really know what we are getting into.”
Heading towards smooth flying: future programs
Next iterations of the program are already set at Agnes Scott College in Decatur, GA and one of the Claremont McKenna Colleges in southern California starting at the end of May. McAdams and Gheno will be continuing their work informing the programs.
“I see scaling to many more people, so those plans are still in the works, but taking what we learned in the spring and summer pilot and trying to scale that in 2020 to a bigger audience,” Gheno said.
When asked how he felt the program went, McAdams called the outcome “acceptable” as he looked forward to more editions of the intensive.
“I have a high bar for what I want this class to be and I’m not embarrassed by what we did, I think we pulled it off and made it acceptable for these students. I think it will be good this next iteration but I think it could be great by the third iteration,” McAdams said. “It’s going to better in the summer, it’s going to be better in the summer of 2021, it’s starting to become a solid course.”
