Aftermath of MLSE 2018

This week marked the second annual Machine Learning in Science and Engineering conference, jointly organized by Georgia Tech and Carnegie Mellon University, and hosted by Carnegie in Pittsburgh. I was lucky enough to be able to attend and spend 3 days exploring all the ways that machine learning has been used across many different disciplines. The next several posts will be devoted to researching some of the techniques that were discussed and the kinds of applications that were both well suited and perhaps less well suited to using machine learning (ML).

In this post I wanted to summarize my overall takeaways from the conference. It was divided into a series of tracks by discipline - biomedical engineering, chemical engineering, physics, mechanical engineering, etc. There was even an entire track devoted to public policy as relates to technology in today's society. The speakers ranged from current graduate students to senior faculty and industry experts. The track talks were largely focused on describing different problems that researchers had attempted to solve with machine learning. There were also several plenary talks that provided a broader overall commentary on the state of ML today, as well as its future.

Patrick Riley (a principal engineer and senior researcher at Google) gave a plenary talk that discussed the uses and dangers of ML [1]. First, he explained where ML is today. He referenced the hype cycle, see the figure below:

Figure 1: The Hype Cycle (Source: https://en.wikipedia.org/wiki/Hype_cycle)

Based on the title of Riley's talk - The Promise and Perils of Machine Learning - suggests that we might be slide off the Peak of Inflated Expectations and towards the Trough of Disillusionment [1]. (Although, as I have learned from Pedro Domingos' The Master Algorithm [2], various techniques in ML have been around for decades and they have risen and fallen in popularity as they have been introduced.) His team at Google has been using ML in various ways and he summarized the lessons that they had learned through their experiences. Some of my favorite points were [1]:

It can be hard to predict when generalization will work. Riley explained that there were applications where his team would not be able to build a good ML model with the data they had available. For reasons that they could not have identified a priori, the team would not be able to build a model that was general enough to fit their data set. He warns that because it can be difficult to predict when a ML model will fail to generalize, researchers should be prepared to find out that their models simply don't work well for a given application.
Begin with a simple model first. Riley's argument here is that beginning with a complex model can hide valuable insights. If you are lost in the details of building a complex model, you may lose sight of which variables are the most critical or you may miss out on understanding the phenomenon that you are studying.
Examine your data closely before starting. Understand where your data is coming from, try to understand what the possible sources of variation are and how they will affect your model. Besides Riley's discussion on this topic, Prof. Mark Styczynski's talk on analyzing errors in metabolic analyses was another excellent illustration of this peril [3].
Approximately right can be totally wrong. I liked this point a lot - basically, you need to define how right you need your model to be. It is not enough to say "my model is accurate to within +/- 5%" - put the error in context. Can you accept a 5% error? Or do you really need the model to be accurate to within +/- 1% in order for it to be useful?

I plan to write a couple of posts that go into more detail on some of my favorite talks. I will also write at least one post that explains a lot of the words I heard in the talks but didn't understand - what's the difference between a random forest and a decision tree, anyway?

[1] P. Riley, "The Promise and Perils of Machine Learning for Science", Machine Learning in Science and Engineering Conference, 2018.

[2] P. Domingos, The Master Algorithm. New York: Basic Books, 2015.

[3] M. Styczynski, "Machine Learning in Systems-Scale Metabolic Analysis and Modeling", Machine Learning in Science and Engineering Conference, 2018.