Crowdsourcing and Machine Learning at Pinterest

We recently caught up with James Rubinstein, data scientist at Pinterest. James, firstly thank you for the interview. Let's start with your background and how you became interested in data science.

What is your 30-second bio?

I am a data scientist on the Discovery team at Pinterest, but I prefer to be called a “Disco Scientist”.

Prior to joining Pinterest I was a statistician at Apple on the Maps and iTunes teams. Before that I worked as a Product Manager for the bidding team and Search Metrics at eBay, where I introduced a betting market on experiments, a new Bid Layer, inline surveys, and crowdsourced search quality evaluations.

In my free time, I build audio components, hack clothes dryers, brew beer, and hang out with my family.

What was the first data set you remember working with? What did you do with it?

They say you never forget your first time, which is entirely false. I do however remember being an undergrad at Clemson and handing out surveys to students as well as customers at the store I worked at. I’d then enter that data manually and run it through SPSS using t-tests or some other basic stats to look for differences in the populations.

You’ve worked with some of the most innovative companies. Any highlights you want to share? What in your career are you most proud of so far?

I’m extremely proud of the work I (and my team, because data science is a team sport) accomplished at eBay. We created new channels for teams at eBay to listen to ‘the voice of the customer’ by creating a survey platform.

We also drove a lot of decision making within the search team by providing data separate from live site metrics - i.e. relevance. Those insights are still driving decisions at the company now, albeit in a different way, with a different team.

I’m really proud to work as part of the team here at Pinterest, too. I work with amazing people to bring an amazing product to the world. Our team works very hard to make sure that every algo release for the discovery team is “Pinterest quality”, and I’m proud of that, too.

Where do you think crowdsourcing and data science and ML are going? What’s the near future and then the far future?

There are two things that we use human evaluations for: steering and scoring. Steering is providing the input to tell you where things are bad/good, providing the input for ML. Scoring is about measuring how well things are going. In the near term, I think we’ll see more of a shift from scoring to steering using these human labeled data.

In the future, we’ll probably see a closer loop between people and machines, where machine learning, training data collection, and measurement are all happening in near real time. In the far future, we’ll have done such a good job training the machines that we’ll all be able to relax on a beach with no human oversight at all.

Can you tell us about the work you’re doing at Pinterest?

I’m mostly working on measuring the performance of various discovery algorithms. The Discovery team is responsible for helping you find what you love on Pinterest, whether that’s through Search, Homefeed, or Recommendations. We measure that experience using human evaluation as well as through experimentation on the site. I also work on measuring Pin quality to try to train ML algos on what is good Pin vs a not-so-good Pin.

What are some other areas of opportunity / questions you would like to tackle?

A big challenge is how do we know when a Pin is good. What makes it good? Why do people engage with some Pins and not others? How can we know when we first see a Pin creation if that Pin will be something interesting for Pinners?

The other subject I’m most interested in is building the pipelines for evaluation so that it’s easy for anyone in the company to get measurement of all the different algorithms we use on the site.

You have an interesting background in experimental design and statistics, can you talk a little bit about how your background gave you some advantages in your career as a data scientist?

I view a lot of what I do as designing surveys. In a way, it’s not so different from what I did as an undergraduate, handing out surveys to students in Brackett Hall.

You need proper survey design that doesn’t bias the respondent, and you need to make sure that the results are because of the data, not because of the bias of the users.

Setting up proper experimental controls is also vital, which is something that’s not always appreciated as being a critical component of experimentation.

Finally, there are the statistics -- understanding how and when to apply them is important. I will admit though, that there are many people better with a wider range of statistics than I am, and I am only too happy to ask for help or a second opinion. That’s one of the great things about working at Pinterest: despite having a small team, we have people who are really excellent in their disciplines and they are willing to help me find the right answer.

What are your favorite tools / applications to work with?

Well, on the crowdsourcing side, we are usually using Crowdflower or Mechanical Turk. We do use different vendors in different markets, though. You have to match the right crowd to the right task.

When it comes to data analysis, I’ve been working with R for a lot of things. I’ve been learning more about Python over the past couple of years though, which has been great.

Any words of wisdom for Data Science / Machine Learning students or practitioners starting out?

Yes, learn about science. Go do some real experiments with real people. Figure out the confounds, ask about sources of noise, try to get significance from as few participants as possible. Data science doesn’t replace science, it complements it.

Follow James on Twitter and Pinterest. Keep up with Data Science at Pinterest on the Pinterest Engineering Blog.