On April 11, there was a Releng workshop held at Google in Mountain View. The two keynote talks and panel at the end of the day were recorded and made available on the Talks at Google channel on YouTube. Thank you Google!
Moving to mobile: The challenges of moving from web to mobile releases, Chuck Rossi, Facebook https://www.youtube.com/watch?v=Nffzkkdq7GM
Some interesting notes from Chuck’s talk:
- Facebook has more users on their beta apps than most companies have on their release apps. Number one app for IOS, number one non-Google app for Android. They have more beta users on Android than most people have users.
- There are three things we look act as software developers: features, quality, schedule. As release engineers we focus on quality and schedule. Because we ship on time. New features can go in the next mobile release which is only four weeks in the future.
- Developers have karma, if you release to many bad changes you lose karma. Developers start out with four Karma stars. You can only see you own karma. The release engineering team can reduce your karma if you release too many breaking changes.
- The release engineering team has a lot of respect within Facebook, they are the ones who determine if your feature gets in a release. He notes that you should make sure that you work for a organization that respects you for your judgement and skillset as as release engineer. As an aside, it seems the release engineering at Facebook has responsibility that corresponds to release engineering + release management at Mozilla.
- Compute power and storage are infinite at Facebook for their build and test infrastructure. It’s not a contraint.
- They run 28 – 30,000 builds a day. Wow.
- They have a ui for developers to look at their build and test results called shrubbery. It looks like a fork of Mozilla’s tbpl (Facebook’s release engineering team has Mozilla alumni 🙂 They also run Buildbot as their continuous integration engine too.
- Like Mozilla they have sheriffs to monitor builds but this role is rotated among the developer teams. If you put developers on call, they will make it less likely that they will get paged by being more careful when releasing changes because they share your operational burden.
- They have 17 apps to upload to the Google Play store. In the presentation, he asked Google to provide and API to be able to do so. Right now they have to a person interact with the UI and press buttons for all these apps they need to upload. Very painful.
- 15 minutes to deploy a new (web) facebook.com. Of course, this doesn’t happen all at once. They deploy to a certain percentage of their users, and evaluate the data and then deploy to more users.
The 10 Commandments of Release Engineering, Dinah McNutt, Google
- Release engineering is accelerating the path for development to operations
- You want to be able to reproduce your build environment and source code management system if you have to recreate a very old build
- Configuration management and release engineering as disciplines will probably merge as the next few years
- Reproducibility is a virtue. Binaries don’t belong in SCMs. However, it’s important to be able to reproduce binaries. If you do need a repo with binaries, put them in a separate repo.
- Use the right tool for the job, you will have multiple tools. Both commercial and open source.
- View the job of a release engineer and making a developers job easier. By setting up tooling and best practices.
- Package management. Provides auditing, upgrading, installation, removals. Tars and jars are not package managers.
- You need to think about your upgrade process before you release 1.0.
- Customers find problems we cannot find ourselves. Even if we’re dogfooding.
- As release engineers, step back and look at the big picture. Look and see how we can make things better from a cost perspective so we have the resources we need to do our jobs.
- It’s a great year to be a release engineer. Dinah is the organizing committee for Release Engineering Summit June 20 in Philadelphia as part of USENIX. There is also one as a part of LISA in Seattle in November. Overwhelming interest for a first time summit in terms of submissions!
Stephanie Bellomo, one of my colleagues from the organizing committee for this workshop moderated the panel. Really interesting discussions, well worth a listen. I like that the first question of “What is your worst operational nightmare and how did you recover from it?” I love war stories.
As an aside, we charged $50 per attendee for this workshop. We talked to other people who had organized similar events and they suggested this would be an appropriate fee. I’ve read that if you don’t charge a fee for an event, you have more no-shows on the day of the event because psychologically, they attach a lesser value to the event since they didn’t pay for it. However, we didn’t have many expenses to pay for the workshop other than speaker gifts, eventbrite fees and badges. Google provided the venue and lunch, again, thank you for the sponsorship. So we donated $1,531.00 USD to each of the following organizations from the remaining proceeds.
YearUp is an organization that in their words “Year Up empowers low-income young adults to go from poverty to professional careers in a single year.” I know Mozilla has partnered with YearUp to provide mentoring opportunities within the IT group and it was an amazing experience for all involved.
The second organization we donated to is the Tanzania Education Fund. This organization was one that Stephany mentioned since she had colleagues that were involved with for many years. They provide pre-school, elementary, secondary and high school education for students in Tanzania. Secondary education is not publicly funded in Tanzania. In addition, 50% of their students are girls, in an area where education for girls is given low priority. Education is so important to empower people.
Thanks to all those that attended and spoke at the workshop!