Rapidly scaling to tens of millions of users
Shilei started working on PayPay before it launched, and has seen it grow into a product used by tens of millions of users. He explains how diversity has helped them overcome challenges, and what PayPay is looking for in candidates.
In 2016, Shilei Long joined Paytm, one of India’s biggest mobile wallet companies, to work at their research and development centre in Toronto, Canada. A couple of years in, he started working on what was at that time a secret project, a joint venture between Paytm, Yahoo! Japan, and SoftBank. After just three months of development, they would launch PayPay, a mobile wallet for Japan.
Shilei began as a tech lead on one of the modules, and over time started to take on more responsibility on the project. He said, “Eventually I decided that for all the things that I was doing at the time, and also the fast growing reality of PayPay, the best thing for me would be to move to Tokyo, at least for some time, to look after this project from a technical perspective.” Now he’s a Senior Manager and Architect of Product Engineering at PayPay, working with several different engineering teams to oversee the technical architecture of different projects.
When the product is used by tens of millions of customers, every simple problem can become quite big. Coming up with solutions to solve those problems with quality and also speed is something that I really like.
At PayPay, he enjoys that they’re always building new things. He said, “We are far from being in maintenance mode. Every day we get new requirements from our product manager. Our job is to translate those into products that are reliable, scalable, and usable for our customers. It sounds like an easy transition, but trust me, it’s not that easy. When the product is used by tens of millions of customers, every simple problem can become quite big. Coming up with solutions to solve those problems with quality and also speed is something that I really like.”
An example of such a solution is a “Canary Production” environment. He said, “We want to deliver fast. At the same time, the Japanese quality requirements require us to make zero mistakes if possible. Any mistake that we make, we have to make a public apology. At the same time, the other part of culture requires us to deliver as fast as we can. These things are usually conflicting things. So we came up with a solution called production canary.”
While canary deployments, where a change is released first to a small set of users, are an established practice, this approach still results in some users potentially experiencing errors, something that is unacceptable to them. Their approach takes inspiration from canary deployments, but rather than releasing them to a small percentage of real users, they instead only route requests from PayPay employees to the environment. This allows them to do real money transactions and run automated tests against an environment that’s identical to production. If anything goes wrong, the deployment will automatically be rolled back, ensuring that it never reaches production.
Shilei believes that this kind of solution, bringing together the disparate goals of development speed and avoiding errors, is only made possible by PayPay’s diverse engineering culture, which has its roots in the Indian, Japanese, and Canadian engineering teams that started it, and has continued to grow more diverse as engineers from all over the world join it. He goes into more detail about this in his article “How PayPay’s Diverse Engineering Teams Achieve Scale, Speed And Reliability”.
This diversity is facilitated by adopting English as the main language for the Engineering team. Despite being in a senior management position, Shilei himself doesn’t speak Japanese, but he doesn’t feel that has held him back at all in his role. He estimates about 80% of the product team speaks English at least at a business level, and for the times when language does become an issue, PayPay has interpreters on staff that can provide simultaneous translation at meetings.
In his day to day work, he dedicates the most time to architecture design, going through design sessions with various teams. He said, “I encourage the team to propose the actual design on their own. This way I think the engineers and leads of each team get to practice and have motivation on their own. By doing things in the way they prefer, they gradually come up with better and better solutions. I get involved in reviewing the design, pointing out things that should be paid attention to. A lot of my work comes down to asking questions.”
Engineers here have quite a lot of liberty to make technical decisions, choosing the right technology, be it MySQL, Aurora, DynamoDB, or TiDB for example.
This approach lets each team choose which technology they use. He said, “Engineers here have quite a lot of liberty to make technical decisions, choosing the right technology, be it MySQL, Aurora, DynamoDB, or TiDB for example. We do have recommendations, and a more standardized platform and recommended languages, but people do have a lot of freedom in choosing other things.”
Another thing that he’s heavily involved in is interviewing developers. He said, “On average, I do maybe one or two interviews every day. We are still very aggressively hiring, trying to double the size of our engineering team in the next year or so. So hiring is a very important part of my job.”
When hiring, PayPay is looking for people who are motivated, responsible, and smart. He elaborated, “Motivated means you should be interested in PayPay, and motivated by what PayPay is trying to do, revolutionizing the way people make payments in Japan and also building a better product for our customers. After that, we care a lot about ownership and responsibility. We want people who are mature in terms of personality, in terms of work attitude, and who are willing to take ownership, where if you’re working on something, you take care of it end-to-end. Lastly, the smart part. We have quite a bit of technically challenging problems here at PayPay, so we are looking for people who are smart to come solve these problems.”
We don’t require any specific language or framework. Rather we are looking for someone who has good computer science and software engineering basics.
For their technical evaluation, they’re looking for well-rounded candidates. He explained, “We don’t require any specific language or framework. Rather we are looking for someone who has good computer science and software engineering basics. For example, algorithms, data structures, and understanding of operating systems if you are more towards the backend. In the first two rounds of interviews, we ask people to do live coding questions. So you should have very good skills when you are solving problems on the whiteboard or using the computer without an IDE.”
He continued, “In the final two rounds, we care more about peoples’ overall knowledge structure. As I mentioned, the basics, things like databases, understanding of networking, understanding of operating systems, and understanding of security. You don’t need to be an expert in one particular area, or even to know some of everything, but we do expect you to be generally pretty good at most other things, so that you can make well rounded decisions during work. That’s very important in the financial industry. Mistakes are costly, so we want to reduce the chance that people make mistakes like that. We do expect different things from people with different levels of seniority. So we do adjust the interview questions, and our evaluation based on that.”
Summing up why an international engineer would want to work at PayPay, he said, “If you want to make an impact, here is the place. You get a chance to work with people from different cultures, and you do not have to speak Japanese. If you want to solve hard problems using the latest technology, this is the place.”