Transfers, Trust, and Teamwork on the SRE Team at Komoju
Muhammad Denaw, Senior Site Reliability Engineer at Komoju, talks about his work and shares how Komoju's trust and support propelled him to a promotion.
“I think it’s a very trusting environment,” said Muhammad “Doni” Denaw of Komoju, where he’s worked as an engineer for the last two years. He was particularly referring to Komoju’s willingness to let him transfer from his original role to the SRE team, where he quickly became the lead on several large, important projects.
So when I joined. . . one of my first tasks was to build the data warehouse, which is also a very big [project]. It really helped me that the manager trusted me with this. . . . And then in Komoju, we don’t really have strict deadlines. It’s very chill. You can set your own deadlines in a sense. Like, ‘We can finish this by Q3 or Q4, so what are the things that we need to do to attain this solution?
The data warehouse was urgently required, Doni said, “because we have many services, and each service has its own database. . . . For example, if a marketing team wants to do a query joining table A from service A to service B, we could not really do that at that time. So we wanted to have a data warehouse, which we built.
“Now we can do all sorts of queries from this database—the pipeline of data, the CDC process, and everything. . . . Also, the data warehouse that we are using now is built using an actual data warehouse solution, like Redshift, so it is a proper analytical database. The query becomes fast and also allows us to build a machine learning solution on top of it. We now have Looker.
Again, the goal was to make the team more data-driven. Ideally everyone could do the query to this database without having to understand SQL. Now with Looker, everyone can just do that query with some strings and then it’s converted into SQL.”
While Doni was in charge of executing this project, he did have plenty of support. “When we were building the data warehouse, we had a weekly meeting,” he said. At those meetings, Doni was able to break the work down into tasks which other members of the team could pick up. “It’s very collaborative, each task,” he confirmed.
This project was Doni’s second assignment after he transferred to the SRE team. Originally he was hired as a full-stack engineer for the platform team and worked remotely from Indonesia, with the understanding that he’d eventually move to Japan once he’d accrued the 10 years of experience required to receive a work visa without a university degree.
”[Komoju is] very accommodating,” he explained. “I have many examples of it. The first one is that they allowed me to move to Japan at my own pace. They also [helped me bring over] my wife as well. They supported the Dependent visa and they were okay with me not having a bachelor degree. They were okay with me waiting for one year [to get the work visa].”
Doni requested the transfer to SRE because of his previous experience with infrastructure: “That’s the kind of thing that I wanted to work on.”
So I asked my manager, “Can I move to SRE if there’s a possibility?” And not long after that, maybe two weeks after that, the manager from the SRE team approached me [and said] that there is an opening if I want it.
The transfer itself was fairly smooth, although for a while Doni was “still finishing up the platform stuff, but also building the incident workflow and the data warehouse.” Transitioning fully into the SRE mindset, however, took time.
“It was actually the opposite of easy,” Doni said. “It was very, very challenging, especially for me because my role in my previous company was not SRE and DevOps. It was infrastructure-heavy, but I wasn’t fully in that [SRE] role. So I had to learn a lot of things.”
But thankfully, when I joined the SRE team, there was also this new guy who’s now the tech lead of SRE. He’s been very, very helpful. He set up one-on-ones with me, to mentor me.
Having successfully settled into his new role, Doni is now working on an even more massive task: rebuilding Komoju’s deployment system.
“The initial trigger was that we wanted to move to Fargate,” Doni said. Until now, the team has been managing their own EC2 instances, but maintaining these and keeping them up to date has significant overhead. “We’ve been wanting to move to Fargate all this time, but we cannot do it with Barcelona.”
Doni explained that Barcelona is their own in-house infrastructure management. “It’s not formalized infrastructure as code. If you don’t understand Barcelona, you cannot understand the infrastructure. . . . Staging and production could be very different.”
Their new system uses Terraform, which gives the team better control over the states of different environments. “With Terraform now,” he said, “we have three repos—Terraform modules, Terraform staging, and production. We are doing everything on these modules, and then just use them on staging and production. So now there are no discrepancies.”
With the new repositories, the SRE team has empowered other teams to manage their own infrastructure. “Everyone can create PRs, that’s the idea. Everyone on the team, not only SREs—like [those] on the app side—can also do some things with infrastructure because they can just create PRs for Terraform.”
“The task has been there for a long time,” Doni said. Until recently, however, the SRE team was too short-handed to work on migrating the infrastructure. “But when I moved to SRE . . . and the tech lead guy joined SRE, we had more manpower, so we could take this.”
“A lot of challenges come from this,” he added. “I think if I have to rank it . . . it will rank as one of the hardest tasks that I have ever done in my life, actually.”
Yet it’s a task Doni volunteered for. “I said that I think I prefer to take this one, because it doesn’t have any deadlines, and it kind of warms me up to the infrastructure side of Komoju because it touches a lot of things. It’s almost everything in Komoju because we are moving all of our applications. We have many services. . . . The plan is to have all the services with the new deployment system. So this is a very good opportunity for me to learn all the other services as well. . . . That’s why I took it.”
This single project doesn’t occupy all of Doni’s work day, though.
In SRE, at least compared to when I worked in software engineering, the day-to-day [has a lot of] variety. . . . It’s a very flexible sort of role.
“We also create our own tasks because there is always something to be done,” he said. “There are always things that need to be improved. [Or] maybe other teams want to deploy this thing and they ask us to create the infrastructure side.”
Komoju uses a tool called Rootly to help navigate incident response and conduct postmortems.
It’s very blameless. They’re very healthy postmortems. That’s why we use Rootly, so the discussions and everything will be centralized on one channel.
“Once the incident is mitigated or resolved, it will automatically generate postmortems with the timelines of what happened. Then, usually, the one commander who’s leading the incident will fill out the postmortem more. But Rootly helps us get the timeline and everything accurate.”
Since they’re critical to the company’s functioning, the SRE team has been the first to experiment with compensated on-call rotation. “Currently, I think the plan is for every team to have on-call rotation, but now it’s being tested on SRE only.”
It’s clear that Doni has been hard at work, but thanks to his self-confessed imposter syndrome, he hasn’t done much to promote his accomplishments. Luckily, parading his achievements wasn’t required.
“There’s [another] good thing about Komoju,” said Doni. “They always notice your achievement. It was my manager, who since the beginning told me that if you finish this new deployment system, you will be promoted to senior [engineer]. But I haven’t finished it. It should be finished this week, but I have been promoted already.”
I’ve never asked because I [have] imposter syndrome. I felt like I didn’t deserve the promotion at that time. It’s just, they noticed me, and then they promoted me. That’s what happened.
Even before his promotion, when he was on the software engineering team, Doni’s achievements were noticed by management. “Due to my performance, the bonus was quite high,” he said.
The ultimate result of Komoju’s “trusting environment,” accommodating nature, and thoughtful managers, is that Doni is highly content in his new role.
“I think you can also notice this if you see my LinkedIn,” he said, “because this is actually the longest I’ve stayed at any company. Usually I stay less than one year, but I have been here for two years at Komoju, and I don’t have any plans to move, because it’s very healthy. I’m really happy with Komoju.”