Using the Elixir programming language to fight crime
Eurico is the CTO of Spider Labs, whose ad fraud detection tool handles billions of requests and terabytes of data on a daily basis. The Elixir programming language has been instrumental in helping them reach this scale.
While undertaking his PhD in Portugal, Eurico Doirado came to Tokyo in 2010 for a six month exchange program at Japan’s National Institute of Informatics, where he researched AI applied to games. He found the city to be incredibly vibrant, with everything always open, and being blown away by the quality of the service industry. He enjoyed his experience so much that he ended up never returning to Portugal, first extending his research in Japan, then applying to continue his PhD in the country.
After a couple of years, he discovered that he was more interested in the industry than the academic track. Eurico chose to not finish his PhD, instead joining a Japanese game company, tri-Ace. He said, “They had done a lot of RPGs that I liked. I was working there with a lot of people that I’d seen their names in the credits of games. It was quite interesting, seeing the industry from the inside.”
A year later, he wanted to try building his own company. Through a mutual acquaintance, he was introduced to Satoko Ohtsuki, the CEO of Phybbit, which would later be rebranded as Spider Labs. Rather than starting his own thing, she convinced him to join her company as the CTO.
While at Phybbit, they tried many different projects and services across diverse industries, such as digital entertainment, virtual reality, and a C2C marketplace. None of them worked out. As they continued to try new things, they began to work more closely with the ad industry, doing consulting projects. As a part of this, they built a machine learning framework that was used as an internal tool to help their customers. One of their customers was having problems with ad fraud, and by applying their framework to it, they realized they could launch a new service to help solve it. This would become SpiderAF.
Initially Eurico was the only developer that could focus on the project, while the other members were on consulting projects that helped keep up the cash flow. But as it became obvious this product was different from their past attempts, they raised funding, securing rounds in 2018 and 2019. This allowed the rest of the company to also concentrate on the project. To reflect this change, they renamed the company to Spider Labs.
We handle an amount of traffic that you don’t handle in other contexts. Many terabytes of data on a daily basis. Terabytes became the default unit. Billions of requests a day became the new normal as well.
The additional focus paid off, and they were able to achieve incredible growth. This growth has brought scaling challenges though. Eurico said, “We handle an amount of traffic that you don’t handle in other contexts. Many terabytes of data on a daily basis. Terabytes became the default unit. Billions of requests a day became the new normal as well.”
Initially, the early prototype was written in Ruby, but soon performance became an issue. Eurico said, “We started to have memory issues. For a while, we worked on those issues, but we were fighting and fighting. That was a good point to switch to another stack.”
They decided to choose Elixir. Eurico said, “Elixir was created by somebody involved in the Ruby community, and so it had some of the same appeal. We ran a pilot project on it. Initially there was a lot we couldn’t understand, so we focused on the good parts: the compiler support, it’s much faster than Ruby and has massive concurrency support. We can do things like Sidekiq out of the box by spawning a process here and sending it to the background. That’s the stuff we could grasp and start to use immediately. The other stuff we could learn later.”
Eurico was initially concerned about deploying a production Elixir app. He said, “I was very worried about operational knowledge. If you deploy something in production, and it blows up, what do you do? How do you investigate the problem, the errors? But that pilot project helped us a lot to understand how to deploy an application built in Elixir and prepare on the operational side.”
Emboldened by the success of their pilot project, they decided to rewrite their web application in Elixir, a decision they haven’t regretted. Eurico said, “It’s still a very niche language and ecosystem, but the documentation is amazing, the toolset is amazing, and the libraries are well defined and very stable. On top of that, it’s built upon the Erlang virtual machine, which in itself gives a lot of tooling and support.”
Most of their new hires don’t have Elixir experience, but that hasn’t been a problem for them. Eurico said, “A lot of the web development part is centered around the concept of a request. People that have been using frameworks like Sinatra or Ruby on Rails understand that lifecycle. We don’t look explicitly for Elixir experience. I think functional programming has some concepts that are hard to grasp initially. But after implementing them a couple of times, they become second nature.”
Every time we pick up on something, there’s something else they can use. The type of fraud we see today is not the same as we saw early on.
Besides performance, the other big technical challenge they’ve faced is the constant arms race between them and the scammers. Eurico said, “Five years ago, when we released SpiderAF more publicly, we were catching a lot of traffic that was disguised as a Japanese user, but was actually from other countries. We picked up on that. They learned, and the next thing they did was to buy access through a data center in Japan. Now they had an IP in Japan, but we were still able to detect it. Every time we pick up on something, there’s something else they can use. The type of fraud we see today is not the same as we saw early on.”
When a fraudster bypasses their detection countermeasures, they conduct an investigation and come up with new countermeasures. They’re not only being passive though, but are also working to proactively prevent next-generation exploits. Eurico said, “We have research that may translate into a product down the line about two years. What if we can’t rely on that piece of information? What if, for example, people running bots will use this approach to be more like a human? We’re doing a lot of research, so when those shifts happen, we’re ready to still counter an attack.”
When we’re able to help [our customers] recover and protect themselves, and that brings an impact to their business, they often say thank you. Seeing the results, the comments, the feedback, that always gives me a feeling of satisfaction and drives me a lot.
Not only is overcoming fraud interesting from a technical perspective, but it’s also rewarding to work on because of the quantifiable effect it has on their customers’ businesses. Eurico said, “Sometimes businesses come to us with urgent situations. People are at a loss what to do. Reaching out to professionals that can help them solve the situation makes them very relieved. When we’re able to help them recover and protect themselves, and that brings an impact to their business, they often say thank you. Seeing the results, the comments, the feedback, that always gives me a feeling of satisfaction and drives me a lot.” As for what the future holds, he said “we’re focused on bringing the technology that has helped some many in the ad tech industry to a wider audience making it both affordable and easy to access”.
Besides the interesting technical challenges, there are other reasons he recommends an international engineer work at Spider Labs. He said, “As a company, Spider Labs continues to invest in creating a better work life integration so that work becomes both more comfortable and less stressful. Spider Labs is expanding both the Tokyo and Lisbon offices and teams, and hopefully once the COVID situation gets under control, travel across the offices can resume.”