Trends in AI 2019 Edition
AI and Machine Learning are at a tipping point. Today we summarize some of the exciting trends this year.
AI Enabled Chips
Moore’s law is dead and custom chips are here to stay. At the heart of developing chips specifically for AI lies the principle that AI requires a lot of matrix multiplication operations. Speeding those up at the hardware layer significantly speeds up the entire algorithm.
The chart below with the price of a GPU shows just how popular they’ve become in recent times due to the crypto excitement and use of AI.
The second AI Hardware summit took place this year in September where the focus again were DSAs or Domain Specific architecture.
Nvidia announced their enterprise “supercomputer”, the DGX-2 with 16 Tesla Vs for 480 TFLOPs of FP16 operations delivering around 200x the performance of an average CPU.
Intel has definitely been lagging in this space but is on track to building Aurora which is an exascale computer.
Google launched Google TPU Pods V2 and V3 publicly with it breaking scalability records for AI inference. But even more exciting is the fact that they recently announced an edge TPU which can run Tensorflow Lite.
Startups like Cerebras building the CS-1 which is 56x faster than the average GPU.
Machine Learning requiring less data
We all know that even today one of the biggest challenges data scientists face is the lack of data or data that is inaccessible or expensive (or both). Companies that do have access to large datasets are for obvious reasons not opening them up. The need for copious amounts of data only gets worse with the rising popularity of deep learning.
So how do smaller companies and research labs go about building the next big thing? There has been some research ongoing related to relaxing the amount of data needed in various ways: zero-shot learning with less reliance on labeled training data by solving unseen samples in 1 shot; transfer learning where you train on some kind of task and try to generalize the knowledge on some other task; weak supervision or generating lower quality training data cheaply and getting some higher quality training data from SMEs, etc. There are multiple approaches explained by Jyoti Prakash Maheswari in their post.
Research
Deep Learning
Deep Learning is by far the most exciting topic within machine learning today. Low hanging fruits of deep learning have mostly been achieved. DNNs are now everywhere: self driving cars, voice assistances, robots, etc. Various research breakthroughs (Google AI’s BERT, Transformer; Allen Institute’s ELMo; OpenAI’s Transformer, Ruder & Howard’s ULMFiT, Microsoft’s MT-DNN) demonstrated that pretrained language models can substantially improve performance on a variety of NLP tasks. OpenAI’s GPT-2 has been making waves with their pretrained language model (you can try it out at http://transformer.huggingface.co/) and language understanding surpassed human thresholds with GLUE.
At the same time, some of the biggest problems such as explainability of Deep Learning, the amount of data needed, etc are becoming more and more pronounced. An interesting article by @garymarcus on the deepest problems with deep learning highlight the same.
Reinforcement Learning
Shot into the spotlight after AlphaGo beat Lee Sedol in 2016, the artificial intelligence community has been going back to basics with reinforcement learning which is based on rewards and punishment. In October 2018, RL achieved new heights when OpenAI achieved superhuman performance at Montezuma’s revenge with a technique called Random Network Distillation and again in 2019 with their performance in Dota. Deepmind’s work on deep reinforcement learning for robotic manipulation also made some new strides. Some people have gone so far as to predict the death of deep learning, while the amount of research in reinforcement learning has seen an uptick.
OpenAI has been making tremendous breakthroughs in this space with the open sourcing of their OpenAI Gym, . They recently released Spinning up in Deep RL designed to let anyone become an expert in deep reinforcement learning. Also this is here. .
Generative Adversarial Networks (GANs)
First conceived in a 2014 with the groundbreaking research paper “Generative adversarial networks,” by Ian Goodfellow et al. GANs have since been a hot area for researchers despite real life applications still missing. The most popular real life application has been around image to image style transfer. State of the art GANs continue to evolve from grainy to GANgsta.
Deep fakes have aroused as much curiosity as fear, since they have crushed the most trustable source of information: the video footage.
China gaining ground in research
The maximum number of research papers have historically been from the US only. Even though the number of papers from China have increased, the number of citations are maximum for papers from the US. Europe punches above its weight in terms of citations when it comes to AI research, but China is quickly gaining ground.
DevOps for ML workloads
ML infrastructure is also at an interesting tipping point. The DevOps moment (which is really a period) for software engineering propelled tech to all industries, making them not just tech enabled but tech driven.
We now have a whole conference dedicated to discussing infrastructure for machine learning and AI workloads: Twimlcon.
AutoML
AutoML has never been as exciting as it is right now. Amazon recently announced Sagemaker AutoML platform Autopilot. AutoML funding has been at an all time high. These autoML platforms package the solution conveniently usable as an API for consumption by any generic software engineer who might not have data science skills. Google has its own version with Google Cloud AutoML.
Machine Learning Frameworks Galore
Pytorch has been increasingly dominating research in recent years and has now overtaken Tensorflow. Some of the features driving this include it’s easy to understand and work with API and ease of debugging.
Besides the development framworks, there are now a lot of deployment end to end pipelines such as kubeflow, MLflow, etc are gaining more ground but there’s no clear winner yet.
ML Infra startups making a mark
This is the year in which scale.ai became the first ML infra unicorn that offers annotations as a service using AI and contract workers.
Conclusion
The past year pushed the envelope not just in terms of new techniques and research but also more pragmatic questions like explainability, scaling AI in large enterprises, tooling and finding business value. While we continue to face challenges in industry applications, expect to see a lot more in these spaces in the next year. The dreaded question “Is AI a hype” will only be settled if/when we see more meaningful and successful applications.
Most importantly, deep reinforcement learning now powers Pipernet.
