Should We Use Machine Learning As A Hammer For Every Nail?
Machine learning (as an important subfield of broader field of artificial intelligence), is clearly in vogue. Every week we hear about a new hot startup, proudly having ‘machine learning’ and / or ‘artificial intelligence’ as part of their name or sales pitch, which has been either acquired, or showered with head spinning amount of fresh VC funding. It seems that even blockchain, after several years of being in the prime spotlight on its own, can’t compete anymore – especially after showing signs of struggle to make inroads into traditional enterprises.
Understanding Underlying Technology Fundamentals Is The Key
However, as with any technology, understanding the fundamentals behind it is crucial for meaningful utilization. Being fully clear about the technology’s potential, but also be comfortable with its limitations, is the most important element of any technology rollout and adoption strategy. Lessons learned from blockchain space, clearly teach us that simply believing in ‘magic behind it’, does not help at all. Unfortunately, based on what I read on a daily basis, many AI and machine learning enthusiasts seem to be frequently falling into the similar narrative traps. The hard reality is that there is no ‘magic’ behind any technology. You either understand it or you do not.
Going back to blockchain example, hopefully by now it should have become obvious that any distributed consensus-based system will ALWAYS going to perform visibly slower than the centralized based equivalent solution. Does that mean that blockchain will have no place in our lives? No, not at all. On the contrary, there are areas where blockchain may be the most natural fit, like in trade finance, digital identity, etc. But in the areas where high throughput and low latency are primary requirements (like retail, commercial and transport real time payments, mobile telephony, stock exchange matching engines, etc), blockchain just may not be the right tool for the job – and there is nothing wrong with admitting that.
Similarly, understanding the fundamentals behind machine learning will help you in making educated decisions about whether machine learning is appropriate solution (or not) for business problems and use cases you have at hand. Many business problems can still be better and efficiently solved / automated, by using the traditional deterministic programming techniques. Again, the very same examples of retail and transportation payments, mobile telephony or stock exchange matching engines immediately come to mind here as well. The efficient set of clear and deterministic business rules (of course this is over simplified) is already doing a perfectly good job in these cases.
On the other hand, the machine learning may be the best technology choice and option, for:
Tasks whose underlying rules are too complex to be clearly understood, specified and programmed: There are large number of routine tasks that may look as trivial to a human brain and which we all perform subconsciously on a daily basis, but which are extremely hard, if not impossible, for a system analyst to specify in a set of well-defined steps and conditions that could be precisely coded in software. Everyday examples of such human tasks are walking, running, car driving, speech recognition, and image interpretation / vision. Those activities are simply based on too many interdependent external environmental factors and sensory inputs, connected via large number of often overlapping and not so obvious rules. In such fuzzy and fluid environment, it can be impossible for human mind to fully grasp and articulate the problem domain in a set of precise deterministic specifications, which can be accurately coded. On the other hand, today’s advanced machine learning programs, that “learn from experience,” usually achieve quite acceptable results in these areas, approaching human brain performance, after being trained with sufficiently large set of training data samples.
Systems with highly adaptive nature: One painfully limiting feature of deterministic software systems is their inherent inflexibility – once the software requirements have been understood, captured, designed, implemented and installed, they stay unchanged. For the exact same input data values, deterministic software will always accurately produce the exact same expected output, but it will not be able to flexibly handle slight variations of input parameters, unless its rules have been updated and reprogrammed. However, it is prohibitively expensive to change and retest software every time when new unseen input variation is encountered. Typical examples of such problems, where degree of behavioural flexibility is required, include interpreting handwritten text, recognizing speech, detecting spam, detecting mood and emotions, etc. Good news is that the machine learning systems, with their ability to adapt their behaviour based on input data offer a potentially elegant solution to such challenges; i.e. they are, by their very nature, adaptive to reasonable changes in the input data they process.
What Are The Limits Of Machine Learning?
Even when machines can ‘learn’ by themselves, the preferred patterns of desirable behaviour are still set and defined by humans, primarily those preparing and supplying the labeled training and test data sets. Majority of current generation machine learning applications are highly specialized for a specific narrow domain and trained by skilled engineers who choose the set of input data features and ‘tell’ the machine what is ‘right’ and what is ‘wrong’. For example, the fine-tuned deep neural network that is pre-trained for face recognition, is going to be terrible at detecting email spam or financial fraud. It will at least have to be fully re-trained from scratch for the new problem – which is limiting our ability to randomly apply pre-trained neural network to brand-new problem domains. We experience similar examples in real life. A highly trained ‘English speaking’ human brain, immediately struggles when faced with task of interpreting / understanding the ‘Russian language’, despite both being instance of a well-known human language.
All in all, machine learning is very interesting and powerful systems development approach, which for some types of problem domains, especially those identified and listed above, vastly outperforms the traditional deterministic programs in both - accuracy and flexibility. That is excellent news for business and software engineering community. However, the quality and amount of training and test data is of utmost importance ... otherwise you may inadvertently end up with biased systems, which may behave in discriminatory manner, for example by favouring one gender / demographics / race / over another, etc.
Therefore, despite what you are being told by an over-enthusiastic futurist, random travelling salesman or an eloquent consultant, machine learning isn’t panacea nor universal remedy for every type of business problem. Even if machine learning is determined to be the right approach and tool for a particular business domain, the usefulness and accuracy of the final / trained model would be highly dependent on skill of your data scientists and engineers – i.e. those in charge of identifying the right set of input features, choosing the right type of learning algorithm and properly training your enterprise AI engines without undesired bias.
In other words, highly creative humans will still be required, going forward, even in the new world of advanced artificial intelligence. Machine learning doesn’t eliminate demand for skilled engineers. Humans still need to understand the nature of the problem first and then instruct the machine on how to teach itself, in order to deal with the problem in real world.
To summarize - despite recent hype, nothing fundamentally changes in the way professional software engineers will be approaching solving business problems in the future – only available tools will be different. And as far as HR departments are concerned - they are becoming rapidly aware that the competition for real technology talent is only going to get worse.