Machine learning and deep learning architectures and trends: A review

The arrival of machine learning (ML) together with deep learning (DL) has been revolutionizing many fields through advances in data-driven decision-making, automation, and predictive analytics. This has formed the keystone for the exploration of the most recent architectures and upcoming trends in said domains as to how they are significantly impacting other sectors. Recent ML designs, such as Transformers or graph neural networks (GNNs) in combination with neural differential equations, have found remarkable performance in tasks such as natural language processing (NLP) or recommendation systems and molecular modeling. The birth of big language models (LLMs)-from GPT-4 to BERT-has furthered the understanding and production of human languages to degrees where chatbots, translation, and content generation are advanced. At the same time, DL structures have evolved with the advent of state-of-the-art advancements, e.g., convolutional neural networks (CNNs) and generative adversarial networks (GANs), that play an essential role in areas such as image and video processing, autonomous driving, and synthetic data generation. This work focuses on how these structures may be combined with the advanced technologies of the Internet of Things with that of blockchain and quantum computing to enhance security, efficiency, and scalability for intelligent systems. Increasing trends show a concern with artificial intelligence (AI) and explainable AI (XAI) to deal with crucial problems of transparency, fairness, and accountability. The study also examines how federated learning is likely to influence privacy-driven data analysis and the surge in edge AI, which involves pushing computing closer to the source of data, reducing latency and improving real-time decision-making. This research will identify the importance of ML and DL, which is crucially important in showing the shape of technology and society in the future.

Keywords: Machine learning, Deep learning, Architectures, Artificial intelligence, Convolutional neural networks, Recurrent neural networks, Natural language processing

Citation: Rane, N. L., Mallick, S. K., Kaya, O., & Rane, J. (2024). Machine learning and deep learning architectures and trends: A review. In Applied Machine Learning and Deep Learning: Architectures and Techniques (pp. 1-38). Deep Science Publishing. https://doi.org/10.70593/978-81-981271-4-3_1

1.1 Introduction

The fields of healthcare and finance have seen significant changes as a result of machine learning (ML) and deep learning (DL), which enable machines to learn from data and make decisions with minimal human intervention (Chauhan & Singh, 2018; Shrestha & Mahmood, 2019). These technologies' adoption and expansion have been accelerated by the rapid advancement of processing power and the wealth of available data (Shrestha & Mahmood, 2019; Shinde & Shah 2018; Dargan et al., 2020). The ML and DL architectures, which are the foundation of these technologies, have made significant progress and shown remarkable capabilities in tasks such as natural language processing, autonomous systems, and image and audio recognition. ML models come in a variety of architectures, from basic linear regression models to intricate neural networks, designed for different tasks and types of data (Chauhan & Singh, 2018; Sengupta et al., 2020; Alzubaidi et al., 2021). DL, a branch of ML, utilizes neural networks with multiple layers to capture complex patterns and features in data (Dargan et al., 2020; Alzubaidi et al., 2021; Minar & Naher, 2018). Architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs) have expanded the capabilities of machines, resulting in advancements in computer vision, speech synthesis, and generative art.

Research trends in machine learning and deep learning are driven by continuous advancements and growing demands from various applications (Shrestha & Mahmood, 2019; Janiesch et al., 2021; Topuz & Alp, 2023). Better algorithms, integrating ML and DL with other technologies like blockchain and IoT, and emphasising ethical AI and responsible use of these technologies are some recent developments (Bal & Kayaalp, 2021-Thakkar & Lohiya, 2021). To improve the transparency and interpretability of ML and DL models for users, there is also a developing trend towards explainable AI (XAI) (Wang et al., 2020; Mu & Zeng, 2019; Miotto et al., 2018). This research explores the complex structures of ML and DL, analyzing their progress, present developments, and upcoming paths. By conducting a thorough review of the literature, we pinpoint important developments and upcoming trends in these areas. Moreover, we utilize keyword co-occurrence and cluster analysis to reveal the key topics and research groups, offering a detailed insight into the present scenario and possible future advancements (Fig. 1.1).

Machine learning (ML) continues to evolve rapidly along with advances in data-driven technologies. As a sub-discipline of artificial intelligence (AI), this field is revolutionizing many industries with its abilities to learn and make predictions from data. Current ML trends are expanding the application areas of the technology and increasing the effectiveness of existing methods.

Fig. 1.1 Recent trends in Machine learning and deep learning architectures

Foundation Models are a popular trend; these are models that can be trained on large-scale datasets and are versatile enough to perform a range of tasks. Artificial intelligence (AI) solutions that maximise human-machine cooperation and augment human capabilities are referred to as augmented intelligence. IoT devices in particular use embedded machine learning (ML), which describes models that are housed inside the device and have the ability to interpret data in real time.

Metaverses create new interaction and economy models with the use of ML algorithms in virtual and augmented reality environments. In the healthcare industry, machine learning in healthcare offers innovative solutions in critical areas such as disease diagnosis and patient care. Data Security and Regulations enable the development of reliable AI systems by addressing privacy and ethical issues in the use of ML models.

Algorithmic Decision-Making enables automated and optimized decisions to be made in business processes and daily life. Transformers or Seq2Seq Models are models that show strong performance in language processing and other sequential data tasks. Multi-modal ML deals with models that can perform more comprehensive analyses inter-jointly, considering different styles of information, such as text, image, or audio. Low-Code/No-Code Innovations, on their part, provide tools that will let non-technical users develop models for machine learning. Natural Language Processing does the same but in developing models that understand and produce human language.

These trends are shaping the future direction of machine learning and opening up wider application possibilities across various industries. The increase in data-based decision-making processes and automation increases the importance of ML day by day. These advances are paving the way for smarter, more effective and more accessible AI solutions (Table 1.1).

Table 1.1 Deep learning trends in emerging cloud computing architectures

Deep Learning Technique

Computing Architecture

Devices

Deep Belief Network

Mobile Edge Computing

Medical wearables

Deep Belief Network

Mobile Edge Computing

Smart phones

Deep Belief Network

Mobile Edge Computing

Smart cameras

Deep Belief Network

Mobile Edge Computing

Voice controllers

Deep Reinforcement Learning

Fog Computing

Gateways

Deep Reinforcement Learning

Fog Computing

Computers

Deep Reinforcement Learning

Fog Computing

Switches

Deep Reinforcement Learning

Fog Computing

Automated vehicles

Convolutional Neural Network

Edge Computing

Desktops

Convolutional Neural Network

Fog Computing

Routers

Convolutional Neural Network

Mobile Edge Computing

Smart watches

Convolutional Neural Network

Volunteer Computing

Desktops

Modified Convolutional Neural Network

Edge Computing

IoT sensors

Modified Convolutional Neural Network

Mobile Edge Computing

Smart phones

Modified Convolutional Neural Network

Fog Computing

Gateways

Deep Neural Network

Edge Computing

IoT sensors

Deep Neural Network

Mobile Edge Computing

Smart phones

Deep Neural Network

Fog Computing

Routers

Deep Neural Network

Serverless Computing

Smart speakers

Recurrent Neural Network

Edge Computing

Smart phones

Recurrent Neural Network

Edge Computing

ECG device

Long Short-Term Memory

Edge Computing

Medical devices

Long Short-Term Memory

Edge Computing

Smart wearable devices

Long Short-Term Memory

Fog Computing

Switches

Long Short-Term Memory

Fog Computing

Servers

Long Short-Term Memory

Fog Computing

Routers

 

Deep learning (DL) is a sub-field of machine learning that uses multi-layer artificial neural networks to learn intricate patterns in data. Assume that deep learning is currently being included into developing cloud computing architectures to offer a theoretical framework across several domains. Then, determining the future course of the technology will require an understanding of how deep learning techniques interact with other cloud computing infrastructures and devices.

Deep learning approaches often improve data processing capacities and optimise device interactions when combined with cloud computing infrastructures. These trends provide notable benefits in applications needing massive data amounts and great computational power by enabling smart systems to function more effectively and efficiently. Future developments of increasingly intelligent and autonomous systems will be facilitated by advances in this dynamic and quickly developing field of deep learning (Balaji et al., 2018; Alzoubi et al., 2024; Deng et al., 2024; Lundberg, et al., 2024).

Significance of the research work:

• Performing an extensive review of literature to outline the progression and present status of ML and DL structures.
• Applying keyword co-occurrence analysis to recognize significant trends and upcoming subjects within the field.
• Conducting cluster analysis to reveal the main research clusters and how they are connected, offering insight into potential future research paths.

 1.2 Methodology

In this research, we utilized an extensive literature review, keyword analysis, co-occurrence analysis, and cluster analysis to examine the structures and patterns in the fields of ML and DL. The research was based on a literature review that included a methodical examination of peer-reviewed articles, conference papers, and academic publications on ML and DL architectures and trends. We made use of various academic databases such as IEEE Xplore, Scopus, and Google Scholar to compile a comprehensive selection of pertinent research. The criteria for selection included works published within the last ten years, guaranteeing a contemporary comprehension of the subject. Our attention was directed towards articles covering new ML and DL architectures, their uses, evaluations of how well they perform, and upcoming trends. This thorough examination enabled us to pinpoint key themes and progressions in the literature.

In order to explore more deeply the specific areas of ML and DL research, we carried out a keyword analysis. This included the extraction and analysis of keywords from the chosen literature in order to pinpoint the terms and concepts that appeared most frequently. Automated tools were used in conjunction with manual verification during the keyword extraction process to guarantee accuracy. We proceeded to measure how often each keyword appeared in order to identify the main topics of interest and research focus in the ML and DL community. This examination gave us a deeper understanding of the main subjects and aided in visualizing the research field. Expanding on the keyword analysis, we conducted a co-occurrence analysis to investigate the connections among various keywords and concepts within the field. This technique required forming a co-occurrence matrix to record the frequency of pairs of keywords appearing together in the articles. By utilizing network analysis methods, we depicted these connections through co-occurrence graphs. These charts emphasized groupings of similar keywords, uncovering connections and thematic focuses in the realm of ML and DL research. This was a crucial measure in recognizing connections between different disciplines and merging multiple sub-fields. In the end, we used cluster analysis to group the identified themes and trends into cohesive categories. We used co-occurrence data to apply clustering algorithms like K-means and hierarchical clustering for grouping related keywords and research topics. This examination allowed us to identify separate groups that represent various areas of research, architectural advancements, and trend trends in ML and DL. Through analysis of these groupings, we were able to offer a systematic review of the present condition of the field, emphasizing key research topics and up-and-coming patterns.

 1.3 Results and discussions

Co-occurrence and cluster analysis of the trending keywords in ML

The network diagram in Fig. 1.2, illustrating the co-occurrence and clustering of keywords in ML trends, provides important insights into the current trends and connections in the field. This examination investigates important groupings and the connections among main ideas, demonstrating how various aspects of ML study are intertwined and developing. The network diagram shows various separate clusters, each indicating a different thematic area in ML research. Each cluster consists of keywords that often appear together in academic papers, demonstrating a clear thematic relationship. The main clusters that were identified are:

Key Methods and Algorithms in ML (Red Cluster)

The fundamental ideas and techniques of machine learning constitute the heart of the red group. In this group, terms like "classification," "learning systems," "predictive models," "machine learning," and "optimisation" are essential. This team exemplifies the fundamentals of machine learning research, focussing on developing and optimising algorithms that can learn from data and make decisions or predictions. Notable terms in this category are also "decision making," "performance," "forecasting," and "support vector machines." These words designate study fields that aim to improve machine learning models' accuracy and efficacy. Phrases like "internet of things" and "cybersecurity" are included because they imply the application of these techniques in specific domains, demonstrating the interdisciplinary nature of current machine learning research.

Healthcare and Diagnostic Uses (Green Cluster)

The significance of machine learning for the healthcare sector and medical testing is emphasised by the green group. The terms "human," "algorithm," "diagnosis," "diseases," and "controlled study" are essential elements of this category. This demonstrates a focused attempt to analyse medical data using machine learning techniques, identify diseases, and improve patient outcomes. Terms like "sensitivity and specificity," "comparative study," and "major clinical study" refer to the exacting assessment methods used in medical research to ensure the precision and reliability of machine learning models. Demographic terms such as "male," "female," "aged," and "adult" ensure that machine learning algorithms are broadly applicable and effective in healthcare research because they point to a focus on a range of patient populations.

Blue Cluster consists of Neural Networks and DL

The blue cluster is mostly focused on topics related to deep learning and neural networks. Within this category, terms like "deep learning," "artificial neural network," "convolutional neural networks," "training," and "feature extraction" are essential terms. This highlights the significance of neural networks in current machine learning research because of their superior ability to recognise complex patterns in large datasets. Keywords like "image processing," "image segmentation," and "image enhancement" point to specific uses of deep learning in computer vision problems. The persistent emphasis on "precision" and "algorithmic learning" indicates the dedication to improving the efficiency and efficacy of deep learning models. This group demonstrates how important deep learning techniques are to improving machine learning systems' capabilities.

Fig. 1.2 Co-occurrence analysis of the trending keywords in ML

Natural Language Processing and Computational Modeling (Yellow Group)

The yellow cluster is primarily concerned in computational modelling and natural language processing (NLP). Important terminology like "natural language processing," "natural languages," "computational modelling," and "neural networks" are central to this cluster. This indicates that there is a great deal of interest in the study and development of algorithms that can understand, interpret, and produce human language. Words like "e-learning" and "data mining" refer to the application of natural language processing (NLP) techniques to improve educational systems and extract meaningful information from large text datasets. The integration of "semantics" with "feature extraction" highlights the difficulties and developments in comprehending and expressing the meaning of words and phrases in a computationally effective way.

Cybersecurity and Internet of Things (Purple Cluster) focus on protecting connected devices from online threats.

The purple cluster signifies the overlap of machine learning, cybersecurity, and the Internet of Things (IoT). Terms like "cybersecurity," "network security," "intrusion detection," and "internet of things" are fundamental within this group. This shows a strong research emphasis on utilizing machine learning to improve security protocols in connected devices and networks. The use of terms such as "optimization" and "predictive models" indicates the utilization of machine learning methods to forecast and address security risks. This cluster highlights the important role that machine learning plays in creating strong cybersecurity solutions in a world becoming more interconnected.

Connections between different disciplines and collaborative research involving multiple fields.

The network diagram shows both separate clusters and connections between various areas of machine learning research. For example, terms like "optimization," "predictive models," and "performance" are present in numerous groups, showing their wide relevance in different fields. The interconnected clusters demonstrate the interdisciplinary method commonly seen in contemporary machine learning studies. Progress in fundamental methods and algorithms (red group) are utilized in specialized sectors like healthcare (green group) and cybersecurity (purple group), with advancements in deep learning (blue group) and NLP (yellow group) pushing advancements in various areas.

Current patterns and upcoming paths

In machine learning research, the network diagram provides an insight into emerging trends and possible future directions. Deep learning, healthcare, and cybersecurity are highly frequented disciplines, suggesting that these areas will continue to be important areas of study. A step towards developing intelligent systems that can effortlessly connect with both humans and machines is suggested by the combination of machine learning, IoT, and NLP. Furthermore, efforts to improve the efficiency and accuracy of machine learning models are continually demonstrated by the emphasis on optimising performance, using predictive modelling, and extracting features. As machine learning advances, researchers will likely focus on addressing challenges such as interpretability, scalability, and ethical difficulties to ensure that machine learning systems are not only robust but also just and dependable.

Co-occurrence and cluster analysis of the trending keywords in DL

Fig. 1.3 displays an examination of different keywords through co-occurrence and cluster analysis, providing understanding of the connections and importance of various concepts in deep learning studies. Central to the network diagram is the term "deep learning," situated prominently and linked to many other keywords, highlighting its pivotal position in the field. Deep learning, a component of machine learning, employs deep neural networks with multiple layers to examine diverse forms of data. The strong links surrounding "deep learning" demonstrate its essential significance and broad usage in many different fields.

Crimson Group: Educational Platforms and Usage

The red cluster has a high concentration of terms associated with learning systems and how they are used. The close connection between key terms such as "learning algorithms," "reinforcement learning," "learning systems," and "computational modeling" underscores the focus on advancing complex learning methods. Reinforcement learning is well-known for its ability to train models using reward-based learning methods, which are essential for tasks that involve making decisions in situations of uncertainty. This group also includes "neural networks" and "convolutional neural networks," which are important structures in deep learning. The importance of "convolutional neural networks" (CNNs) is highly notable, especially because of their extensive utilization in image and video recognition assignments. CNNs are crucial in handling visual data as they are linked to tasks such as "image enhancement," "object detection," and "object recognition."

Green Cluster: Applications Focused on Humans and Diagnostics

The green group is defined by key terms related to applications centered around humans and diagnostics. Words like "human," "adult," "female," "male," "diagnostic imaging," and "procedures" indicate a clear emphasis on medical and healthcare uses. This grouping suggests that deep learning methods are widely used in the examination of medical images, detecting illnesses, and enhancing healthcare results. Deep learning plays a crucial role in medical imaging, as evidenced by the frequent mention of terms such as "image segmentation," "diagnosis," and "nuclear magnetic resonance imaging" (MRI). Words like "important medical research," "group analysis," and "accuracy in identifying conditions" emphasize the use of deep learning in analyzing large data sets to find patterns and enhance diagnostic precision in clinical studies. The word "algorithm" is also often seen in this group, highlighting the creation of specific algorithms designed for medical use.

Fig. 1.3 Co-occurrence analysis of the trending keywords in DL

Blue Cluster: Processing and Analysis of Images

Terms like "image processing," "segmentation," "image classification," and "computerised tomography" are concentrated in the blue cluster because they are related to image processing and analysis. In domains like automated surveillance, remote sensing, and medical imaging, technological aspects of managing and interpreting visual data are the focus of this group. To increase the precision and effectiveness of picture analysis, sophisticated methods including "semantic segmentation" and "attention mechanisms" are used. For tasks requiring detailed understanding of visual scenes, semantic segmentation—such as identifying each pixel in an image—is crucial. Attention mechanisms, on the other hand, enable models to focus on significant portions of the data, increasing their efficacy in difficult tasks.

Golden Group: Fundamentals of Machine Learning

The small yellow group is important, emphasizing basic machine learning ideas. The central focus of this cluster is on keywords such as "machine learning," "learning," "prediction," and "algorithm." This indicates a close connection between deep learning and general machine learning principles, emphasizing how progress in deep learning is rooted in core machine learning theories and methods.

The link of this group to "forecasting" and "decision making" suggests the use of deep learning methods in predictive analytics and strategic decision-making processes, crucial in multiple sectors like finance, marketing, and operations management.

Connections and Evolving Patterns

The interconnectivity of various deep learning subfields is further illustrated by the network diagram, which displays multiple noteworthy linkages amongst clusters. One example of this is the association between "image segmentation" in the blue group and "CNNs" in the red group, which highlights the critical role that CNNs play in image processing tasks. Similarly, "diagnostic imaging" in the green cluster and "machine learning" in the yellow cluster are related, indicating that machine learning techniques are being used in medical diagnostics. Emerging trends in deep learning are also depicted in the network diagram. Keywords like "attention mechanisms," "auto encoders," and "transformer" demonstrate the growing importance of sophisticated designs and methods. Transformers, originally made popular in the field of natural language processing, are now being used more and more for different purposes such as image processing and analyzing time series data. Attention mechanisms are increasingly essential for improving model performance in various applications by enabling models to selectively emphasize relevant input components. Autoencoders, which are utilized for unsupervised learning and data compression, demonstrate the continual striving for enhanced model efficiency and effectiveness.

Emerging architectures in machine learning

Models based on transformer architecture

Transformer models, with self-attention mechanisms at their core, heralded a breakthrough in developing ML infrastructures (Sengupta et al., 2020; Bachute & Subhedar, 2021; Penney & Chen 2019). Initially designed for NLP, they have found promisingly effective use in wide applications. Their design based on self-attention mechanisms allows for the efficient handling of dependencies for both long and short distances in sequential data, something that recurrent neural networks (RNNs) had previously struggled with.  A good example is the Transformer-based architecture which has enabled the recent models BERT and GPT to set state-of-the-art results for many NLP tasks, including translation, summarization, and question answering (Sengupta et al., 2020; Voulodimos et al., 2018; Jordan & Mitchell, 2015). These models were pre-trained on large enough datasets and fine-tuned on the target tasks. Transformer architectures have also been extrapolated to cater to visual tasks, showcasing scalability and generalization features. Vision Transformers (ViTs) apply the Transformer body on patches within images and perform equally with the CNN structure for tasks like image classification and object detection. The modularity and reusable concept bring to the fore the prospect of change within Transformer architectures in ML.

Graph Neural Networks (GNN)

GNN are powerful tools that have become popular for analyzing data structured in graphs (Shrestha & Mahmood, 2019; Alzubaidi et al., 2021; Deng, 2014; Sarker, 2021). GNNs differ from traditional neural networks in that they can process data in the form of graphs, which are commonly found in social networks, biological networks, and recommendation systems (Shinde & Shah, 2018; Özerol & Arslan Selçuk, 2023; Thakkar & Lohiya, 2021; Miotto et al., 2018). GNNs use message-passing methods, where nodes exchange information with neighboring nodes in a graph, enabling the network to understand the data's underlying structure. The capability to represent connections and interactions among entities makes GNNs highly efficient for tasks such as node classification, link prediction, and graph classification. Recent developments in GNN designs have been concentrating on enhancing scalability and efficiency. Strategies like GraphSAGE, which collect data from a set number of neighbors, and attention mechanisms in Graph Attention Networks (GATs), which give varying importance to different neighbors, have greatly improved the effectiveness and usefulness of GNNs in large-scale environments.

Neural Architecture Search (NAS)

NAS is a revolutionary change in neural network design as it automates the process of finding optimal architectures (Thakkar & Lohiya, 2021; Bashar, 2019; Lee et al., 2017; Moein et al., 2023). In the past, creating successful neural network structures involved a great deal of skill and hands-on trial and error (Dargan et al., 2020; Avci et al., 2021; Nguyen et al., 2018; Yap et al., 2019). NAS uses reinforcement learning, evolutionary algorithms, or gradient-based optimization to find the best architectures designed for specific tasks. One major accomplishment of NAS is the creation of EfficientNet, a series of models that excel in image classification benchmarks by using fewer parameters and requiring less computational resources. EfficientNet models were developed using a compound scaling technique found through NAS, which evenly scales depth, width, and resolution dimensions. NAS has also been expanded to uncover designs for specific tasks like object detection and semantic segmentation, resulting in models that surpass those designed by humans. As NAS techniques progress further, they offer the potential to make the design of neural network structures more widely available, thereby increasing accessibility to advanced ML models.

Architectures for Federated Learning

Federated Learning (FL) is a new approach that tackles privacy and data security issues in ML by allowing joint model training across various devices or organizations without consolidating data (Shrestha & Mahmood, 2019; Alzubaidi et al., 2021; Emmert-Streib et al., 2020; Patil et al., 2020; Kassem et al., 2021). In Florida, a worldwide model is trained by combining updates from local models trained on distributed data sources. Federated learning systems consist of important elements: client devices conduct local training, the server aggregates updates, and communication protocols guarantee secure and efficient data exchange. Recent developments in FL designs prioritize enhancing communication efficiency, managing non-IID data, and guaranteeing resilience against adversarial attacks. Federated learning is being used in different areas such as healthcare for collaborative research while protecting patient confidentiality, and finance for fraud detection without sharing customer data. With the increasing strictness of data privacy regulations, federated learning architectures are expected to have a significant impact on the future of ML.

Capsule Networks

Geoffrey Hinton and his colleagues created Capsule Networks, a novel architecture designed to get beyond the limitations of traditional CNNs. While standard pooling strategies employed in CNNs may not preserve the hierarchical connections between features, Capsule Networks aim to do just that (Dargan et al., 2020; Alzubaidi et al., 2021; Yadav & Vishwakarma, 2020; Angulakshmi & Deepa, 2021; Deng, 2019). The idea behind capsule networks is to represent different aspects of items or parts of objects with accuracy by using capsules, which are collections of neurones. Rather than generating single values, these capsules generate vectors, or matrices, that contain data about the presence and location of features. The capsules have a dynamic routing mechanism that allows higher-level capsules to get input from lower-level capsules that are relevant, which helps in preserving spatial and hierarchical relationships. Despite being in the early stages, Capsule Networks have demonstrated potential in tasks that involve intricate spatial comprehension, such as image recognition and 3D object reconstruction. Further investigation and improvement of Capsule Network structures may result in ML models that are more resilient and easier to interpret.

Self-Supervised Learning (SSL) architectures

The use of SSL is becoming more popular as a useful method for using unlabelled data to train machine learning models (Dargan et al., 2020; Özerol & Arslan Selçuk, 2023; Bashar, 2019). SSL architectures are made to solve automatically produced challenges called pretext tasks, which do not require human labelling, in order to acquire valuable representations from the data itself. Self-supervised learning architectures often follow a two-step process: pre-training on a large dataset with pretext tasks (e.g., guessing the next word in a sentence or missing portions of an image) and fine-tuning on a smaller labelled dataset for the target job. Simple Framework for Contrastive Learning of Visual Representations (SimCLR) and BERT models are two examples of how effective SSL is in identifying significant characteristics from large volumes of unlabelled data. The benefits of self-supervised learning architectures include improved data efficiency, as models can leverage large-scale unlabeled datasets, and enhanced generalization, as the learned representations capture more diverse and robust features. As the volume of unlabelled data continues to grow, self-supervised learning architectures are set to become increasingly important in the ML landscape.

Emerging architectures in deep learning

Transformers and Attention Mechanisms

Transformers have greatly altered the field of NLP (Shrestha & Mahmood, 2019; Dargan et al., 2020; Kassem et al., 2021). Transformers can process sequences in parallel, unlike RNNs and LSTMs, which depend on sequential data processing (Mishra et al., 2021; Sarker, 2021; Lee et al., 2017). Self-attention mechanisms allow the model to assess the importance of various words in a sentence without being limited by their positions. The self-attention mechanism computes attention scores between each word in a sentence, aiding in understanding context and improving accuracy in translations and text generation. Transformers have found success in areas other than NLP. Vision Transformers (ViTs) were introduced to utilize transformer models with image data. ViTs view an image as a series of patches and demonstrate top-notch results on multiple image recognition tests. This change in architecture showcases the adaptability of transformers and their ability to bring together different modalities within one system.

Graph Neural Networks (GNNs)

GNNs are an effective tool for extracting knowledge from data structured as graphs, commonly found in areas like social networks, molecular biology, and recommendation systems (Dargan et al., 2020; Özerol & Arslan Selçuk, 2023; Alzubaidi et al., 2021; Janiesch et al., 2021). Conventional neural networks face challenges when dealing with graph data because of its non-Euclidean characteristics. GNNs tackle this issue by working on the graph structure itself and utilizing message passing to collect data from nearby nodes. Recent developments in GNN structures involve Graph Attention Networks (GATs), which use attention mechanisms to dynamically assess the significance of nearby nodes. Another significant advancement is the introduction of Graph Convolutional Networks (GCNs), which extend the idea of convolution to graph information, allowing for the detection of nearby characteristics. These advancements have greatly enhanced the efficiency and flexibility of GNNs, establishing them as a crucial element of contemporary DL resources (Özerol & Arslan Selçuk, 2023; Alom et al., 2019; Khan & Yairi, 2018; Mu & Zeng, 2019; Miotto et al., 2018).

Neural Architecture Search (NAS)

Creating the best neural network structures typically involves a lot of knowledge and experimentation (Voulodimos et al., 2018; Jordan & Mitchell, 2015; Avci et al., 2021). Neural Architecture Search (NAS) speeds up this process by employing algorithms to seek out the most effective architectures. NAS methods utilize reinforcement learning, evolutionary algorithms, or gradient-based techniques to investigate a wide range of possible architectures. Recent developments in NAS are centered on improving efficiency and scalability. Sharing parameters among different architectures during training in Efficient Neural Architecture Search (ENAS) lowers the computational cost of NAS. DARTS (Differentiable Architecture Search) enhances efficiency by making the architecture search process differentiable, which enables gradient-based optimization. These methods make the design of neural networks accessible to more people, allowing them to find advanced structures without needing to manually adjust them extensively.

References

Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., ... & Asari, V. K. (2019). A state-of-the-art survey on deep learning theory and architectures. electronics, 8(3), 292.

Alzoubi, Y.I., Mishra, A. & Topcu, A.E. (2024). Research trends in deep learning and machine learning for cloud computing security. Artif Intell Rev 57, 132.

Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., ... & Farhan, L. (2021). Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. Journal of big Data, 8, 1-74.

Angulakshmi, M., & Deepa, M. (2021). A review on deep learning architecture and methods for MRI brain tumour segmentation. Current Medical Imaging, 17(6), 695-706.

Avci, O., Abdeljaber, O., Kiranyaz, S., Hussein, M., Gabbouj, M., & Inman, D. J. (2021). A review of vibration-based damage detection in civil structures: From traditional methods to Machine Learning and Deep Learning applications. Mechanical systems and signal processing, 147, 107077.

Aziz, L., Salam, M. S. B. H., Sheikh, U. U., & Ayub, S. (2020). Exploring deep learning-based architecture, strategies, applications and current trends in generic object detection: A comprehensive review. Ieee Access, 8, 170461-170495.

Bachute, M. R., & Subhedar, J. M. (2021). Autonomous driving architectures: insights of machine learning and deep learning algorithms. Machine Learning with Applications, 6, 100164.

Bal, F., & Kayaalp, F. (2021). Review of machine learning and deep learning models in agriculture. International Advanced Researches and Engineering Journal, 5(2), 309-323.

Balaji, K., & Lavanya, K. (2018). Recent Trends in Deep Learning with Applications. In: Sangaiah, A., Thangavelu, A., Meenakshi Sundaram, V. (eds) Cognitive Computing for Big Data Systems Over IoT. Lecture Notes on Data Engineering and Communications Technologies, vol 14 . Springer, Cham. https://doi.org/10.1007/978-3-319-70688-7_9

Bashar, A. (2019). Survey on evolving deep learning neural network architectures. Journal of Artificial Intelligence, 1(02), 73-82.

Chauhan, N. K., & Singh, K. (2018). A review on conventional machine learning vs deep learning. In 2018 International conference on computing, power and communication technologies (GUCON) (pp. 347-352). IEEE.

Dargan, S., Kumar, M., Ayyagari, M. R., & Kumar, G. (2020). A survey of deep learning and its applications: a new paradigm to machine learning. Archives of Computational Methods in Engineering, 27, 1071-1092.

Deng, L. (2014). A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA transactions on Signal and Information Processing, 3, e2.

Deng, Y. (2019). Deep learning on mobile devices: a review. Mobile Multimedia/Image Processing, Security, and Applications 2019, 10993, 52-66.

Deng, Z., Wang, T., Zheng, Y., Zhang, W., & Yun, Y. H. (2024). Deep learning in food authenticity: Recent advances and future trends. Trends in Food Science & Technology, 104344.

Dimiduk, D. M., Holm, E. A., & Niezgoda, S. R. (2018). Perspectives on the impact of machine learning, deep learning, and artificial intelligence on materials, processes, and structures engineering. Integrating Materials and Manufacturing Innovation, 7, 157-172.

Emmert-Streib, F., Yang, Z., Feng, H., Tripathi, S., & Dehmer, M. (2020). An introductory review of deep learning for prediction models with big data. Frontiers in Artificial Intelligence, 3, 4.

Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electronic Markets, 31(3), 685-695.

Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255-260.

Kassem, M. A., Hosny, K. M., Damaševičius, R., & Eltoukhy, M. M. (2021). Machine learning and deep learning methods for skin lesion classification and diagnosis: a systematic review. Diagnostics, 11(8), 1390.

Khan, S., & Yairi, T. (2018). A review on the application of deep learning in system health management. Mechanical Systems and Signal Processing, 107, 241-265.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. nature, 521(7553), 436-444.

Lee, J. G., Jun, S., Cho, Y. W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017). Deep learning in medical imaging: general overview. Korean journal of radiology, 18(4), 570.

Lundberg, L., Boldt, M., Borg, A., & Grahn, H. (2024). Bibliometric Mining of Research Trends in Machine Learning. AI, 5(1), 208-236.

Minar, M. R., & Naher, J. (2018). Recent advances in deep learning: An overview. arXiv preprint arXiv:1807.08169.

Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018). Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6), 1236-1246.

Mishra, R. K., Reddy, G. S., & Pathak, H. (2021). The understanding of deep learning: A comprehensive review. Mathematical Problems in Engineering, 2021(1), 5548884.

Moein, M. M., Saradar, A., Rahmati, K., Mousavinejad, S. H. G., Bristow, J., Aramali, V., & Karakouzian, M. (2023). Predictive models for concrete properties using machine learning and deep learning approaches: A review. Journal of Building Engineering, 63, 105444.

Mu, R., & Zeng, X. (2019). A review of deep learning research. KSII Transactions on Internet and Information Systems (TIIS), 13(4), 1738-1764.

Nguyen, H., Kieu, L. M., Wen, T., & Cai, C. (2018). Deep learning methods in transportation domain: a review. IET Intelligent Transport Systems, 12(9), 998-1004.

Özerol, G., & Arslan Selçuk, S. (2023). Machine learning in the discipline of architecture: A review on the research trends between 2014 and 2020. International Journal of Architectural Computing, 21(1), 23-41.

Patil, T., Pandey, S., & Visrani, K. (2020). A review on basic deep learning technologies and applications. Data Science and Intelligent Applications: Proceedings of ICDSIA 2020, 565-573.

Penney, D. D., & Chen, L. (2019). A survey of machine learning applied to computer architecture design. arXiv preprint arXiv:1909.12373.

Sarker, I. H. (2021). Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions. SN Computer Science, 2(6), 420.

Sengupta, S., Basak, S., Saikia, P., Paul, S., Tsalavoutis, V., Atiah, F., ... & Peters, A. (2020). A review of deep learning with special emphasis on architectures, applications and recent trends. Knowledge-Based Systems, 194, 105596.

Sharma, N., Sharma, R., & Jindal, N. (2021). Machine learning and deep learning applications-a vision. Global Transitions Proceedings, 2(1), 24-28.

Shinde, P. P., & Shah, S. (2018). A review of machine learning and deep learning applications. In 2018 Fourth international conference on computing communication control and automation (ICCUBEA) (pp. 1-6). IEEE.

Shrestha, A., & Mahmood, A. (2019). Review of deep learning algorithms and architectures. IEEE access, 7, 53040-53065.

Thakkar, A., & Lohiya, R. (2021). A review on machine learning and deep learning perspectives of IDS for IoT: recent updates, security issues, and challenges. Archives of Computational Methods in Engineering, 28(4), 3211-3243.

Topuz, B., & Alp, N. Ç. (2023). Machine learning in architecture. Automation in Construction, 154, 105012.

Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018(1), 7068349.

Wang, X., Zhao, Y., & Pourpanah, F. (2020). Recent advances in deep learning. International Journal of Machine Learning and Cybernetics, 11, 747-750.

Wu, N., & Xie, Y. (2022). A survey of machine learning for computer architecture and systems. ACM Computing Surveys (CSUR), 55(3), 1-39.

Yadav, A., & Vishwakarma, D. K. (2020). Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review, 53(6), 4Avci et al., 20215-4385.

Yap M. M., Tekerek, A., & Topaloğlu, N. (2019). Literature review of deep learning research areas. Gazi Mühendislik Bilimleri Dergisi, 5(3), 188-215.