• Home
  • About
  • Contact
  • Privacy
  • Terms
  • DCMA
  • Write For Us / Submit
Tech News, Magazine & Review WordPress Theme 2017
  • Tech
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

  • Gear
    • All
    • Camera
    • Laptop
    • Smartphone
    The New Galaxy Watch Ultra and Galaxy Ring Are Announced by Samsung

    The New Galaxy Watch Ultra and Galaxy Ring Are Announced by Samsung

    Exploring the Innovative Features of Pear Phones

    Exploring the Innovative Features of Pear Phones

    Latest Smart Home Gadgets for a Connected Life

    Latest Smart Home Gadgets for a Connected Life

    Eco-Friendly Products for Students

    Eco-Friendly Products for Students

    Essential Photography Equipment and Gadgets

    Mastering Your Shots: Essential Photography Equipment and Gadgets

    Level Up Your Game: Must-Have Gaming Gear!

    Level Up Your Game: Must-Have Gaming Gear!

    Trending Tags

    • Best iPhone 7 deals
    • Apple Watch 2
    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • iOS 10
    • iPhone 7
    • Sillicon Valley
  • Gaming
    Master the World of Online Gaming with the Best Gaming VPN

    Master the World of Online Gaming with 3 Best Gaming VPN

    7 Powerful Strategies to Overcome Video Game Addiction and Reclaim Your Life

    7 Powerful Strategies to Overcome Video Game Addiction and Reclaim Your Life

    Powerful Ways Xbox Cloud Gaming is Revolutionizing the Gaming World

    5 Powerful Ways Xbox Cloud Gaming is Revolutionizing the Gaming World

    9 Best Alternative Games Like Kahoot

    9 Best Alternative Games Like Kahoot

    The Top 8 Free Bubble Shooter Games for Endless Entertainment

    The Top 8 Free Bubble Shooter Games for Endless Entertainment

    Cloud Gaming Revolution: How Streaming is Changing the Future of Gaming

    Cloud Gaming Revolution: How Streaming is Changing the Future of Gaming

  • Crypto
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

  • Business
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

No Result
View All Result
Geeky Insider
  • Tech
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

  • Gear
    • All
    • Camera
    • Laptop
    • Smartphone
    The New Galaxy Watch Ultra and Galaxy Ring Are Announced by Samsung

    The New Galaxy Watch Ultra and Galaxy Ring Are Announced by Samsung

    Exploring the Innovative Features of Pear Phones

    Exploring the Innovative Features of Pear Phones

    Latest Smart Home Gadgets for a Connected Life

    Latest Smart Home Gadgets for a Connected Life

    Eco-Friendly Products for Students

    Eco-Friendly Products for Students

    Essential Photography Equipment and Gadgets

    Mastering Your Shots: Essential Photography Equipment and Gadgets

    Level Up Your Game: Must-Have Gaming Gear!

    Level Up Your Game: Must-Have Gaming Gear!

    Trending Tags

    • Best iPhone 7 deals
    • Apple Watch 2
    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • iOS 10
    • iPhone 7
    • Sillicon Valley
  • Gaming
    Master the World of Online Gaming with the Best Gaming VPN

    Master the World of Online Gaming with 3 Best Gaming VPN

    7 Powerful Strategies to Overcome Video Game Addiction and Reclaim Your Life

    7 Powerful Strategies to Overcome Video Game Addiction and Reclaim Your Life

    Powerful Ways Xbox Cloud Gaming is Revolutionizing the Gaming World

    5 Powerful Ways Xbox Cloud Gaming is Revolutionizing the Gaming World

    9 Best Alternative Games Like Kahoot

    9 Best Alternative Games Like Kahoot

    The Top 8 Free Bubble Shooter Games for Endless Entertainment

    The Top 8 Free Bubble Shooter Games for Endless Entertainment

    Cloud Gaming Revolution: How Streaming is Changing the Future of Gaming

    Cloud Gaming Revolution: How Streaming is Changing the Future of Gaming

  • Crypto
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

  • Business
    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    How Drones and 360-Degree Cameras Are Changing Bridal Photoshoots

    120+ Best Creative Names for Team at Work

    120+ Best Creative Names for Team at Work

    Can You Repair Printed Circuit Board ?

    Mastering PCB Board Repair Comprehensive Guide

    Breaking Barriers Federal Inmate Texting Service

    Breaking Barriers Federal Inmate Texting Service

    transfer whatsapp from android to iphone

    How to Transfer WhatsApp from Android to iPhone!

    Understanding the Role of Humidity Chambers in Climate Testing

    Understanding the Role of Humidity Chambers in Climate Testing

    Everything You Get to Know About Movember Beard Memes

    Everything You Get to Know About Movember Beard Memes

    Best Book Recommendation Apps

    11 Best Book Recommendation Apps

    How to quickly divide or Split PDF files

    How to Quickly Divide or Split PDF Files

Submit
Geeky Insider
No Result
View All Result

The importance of monitoring machine learning models

by Staff Writer
July 21, 2023
in Education
Reading Time: 5 mins read
security monitoring
Share on FacebookShare on Twitter

 

Agile development teams must ensure that microservices, applications, and databases are observable, have monitoring in place to identify operational issues, and use AIops to correlate alerts into manageable incidents. When users and business stakeholders want enhancements, many DevOps teams follow agile methodologies to process feedback and deploy new versions.

Even if there are few requests, DevOps teams know they must upgrade apps and patch underlying components; otherwise, the software developed today will become tomorrow’s technical debt.

Table of Contents

Toggle
    • READ ALSO
    • Study: Autistic People Recognize Emotions Better Than Neurotypical in Cartoons
    • Feeding the world by AI, machine learning and the cloud
  • Why monitor machine learning models?
  • Best practices in ML monitoring

READ ALSO

How to protect yourself from phishing

Phishing: The nasty tricks of the cybercriminals – this is how they protect themselves

July 28, 2023
Idaho university murders - latest: Police admit errors and backtrack as they reveal other roommates were in home

Idaho university murders – latest: Police admit errors and backtrack as they reveal other roommates were in home

November 17, 2022

The agile transformation in machine learning models’ life-cycle management presents unique challenges. Andy Dang, co-founder and head of engineering at WhyLabs, highlights the complexity of the process. While similar to software development, ML models rely on complex and unpredictable data, making the transformation even more intricate.

In addition to code, components, and infrastructure, models are built using algorithms, configuration, and training data sets. These are selected and optimized at design time but need updating as assumptions and the data change over time.

Why monitor machine learning models?

Like monitoring applications for performance, reliability, and error conditions, machine learning model monitoring provides data scientists visibility on model performance. ML monitoring is especially important when models are used for predictions or when the ML runs on datasets with high volatility.

Dmitry Petrov, cofounder and CEO of Iterative, says, “The main goals around model monitoring focus on performance and troubleshooting as ML teams want to be able to improve on their models and ensure everything is running as intended.”

Rahul Kayala, principal product manager at Moveworks, shares this explanation on ML model monitoring. “Monitoring can help businesses balance the benefits of AI predictions with their need for predictable outcomes,” he says. “Automated alerts can help ML operations teams detect outliers in real time, giving them time to respond before any harm occurs.”

Stu Bailey, cofounder of ModelOp, adds, “Coupling robust monitoring with automated remediation accelerates time to resolution, which is key for maximizing business value and reducing risk.”

In particular, data scientists need to be notified of unexpected outliers. “AI models are often probabilistic, meaning they can generate a range of results,” says Kayala. “Sometimes, models can produce an outlier, an outcome significantly outside the normal range. Outliers can be disruptive to business outcomes and often have major negative consequences if they go unnoticed. To ensure AI models are impactful in the real world, ML teams should also monitor trends and fluctuations in product and business metrics that AI impacts directly.”

For example, let’s consider predicting a stock’s daily price. When there’s low market volatility, algorithms such as the long short-term memory (LSTM) can provide rudimentary predictions, and more comprehensive deep learning algorithms can improve accuracy. But most models will struggle to make accurate predictions when markets are highly volatile, and model monitoring can alert for these conditions.

Another type of ML model performs classifications, and precision and recall metrics can help track accuracy. Precision measures the true positives against the ones the model selected, while recall tracks a model’s sensitivity. ML monitoring can also alert on ML model drift, such as concept drift when the underlying statistics of what’s being predicted change, or data drift when the input data changes.

A third concern is explainable ML, where models are stressed to determine which input features contribute most significantly to the results. This issue relates to model bias, where the training data has statistical flaws that skew the model to make erroneous predictions.

These issues can erode trust and create significant business issues. Model performance management aims to address them across the development, training, deployment, and monitoring phases.

Krishnaram Kenthapadi, chief scientist at Fiddler, believes that explainable ML with reduced risk of biases requires model performance management. “To ensure ML models are not unduly discriminating, enterprises need solutions that deliver context and visibility into model behaviors throughout the entire life cycle—from model training and validation to analysis and improvement,” says Kenthapadi. “Model performance management ensures models are trustworthy and helps engineers and data scientists identify bias, monitor the root cause, and provide explanations for why those instances occurred in a timely manner.”

Best practices in ML monitoring

Modelops, ML monitoring, and model performance management are terms for practices and tools to ensure machine learning models operate as expected and provide trustworthy predictions. What underlying practices should data science and devops teams consider in their implementations?

Josh Poduska, chief field data scientist at Domino Data Lab, says, “Model monitoring is a critical, ongoing process. To improve future accuracy for a model that has drifted, retrain it with fresher data, along with its associated ground truth labels that are more representative of the current reality.“

Ira Cohen, chief data scientist and cofounder at Anodot, shares important factors in ML model monitoring. “First, models should monitor output and input features’ behavior, as shifts in the input features can cause issues,” he says. He suggests using proxy measures when model performance cannot be measured directly or quickly enough.

Cohen says data scientists need tools for model monitoring. He says, “Monitoring models manually is not scalable, and dashboards and reports are not equipped to handle the complexity and volume of the monitoring data generated when many AI models are deployed.”

Here are some recommended practices for ML model monitoring and performance management:

  • Petrov recommends, “Ensure you have the tools and automation in place upstream at the beginning of the model development life cycle to support your monitoring needs.”
  • Dang says, “Data engineers and scientists should run preliminary validations to ensure their data is in the expected format. As the data and the code move through a CI/CD pipeline, they should enable data unit testing through validations and constraint checks.”
  • Cohen suggests, “Use scalable anomaly detection algorithms that learn the behavior of each model’s inputs and outputs to alert when they deviate from the norm, effectively using AI to monitor AI.”
  • Kayala says, “Track the drift in the distribution of features. A large change in distribution indicates the need to retrain our models to achieve optimal performance.”
  • Bailey adds, “Increasingly, organizations are looking to monitor model risk and ROI as part of more comprehensive model governance programs, ensuring that models meet business and technical KPIs.”

Software development largely focuses on maintaining the code, monitoring application performance, improving reliability, and responding to operational and security incidents. In machine learning, ever-changing data, volatility, bias, and other factors require data science teams to manage models across their life cycle and monitor them in production.

Copyright © 2022 IDG Communications, Inc.

Source by www.infoworld.com

Related Posts

Business Lessons You Should Learn
Business

The 8 Business Lessons You Should Learn

February 28, 2023
How to quickly divide or Split PDF files
Apps And Softwares

How to Quickly Divide or Split PDF Files

February 28, 2023
How to protect yourself from phishing
How To

Phishing: The nasty tricks of the cybercriminals – this is how they protect themselves

July 28, 2023
Idaho university murders - latest: Police admit errors and backtrack as they reveal other roommates were in home
Education

Idaho university murders – latest: Police admit errors and backtrack as they reveal other roommates were in home

November 17, 2022

Recommended.

QAnon Candidates Aren’t Thriving, but Some of Their Ideas Are

QAnon Candidates Aren’t Thriving, but Some of Their Ideas Are

July 26, 2022
QueerTech Qonference 2022: 17-year-old Harsehaj Dhami is addressing education gaps in the tech sector

QueerTech Qonference 2022: 17-year-old Harsehaj Dhami is addressing education gaps in the tech sector

November 14, 2022

Trending.

No Content Available
  • Home
  • About
  • Contact
  • Privacy
  • Terms
  • DCMA
  • Write For Us / Submit
Contact us for submission queries. editor[at]geekyinsider.com.
No Result
View All Result
  • Home
  • Review
  • Apple
  • Gaming
  • Gadget and Gear
    • Camera
    • Smartphone
  • Microsoft
  • Security