Skip to main content

Model Maintainance

Updated May 14, 2023 ·

Overview​

When a model's performance declines, it's essential to take action to bring it back to acceptable levels. There are two main strategies: improving the model itself or improving the data used to train it.

Improving the Model​

Sometimes, the issue lies within the model itself, and adjustments are needed to improve its performance.

  • Also called the Model-centric Approach
  • Experiment with different models or combinations of models.
  • Refine the features used by the model.
  • Tune model parameters for better accuracy.

In competitions, improving the model is often the primary solution since datasets are fixed. However, in real-life scenarios, this might not be enough.

Improving the Training Data​

In many cases, enhancing the training data can have a greater impact than changing the model.

  • Also called the Data-centric Approach
  • Use tools to label data more accurately and efficiently.
  • Consider human-in-the-loop systems where experts help label data.

Improving data quality can lead to better model performance, as the model learns from better examples. This approach has been shown to outperform model-centric changes in many real-world use cases.

Human-in-the-Loop​

This is a new ML design where a human expert steps in when the model is unsure, making more accurate decisions. This allows for continuous labeling of new data, supporting ongoing model improvement.

Tracking and Experimentation​

Keeping track of experiments and model performance helps avoid wasting resources on repetitive tasks.

  • Use tools like MLFlow to track experiments and models.
  • Document model changes and their outcomes.
  • Compare new models with previous versions to ensure improvements.

Experiment tracking ensures we don't repeat unsuccessful attempts. It's vital for maintaining a clear history of what works and what doesn’t.