End-to-End Machine Learning

Practical project workflow notes covering data preparation, validation, model training, and evaluation.

Goal

Look at the big picture
Get the data
Explore and visualize the data to gain insights
Prepare the data for machine learning algorithms
Select a model and train it
Fine-tune your model
Present your solution
Launch, monitor, and maintain your system

Repositories

Open data repositories:

Machine Learning Project Checklist

Frame the Problem and Look at the Big Picture

Knowing the objective is important because it will determine how you frame the problem, which algorithms you will select, which performance measure you will use to evaluate your model, and how much effort you will spend tweaking it.
Know how and where your solution will be used.
The current situation will often give you a reference for performance, as well as insights on how to solve the problem.

Pipeline

A data pipeline is a series of asynchronous, self-contained components used to process and transform data, which is especially common in machine learning. In this architecture, each component pulls data from a data store, processes it, and then writes the result to another store for the next component to use. This modular design is simple to grasp and allows different teams to work on separate components in parallel. It's also robust, as a system can often continue running on the last valid output if one component breaks.

Determine what kind of training supervision the model will need: is it a supervised, unsupervised, semi-supervised, self-supervised, or reinforcement learning task? And is it a classification task, a regression task, or something else? Should you use batch learning or online learning techniques? Before you read on, pause and try to answer these questions for yourself.
Select a performance measure. In regression problems, Root-mean-square error (RMSE) and Mean-absolute-error (MAE) are often used. Here, we note that RMSE is more sensitive to outliers than the MAE.
Check the assumptions. List and verify the assumptions that have been made so far (by you or others); this can help you catch serious issues early on.

Get the Data

Explore the Data

Prepare the Data for Machine Learning Algorithms

One should write functions in this part because:

This will allow you to reproduce these transformations easily on any dataset (e.g., the next time you get a fresh dataset).
You will gradually build a library of transformation functions that you can reuse in future projects.
You can use these functions in your live system to transform the new data before feeding it to your algorithms.
This will make it possible for you to easily try various transformations and see which combination of transformations works best.

Note. > ### Scikit-Learn's Design Principles

Consistency & Core Objects

All objects in the library share a consistent, simple interface built around three main types:

Estimators This is the base object for any algorithm that learns from data. Every estimator implements a fit(X, y) method to perform the learning, where X is the dataset and y are the labels (for supervised learning). Hyperparameters are passed to the estimator's constructor when it is created.

Transformers These are estimators that can also modify or transform a dataset. In addition to fit(), they have a transform() method to apply the transformation. A convenient fit_transform() method performs both steps at once and is often more optimized.

Predictors These are estimators capable of making predictions on new data. They provide a predict(X) method that returns predictions for a new dataset X and often include a score(X, y) method to evaluate their performance.

Inspection

The library is designed to be transparent. All hyperparameters are accessible as public attributes (e.g., model.strategy). All parameters learned during fit() are also stored as public attributes, distinguished by a trailing underscore (e.g., model.statistics_).

Composition

Scikit-Learn encourages reusing existing components. It is easy to chain multiple steps into a single workflow using the Pipeline object, which can consist of a sequence of transformers followed by a final estimator.

Nonproliferation of Classes

To maintain simplicity, datasets are represented using standard data types like NumPy arrays or SciPy sparse matrices, not custom classes. Hyperparameters are also simple Python types (strings, numbers).

Sensible Defaults

Most model parameters have reasonable default values. This makes it easy to quickly create and test a baseline working model without having to manually specify every single parameter.

If a categorical attribute has a large number of possible categories (e.g., country code, profession, species), then one-hot encoding will result in a large number of input features. This may slow down training and degrade performance. If this happens, you may want to replace the categorical input with useful numerical features related to the categories.

Feature Scaling and Transformation

There are two common ways to get all attributes to have the same scale: min-max scaling and standardization.
Before scaling, we should first transform it to shrink the heavy tail, and if possible, to make the distribution roughly symmetrical.
- Replace the feature with its square root.
- Replace the feature by raising it to a power between 0 and 1.
- Replace the feature with its logarithm.
Another approach to handle heavy-tailed features is bucketizing the feature. This chops the distribution into roughly equal-sized buckets, and replacing each feature value with the index of the bucket it belongs to. This is simalar to pd.cut which labels numerical values.
- Bucketizing with equal-sized buckets results in a feature with an almost uniform distribution.
When a feature has multimodal distribution, we can also bucketize it, but treat the bucket IDs as categories, rather than as numerical values. And after use encoding.
- Another approach is to add a feature for each of the modes, representing the similarity between the housing median age and that particular mode. This similarity measure is typically computed using a radial basis function (RBF). The most common used is the Gaussian RBF.
Most of Scikit-Learn's transformers have an inverse_transform() method, making it easy to compute the inverse of their transformations. For example,

from sklearn.linear_model import LinearRegression
target_scaler = StandardScaler()
scaled_labels = target_scaler.fit_transform(housing_labels.to_frame())
model = LinearRegression()
model.fit(housing"median_income", scaled_labels)
some_new_data = housing"median_income".iloc[:5] # pretend this is new data
scaled_predictions = model.predict(some_new_data)
predictions = target_scaler.inverse_transform(scaled_predictions)

A simpler option is to use TransformedTargetRegressor.

from sklearn.compose import TransformedTargetRegressor
model = TransformedTargetRegressor(LinearRegression(), transformer=StandardScaler())

model.fit(housing"median_income", housing_labels)
predictions = model.predict(some_new_data)

Custom Transformers

One often needs to write their own transformers for tasks such as custom transformations, cleanup operations, or comibining specific features.

A custom transformer can (and often does) use other estimators in its implementation. For example,

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils.validation import check_array, check_is_fitted

class StandardScalerClone(BaseEstimator, TransformerMixin):
    def __init__(self, with_mean=True): # no *args or **kwargs!
        self.with_mean = with_mean
    def fit(self, X, y=None): # y is required even though we don't use it
        X = check_array(X) # checks that X is an array with finite float values
        self.mean_ = X.mean(axis=0)
        self.scale_ = X.std(axis=0)
        self.n_features_in_ = X.shape[1] # every estimator stores this in fit()
        return self # always return self!
    def transform(self, X):
        check_is_fitted(self) # looks for learned attributes (with trailing _)
        X = check_array(X)
        assert self.n_features_in_ == X.shape[1]
        if self.with_mean:
            X = X - self.mean_
        return X / self.scale_

from sklearn.cluster import KMeans

class ClusterSimilarity(BaseEstimator, TransformerMixin):
	def __init__(self, n_clusters=10, gamma=1.0, random_state=None):
		self.n_clusters = n_clusters
		self.gamma = gamma
		self.random_state = random_state
		
	def fit(self, X, y=None, sample_weight=None):
		self.kmeans_ = KMeans(self.n_clusters, random_state=self.random_state)
		self.kmeans_.fit(X, sample_weight=sample_weight)
		return self # always return self!
		
	def transform(self, X):
		return rbf_kernel(X, self.kmeans_.cluster_centers_, gamma=self.gamma)
		
	def get_feature_names_out(self, names=None):
		return [f"Cluster {i} similarity" for i in range(self.n_clusters)]

Transformation Pipelines

The Pipeline constructor takes a list of name/estimator pairs (2-tuples) defining a sequence of steps. The names can be anything you like, as long as they are unique and don’t contain double underscores (__).

When you call a pipeline’s fit() method, it applies fit_transform() sequentially to each transformer, passing the output to the next step. For the final estimator, only fit() is called. The pipeline exposes the same methods as its final step—if it's a transformer, the pipeline will have a transform() method; if it’s a predictor, it will have a predict() method. These methods apply all transformations before the final action.

Pipelines in Scikit-Learn support indexing: pipeline[1] accesses the second estimator, and slicing like pipeline[:-1] returns a new Pipeline with all but the last step. We can also access estimators via the steps attribute (a list of name/estimator pairs) or via the named_steps dictionary for name-based access—for example, num_pipeline["simpleimputer"] returns the step named "simpleimputer".

To handle both numerical and categorical columns together, it is better to use a ColumnTransformer. This allows applying different preprocessing pipelines to different column subsets. In the example below, num_pipeline is applied to numerical attributes and cat_pipeline to categorical ones.

ColumnTransformer requires a list of triples (3-tuples), each containing a name, a transformer, and a list of names (or indices) of columns that the transformer should be applied to.

In summary,

Numerical missing values → imputed with median
Categorical missing values → filled with most frequent category
Categorical features → one-hot encoded
New ratio features added: bedrooms_ratio, rooms_per_house, people_per_house
Cluster similarity features added (more useful than raw latitude/longitude)
Skewed features → log-transformed
All numerical features → standardized to same scale

Select and Train a Model

When RMSE is very high, the model might underfit the training data. When this happens it can mean that the features do not provide enough information to make good predictions, or that the model is not powerful enough. The main ways to fix underfitting are to select a more powerful model, to feed the training algorithm with better features, or to reduce the constraints on the model.

Evaluation

A basic way to evaluate a decision tree model is to use the train_test_split() function to manually split the training data into a smaller training set and a validation set.
You then train the model on the smaller training set and evaluate it on the validation set.
While this method works reasonably well, a better approach is to use Scikit-Learn’s k-fold cross-validation.
- This technique splits the training data into k non-overlapping subsets (folds), then trains and evaluates the model k times—each time using a different fold for validation and the remaining folds for training. The result is an array of k evaluation scores.

What cross_val_score from scikit-learn expects is that it is maximizing a utility (scoring) function. But since RMSE is a loss function, we want to minimize the scores. Hence, we apply the negative multiplication.

We can try different models from various categories of machine learning algorithms without spending too much time tweaking the hyperparameters. The goal, really, is to shortlist a few (two to fie) promising models.

Fine-Tune Your Model

Grid Search

Instead of manually tuning hyperparameters, we can use Scikit-Learn’s GridSearchCV to automatically search over a grid of hyperparameter combinations using cross-validation. We just define which parameters to search and what values to try, and GridSearchCV evaluates all combinations using CV. It also supports nested parameters inside pipelines by using double underscores (__) to traverse estimators and transformers.

In this example, GridSearchCV searches over two grids of n_clusters and max_features for a RandomForestRegressor wrapped inside a pipeline. The total combinations are 15, and with 3-fold CV, this results in 45 training runs. Scikit-Learn can also cache fitted transformers during the process by setting the memory argument in the pipeline.

from sklearn.model_selection import GridSearchCV

full_pipeline = Pipeline([
    ("preprocessing", preprocessing),
    ("random_forest", RandomForestRegressor(random_state=42)),
])

param_grid = [
    {'preprocessing__geo__n_clusters': [5, 8, 10],
     'random_forest__max_features': [4, 6, 8]},
    {'preprocessing__geo__n_clusters': [10, 15],
     'random_forest__max_features': [6, 8, 10]},
]

grid_search = GridSearchCV(full_pipeline, param_grid, cv=3, scoring='neg_root_mean_squared_error')

grid_search.fit(housing, housing_labels)

Once GridSearchCV finishes, you can:

Access the best model using grid_search.best_estimator_.
Access full evaluation results with grid_search.cv_results_, a dictionary containing metrics for each parameter combination and fold.
Wrap the results in a pandas.DataFrame for easier sorting and inspection.
Sort by "mean_test_score" (which is negative RMSE in this case).
The best mean test RMSE achieved was 44,042, which is better than the earlier baseline of 47,019.

Since n_clusters=15 is the upper boundary of the search, you might try a larger value to improve results further.

Randomized Search

Unlike GridSearchCV, which tries all combinations, RandomizedSearchCV randomly samples a fixed number of combinations, making it much more efficient when the hyperparameter space is large. This is especially useful for:

Continuous/discrete hyperparameters with many possible values.
Ignoring unimportant hyperparameters without wasting compute.
Exploring high-dimensional search spaces efficiently. You define hyperparameter distributions using scipy.stats (e.g., randint) and set how many iterations you want (n_iter).

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_distribs = {
    'preprocessing__geo__n_clusters': randint(low=3, high=50),
    'random_forest__max_features': randint(low=2, high=20)
}

rnd_search = RandomizedSearchCV(
    full_pipeline, param_distributions=param_distribs, n_iter=10, cv=3,
    scoring='neg_root_mean_squared_error', random_state=42
)

rnd_search.fit(housing, housing_labels)

Halving Grid Search

Scikit-Learn also provides HalvingGridSearchCV and HalvingRandomSearchCV to further optimize computation. These methods:

Start by evaluating many candidates with fewer resources (e.g., on a small data subset).
Iteratively keep the best candidates and allocate more resources.
End with top candidates evaluated fully.
Great for reducing time in large search spaces.

Ensemble Methods

You can combine several models (e.g., via ensemble techniques like random forests or voting classifiers) to boost performance, especially when individual models make diverse errors.

Bonus section: how to choose the sampling distribution for a hyperparameter

scipy.stats.randint(a, b+1): for hyperparameters with discrete values that range from a to b, and all values in that range seem equally likely.
scipy.stats.uniform(a, b): this is very similar, but for continuous hyperparameters.
scipy.stats.geom(1 / scale): for discrete values, when you want to sample roughly in a given scale. E.g., with scale=1000 most samples will be in this ballpark, but ~10% of all samples will be <100 and ~10% will be >2300.
scipy.stats.expon(scale): this is the continuous equivalent of geom. Just set scale to the most likely value.
scipy.stats.loguniform(a, b): when you have almost no idea what the optimal hyperparameter value's scale is. If you set a=0.01 and b=100, then you're just as likely to sample a value between 0.01 and 0.1 as a value between 10 and 100.

Evaluate Your System on the Test Set

After finalizing your model, you evaluate it on the held-out test set to estimate its real-world performance. This involves generating predictions and computing a final RMSE. For more reliability, you can also compute a 95% confidence interval of the generalization error using scipy.stats.t.interval(). This interval gives an estimate of the range in which the true RMSE likely falls.

Hyperparameter tuning may cause the test set performance to be slightly worse than validation performance due to overfitting. Therefore, avoid tweaking hyperparameters just to boost test results. Finally, present your solution clearly, documenting your process, assumptions, and findings. Use clear visualizations and memorable takeaways (e.g., "median income is the #1 predictor of housing prices").

Launch, Monitor, and Maintain Your System

Once your model is ready, you can launch it by saving it with joblib, deploying it to production (locally or via the cloud), and monitoring it regularly. Saving your model allows easy reuse, deployment, and reproducibility.

To save a model locally:

import joblib

joblib.dump(final_model, "my_california_housing_model.pkl")

Models can be deployed as web services (e.g., via REST APIs) or in the cloud using tools like Google’s Vertex AI. Just upload the .pkl file to Google Cloud Storage, then create a model version in Vertex AI that points to the file.

However, deployment is just the beginning. You must monitor your model's live performance to detect degradation over time (data drift, broken pipelines, etc.). You can:

Use downstream metrics to infer model quality.
Trigger alerts if metrics drop.
Evaluate performance with human-labeled data.
Set up automated retraining pipelines.

You should also:

Automate retraining and hyperparameter tuning.
Periodically evaluate both new and old models on updated test sets.
Maintain model and dataset backups for rollback and reproducibility.

These practices help ensure your deployed model remains accurate, reliable, and maintainable over time.