Automatic Model Tuning with Amazon SageMaker

In this article, we will explore how to perform automatic model tuning using Amazon SageMaker. This process helps optimize the performance of machine learning models by adjusting their hyperparameters. If you haven't already, please check out my previous articles on Building a Machine Learning Model with AWS SageMaker and Deploying the Trained XGBoost Model as a Real-Time Endpoint, as they provide essential background information.

Setting Up Hyperparameter Ranges for SageMaker Model Tuning

To begin with, we need to define the hyperparameter ranges that SageMaker will explore during the tuning process. Hyperparameters are crucial as they control the learning process of our model. In this example, we will be tuning an XGBoost model.Here’s how we can set up our hyperparameter ranges:

from sagemaker.tuner import IntegerParameter, ContinuousParameter, HyperparameterTuner

# Define the range of hyperparameters to tune
hyperparameter_ranges = {
    'eta': ContinuousParameter(0, 1),               # Learning rate, from 0 to 1
    'min_child_weight': ContinuousParameter(1, 10), # Minimum sum of weights for child nodes, from 1 to 10
    'alpha': ContinuousParameter(0, 2),             # L1 regularization term, from 0 to 2
    'max_depth': IntegerParameter(1, 10)            # Maximum depth of trees, from 1 to 10
}

# Set the objective metric for tuning
objective_metric_name = 'validation:auc' # The metric to optimize during tuning

Initializing Hyperparameter Tuner for XGBoost

Next, we’ll create a HyperparameterTuner object that will manage the tuning process. This includes specifying the model we want to tune and the metrics we want to optimize.

# Initialize the Hyperparameter Tuner
tuner = HyperparameterTuner(
    xgb,                         # XGBoost estimator
    objective_metric_name,       # Metric to optimize (validation:auc)
    hyperparameter_ranges,       # Hyperparameter ranges to explore
    max_jobs=20,                 # Total tuning jobs
    max_parallel_jobs=3          # Concurrent jobs allowed
)

Launching Hyperparameter Tuning Job

Once our tuner is set up, we can launch the tuning job. This step requires specifying where our training and validation datasets are located in S3.

# Launch the hyperparameter tuning job
tuner.fit({'train': s3_input_train, 'validation': s3_input_validation})

Checking Hyperparameter Tuning Job Status

After launching the job, it’s important to monitor its status. We can retrieve the current state of our tuning job using the following code:

import boto3

# Check the status of the latest hyperparameter tuning job
status = boto3.client('sagemaker').describe_hyper_parameter_tuning_job(
    HyperParameterTuningJobName=tuner.latest_tuning_job.job_name
)['HyperParameterTuningJobStatus']

print(status)  # Outputs: 'Completed'

Retrieving the Best Training Job from Hyperparameter Tuning

Once the tuning job is completed, we can identify which training job performed best based on our specified metric.

# Get the name of the best-performing training job
best_job_name = tuner.best_training_job()
print(best_job_name)

Deploying the Best Model from Hyperparameter Tuning

Now that we have identified our best model, it's time to deploy it. We will create an endpoint on Amazon SageMaker that allows us to make predictions in real-time.

# Deploy the best model from the tuning job to a SageMaker endpoint
tuner_predictor = tuner.deploy(
    initial_instance_count=1,          # Number of instances to deploy
    instance_type='ml.m4.xlarge'      # Type of instance for hosting
)

Setting the Serializer for Model Inference

To ensure that our input data is correctly formatted when sending requests to our endpoint, we need to set up a serializer.

# Set the serializer to handle CSV input format for inference
tuner_predictor.serializer = sagemaker.serializers.CSVSerializer()

Load Test Data for Inference: Features and Labels

Next, we will load our test data from S3. This includes both feature data and actual labels.

test_data_x = pd.read_csv(os.path.join(test_path, 'test_script_x.csv'), header=None)
test_data_y = pd.read_csv(os.path.join(test_path, 'test_script_y.csv'), header=None)

Making Predictions with the Deployed Model

With our model deployed and test data loaded, we can now make predictions.

# Make predictions using the deployed model and convert the result to a NumPy array
predictions = predict(test_data_x.to_numpy(), tuner_predictor)

Generate Confusion Matrix for Model Predictions

To evaluate how well our model is performing, we can generate a confusion matrix that compares predicted values with actual labels.

# Generate a confusion matrix to evaluate the model's performance by comparing actual vs predicted values
pd.crosstab(index=test_data_y[0], columns=np.round(predictions), 
            rownames=['actuals'], colnames=['predictions'])

Delete Endpoint

Finally, once we're done with predictions and evaluations, it's good practice to clean up by deleting our endpoint.

tuner_predictor.delete_endpoint(delete_endpoint_config=True)

Conclusion

In this article, we've walked through setting up automatic model tuning using Amazon SageMaker. By defining hyperparameter ranges and utilizing SageMaker's powerful tuning capabilities, you can significantly improve your model's performance. For further details on deploying your trained models or exploring more advanced topics in machine learning with AWS SageMaker, feel free to check out my previous articles! Additionally, you can find the complete code and resources in my GitHub repository.