Forecasting Commodity Prices and Arrivals in Telangana: A Machine Learning and Time Series Analysis Approach

I'm thrilled to share a project that has been both challenging and enlightening! Although the journey had its obstacles, it was a rewarding path of personal and professional growth. Here are some key highlights of the process, insights gained, and lessons learned:

✨ Project Selection:
I chose a project that aligned with my passion, making the journey more engaging—even through the toughest challenges.

📊 Data Collection:
By sourcing real-time data directly from Telangana’s official data portal (data.telangana.gov.in), I could streamline the collection process, building a robust dataset.

🗂️ Data Preparation:
The dataset covered 2016 to the present, so I combined everything in a single set with Power Query in Excel, tackling tasks like:
📅 Standardizing timestamps,
🏙️ Adding district insights, and
🏷️ Integrating market type information.

📈 Initial Power BI Analysis:
Although new to Power BI, I dove in with the help of YouTube tutorials, gaining basic analytical insights. While I could create initial graphs, I quickly realized the full picture needed deeper exploration.

🧹 Managing Outliers:
Identifying and handling outliers was key. Familiar with Python’s Z-Score, I learned to apply DAX functions and the IQR method in Power BI for outlier detection—enhancing my analytical toolkit!

🖥️ Moving to Kaggle for Advanced Analysis:
Switching to Kaggle opened the door to time series forecasting, where I experimented with models like:
📉 Linear Regression,
🌲 Random Forest,
⚡ XGBoost,
🔄 ARIMA,
🧠 RNN, and
💾 LSTM.

📊 Model Selection and Forecasting:
By splitting the data (80% training, 20% testing), I compared models based on RMSE values, identifying the best-performing one. This model then forecasted arrivals or prices over a custom timeframe, helping users anticipate peak trends.

This project wasn’t just about coding—it was about adapting, learning, and finding solutions to challenges along the way. Like Diwali, it’s a reminder that perseverance and passion can turn every challenge into a new skill!

Links to Explore:

Kaggle: https://www.kaggle.com/code/savullavaishnavi/final-crop-predicition-model

Power Bi:

https://app.powerbi.com/view?r=eyJrIjoiZTUxZjRlYTEtNjg1ZS00OGY0LTgxNzktYTM2ZjFjOWFlNTdhIiwidCI6IjhjYWI5ZTZmLTRlMTQtNDcyYS1hNmVjLTFiZDYyOGE0YjZhNCJ9

In conclusion, this project on forecasting commodity prices and arrivals in Telangana using machine learning and time series analysis was a journey of growth and learning. By selecting a project aligned with personal passion, the process remained engaging despite challenges. The project involved meticulous data collection and preparation, leveraging tools like Power Query in Excel and Power BI for initial analysis. Transitioning to Kaggle allowed for advanced time series forecasting with various models, ultimately identifying the best-performing model through RMSE comparison. This experience underscored the importance of perseverance, adaptability, and continuous learning, transforming challenges into valuable skills.