How to make an AWS Resource Inventory?

As working professionals, we usually handle a variety, or I must say, tons of different workloads, each sitting in separate accounts, regions, or with different cloud/on-prem providers.

Navigating among these or having a sense of which thing is where, running on what type of machine, or which version is often a problem we DevOps guys face a lot. These workloads/resources serve different use cases and are often made by separate individuals, and it's easy to lose track of them soon. These also run on different versions, and these versions have different support cycles. If you don’t track these support cycles, it costs you a ton more than usual to use extended support services for stale versions.

I was also facing this problem and came upon a challenge to inventorize our whole workload. So what I needed is a method that.

1. Isn’t manual.

2. Updates automatically.

3. Only needs a one-time setup.

4. Is secure.

5. It completely wraps around all the different accounts/providers I have.

6. Is free.

7. Doesn’t require many resources.

8. Cheap fixed monthly cost.

Pilot.

So, when I searched for a tool or a service for it, it came as a shocker that AWS doesn’t provide a service that can be used to inventory my resources org-wide. I searched for a day or two and couldn’t find anything on it. I left the idea there and thought of maintaining an Excel document to store which of my EKS clusters or RDS clusters are running on which version, where, and when their expiration date is reaching for that version.

But one fine day…

While searching for a security and compliance tracking tool, I came across Steampipe, an open-source tool.

So what is Steampipe

The Steampipe exposes APIs and services as a high-performance relational database, enabling you to write SQL-based queries to explore dynamic data. The Steampipe CLI is a turnkey solution that includes its own PostgreSQL database and plugin management.

In simple terms, it's a way to query your different API’s in an SQL-like manner.

I looked for the AWS and other cloud provider integrations they have. After reading their docs, I thought this was the best choice for my inventory gig.

It was the most formidable choice to go with because it allows you to query most of the cloud agnostic APIs' including the listing and description functionalities.

How it looks,

So, architecturally speaking, having just a query engine wasn’t enough. I needed a good cloud BI tool, for which I used QuickSight.

I made the steampipe run on a simple EC2, which has an IAM Profile that gives me access to all my resources in different accounts through IAM roles. I made it run as a runtime Postgres server, connected it as a data source to my QuickSight account, and made dashboards out of it, explaining the What, When, Where, and How Many?

Voila! problem solved.

More to this

So I have gone with using Steampipe, but you can also go with using Cloudquery engine. It also provides the same thing, but it's just a little expensive, but yes, it's better, too.

For the BI tool, you can also choose Powerpipe, which is open-source/free and made by the same people who made Steampipe. By this, you can even run this locally, effectively for ZERO $