How I Saved ~₹24 Lakhs a Year on Azure with a Custom Cost-Optimization Pipeline (Built with AI as a Co-Pilot)

The Problem

Working as a DevOps/Cloud Infrastructure engineer supporting production Kubernetes workloads on Azure, I ran into a recurring cost problem: Azure Application Gateway diagnostic logging into Log Analytics was quietly becoming one of the largest line items on the monthly Azure bill.

A KQL-based cost analysis of the Log Analytics workspace confirmed it: AzureDiagnostics (specifically, Application Gateway Access Logs) was the single largest ingestion source, at roughly 28.5 GB/day. At Log Analytics' per-GB ingestion pricing, this alone was costing over ₹2 lakh (~$2,500 USD) per month — just to store and query gateway access logs that, in practice, only a handful of engineers needed to look at for response-time monitoring.

The catch: Log Analytics wasn't providing much value beyond basic aggregate metrics (avg/min/max/P95 response time, request volume by hour). We didn't need its full-text search, alerting integrations, or long-term retention for this specific use case — we just needed fast, reliable access to response-time trends.

The Options I Evaluated

Before building anything custom, I looked at a few alternatives:

• Data Collection Rule (DCR) transformations — could filter/reduce ingested volume, but added complexity and didn't eliminate the core cost driver
• Azure Functions-based log processing — flexible, but added its own compute and maintenance overhead
• Third-party log platforms (self-hosted) — evaluated for feature completeness, but overkill for a narrow response-time monitoring use case
• Azure Data Explorer / Synapse Serverless — good for ad-hoc querying against blob storage, but not ideal as the primary always-on dashboard backend given the recurring query pattern

Ultimately, the simplest and most cost-effective answer was: the Application Gateway can already archive logs directly to Blob Storage. Why not build a lightweight custom dashboard that reads from there instead?

The Solution: A Blob-Based Analytics Dashboard

I designed and built a Flask-based analytics dashboard that reads Application Gateway access logs directly from blob storage, with a three-tier caching architecture to keep queries fast without needing a query engine like Log Analytics at all:

1. cron_aggregate.py — a scheduled job that pre-aggregates historical log data into compact cache files (hourly buckets: total requests, avg/min/max/P95 response time, grouped by endpoint and HTTP status)
2. live_aggregate.py — handles near-real-time aggregation for the current day, bridging the gap until the next scheduled cron run
3. live_fetch.py — serves live queries against the cache layer, with fallback to raw blob reads for edge cases

The dashboard supports the same query parameters teams were used to from Log Analytics/KQL — date range, timezone (UTC/local), endpoint filter, HTTP status filter — and returns the same core metrics: total requests, avg/min/P95/max response time, plus hourly charts for response time trends and request volume.

Validating It Was Actually Correct

Before switching off Log Analytics ingestion, I ran a 14-day parallel validation: every query run against the new dashboard was cross-checked against the equivalent KQL query in Log Analytics, comparing total request counts, average response times, and percentile calculations side by side. Results consistently matched within a fraction of a percent — small residual differences were traceable to caching/timing boundaries rather than any real data discrepancy.

Only once this validation held up consistently did I proceed to disable the Log Analytics diagnostic setting for the Application Gateway (while keeping the blob storage export, which the dashboard depends on, fully intact).

The Result

Comparing actual billed cost the day before and the day after the cutover:

	Before	After	Change
Daily Log Analytics cost	~₹6,800	~₹320	−95.3%

Projected out, that's a reduction of roughly ₹1.95–2.1 lakh per month (~$2,300–2,500 USD) — or ~₹23–25 lakh annualized — in Log Analytics ingestion cost, while every other service on the subscription (compute, storage, the gateway itself) remained completely unaffected. No downtime, no impact to production traffic, and the team retained full visibility into gateway performance metrics, just through a purpose-built tool instead of a general-purpose (and expensive) logging platform.

Where AI Fit Into This

A good chunk of this project moved faster because I used AI (Claude) as a working partner throughout — not to replace the engineering judgment, but to compress the time spent on repetitive or exploratory work:

• KQL query generation and iteration — instead of hand-writing every diagnostic query from scratch, I described what I needed (time-range filters, IST/UTC conversions, percentile aggregations) and iterated on generated KQL until it matched exactly what the dashboard needed to validate against.
• Cross-checking dashboard output against Log Analytics — during the 14-day validation window, I used AI to help spot-check discrepancies between the two data sources, flag where numbers diverged, and reason through why (caching lag vs. genuine data gaps) rather than just staring at two numbers that didn't match.
• Cost analysis and before/after comparisons — once the cutover happened, I used AI to help structure the before/after cost comparison cleanly, catch what mattered (Log Analytics dropped ~95%) versus what was just normal billing noise (Storage, VM, Bandwidth), so I could report a number I was actually confident in.
• Documentation and communication — ticket updates, rollback plans, and stakeholder-facing summaries were drafted with AI assistance and then reviewed/edited by me, which meant less time writing boilerplate and more time on the actual infrastructure work.

The underlying architecture decisions, the validation rigor, and the judgment call on when it was safe to flip the switch in production were mine — AI just meant I could move through the exploratory and repetitive parts of the work faster, and spend more time on the parts that actually needed engineering judgment.

Key Takeaways

• Not every log needs a full observability platform. If you only need a narrow set of aggregate metrics, a lightweight custom solution reading from cheap storage can outperform a general-purpose platform on cost, without sacrificing the metrics that actually matter.
• Validate before you cut over. Running the new system in parallel with the old one for a meaningful window (in this case, two weeks) is what made it safe to flip the switch with confidence.
• Diagnostic settings are just data routing, not data processing. Understanding that AppGw's diagnostic export config is completely decoupled from its actual traffic-serving path made it clear from the start that this change carried zero risk to production traffic — only to where telemetry was shipped.