The Case for a Dashboard Recommendation System in Observability

Most observability programs eventually accumulate a familiar kind of debt: not broken dashboards, but undiscoverable ones.

Teams build valuable views during high-pressure incidents, complex migrations, and release retrospectives. But once the crisis fades, those dashboards often disappear into an endless list of folders, inconsistent tags, and tribal knowledge. This “dashboard sprawl” doesn’t just waste storage; it wastes time when minutes matter most.

A dashboard recommendation system is a pragmatic response to this problem. It applies the same mechanics that drive product discovery, search, recommendations, sharing, and personalization, to observability assets.

What “Recommendations” Mean in Observability

A useful dashboard recommender isn’t just a “trending now” list. It must answer operationally specific questions:

• On-call Context: “I’m on call for Service X; what should I look at first?”
• Deployment Context: “We just deployed; show me the dashboards that caught regressions in this domain last time.”
• Incident Context: “This alert fired; what dashboards helped the last three times it happened?”
• Onboarding: “I’m new to the team; what are the canonical dashboards for our stack?”

The Engine: Dashboard Attributes

A recommender system rises or falls on the quality of its features. To build a robust engine, we categorize attributes into six key areas:

1. Structural and Semantic (Content-Based)

• Title/Description: Utilizing NLP to parse jargon and synonyms.
• Tags/Labels: Service, environment, SLO, cost, or security tags.
• Layout Patterns: Distinguishing between dense troubleshooting views and executive summaries.

2. Data-Source and Query Signals

• Query Fingerprints: Normalized templates that ignore literal thresholds but recognize the underlying metrics (for example, Prometheus or Datadog queries).
• Entity Coverage: Which pods, nodes, or regions does this dashboard cover?

3. Ownership and Lifecycle

• Trust Score: Is the dashboard verified? Who is the maintainer?
• Staleness: Recency of edits and data source validity.

4. Contextual Operational Data

• Active Incidents: Correlation with current PagerDuty or Opsgenie alerts.
• Deployment Logs: Recent CI/CD events related to the viewer’s domain.

5. Usage and Engagement (Collaborative Signal)

• Co-view patterns: Dashboards frequently opened in the same troubleshooting session.
• Interaction Depth: Did the user just open it, or did they apply filters and drill down?

6. Outcome and Value (The Feedback Loop)

• Incident Association: Was this dashboard linked in a post-mortem?
• Explicit Feedback: We must request users to provide feedback, simple “Helpful/Not Helpful” prompts, to refine the “Taste Profile.”

Neighborhood Algorithms to Consider

Neighborhood methods are ideal for observability because they are explainable. When a system says, “Recommended because you are viewing Service Y,” users trust it more.

1. Item-based k-Nearest Neighbors (Item-kNN): Finds dashboards similar to the one currently being viewed based on features like query overlap or shared tags.
   • Similarity Metric: Often uses Cosine Similarity: $similarity = \frac{A \cdot B}{\|A\| \|B\|}$
2. User-based k-Nearest Neighbors (User-kNN): Finds “peers” (engineers on the same team or with similar on-call rotations) and recommends what they find useful.
3. Session-based Co-occurrence: Uses graph theory to recommend dashboards that appear in the same investigation “session” or incident timeline.
4. Content Neighborhoods (Embeddings): Representing dashboards as vectors in a continuous space using LLMs to find neighbors even when usage data is sparse (the “cold start” problem).

Implementation: GitHub vs. Observability Platform

Where should this logic live? There are two primary schools of thought:

The Bridge: Private MCP Catalogs

The Model Context Protocol (MCP) provides a transformative way to bridge these two worlds. Private MCP catalogs act as the control plane for dashboard discovery.

Today: Curated, Verified Collections

Currently, Private MCP catalogs serve as a “Source of Truth.” They host approved dashboard packs with verified metadata, linted queries, and clear ownership. They solve the “trust” problem by ensuring that recommendations only surface high-quality, non-broken assets.

Tomorrow: Self-Improving Discovery Systems

As these catalogs mature, they become the foundation for algorithmic evolution.

• Feedback Integration: The catalog ingests usage events from the platform.
• Automated Lifecycle: The system suggests “Dashboards to Verify” (high usage, low trust) or “Dashboards to Retire” (stale/unused).
• AI-Driven Discovery: LLMs utilize the MCP catalog as a “knowledge base” to answer natural language queries like, “What’s the best dashboard for debugging latency in the EU-West region?”

Benefits and Challenges

The Upside

• Faster MTTR: Reduces “search time” during outages.
• Reduced Duplication: Engineers reuse proven views instead of rebuilding them.
• Smoother Onboarding: New hires get a guided map of canonical tools.

The Hurdle

• Metadata Hygiene: Inconsistent tagging can ruin recommendation quality.
• The “Rich Get Richer” Bias: Popular dashboards stay popular, potentially hiding newer, better views. This requires “serendipity” algorithms to inject variety.

A practical rollout plan (how to start without boiling the ocean)

• Define a dashboard metadata schema: owners, services/entities, environment, purpose, runbook links, lifecycle state
• Stand up a Private MCP catalog: enforce schema and verification checks; ingest verified dashboards (initially curated)
• Implement item-based neighbourhood recommendations: start with entity overlap + data source similarity + tags + query fingerprints
• Add usage telemetry and session co-occurrence: views, shares, incident links (with strict RBAC)
• Introduce taste profiles with decay and governance constraints: personalize within safe boundaries; never recommend deprecated/unverified by default
• Operationalize lifecycle workflows: “recommend dashboards to verify,” “recommend dashboards to retire,” “recommend missing dashboards for critical services”

Conclusion

A dashboard recommendation system turns static artifacts into an evolving knowledge system. By moving from simple lists to an algorithmic discovery model powered by Private MCP catalogs, organizations can ensure that the right data finds the right engineer at the right time.