BLiNQ NetworksNetLiNQ · Monitoring All guides/Overview

Customer Monitoring Guide

Monitor your NetLiNQ deployment.

Metrics, dashboards and alert email are installed for you as a single package. This guide shows what you get, how to reach each tool, and how long your data is kept.

35 dashboards 8 components 15-day metrics Email alerts

Overview

What gets installed

Monitoring ships as one Helm release named monitoring, built on kube-prometheus-stack 87.19.2. Three tools do the work you will actually interact with; the rest feed them data.

You use these

Grafana

Dashboards and graphs. This is where you look at the health of the platform day to day.

You use these

Alertmanager

Sends you email when something needs attention.

Every component

PrometheusCollects and stores metrics from the cluster and from NetLiNQ services.
AlertmanagerSends alert email when a rule fires.
GrafanaDashboards and graphs, backed by the EMS PostgreSQL database.
kube-state-metricsReports the state of Kubernetes objects such as pods and deployments.
node-exporterReports CPU, memory, disk and network for each server.
prometheus-operatorManages the Prometheus and Alertmanager instances.
es-exporterExposes Elasticsearch health as metrics. Installed as its own release.
jsonScrapes NetLiNQ JSON metric endpoints. Installed as its own release.

Where things run

Everything runs in the monitoring namespace — except Grafana, which runs in netlinq.

Grafanadashboards you log in tografana, grafana-headless netlinq
Metrics and alertingPrometheus, Alertmanager, operatorprometheus-server, alertmanager monitoring
Collectorscluster and per-server metricskube-state-metrics, node-exporter monitoring

Grafana deliberately stays in netlinq so the existing secrets, database mounts and web routes keep working unchanged. This is expected, not a misconfiguration.

Step 1

Before you start

Deploy monitoring after NetLiNQ EMS and PostgreSQL are running. Grafana keeps its dashboards, users and datasources in the EMS PostgreSQL database, so that database has to exist first.

This stack currently needs access to public container registries. Air-gapped and private-registry installs will fail with ImagePullBackOff until the internal image mirror lists are updated. Check with BLiNQ before deploying in a disconnected environment.
If you ran the main installer, monitoring was already deployed as its final stage. You only need the steps below when redeploying or upgrading.

Step 2

Deploy monitoring

deploy-monitoring
$ cd blinq-ems-autom-configs/monitoring
$ source deploy_monitoring.sh
Use source, not bash. The script reads your web address, device IP, setup type and SMTP settings from the shell. Running it with bash starts a new shell, those values are lost, and the deploy ends up misconfigured.

What the script does

Fill in your settings
Install the stack
Add exporters
Load dashboards
Ready
Your settings are written into a temporary copy of the configuration, so the file in the repository is never modified. For singlenode_demo, resource requests are halved automatically.

Step 3

Reach the tools

Grafana is published at the grafana subdomain of the web address you set during installation.

grafana address
https://grafana.<your endpoint>
# e.g. endpoint ems.example.org becomes
https://grafana.ems.example.org

Prometheus and Alertmanager are published through the platform's web routes rather than by the monitoring package itself. List the exact addresses, then add the ones you need to your DNS.

list addresses
$ kubectl get httproutes -A

Signing in to Grafana

Grafana uses a generated administrator account held in the internal-grafana secret. Retrieve it with secretsctl — the same tool you used during installation — or read it straight from the cluster.

browser form
$ secretsctl ui   # open the Internal section, then Reveal
read from cluster
$ kubectl get secret internal-grafana -n netlinq \
    -o jsonpath='{.data.grafana-username}' | base64 -d; echo
$ kubectl get secret internal-grafana -n netlinq \
    -o jsonpath='{.data.grafana-password}' | base64 -d; echo
These credentials are generated at install time and are unique to your deployment. There is no default password, so retrieve them rather than guessing.

Step 4

Dashboards

NetLiNQ ships six custom dashboards, plus 29 standard Kubernetes dashboards from the upstream project.

Cluster monitoringOverall cluster health, node and pod resource use.
Certificate expirationCertificates approaching expiry.
ElasticsearchElasticsearch cluster health, indices and storage.
PostgreSQLDatabase connections, queries and performance.
Akka / KamonNetLiNQ application internals and message throughput.
Spark / KamonDevice-facing service metrics.
These dashboards are read-only in the Grafana interface. To change one, edit its JSON file and re-apply it. Dashboards you create yourself stay editable and are saved in PostgreSQL.

Add your own dashboard

Export the dashboard as JSON from Grafana, drop it into the dashboards folder, then re-apply.

add dashboard
$ cp my-dashboard.json blinq-ems-autom-configs/monitoring/grafana/dashboards/
$ cd blinq-ems-autom-configs/monitoring
$ ./apply_grafana_dashboards.sh

Step 5

Alert email

Alertmanager sends email using the SMTP details collected during installation.

If you skipped SMTP during installation, no email is sent. Alerts still fire and appear in the interface, but nothing reaches your inbox. Set your SMTP values, then redeploy monitoring.

Alert routing and recipients live under alertmanager.config in monitoring/values.yaml. After editing, run the deploy command again.

Step 6

Data retention

Prometheus metrics15 days by default. Change prometheus.prometheusSpec.retention in monitoring/values.yaml.
Elasticsearch indices14 days by default. Change retention_realtime in netlinq_onprem_install.vars.
Grafana contentDashboards and users are kept indefinitely in the EMS PostgreSQL database.

Prometheus storage size is set by prometheus.prometheusSpec.storageSpec. Increase the retention period and the volume size together — a longer retention on an unchanged volume simply fills up and starts dropping the oldest data.

Retention and volume sizing are an estimate and still need to be validated against your device count and how much history you want. Confirm with BLiNQ before committing to hardware.

Step 7

Common tasks

Check that monitoring is healthy

health check
$ kubectl get pods -n monitoring
$ kubectl get pods -n netlinq -l app.kubernetes.io/name=grafana

Every pod should be Running or Completed.

Add a metrics target

Add it under prometheus.prometheusSpec.additionalScrapeConfigs in monitoring/values.yaml, then redeploy.

values.yaml
prometheus:
  prometheusSpec:
    additionalScrapeConfigs: |
      - job_name: my-exporter
        static_configs:
          - targets:
            - my-exporter.netlinq.svc.cluster.local:9100

Reinstall or remove monitoring

lifecycle
$ ./reinstall_monitoring.sh              # remove, then deploy again
$ ./uninstall_monitoring.sh              # interactive, keeps stored data
$ ./uninstall_monitoring.sh -y           # no prompts, keeps stored data
$ ./uninstall_monitoring.sh -y --delete-pvc # also deletes stored metrics
Deleting the volumes discards all Prometheus metric history, and it cannot be recovered. Grafana dashboards and users survive, because they live in PostgreSQL.

Help

Troubleshooting

Monitoring pods stuck in ImagePullBackOff›
The stack needs access to public container registries. Air-gapped and private-registry installs are not yet supported — contact BLiNQ.
Grafana redirects to the wrong address›
The Grafana domain must match the web route hostname. Confirm your endpoint was correct when you deployed, then redeploy monitoring.
No alert email arriving›
SMTP was skipped during installation. Set the SMTP values and redeploy monitoring. Alerts will still be visible in the interface in the meantime.
A new dashboard does not appear›
Re-run ./apply_grafana_dashboards.sh and confirm the JSON file is valid. Provisioned dashboards can take a few seconds to be picked up.
Prometheus history disappeared›
The metrics volume was recreated — for example by an uninstall using --delete-pvc. Metric history cannot be recovered.
Errors about Prometheus custom resources after an upgrade›
A Helm upgrade does not update the operator's custom resource definitions. After a chart version change they must be applied manually. Contact BLiNQ if you are unsure.
Grafana pod will not start›
Check that PostgreSQL is running in the netlinq namespace. Grafana depends on it for dashboards, users and datasources.

Reference

At a glance

Helm releasemonitoring
Chartkube-prometheus-stack 87.19.2
Stack namespacemonitoring
Grafana namespacenetlinq
Grafana addresshttps://grafana.<your endpoint>
Grafana credentialsinternal-grafana secret in netlinq
Prometheus retention15 days
Dashboards6 custom, plus 29 upstream
Extra releaseses-exporter, json