Speaker Notes:

Welcome everyone. Today I'm going to walk you through how we built a unified observability platform for our Cloud Foundry landscape at Swisscom. We run dozens of Spring Boot, Spring Integration, and Reactor-based applications across multiple Cloud Foundry spaces and orgs — some even in remote app clouds with limited connectivity. The challenge was clear: we needed a single pane of glass for both logs and metrics, across all of these environments. What I'll show you is the architecture we landed on, from CF log drains through Logstash all the way to Kibana and Grafana dashboards, including the full Logstash pipeline configuration that you can take and adapt for your own setup.

Speaker Notes:

Let me set the stage. We have a fairly complex Cloud Foundry landscape. Multiple orgs, multiple spaces — dev, staging, production — each running anywhere from a handful to dozens of application instances. These aren't all the same kind of app either. We have traditional Spring Boot REST services, Spring Integration flows for message-driven processing, and Reactor-based reactive applications. Each of these frameworks logs differently. Spring Integration has its own channel logging, Reactor has the challenge of context propagation across reactive streams. On top of that, some of our CF deployments are in remote app clouds that can't directly reach our central infrastructure. So we needed a solution that handles all of this — diverse log formats, multiple instances, and even network-isolated environments — and brings it all together into a single observability platform.

Speaker Notes:

Here's the big picture. We have two parallel pipelines that converge in Grafana. On the left, the logging pipeline: our CF apps produce logs that flow through CF's built-in log-drain mechanism via syslog to a Logstash endpoint. Logstash parses the CF syslog envelope, extracts metadata like org name, space name, app name, and instance index, then enriches and forwards everything to Elasticsearch. From there, Kibana gives us powerful log search and visualization. On the right, the metrics pipeline: Spring Boot Actuator exposes a Prometheus endpoint on each app. Promregator — a fantastic open-source tool — automatically discovers all our CF apps and their instances via the CF API, scrapes each one, and feeds the metrics to Prometheus. Grafana then pulls from both Prometheus and Elasticsearch, giving us unified dashboards with metrics and log data side by side. For remote app clouds that can't reach Logstash directly, we use a shared S3 bucket as an intermediate store. The remote apps write logs to S3, and our central Logstash polls S3 for new data. Same pipeline, same enrichment — just a different transport.

Speaker Notes:

Let's dive into the first piece: Cloud Foundry log drains. CF has a built-in mechanism for streaming application logs out of the platform. Every app instance — and remember, in production you might have 3, 5, or even 10 instances of the same app — sends its stdout and stderr via syslog to a configured drain endpoint. You set this up by creating a user-provided service with a syslog URL pointing to your Logstash instance, then binding it to your app. What's great is that CF automatically enriches the syslog messages with metadata: the org name, space name, app name, app GUID, and crucially the instance index. This means that even though all instances of an app send logs to the same endpoint, we can distinguish which instance produced which log line. The challenge, though, is that these logs arrive interleaved and in CF's syslog envelope format. We need to parse that envelope, extract the metadata, and then parse the actual application log message inside. That's where Logstash comes in.

Speaker Notes:

Before we look at the Logstash pipeline, let's talk about what we do on the application side. The key decision we made early on was to use structured JSON logging everywhere. We use the Logstash Logback encoder, which outputs each log line as a JSON object. This is critical because it means Logstash can parse the application log body as JSON directly, preserving all the fields — log level, logger name, message, stack traces, and importantly any MDC fields we've set. We use MDC extensively for request tracing. Every incoming request gets a correlationId set in the MDC, which then appears in every log line for that request. For Spring Integration flows, we use wire-tap patterns and logging channel interceptors to trace messages through the flow. Reactor is trickier because reactive streams don't have a thread-local context like MDC. We use Hooks.onEachOperator with context propagation to carry our tracing context through the reactive pipeline. The result: consistent, structured, correlation-ready logs from every app type.

Speaker Notes:

Now let's look at the heart of the system: the Logstash pipeline. When a CF syslog message arrives, it's wrapped in the RFC 5424 syslog format. We need to unwrap it. The first grok pattern strips apart the syslog envelope — priority, version, timestamp, host, app ID, process ID, message ID, structured data, and finally the actual message. The structured data field is where CF puts the good stuff: org name, space name, app name, and instance index. We extract those with a second grok pattern, then use mutate to rename them into a prefix document under the "app" key. This prefix doc pattern is really powerful in Elasticsearch and Kibana — it means all CF metadata lives under app.org, app.space, app.name, app.instance, which makes filtering and dashboarding very intuitive. Finally, we check if the syslog message body is JSON. If it is — which it will be if the app uses our Logstash Logback encoder — we parse it into a "log" prefix document. If it's plain text, we just store it as log.message. Let me show you the full pipeline config on the next slide.

Speaker Notes:

Here's the complete Logstash pipeline configuration. I'm going to walk through it section by section. The input section has two sources: a syslog listener on port 5514 for direct CF log drains, and an S3 input that polls a shared bucket for logs from remote app clouds. In the filter section, for syslog messages, we first grok the syslog envelope, then extract CF metadata from the structured data field. We rename everything into our prefix document pattern — app.org, app.space, app.name, app.instance. Then we parse the message body as JSON if it starts with a curly brace, otherwise store it as plain text. After the type-specific parsing, we do common enrichment: we clean up temporary fields, and we add an environment tag based on the space name. If the space contains "prod", we tag it as production; "staging" gets staging; everything else is development. The output sends everything to Elasticsearch with a daily index pattern. This config is copy-pasteable — you can take this, adjust the hostnames and credentials, and have a working CF log pipeline. Feel free to grab it from the presentation or the PDF version.

Speaker Notes:

One of the more interesting challenges we faced was remote app clouds. Some of our CF deployments are in environments that can't establish a direct syslog connection to our central Logstash instance — maybe it's a different network segment, maybe it's a partner's infrastructure. Our solution was simple but effective: a shared S3 bucket. Both the remote environment and our central infrastructure can access S3. On the remote side, a sidecar process or a scheduled Cloud Foundry task collects logs and writes them to the S3 bucket in JSON lines format, with CF metadata included. On the central side, our Logstash pipeline includes an S3 input plugin that polls the bucket for new files. The beauty is that the same filter pipeline processes both sources. Whether a log arrived via direct syslog or via S3, it goes through the same enrichment and ends up in the same Elasticsearch index. The user looking at Kibana doesn't even know the difference. This pattern has been very reliable — the only trade-off is latency. S3 logs are near-real-time rather than real-time, with a delay of usually 1-2 minutes depending on your polling interval.

Speaker Notes:

Now let's talk about the metrics side. Every Spring Boot app we deploy includes the Actuator starter with the Prometheus endpoint enabled. This gives us a wealth of metrics out of the box — JVM memory, garbage collection, thread pools, HTTP request counts and latencies, and for Reactor apps, scheduler and subscriber metrics. We also add custom business metrics via Micrometer — things like orders processed, messages handled, or integration flow completion times. The challenge with CF is that you can't just point Prometheus at your apps. CF doesn't give you direct network access to individual app instances — they're behind the GoRouter. This is where Promregator comes in. Promregator is a Java application that sits between Prometheus and your CF apps. It uses the CF API to discover all your applications, resolves the individual instance routes using CF's instance routing header, and presents a single scrape endpoint to Prometheus that includes metrics from all instances, properly labeled with app name, space, org, and instance index. It's essentially a Prometheus federation proxy that understands Cloud Foundry. Without Promregator, you'd have to manually configure Prometheus targets for every app and somehow handle instance routing yourself. With it, you just configure which orgs and spaces to monitor, and it handles the rest.

Speaker Notes:

Finally, Grafana brings it all together. We configure two main datasources: Prometheus for metrics and Elasticsearch for logs. This lets us build dashboards that show both metrics and log-derived data on the same page. For example, our main application dashboard has panels showing request rate and latency from Prometheus right next to error log counts from Elasticsearch. When you see a latency spike, you can immediately see if there's a corresponding increase in error logs. Template variables are essential. We define variables for org, space, app, and instance, which cascade — select an org and the space dropdown filters to spaces in that org, and so on. These variables are used across both Prometheus queries and Elasticsearch queries, so everything filters consistently. Some of our most useful panels include JVM heap usage per instance from Prometheus, which helps us right-size our app memory, and log level distribution from Elasticsearch, which gives us a quick health signal — a sudden increase in WARN or ERROR logs is often the first sign of trouble. The combination of both data sources in Grafana is really what makes this a true observability platform rather than just a logging system or just a metrics system.

Speaker Notes:

Let me wrap up with six key takeaways. First: invest in structured logging from day one. JSON logs with MDC fields pay dividends throughout the entire pipeline — they're parseable by Logstash, searchable in Kibana, and correlatable across services. Second: CF log drains combined with Logstash grok patterns give you a metadata-rich pipeline almost for free. CF already adds org, space, app, and instance metadata to every log line — you just need to extract it. Third: the prefix document pattern — nesting all CF metadata under the "app" key in Elasticsearch — makes Kibana filtering incredibly intuitive. You can filter by app.org, app.space, app.name, app.instance with dropdowns. Fourth: Promregator is the key to metrics in CF. Without it, you're flying blind on individual instance metrics. Fifth: when you have remote or network-isolated CF deployments, a shared S3 bucket is a simple, reliable bridge. And sixth: Grafana with both Prometheus and Elasticsearch datasources gives you that single pane of glass that every operations team needs. Thank you for your attention.

Speaker Notes:

Thank you all for listening. I'm happy to take any questions. If you want to get in touch later, my email is on the slide — [email protected]. The Logstash pipeline config from this presentation is fully copy-pasteable, and you can download a PDF version of the slides including speaker notes. Thanks again.