Data Prepper

Data Prepper는 추적 및 로그를 OpenSearch로 수집하는 기능을 제공한다. Data Prepper 1.4.0 부터는 OpenTelemetry를 통해 수신된 지표에 대한 지원을 한다.  

 

Data Prepper 에서 OTel Collector에서 수집되는 Merices을 OpenSearch로 저장하는 방법에 대해 알아보자.

 

아키텍처

 

https://github.com/opensearch-project/data-prepper/blob/main/docs/trace_analytics.md

설치

1) data prepper 설치

apiVersion: v1
kind: Namespace
metadata:
  name: data-prepper
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: data-prepper
  name: data-prepper-config
  namespace: data-prepper
data:
  pipelines.yaml: |
    entry-pipeline:
      delay: "100"
      source:
        otel_trace_source:
          ssl: false
      sink:
        - pipeline:
            name: "raw-pipeline"
        - pipeline:
            name: "service-map-pipeline"
    raw-pipeline:
      source:
        pipeline:
          name: "entry-pipeline"
      processor:
        - otel_trace_raw:
      sink:
        - stdout:
    service-map-pipeline:
      delay: "100"
      source:
        pipeline:
          name: "entry-pipeline"
      processor:
        - service_map_stateful:
      sink:
        - stdout:
  data-prepper-config.yaml: |
    ssl: false
    peer_forwarder:
      discovery_mode: dns
      domain_name: "data-prepper-cluster.my-domain.net"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: data-prepper
  name: data-prepper-headless
  namespace: data-prepper
spec:
  clusterIP: None
  ports:
    - name: "21890"
      port: 21890
      targetPort: 21890
  selector:
    app: data-prepper
status:
  loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: data-prepper
  name: data-prepper-metrics
  namespace: data-prepper
spec:
  type: NodePort
  ports:
    - name: "4900"
      port: 4900
      targetPort: 4900
  selector:
    app: data-prepper
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: data-prepper
  name: data-prepper
  namespace: data-prepper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: data-prepper
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: data-prepper
    spec:
      containers:
        - image: opensearchproject/data-prepper:2
          name: data-prepper
          ports:
            - containerPort: 21890
          resources: {}
          volumeMounts:
            - name: prepper-configmap-config
              mountPath: /usr/share/data-prepper/config/data-prepper-config.yaml
              subPath: data-prepper-config.yaml
            - name: prepper-configmap-pipelines
              mountPath: /usr/share/data-prepper/pipelines/pipelines.yaml
              subPath: pipelines.yaml
      restartPolicy: Always
      serviceAccountName: ""
      volumes:
        - name: prepper-configmap-config
          configMap:
            name: data-prepper-config
            items:
              - key: data-prepper-config.yaml
                path: data-prepper-config.yaml
        - name: prepper-configmap-pipelines
          configMap:
            name: data-prepper-config
            items:
              - key: pipelines.yaml
                path: pipelines.yaml
status: {}
---

2) OpenSearch 및 Dashboard 설치

https://kmaster.tistory.com/130

OpenSaerch 설치

3) OpenTelemetry 설치 및 설정

OpenTelemetry에서 자동으로 애플리케이션에 대한 관찰을 지원하기 위한 Auto instrumentation 방식으로 배포해 보자.

# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
namespace/opentelemetry-operator-system created
customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created
customresourcedefinition.apiextensions.k8s.io/opentelemetrycollectors.opentelemetry.io created
serviceaccount/opentelemetry-operator-controller-manager created
role.rbac.authorization.k8s.io/opentelemetry-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/opentelemetry-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/opentelemetry-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/opentelemetry-operator-proxy-rolebinding created
service/opentelemetry-operator-controller-manager-metrics-service created
service/opentelemetry-operator-webhook-service created
deployment.apps/opentelemetry-operator-controller-manager created
certificate.cert-manager.io/opentelemetry-operator-serving-cert created
issuer.cert-manager.io/opentelemetry-operator-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/opentelemetry-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/opentelemetry-operator-validating-webhook-configuration created

 

Collector 생성

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
EOF

 

Instrumentation 생성

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
EOF

 

4) 샘플 앱 생성

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-petclinic
spec:
  selector:
    matchLabels:
      app: spring-petclinic
  replicas: 1
  template:
    metadata:
      labels:
        app: spring-petclinic
      annotations:
        sidecar.opentelemetry.io/inject: "true"
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: app
        image: springcommunity/spring-framework-petclinic
---
apiVersion: v1
kind: Service
metadata:
  name: spring-petclinic
spec:
  type: NodePort
  selector:
    app: spring-petclinic
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

 

OpenTelemetry 에서 수집된 data를 data prepper로 export해보자.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:
      otlp/data-prepper:
        endpoint: data-prepper-headless.data-prepper.svc.cluster.local:21890
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp/data-prepper]
EOF

 

마지막으로 data prepper의 sink 설정을 해보자.

k edit cm -n data-prepper data-prepper-config

apiVersion: v1
data:
  data-prepper-config.yaml: |
    ssl: false
    peer_forwarder:
      discovery_mode: dns
      domain_name: "data-prepper-headless"
  pipelines.yaml: |
    entry-pipeline:
      delay: "100"
      source:
        otel_trace_source:
          ssl: false
      sink:
        - pipeline:
            name: "raw-pipeline"
        - pipeline:
            name: "service-map-pipeline"
    raw-pipeline:
      delay: "3000"
      source:
        pipeline:
          name: "entry-pipeline"
      processor:
        - otel_trace_raw:
      sink:
        - opensearch:
            hosts: ["https://opensearch-cluster-master-headless.opensearch.svc.cluster.local:9200"]
            insecure: true
            username: admin
            password: admin
            index_type: trace-analytics-raw
    service-map-pipeline:
      delay: "100"
      source:
        pipeline:
          name: "entry-pipeline"
      processor:
        - service_map_stateful:
      sink:
        - opensearch:
            hosts: ["https://opensearch-cluster-master-headless.opensearch.svc.cluster.local:9200"]
            insecure: true
            username: admin
            password: admin
            index_type: trace-analytics-raw

 

실제 데이터가 정상적으로 저장되었는지 Dashboard에서 확인할 수 있다.