Datadog Agent 설치 방법 및 JAVA Trace 설정

CentOS 서버에 Datadog Agent 설치

https://app.datadoghq.com/account/settings/agent/latest?platform=centos

 

1. Datadog의 Yum 저장소를 설정

sudo vi /etc/yum.repos.d/datadog.repo 명령어 실행 후 아래 내용 삽입

[datadog]
name=Datadog, Inc.
baseurl=https://yum.datadoghq.com/stable/7/x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public 
       https://keys.datadoghq.com/DATADOG_RPM_KEY_B01082D3.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_FD4BF915.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_E09422B3.public

※ 참고: dnf의 버그로 인해 CentOS/Red Hat 8.1에서는 repo_gpgcheck=1 대신 repo_gpgcheck=0을 사용하세요.

2-1. Agent 5 또는 6에서 업그레이드하는 경우, 미사용 RPM GPG 키 삭제

sudo sh -c 'if rpm -q gpg-pubkey-4172a230-55dd14f6 >/dev/null; then rpm --erase gpg-pubkey-4172a230-55dd14f6; fi'

2-2. 로컬 yum 저장소 업데이트 후 Agent 설치

sudo yum makecache
sudo yum remove datadog-agent-base
sudo yum install datadog-agent

3-1. Agent 5 (5.17+만 해당)에서 업그레이드 시, 선택적으로 기존 Agent 5 구성 import

sudo -u dd-agent -- datadog-agent import /etc/dd-agent /etc/datadog-agent

3-2. 또는, example config를 복사하고 API 키 연결

sudo sh -c "sed 's/api_key:.*/api_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/' /etc/datadog-agent/datadog.yaml.example > /etc/datadog-agent/datadog.yaml"

4. Datadog 리전 구성

sudo sh -c "sed -i 's/# site:.*/site: datadoghq.com/' /etc/datadog-agent/datadog.yaml"

5. 권한 확인

sudo sh -c "chown dd-agent:dd-agent /etc/datadog-agent/datadog.yaml && chmod 640 /etc/datadog-agent/datadog.yaml"

6. Re-start (CentOS 7)

sudo systemctl restart datadog-agent.service

7. 설치 완료 후 확인

Agent가 설치되면 Datadog Infrastructure List에 등록이됩니다.

https://app.datadoghq.com/infrastructure

+) Agent 삭제 방법

sudo yum remove datadog-agent

로그까지 삭제 시 에이전트를 삭제 후 아래 명령어 실행 

sudo userdel dd-agent \
&& sudo rm -rf /opt/datadog-agent/ \
&& sudo rm -rf /etc/datadog-agent/ \
&& sudo rm -rf /var/log/datadog/

 

 

 

APM에서 불필요한 리소스 무시하기

https://docs.datadoghq.com/ko/tracing/guide/ignoring_apm_resources/?tab=datadogyaml#ignoring-based-on-resources

`datadog.yaml` 에서 apm_config 설정 주석을 해제합니다.

vi /etc/datadog-agent/datadog.yaml
 
apm_config:
 ignore_resources: ["(GET|POST) /healthcheck", "(GET|POST) /resources/**"]	# 맨앞에 공백 중요

 

 

 

Java 어플리케이션 추적

1. dd-java-agent.jar 설치

https://docs.datadoghq.com/ko/tracing/trace_collection/automatic_instrumentation/dd_libraries/java/?tab=dockerfile

mkdir /home/integtool/apps/dd-java-agent
cd /home/integtool/apps/dd-java-agent

wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'
# curl -Lo dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'
# ADD 'https://dtdg.co/latest-java-tracer' dd-java-agent.jar

2. JVM에 Java tracer 추가

java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -XX:FlightRecorderOptions=stackdepth=256 -jar <YOUR_SPRING_BOOT_APP>.jar

3. Springboot 어플리케이션 설정

`application.yml` 구성

dd:
  trace:
    enabled: true
  service:
    name: <SERVICE_NAME>
  env: <ENVIRONMENT>
  version: <VERSION>
  logs:
    injection: true

Maven 의존성 추가

		<dependency>
			<groupId>com.datadoghq</groupId>
			<artifactId>dd-trace-api</artifactId>
			<version>1.35.0</version>
		</dependency>
		<dependency>
			<groupId>io.opentracing</groupId>
			<artifactId>opentracing-api</artifactId>
			<version>0.33.0</version>
		</dependency>
		<dependency>
			<groupId>io.opentracing</groupId>
			<artifactId>opentracing-util</artifactId>
			<version>0.33.0</version>
		</dependency>

Spring Boot 어플리케이션에서 OpenTracing을 사용하여 Datadog 트레이서를 설정하려면, 아래와 같이 Tracer 빈을 구성해야 합니다.

import com.datadoghq.trace.api.Trace;
import datadog.opentracing.DDTracer;
import io.opentracing.Tracer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class TracingConfig {

    @Bean
    public Tracer tracer() {
        return DDTracer.builder().build();
    }
}

OpenTracing API를 사용하여 메서드나 특정 코드 블록을 트레이싱할 수 있습니다. 

import io.opentracing.Span;
import io.opentracing.Tracer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class MyService {

    @Autowired
    private Tracer tracer;

    public void myMethod() {
        Span span = tracer.buildSpan("myMethod").start();
        try {
            // 메서드 로직
        } finally {
            span.finish();
        }
    }
}

 

 

 

Log Appender 설정

데이터 독은 기본적으로 tomcat, java, nginx 로그를 수집할 수 있는 agent 기능이 존재 하나, 서비스별 로그 수집 및 불필요 로그 적재에 따른 비용 증가가 예상됩니다.

(선택적인 로그 적재를 위해서 logAppender를 통한 기록을 사용)

<!-- Configure the Datadog appender -->
<appender name="DATADOG_JSON_TCP" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>intake.logs.datadoghq.com:10516</destination>
    <keepAliveDuration>20 seconds</keepAliveDuration>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="ch.qos.logback.classic.PatternLayout">
                <pattern>API키 %mdc{keyThatDoesNotExist}</pattern>
            </layout>
        </prefix>
    </encoder>
    <ssl />
</appender>
 
 
<!-- 데이터독 로그 적재 -->
<root>
    <level value="INFO" />
    <appender-ref ref="DATADOG_JSON_TCP" />
</root>