Creating the AMP and Grafana and Deploy the CloudWatch agent and Fluent inside the K8s cluster using terraform and helm chart

 In this Blog, we are going to create the AWS-managed Prometheus and grafana using Terraform and We will deploy the Cloudwatch agent with fl...


 In this Blog, we are going to create the AWS-managed Prometheus and grafana using Terraform and We will deploy the Cloudwatch agent with fluent using helm chart and collect the (Container insights) application logs, and data-plane logs.

What is Prometheus?
    Prometheus is an open-source tool that monitors and alerts for systems and is used to collect and store metrics from applications and infrastructure. It's used to monitor cloud-native architectures, such as Kubernetes.

Amazon Managed Service for Prometheus is a fully managed Prometheus-compatible monitoring service that makes it easy to monitor containerized applications securely and at scale.

Benefits and features:
  Prometheus-compatible APIs give you access to remote write metrics from existing Prometheus servers and query metrics using PromQL.
   AMP integrates with AWS Identity and Access Management (IAM) for authentication and fine-grained permissions for users and groups. VPC PrivateLink provides easy and secure access to services hosted on AWS, keeping your network traffic within the AWS network.
    AMP automatically scales as your ingestion and query needs grow, handling millions of unique time series metrics using multi-AZ deployments.

What is grafana
    Grafana is an open-source analytics and monitoring tool that helps users visualize, query, alert, and understand their data.

Amazon Managed Grafana is a fully managed Grafana service that is scalable, secure, and highly available. Using Amazon Managed Grafana, you can analyze, monitor, and alarm on metrics, logs, and traces across multiple data sources.

Before creating the Prometheus and grafana we need an EKS cluster. Check out our blog on the creation of the EKS cluster from scratch.

Now we need to create the resources using Terraform, you can find the Terraform code on my repo https://github.com/maneshmohan124/Techiev-observability

The Backend code Basically we declared the Statefile path as S3.
For grafana and Prometheus, we need to create IAM roles and policies, so we are using iam.tf to create the roles and policies.
In the main.tf file, we are adding the modules for creating the AMP and grafana workspace.
In the grafana workspace, we can add AWS data sources for now we are adding cloudwatch and Prometheus. Later blogs we can add x-ray and elastic search.
In the variables.tf we are declaring the env, region, oidc ARN, URL.


Once we planned and applied the terraform code, it created the AMP and Grafana workspaces.


Now we can log in to the Eks cluster and deploy the cloudwatch agent and fluent using the helm chart.

What is helm?
    Helm is an open-source project that automates the distribution of applications to Kubernetes clusters. It can help simplify and make more efficient the management of Kubernetes applications.

The Helm chart of cloudwatchagent with fluent can be found inside our repo
Repo URL: https://github.com/maneshmohan124/Techiev-observability

In the Helm chart values.yaml we need to type our Cluster name and region.

The Helm chart will create the Namespace, role, role binding, config map, daemonset for cloudwatchagent, and fluent.
Now deploy the helm chart using the below command.

command: helm upgrade amazon-cloudwatch ./cloudwatch-fluent-bit --install -f ./cloudwatch-fluent-bit/values.yaml


Before Checking the Cloudwatch log group, we need to add CloudWatchAgentServerPolicy to our worker node Role.


After deploying the Cloudwatch and Fluent you can find the log groups inside the cloudwatch.


We have successfully created the AMP and grafana and deployed the Cloudwatch agent with fluent using helm chart and collected the (Container insights) application logs, and data-plane logs,

we can collect the Container insights with Cloudatchagent using Eks Add-ons. Later we will see as a blog for another option.


----------------------------------------!!!! Happy Learning with Techiev !!!!!!!!----------------------------------

-------------------------Subscribe our Youtube Channel by clicking the below link---------------------- ----------------------------!!https://www.youtube.com/@techieview729!!---------------------





Name

AWS,19,Devops,29,linux,11,
ltr
item
Techie View: Creating the AMP and Grafana and Deploy the CloudWatch agent and Fluent inside the K8s cluster using terraform and helm chart
Creating the AMP and Grafana and Deploy the CloudWatch agent and Fluent inside the K8s cluster using terraform and helm chart
https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYM7oVaFPPqm5Dfv6WwbpxJ6PzmyaSY3r061TZNegsyJ2xQ77hDZwrLzfWfPB2Uygs06VImVT9dMpuDcZvYuHzGmqdb84rtiuLypnm4W1srSrul0YbaOj8zX6DxQ4bHHZ7NkWWK1sIctGoKEu4-9cbtU54Dvd538pWMqoGf53IoEvGSF6X29ZeGrVp9iB-/w640-h304/AMP%20and%20grafana%20.jpg
https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYM7oVaFPPqm5Dfv6WwbpxJ6PzmyaSY3r061TZNegsyJ2xQ77hDZwrLzfWfPB2Uygs06VImVT9dMpuDcZvYuHzGmqdb84rtiuLypnm4W1srSrul0YbaOj8zX6DxQ4bHHZ7NkWWK1sIctGoKEu4-9cbtU54Dvd538pWMqoGf53IoEvGSF6X29ZeGrVp9iB-/s72-w640-c-h304/AMP%20and%20grafana%20.jpg
Techie View
https://www.techiev.com/2024/10/creating-amp-and-grafana-and-deploy.html
https://www.techiev.com/
https://www.techiev.com/
https://www.techiev.com/2024/10/creating-amp-and-grafana-and-deploy.html
true
7013663511659419322
UTF-8
Loaded All Posts Not found any posts VIEW ALL View Full Article Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share to a social network STEP 2: Click the link on your social network Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy