Hi, everyone, and welcome back.
Now in today's video, we are going to discuss about centralized logging.
Now in production environment, having a centralized logging is extremely important.
Otherwise, if you do not have it and an incident occurs, it can be any incident related to reliability,
security, etc. When an incident occurs, it has been observed that the team will have to jump across
multiple accounts, multiple servers to fetch the log file that can give details related to that specific
incident.
Sometimes it has also been observed that due to the disk space that was full on that server, the log
file was never generated.
So in order to overcome those issues, it is important that you have centralized logging configured.
And in today's video we are going to look into this type of architecture now generally in organizations,
specifically in these stages where they follow the multi account based architecture.
Log monitoring is one of the very crucial things that you will have to take care of.
Now, let's say in this diagram there are three eight place accounts now in place.
Many of the services generate their own set of log files.
Cloud generates its log file guide you to generate the log file.
If you have a set of easy two instances, there will be logs, then there will be logs related to the
config, etc. So this is one part and same applies for other accounts as well.
Now generally many organizations, what they do when they have a centralized monitoring solution like
Splunk, they associate the Splunk with each and every individual account to fetch the logs of each
and every individual services.
So Splunk first connects to development account, fetches the logs of cloud trail.
Guard duty is two instances and so on, same for staging, same for production and so on.
So although this seems to be good as a starting point, but when you have hundreds of database account,
this is not the best of the architectures that can be used.
Now one of the recommended architectures for logging is you have a centralized logging account.
So what really happens here is that every individual account.
So in this case you have an account, you have an account, B, etc. Each account is configured to send
its log files to a central logging account.
So in account you might have a cloud service enabled cloud will send the logs to center logging account
account.
We will have a config service enable again it will send to central logging account and your log monitoring
tool like Splunk.
This log monitoring tool will be associated with only one account, in this case the center logging
account, and it will fetch all of the data from here.
So this way, even if you have 100 database account, it becomes much more simpler because all of those
accounts will be sending the logs to a central logging account here.
Now, when you have a logging based architecture, there are certain considerations that you'll have
to take care of.
First consideration is you will have to define a log retention requirement and the lifecycle policies
early on.
Now, generally in larger organizations, the log size would go in terabytes.
So do you need all of the terabytes of data in your S3 bucket or you can move it to Glacier as well?
Because if you store so much of data in S3, your bill will be also huge.
So many of the organizations have a log retention requirement.
For example, for the first three months the logs will be necessary bucket.
After three months, all of the logs will go to the glacier and so on.
So all of those depends upon the log retention policies in your organization again is you need to incorporate
tools and features to automate the lifecycle policies.
So this in case if you're using S3, this is much more straightforward.
Third important part is you have to automate the installation and configuration of log shipping.
Imagine for example, let's say you launch ten new easy to instances.
Now after that SC two instances are launched.
You also have to ensure that all of those easy to instances are immediately sending all of the logs
to the centralized logging account.
So maybe in the army itself you might have your Splunk agent or a logging agent preconfigured.
So whenever instance starts, the agent will go ahead and start to send the logs to the central account.
And the fourth consideration is to make sure that the solution supports hybrid environments to support
the needs.
It is not necessary that all of the accounts would be based on a database.
It can happen.
Some accounts are based on hybrid architecture, maybe Azure, maybe on prem environment, maybe eight
and so on.
So you have to ensure that your logging architecture supports such kind of hybrid environments as well.
So let's do one thing before we proceed further.
Let's have a quick demo about this centralized logging architecture.
Now for today's demo, I have two accounts in two different types that are running.
First account we'll be referring to as a central logging account, and second account is basically the
production account.
And similar to the architecture that we were discussing, the production account will be sending the
data to a central logging account.
So let's go ahead and look into.
Now in that logging account, let's quickly open up the three service.
Now within here I have one S3 bucket calls, escape, central logs, cloud trail.
Now from the name itself, we can identify that this bucket will be hosting the cloud logs from other
accounts as well.
Now, if you open this bucket, you have a folder of logs.
And at this stage there is one account that is adding the data to this S3 bucket, and this account
ends with four, seven, six three.
And if you look here, it is not the same account.
So currently the account number of this address account where this bucket lies is 9364.
But this is a different account altogether.
So this 4763 account is basically running in a different type.
If it is four, seven, six three.
And our cloud service from this production account is sending all of its logs to our central logging
account.
So if you would look into the configuration of a cloud trail, let's quickly open up a trail here.
If you'll see the trail location escape will have central logs, cloud trail, which is nothing but
this specific bucket in a logging account grid.
So that's the high level overview about centralized logging.
Now a few more important points that we need to note.
First one is that we can make use of multiple managed services to build a centralized logging solution.
You can make use of Elasticsearch service cloud, watch logs, kinases, even S3 bucket to host these
centralized logs.
Along with that, one more important part that we need to notice that the ways in which you can configure
the centralized logging for each services like load trail, VPC, flow logs, guard duty, etc. differs.
Every service has a different way in which you can use it to send the logs to a central logging account.
Some of the ways are pretty straightforward.
Some of the ways requires a little more steps to be configured.
So that's the high level overview and this will conclude today's video.