Apache web server devouring resources

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache web server devouring resources

Darryl Philip Baker

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Jim Albert
On 3/28/2019 12:11 PM, Darryl Philip Baker wrote:

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 


First thing would be to check request logs during time of high load. I assume this is a public web server. Perhaps you were being probed heavily.

Jim

  



	
	
	
	
Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Rose, John B
In reply to this post by Darryl Philip Baker

Regarding the "load increasing quickly after restarting the daemons" ...


I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?


Are there a preponderance of http connections or PHP-FPM processes, or both?


If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources
 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Daniel Ferradal
Have you tried mod_status? That would clearly tell you which threads eat resources for you.


You can share the mpm you are using ,the values you have configured for then, also the list of modules you load, and the actual load you receive. That alone can give great hints about a likely culprit.


El jue., 28 mar. 2019 17:31, Rose, John B <[hidden email]> escribió:

Regarding the "load increasing quickly after restarting the daemons" ...


I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?


Are there a preponderance of http connections or PHP-FPM processes, or both?


If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources
 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Darryl Philip Baker
In reply to this post by Darryl Philip Baker

No PHP on the system at all. The web server was down for 15-20 minutes so anything in the queue should have cleared, right?

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 11:32 AM
To: "[hidden email]" <[hidden email]>
Subject: [users@httpd] Re: Apache web server devouring resources

 

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Darryl Philip Baker
In reply to this post by Daniel Ferradal

Here you go:

mpm_worker

MinSpareThreads 40

ThreadsPerChild 25

StartServers 10

ServerLimit 250

MaxConnectionsPerChild 0

MaxRequestWorkers 6000

MaxSpareThreads 6000

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: Daniel Ferradal <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 12:02 PM
To: "<[hidden email]>" <[hidden email]>
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

Have you tried mod_status? That would clearly tell you which threads eat resources for you.

 

 

You can share the mpm you are using ,the values you have configured for then, also the list of modules you load, and the actual load you receive. That alone can give great hints about a likely culprit.

 

 

El jue., 28 mar. 2019 17:31, Rose, John B <[hidden email]> escribió:

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Rose, John B
In reply to this post by Darryl Philip Baker

I don't think the TCP buffer would be clear if there was a continuing flow of http requests during that time, whether the web server software was down, or maxed out


But maybe I am wrong. 




From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 1:22:59 PM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources
 

No PHP on the system at all. The web server was down for 15-20 minutes so anything in the queue should have cleared, right?

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 11:32 AM
To: "[hidden email]" <[hidden email]>
Subject: [users@httpd] Re: Apache web server devouring resources

 

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Darryl Philip Baker

It does strike me odd if there were continued request why didn’t pick right back up after the reboot. The reboot would take less than 3 minutes.

 

Also a connection attaching to the load balancer, the servers are not directly addressable, would have been routed to one of the other servers while Apache was down.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 12:57 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

I don't think the TCP buffer would be clear if there was a continuing flow of http requests during that time, whether the web server software was down, or maxed out

 

But maybe I am wrong. 

 

 


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 1:22:59 PM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

No PHP on the system at all. The web server was down for 15-20 minutes so anything in the queue should have cleared, right?

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 11:32 AM
To: "[hidden email]" <[hidden email]>
Subject: [users@httpd] Re: Apache web server devouring resources

 

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Muggeridge, Matt
In reply to this post by Rose, John B

You will need to do some more triaging.  Suggestions for things to investigate more deeply:  http Log files, system log files, system performance monitoring, connection statistics, source of traffic, TCP performance tuning, firewall control, protection against DOS attacks… and that’s just off the top.

 

You will need to profile the system using a myriad of tools that suit your need (e.g. tcpdump, lsof, top, netstat, ss, and a large variety of others, depending on what you learn along the way).

 

Matt.

 

From: Rose, John B <[hidden email]>
Sent: Friday, 29 March 2019 3:56 AM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

I don't think the TCP buffer would be clear if there was a continuing flow of http requests during that time, whether the web server software was down, or maxed out

 

But maybe I am wrong. 

 

 


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 1:22:59 PM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

No PHP on the system at all. The web server was down for 15-20 minutes so anything in the queue should have cleared, right?

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 11:32 AM
To: "[hidden email]" <[hidden email]>
Subject: [users@httpd] Re: Apache web server devouring resources

 

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

Reply | Threaded
Open this post in threaded view
|

Re: Apache web server devouring resources

Darryl Philip Baker

Thank you for your response but if this was an ongoing issue, I would have more places to look and tools available. After rebooting the system, it just went back to normal. The network team reviewed the load balancer and firewall logs. I have reviewed the Apache log and system messages. We are reviewing activity using the Splunk logs for the network elements but still no gun smoking or not.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Muggeridge, Matt" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 1:36 PM
To: "[hidden email]" <[hidden email]>
Subject: RE: [users@httpd] Re: Apache web server devouring resources

 

You will need to do some more triaging.  Suggestions for things to investigate more deeply:  http Log files, system log files, system performance monitoring, connection statistics, source of traffic, TCP performance tuning, firewall control, protection against DOS attacks… and that’s just off the top.

 

You will need to profile the system using a myriad of tools that suit your need (e.g. tcpdump, lsof, top, netstat, ss, and a large variety of others, depending on what you learn along the way).

 

Matt.

 

From: Rose, John B <[hidden email]>
Sent: Friday, 29 March 2019 3:56 AM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

I don't think the TCP buffer would be clear if there was a continuing flow of http requests during that time, whether the web server software was down, or maxed out

 

But maybe I am wrong. 

 

 


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 1:22:59 PM
To: [hidden email]
Subject: Re: [users@httpd] Re: Apache web server devouring resources

 

No PHP on the system at all. The web server was down for 15-20 minutes so anything in the queue should have cleared, right?

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674

 

 

From: "Rose, John B" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 28, 2019 at 11:32 AM
To: "[hidden email]" <[hidden email]>
Subject: [users@httpd] Re: Apache web server devouring resources

 

Regarding the "load increasing quickly after restarting the daemons" ...

 

I do not believe just restarting the daemons clears the TCP queue. Nor does it prevent new TCP requests. If it is an attack, then the load would ramp back up immediately. That is why you have to reboot I am guessing.

Do you utilize PHP? PHP-FPM? Do you use TCP or Unix Domain sockets?

 

Are there a preponderance of http connections or PHP-FPM processes, or both?

 

If PHP-FPM do you use "static" "dynamic" or "ondemand"?


From: Darryl Philip Baker <[hidden email]>
Sent: Thursday, March 28, 2019 12:11:27 PM
To: [hidden email]
Subject: [users@httpd] Apache web server devouring resources

 

Gentlefolk,

I had an incident yesterday where the Apache web server host had a load average of over 170 and was performing very slowly. Stopping the web server did fix the issue but when I restarted the daemons the load started to increase very quickly. I ended up having to reboot the system to fix the issue. I don’t like that one bit, this is a Linux system not a Windows server. (Editorial remark: I have found that systems need reboots to fix stuff much more frequently since the adoption of systemd) I have been asked to do a root cause analysis, but I have not found anything as of yet. I am reaching out for help in this matter.

 

The system is a RHEL7 ESX VM with the Red Hat’s main line distribution of Apache 2.4 as opposed to the RHSCL version. The configuration is quite complex and a bit sensitive so I cannot share all of that. What I’m looking for is technics to look at what happened rather than being given the answer anyway.

 

Darryl Baker  (he/him/his)

Sr. System Administrator

Distributed Application Platform Services

Northwestern University

1800 Sherman Ave.

Suite 6-600 – Box #39

Evanston, IL  60201-3715

[hidden email]

(847) 467-6674