Apache rewritemap condition that will CDIR-ipmatch against returned value from the map?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache rewritemap condition that will CDIR-ipmatch against returned value from the map?

Wendell Hatcher

Hello, I have a ipfileblocklist text file that has over 10,000 ip addresses that need to be matched against. We managed to do a RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )” on the cdir ipmatch that comes back from the F5 but there are too many single rewrite cond exprs that are being called and it caused apache to slow down or lock  due to each singular condition call within the file. 


Over 60000 plus single condition calls in our separate ipblocking file. This causes a huge load on the apache server and locks the system. 

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

     RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

 


We want to load the ips into a rewritemap text or db and call them but the mapping condition on the CDIR seems to be off somewhat and doesn't work. 

Also, as the file grows do you have any recommendations on iobuffering or slow down loading the data into a map?   


Our file has a list of ips like to so. The condition will check each beginning and end with b$

142.255.85.0/17   b

65.144.129.15/1   b

 

RewriteMap ipblocklist “txt:/path/to/ipblocklist.txt”

  RewriteCond "${ipblocklist:% ({HTTP:x-etsc-client-ip} -ipmatch  ^b$)”   [NC]

  RewriteCond ${ipblocklist:%{HTTP:x-etsc-client-ip} -ipmatch}}  ^b$            [NC]

RewriteRule ^/[^/]+/Main/ad/ - [F,L]

Reply | Threaded
Open this post in threaded view
|

Re: Apache rewritemap condition that will CDIR-ipmatch against returned value from the map?

angel Hall-Coulston
Depends on not ’smarter thinking’ but 'out of the box' thinking… I can block a few hundred thousand IP’s with just THREE characters ! Not boasting but I have half of China and Russia blocked if not 3/4 with less than 30 characters… So I suggest (with the aid of the 'GREP' command) to ascertain which countries you won’t deal with and block whole IP blocks or domain suffixes…. ONE possible solution and a LOT less work :) Anyone blocked can ALWAYS email me if they are a genuine customer and ask WHY they are blocked if they feel it’s a wrong decision, 99.99% don’t because MOST bad IP requests are automated, so no loss there…. 

Angel
Scotland.

On 1 Jul 2020, at 15:33, Wendell Hatcher <[hidden email]> wrote:

Hello, I have a ipfileblocklist text file that has over 10,000 ip addresses that need to be matched against. We managed to do a RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )” on the cdir ipmatch that comes back from the F5 but there are too many single rewrite cond exprs that are being called and it caused apache to slow down or lock  due to each singular condition call within the file. 


Over 60000 plus single condition calls in our separate ipblocking file. This causes a huge load on the apache server and locks the system. 

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

     RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

 


We want to load the ips into a rewritemap text or db and call them but the mapping condition on the CDIR seems to be off somewhat and doesn't work. 

Also, as the file grows do you have any recommendations on iobuffering or slow down loading the data into a map?   


Our file has a list of ips like to so. The condition will check each beginning and end with b$

142.255.85.0/17   b

65.144.129.15/1   b

 

RewriteMap ipblocklist “txt:/path/to/ipblocklist.txt”

  RewriteCond "${ipblocklist:% ({HTTP:x-etsc-client-ip} -ipmatch  ^b$)”   [NC]

  RewriteCond ${ipblocklist:%{HTTP:x-etsc-client-ip} -ipmatch}}  ^b$            [NC]

RewriteRule ^/[^/]+/Main/ad/ - [F,L]



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apache rewritemap condition that will CDIR-ipmatch against returned value from the map?

Wendell Hatcher
Angel, you are the man what you did in your solution is exactly what we are trying to do block certain countries. I am leaning towards using Perl to loop through the bad ips list on a text file and then take the blocked ip match blocking similar to the solution in the link below.  Can you please take a look and provide me with a high-level code set of how you completed this and got it working with Apache?

https://www.ispcolohost.com/2017/02/03/keeping-amazon-ec2-crap-off-your-website/

Wendell

On Jul 1, 2020, at 1:42 PM, angel Hall-Coulston <[hidden email]> wrote:

Depends on not ’smarter thinking’ but 'out of the box' thinking… I can block a few hundred thousand IP’s with just THREE characters ! Not boasting but I have half of China and Russia blocked if not 3/4 with less than 30 characters… So I suggest (with the aid of the 'GREP' command) to ascertain which countries you won’t deal with and block whole IP blocks or domain suffixes…. ONE possible solution and a LOT less work :) Anyone blocked can ALWAYS email me if they are a genuine customer and ask WHY they are blocked if they feel it’s a wrong decision, 99.99% don’t because MOST bad IP requests are automated, so no loss there…. 

Angel
Scotland.

On 1 Jul 2020, at 15:33, Wendell Hatcher <[hidden email]> wrote:

Hello, I have a ipfileblocklist text file that has over 10,000 ip addresses that need to be matched against. We managed to do a RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )” on the cdir ipmatch that comes back from the F5 but there are too many single rewrite cond exprs that are being called and it caused apache to slow down or lock  due to each singular condition call within the file. 


Over 60000 plus single condition calls in our separate ipblocking file. This causes a huge load on the apache server and locks the system. 

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

     RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

 


We want to load the ips into a rewritemap text or db and call them but the mapping condition on the CDIR seems to be off somewhat and doesn't work. 

Also, as the file grows do you have any recommendations on iobuffering or slow down loading the data into a map?   


Our file has a list of ips like to so. The condition will check each beginning and end with b$

142.255.85.0/17   b

65.144.129.15/1   b

 

RewriteMap ipblocklist “txt:/path/to/ipblocklist.txt”

  RewriteCond "${ipblocklist:% ({HTTP:x-etsc-client-ip} -ipmatch  ^b$)”   [NC]

  RewriteCond ${ipblocklist:%{HTTP:x-etsc-client-ip} -ipmatch}}  ^b$            [NC]

RewriteRule ^/[^/]+/Main/ad/ - [F,L]


Reply | Threaded
Open this post in threaded view
|

Re: Apache rewritemap condition that will CDIR-ipmatch against returned value from the map?

angel Hall-Coulston
Hi Wendell,
I tend to use Apache configuration for blocking, a mix of hosts and also IP’s. Sometimes it’s as easy as blocking say '*.ru' as it’s highly improbable that some English company has a Russian suffix (I could be wrong of course, but if they DO they prob have a UK based website too..). You have to be careful though if you have say two genuine Russian companies you do business with (for example) to 'Require' their domain/IP address. Hostnames is sometimes better… sometimes IP ranges. A mix of both seems a good idea. 'GREP' command will come in really useful perhaps 'PIPED' with ’SORT' to sift through so many IP’s to at least first group them and so break down into smaller chunks the work. There are many sites that can help too:


Not forgetting the great (no, honestly) manual pages of Apache that you can enable through Apache configurations contained in httpd.conf and the 'extra' directory so you can access through your site(s), much quicker and easier to navigate. For example I can type 'open http::/manual' from my terminal, and I’m instantly there in the manual home index page. Hope this helps, have a nice day.

Angel
Scotland


On 1 Jul 2020, at 19:05, Wendell Hatcher <[hidden email]> wrote:

Angel, you are the man what you did in your solution is exactly what we are trying to do block certain countries. I am leaning towards using Perl to loop through the bad ips list on a text file and then take the blocked ip match blocking similar to the solution in the link below.  Can you please take a look and provide me with a high-level code set of how you completed this and got it working with Apache?

https://www.ispcolohost.com/2017/02/03/keeping-amazon-ec2-crap-off-your-website/

Wendell

On Jul 1, 2020, at 1:42 PM, angel Hall-Coulston <[hidden email]> wrote:

Depends on not ’smarter thinking’ but 'out of the box' thinking… I can block a few hundred thousand IP’s with just THREE characters ! Not boasting but I have half of China and Russia blocked if not 3/4 with less than 30 characters… So I suggest (with the aid of the 'GREP' command) to ascertain which countries you won’t deal with and block whole IP blocks or domain suffixes…. ONE possible solution and a LOT less work :) Anyone blocked can ALWAYS email me if they are a genuine customer and ask WHY they are blocked if they feel it’s a wrong decision, 99.99% don’t because MOST bad IP requests are automated, so no loss there…. 

Angel
Scotland.

On 1 Jul 2020, at 15:33, Wendell Hatcher <[hidden email]> wrote:

Hello, I have a ipfileblocklist text file that has over 10,000 ip addresses that need to be matched against. We managed to do a RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )” on the cdir ipmatch that comes back from the F5 but there are too many single rewrite cond exprs that are being called and it caused apache to slow down or lock  due to each singular condition call within the file. 


Over 60000 plus single condition calls in our separate ipblocking file. This causes a huge load on the apache server and locks the system. 

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

     RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

    RewriteCond expr  " (%{HTTP:x-etsc-client-ip} -ipmatch 'X.X.X.X' )”

 


We want to load the ips into a rewritemap text or db and call them but the mapping condition on the CDIR seems to be off somewhat and doesn't work. 

Also, as the file grows do you have any recommendations on iobuffering or slow down loading the data into a map?   


Our file has a list of ips like to so. The condition will check each beginning and end with b$

142.255.85.0/17   b

65.144.129.15/1   b

 

RewriteMap ipblocklist “txt:/path/to/ipblocklist.txt”

  RewriteCond "${ipblocklist:% ({HTTP:x-etsc-client-ip} -ipmatch  ^b$)”   [NC]

  RewriteCond ${ipblocklist:%{HTTP:x-etsc-client-ip} -ipmatch}}  ^b$            [NC]

RewriteRule ^/[^/]+/Main/ad/ - [F,L]




signature.asc (849 bytes) Download Attachment