Cannot troubleshoot php-fpm with apache

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cannot troubleshoot php-fpm with apache

Nikolaos Milas
Hello,

I am running httpd-2.4.6-45.el7.centos.x86_64 with
php-fpm-7.0.22-1.el7.remi.x86_64 (on CentOS 7).

My main problem: On this httpd server I have several vhosts running, but
apparently I am facing intermittent problems with php-fpm communication
on only one of them.

Most of the sites are WordPress or Joomla Applications (running with
mysql). Everythings seems right: low load, all websites are operating
fine, except this one, but only intermittently.

The secondary problem is that I can't seem to be able to log php errors.

When the problem starts, the users receive "503 Service Unavailable"
pages for many hours/days, and in the error log I see only such errors:

    [Tue Aug 08 08:27:31.742640 2017] [proxy_fcgi:debug] [pid 29259]
    mod_proxy_fcgi.c(963): [client 180.76.15.139:30619] AH01076: url:
    fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php
    proxyname: (null) proxyport: 0
    [Tue Aug 08 08:27:31.742642 2017] [proxy_fcgi:debug] [pid 29259]
    mod_proxy_fcgi.c(970): [client 180.76.15.139:30619] AH01078: serving
    URL fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php
    [Tue Aug 08 08:27:31.742645 2017] [proxy:debug] [pid 29259]
    proxy_util.c(2203): AH00942: FCGI: has acquired connection for (*)
    [Tue Aug 08 08:27:31.742661 2017] [proxy:debug] [pid 29259]
    proxy_util.c(2256): [client 180.76.15.139:30619] AH00944: connecting
    fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php to
    127.0.0.1:9000
    [Tue Aug 08 08:27:31.742690 2017] [proxy:debug] [pid 29259]
    proxy_util.c(2422): [client 180.76.15.139:30619] AH00947: connected
    /var/webs/wwwgreekgeo/www/wp/index.php to 127.0.0.1:9000
    [Tue Aug 08 08:27:31.742739 2017] [proxy:debug] [pid 29259]
    proxy_util.c(2798): AH02824: FCGI: connection established with
    127.0.0.1:9000 (*)
    [Tue Aug 08 08:27:52.347960 2017] [proxy_fcgi:error] [pid 29259]
    (104)Connection reset by peer: [client 180.76.15.139:30619] AH01075:
    Error dispatching request to :
    [Tue Aug 08 08:27:52.348021 2017] [proxy:debug] [pid 29259]
    proxy_util.c(2218): AH00943: FCGI: has released connection for (*)

The problem is NOT resolved by restarting apache and/or php-fpm.

The config of the vhost is as follows:

    <VirtualHost *:80>
             DocumentRoot "/var/webs/wwwgreekgeo/www"

             ServerName www.greekgeo.noa.gr

             ServerAdmin [hidden email]

             DirectoryIndex index.php index.html index.htm

             php_flag log_errors on
             php_flag display_errors off
             php_value error_log /var/webs/wwwgreekgeo/log/php_error_log
    #        php_value error_reporting 32767
    #        php_value error_reporting 2147483647
             php_value error_reporting 6135

             Redirect 301 /en.index.html http://www.greekgeo.noa.gr/wp/

             RewriteEngine on

             RedirectMatch ^/index.html$ /wp/

         <Directory "/var/webs/wwwgreekgeo/www">
             Options -Indexes +FollowSymLinks
             AllowOverride All
             Require all granted
         </Directory>

             <Files ~ "^\.ht">
                Require all denied
             </Files>

         <FilesMatch \.php$>
             SetHandler "proxy:fcgi://127.0.0.1:9000"
         </FilesMatch>

         ErrorLog /var/webs/wwwgreekgeo/log/error_log
         LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
    \"%{User-agent}i\"" combined
         CustomLog /var/webs/wwwgreekgeo/log/access_log combined
         LogLevel debug

    </VirtualHost>

As you can see, I have made efforts to enable php debugging, but I still
can't see any php logging, in any php log:

    /var/log/php-fpm/error.log
    /var/log/php-fpm/www-error.log
    The file: /var/webs/wwwgreekgeo/log/php_error_log remains empty

I've also followed these directions to enable Wordpress php logging:
https://codex.wordpress.org/WP_DEBUG but I didn't get any php logging
either.

Please advise: How can I enable full php logging and troubleshoot php on
this vhost?

Also: What would you advise, if possible, on the particular problem (php
processing hanging on this vhost only without any obvious reason)?

I've googled and tried many things for several hours, but I can't find a
solution.

Thanks in advance,
Nick



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Daniel
php_ directives belong to "mod_php" if you are proxying to php-fpm
through fastcgi all related php logging and defined variables belong
in the corresponding FPM pool.

I would start by seeing what php-fpm has to say about those connections.

2017-08-08 7:47 GMT+02:00 Nikolaos Milas <[hidden email]>:

> Hello,
>
> I am running httpd-2.4.6-45.el7.centos.x86_64 with
> php-fpm-7.0.22-1.el7.remi.x86_64 (on CentOS 7).
>
> My main problem: On this httpd server I have several vhosts running, but
> apparently I am facing intermittent problems with php-fpm communication on
> only one of them.
>
> Most of the sites are WordPress or Joomla Applications (running with mysql).
> Everythings seems right: low load, all websites are operating fine, except
> this one, but only intermittently.
>
> The secondary problem is that I can't seem to be able to log php errors.
>
> When the problem starts, the users receive "503 Service Unavailable" pages
> for many hours/days, and in the error log I see only such errors:
>
>    [Tue Aug 08 08:27:31.742640 2017] [proxy_fcgi:debug] [pid 29259]
>    mod_proxy_fcgi.c(963): [client 180.76.15.139:30619] AH01076: url:
>    fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php
>    proxyname: (null) proxyport: 0
>    [Tue Aug 08 08:27:31.742642 2017] [proxy_fcgi:debug] [pid 29259]
>    mod_proxy_fcgi.c(970): [client 180.76.15.139:30619] AH01078: serving
>    URL fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php
>    [Tue Aug 08 08:27:31.742645 2017] [proxy:debug] [pid 29259]
>    proxy_util.c(2203): AH00942: FCGI: has acquired connection for (*)
>    [Tue Aug 08 08:27:31.742661 2017] [proxy:debug] [pid 29259]
>    proxy_util.c(2256): [client 180.76.15.139:30619] AH00944: connecting
>    fcgi://127.0.0.1:9000/var/webs/wwwgreekgeo/www/wp/index.php to
>    127.0.0.1:9000
>    [Tue Aug 08 08:27:31.742690 2017] [proxy:debug] [pid 29259]
>    proxy_util.c(2422): [client 180.76.15.139:30619] AH00947: connected
>    /var/webs/wwwgreekgeo/www/wp/index.php to 127.0.0.1:9000
>    [Tue Aug 08 08:27:31.742739 2017] [proxy:debug] [pid 29259]
>    proxy_util.c(2798): AH02824: FCGI: connection established with
>    127.0.0.1:9000 (*)
>    [Tue Aug 08 08:27:52.347960 2017] [proxy_fcgi:error] [pid 29259]
>    (104)Connection reset by peer: [client 180.76.15.139:30619] AH01075:
>    Error dispatching request to :
>    [Tue Aug 08 08:27:52.348021 2017] [proxy:debug] [pid 29259]
>    proxy_util.c(2218): AH00943: FCGI: has released connection for (*)
>
> The problem is NOT resolved by restarting apache and/or php-fpm.
>
> The config of the vhost is as follows:
>
>    <VirtualHost *:80>
>             DocumentRoot "/var/webs/wwwgreekgeo/www"
>
>             ServerName www.greekgeo.noa.gr
>
>             ServerAdmin [hidden email]
>
>             DirectoryIndex index.php index.html index.htm
>
>             php_flag log_errors on
>             php_flag display_errors off
>             php_value error_log /var/webs/wwwgreekgeo/log/php_error_log
>    #        php_value error_reporting 32767
>    #        php_value error_reporting 2147483647
>             php_value error_reporting 6135
>
>             Redirect 301 /en.index.html http://www.greekgeo.noa.gr/wp/
>
>             RewriteEngine on
>
>             RedirectMatch ^/index.html$ /wp/
>
>         <Directory "/var/webs/wwwgreekgeo/www">
>             Options -Indexes +FollowSymLinks
>             AllowOverride All
>             Require all granted
>         </Directory>
>
>             <Files ~ "^\.ht">
>                Require all denied
>             </Files>
>
>         <FilesMatch \.php$>
>             SetHandler "proxy:fcgi://127.0.0.1:9000"
>         </FilesMatch>
>
>         ErrorLog /var/webs/wwwgreekgeo/log/error_log
>         LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
>    \"%{User-agent}i\"" combined
>         CustomLog /var/webs/wwwgreekgeo/log/access_log combined
>         LogLevel debug
>
>    </VirtualHost>
>
> As you can see, I have made efforts to enable php debugging, but I still
> can't see any php logging, in any php log:
>
>    /var/log/php-fpm/error.log
>    /var/log/php-fpm/www-error.log
>    The file: /var/webs/wwwgreekgeo/log/php_error_log remains empty
>
> I've also followed these directions to enable Wordpress php logging:
> https://codex.wordpress.org/WP_DEBUG but I didn't get any php logging
> either.
>
> Please advise: How can I enable full php logging and troubleshoot php on
> this vhost?
>
> Also: What would you advise, if possible, on the particular problem (php
> processing hanging on this vhost only without any obvious reason)?
>
> I've googled and tried many things for several hours, but I can't find a
> solution.
>
> Thanks in advance,
> Nick
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



--
Daniel Ferradal
IT Specialist

email         dferradal at gmail.com
linkedin     es.linkedin.com/in/danielferradal

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Nikolaos Milas
On 8/8/2017 9:47 πμ, Daniel wrote:

> php_ directives belong to "mod_php" if you are proxying to php-fpm
> through fastcgi all related php logging and defined variables belong
> in the corresponding FPM pool.
>
> I would start by seeing what php-fpm has to say about those connections.

Thank you Daniel,

You mean that the php_flag and php_value settings in the vhost
configuration will not apply?
If so, how can I pass particular php_value settings on a per vhost basis?

As I wrote, the problem is that I don't see any php logging even in the
php-fpm logs for this vhost!

For your reference, I am also including the php-fpm configuration, in
case you (or anyone) can advise me on something:

    # cat /etc/php-fpm.d/www.conf

    include=/etc/php-fpm.d/*.conf

    [global]
    pid = /run/php-fpm/php-fpm.pid

    error_log = /var/log/php-fpm/error.log
    log_level = debug
    daemonize = yes

    [www]

    user = apache
    group = apache

    listen = 127.0.0.1:9000
    listen.allowed_clients = 127.0.0.1

    pm = dynamic
    pm.max_children = 50
    pm.start_servers = 2
    pm.min_spare_servers = 2
    pm.max_spare_servers = 4
    pm.process_idle_timeout = 10s;
    pm.max_requests = 500
    pm.status_path = /fpm-status

    slowlog = /var/log/php-fpm/www-slow.log
    request_slowlog_timeout = 5s
    request_terminate_timeout = 20s

    catch_workers_output = yes

    security.limit_extensions = .php .php3 .php4 .php5 .php7 .htm .html

    php_admin_value[error_log] = /var/log/php-fpm/www-error.log
    php_admin_flag[log_errors] = on
    php_value[session.save_handler] = files
    php_value[session.save_path]    = /var/lib/php/session
    php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache

    php_admin_value[post_max_size] = 40M
    php_admin_value[short_open_tag] = On

    php_value[date.timezone] = "Europe/Athens"

Thanks,
Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Daniel
yes, php values can't be defined in apache config, you define them in
the fpm pool.

IIRC the php-fpm error logs get written with the user the fpm pool has
defined, in your case "apache" should have write permissions in the
/var/log/php-fpm/ to be able to write www-error.log.

about php configurations, yes, just define them in each pool and
create one pool for each different case you need, that's what php-fpm
is all about.

2017-08-08 9:42 GMT+02:00 Nikolaos Milas <[hidden email]>:

> On 8/8/2017 9:47 πμ, Daniel wrote:
>
>> php_ directives belong to "mod_php" if you are proxying to php-fpm
>> through fastcgi all related php logging and defined variables belong
>> in the corresponding FPM pool.
>>
>> I would start by seeing what php-fpm has to say about those connections.
>
>
> Thank you Daniel,
>
> You mean that the php_flag and php_value settings in the vhost configuration
> will not apply?
> If so, how can I pass particular php_value settings on a per vhost basis?
>
> As I wrote, the problem is that I don't see any php logging even in the
> php-fpm logs for this vhost!
>
> For your reference, I am also including the php-fpm configuration, in case
> you (or anyone) can advise me on something:
>
>    # cat /etc/php-fpm.d/www.conf
>
>    include=/etc/php-fpm.d/*.conf
>
>    [global]
>    pid = /run/php-fpm/php-fpm.pid
>
>    error_log = /var/log/php-fpm/error.log
>    log_level = debug
>    daemonize = yes
>
>    [www]
>
>    user = apache
>    group = apache
>
>    listen = 127.0.0.1:9000
>    listen.allowed_clients = 127.0.0.1
>
>    pm = dynamic
>    pm.max_children = 50
>    pm.start_servers = 2
>    pm.min_spare_servers = 2
>    pm.max_spare_servers = 4
>    pm.process_idle_timeout = 10s;
>    pm.max_requests = 500
>    pm.status_path = /fpm-status
>
>    slowlog = /var/log/php-fpm/www-slow.log
>    request_slowlog_timeout = 5s
>    request_terminate_timeout = 20s
>
>    catch_workers_output = yes
>
>    security.limit_extensions = .php .php3 .php4 .php5 .php7 .htm .html
>
>    php_admin_value[error_log] = /var/log/php-fpm/www-error.log
>    php_admin_flag[log_errors] = on
>    php_value[session.save_handler] = files
>    php_value[session.save_path]    = /var/lib/php/session
>    php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache
>
>    php_admin_value[post_max_size] = 40M
>    php_admin_value[short_open_tag] = On
>
>    php_value[date.timezone] = "Europe/Athens"
>
> Thanks,
>
> Nick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



--
Daniel Ferradal
IT Specialist

email         dferradal at gmail.com
linkedin     es.linkedin.com/in/danielferradal

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Nikolaos Milas
On 8/8/2017 11:43 πμ, Daniel wrote:

> about php configurations, yes, just define them in each pool and
> create one pool for each different case you need, that's what php-fpm
> is all about.

But, with my current settings (file php-fpm.d/www.conf), php logging is
*already enabled*, so shouldn't I expect to see all php errors in the
configured file (/var/log/php-fpm/www-error.log)? But I don't, and I
can't understand why!

So, in essence, I believe I should be able to run the server with only
one pool.

Regarding the use of different pools, you mean I should create a new
file, e.g.:  /etc/php-conf.d/wwwdebug.conf  or a new section in www.conf
(like [www]) and specify (I guess I should use a different port for this
pool? - 9001 in this example) all parameters again, like:

[wwwdebug]

    user = apache
    group = apache

    listen = 127.0.0.1:9001
    listen.allowed_clients = 127.0.0.1

    pm = dynamic
    pm.max_children = 50
    pm.start_servers = 2
    pm.min_spare_servers = 2
    pm.max_spare_servers = 4
    pm.process_idle_timeout = 10s;
    pm.max_requests = 500

    slowlog = /var/log/php-fpm/wwwdebug-slow.log
    request_slowlog_timeout = 5s
    request_terminate_timeout = 20s

    catch_workers_output = yes

    security.limit_extensions = .php .php3 .php4 .php5 .php7 .htm .html

    php_admin_value[error_log] = /var/log/php-fpm/wwwdebug-error.log
    php_admin_flag[log_errors] = on
    php_admin_value[error_reporting] = 32767
    php_value[session.save_handler] = files
    php_value[session.save_path]    = /var/lib/php/session
    php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache

    php_admin_value[post_max_size] = 40M
    php_admin_value[short_open_tag] = On

    php_value[date.timezone] = "Europe/Athens"

which actually is exactly the same but sets full reporting and different
log files?

Please advise!

Thanks,
Nick


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Daniel
Like I mentioned earlier, check the permissions of the target directory

namei -mol /var/log/php-fpm/wwwdebug-error.log
or
namei -mx /var/log/php-fpm/wwwdebug-error.log

2017-08-08 11:58 GMT+02:00 Nikolaos Milas <[hidden email]>:

> On 8/8/2017 11:43 πμ, Daniel wrote:
>
>> about php configurations, yes, just define them in each pool and
>> create one pool for each different case you need, that's what php-fpm
>> is all about.
>
>
> But, with my current settings (file php-fpm.d/www.conf), php logging is
> *already enabled*, so shouldn't I expect to see all php errors in the
> configured file (/var/log/php-fpm/www-error.log)? But I don't, and I can't
> understand why!
>
> So, in essence, I believe I should be able to run the server with only one
> pool.
>
> Regarding the use of different pools, you mean I should create a new file,
> e.g.:  /etc/php-conf.d/wwwdebug.conf  or a new section in www.conf (like
> [www]) and specify (I guess I should use a different port for this pool? -
> 9001 in this example) all parameters again, like:
>
> [wwwdebug]
>
>    user = apache
>    group = apache
>
>    listen = 127.0.0.1:9001
>    listen.allowed_clients = 127.0.0.1
>
>    pm = dynamic
>    pm.max_children = 50
>    pm.start_servers = 2
>    pm.min_spare_servers = 2
>    pm.max_spare_servers = 4
>    pm.process_idle_timeout = 10s;
>    pm.max_requests = 500
>
>    slowlog = /var/log/php-fpm/wwwdebug-slow.log
>    request_slowlog_timeout = 5s
>    request_terminate_timeout = 20s
>
>    catch_workers_output = yes
>
>    security.limit_extensions = .php .php3 .php4 .php5 .php7 .htm .html
>
>    php_admin_value[error_log] = /var/log/php-fpm/wwwdebug-error.log
>    php_admin_flag[log_errors] = on
>    php_admin_value[error_reporting] = 32767
>    php_value[session.save_handler] = files
>    php_value[session.save_path]    = /var/lib/php/session
>    php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache
>
>    php_admin_value[post_max_size] = 40M
>    php_admin_value[short_open_tag] = On
>
>    php_value[date.timezone] = "Europe/Athens"
>
> which actually is exactly the same but sets full reporting and different log
> files?
>
> Please advise!
>
>
> Thanks,
> Nick
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



--
Daniel Ferradal
IT Specialist

email         dferradal at gmail.com
linkedin     es.linkedin.com/in/danielferradal

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Nikolaos Milas
On 8/8/2017 1:27 μμ, Daniel wrote:

> namei -mol /var/log/php-fpm/wwwdebug-error.log

(Actually it is www-error.log). Here you are:

    # namei -mol /var/log/php-fpm/www-error.log
    f: /var/log/php-fpm/www-error.log
    dr-xr-xr-x root   root   /
    drwxr-xr-x root   root   var
    drwxr-xr-x root   root   log
    drwxrwx--- apache root   php-fpm
    -rw-r--r-- apache apache www-error.log

Latest information about the case:

I now once more browsed the above file (namei -mol
/var/log/php-fpm/www-error.log) and I found one single PHP entry about
this website, with the following content:

----------------------------------------------------------------------------------------------
...
[08-Aug-2017 05:11:45 Europe/Athens] PHP Fatal error: Uncaught Error:
Call to undefined function get_header() in
/var/webs/wwwgreekgeo/www/wp/wp-content/themes/nirvana/404.php:10
Stack trace:
#0 {main}
   thrown in
/var/webs/wwwgreekgeo/www/wp/wp-content/themes/nirvana/404.php on line 10
[08-Aug-2017 05:11:45 Europe/Athens] PHP Fatal error:  Uncaught Error:
Call to undefined function get_header() in
/var/webs/wwwgreekgeo/www/wp/wp-content/themes/nirvana/404.php:10
Stack trace:
#0 {main}
   thrown in
/var/webs/wwwgreekgeo/www/wp/wp-content/themes/nirvana/404.php on line 10
...
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

So, even this one entry means that THERE IS (at least some) PHP logging
about this website!

But: Why there is only one entry about this website? Why there is no
other error logged? Could php-fpm somehow be "blacklisting" the website
PHP application (WordPress) due to this (and possible other) errors? Why
no debugging info is constantly logged?

Any ideas?

Thanks,
Nick


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Nikolaos Milas
On 8/8/2017 1:55 μμ, Nikolaos Milas wrote:

> Any ideas?

In case this helps, here: http://iweb.noa.gr/files/phpinfo.htm you can
find the php info page of the server.

Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cannot troubleshoot php-fpm with apache

Daniel
It is a php issue, can't tell you much about it.. php-fpm belongs to
php.net. In case nobody else can assist you here,  the php community
will probably help you better regarding php issues.

2017-08-08 13:04 GMT+02:00 Nikolaos Milas <[hidden email]>:

> On 8/8/2017 1:55 μμ, Nikolaos Milas wrote:
>
>> Any ideas?
>
>
> In case this helps, here: http://iweb.noa.gr/files/phpinfo.htm you can find
> the php info page of the server.
>
>
> Nick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



--
Daniel Ferradal
IT Specialist

email         dferradal at gmail.com
linkedin     es.linkedin.com/in/danielferradal

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Loading...