Integration tests running on Docker

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Integration tests running on Docker

Luca Toscano
Hi everybody,

I spent some time reading the previous discussions around the concept
of "CI" and from what I gather, it seems that we didn't reach an
agreement about how to proceed and who is working on it (but I might
be wrong, in case apologies!). From what I can see, there are two
possible working fronts:

1) Simplify the usage of the current perl testing suite, adding
docs/tests/etc.. Some people expressed the desire for a more friendly
framework, especially when adding new tests.
2) Run the testing framework automatically on different environments
to spot anomalies/bugs/etc.. in a timely manner.

I am a bit ignorant about how to run a proper CI but I created a
little prototype of a Dockerfile able to bootstrap a testing
environment on Debian 10 (Buster) and run the perl test suite:

https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile

The above is only an example, it is missing a lot of things and some
follow up work is needed. But with the following commands, on my
laptop I was able to create a docker image and run the test suite:

docker build .
docker run $id-of-the-image make check

Some thoughts:

1) The above Dockerfile is really handy since I can easily switch
between Debian versions (Jessie/Stretch/Buster to name the last three)
and run the test suite with different package versions (openssl,
nghttp2, etc..). It should be also easy to create Dockerfiles for
other OS/environments, and run make check in a similar way.
1-bis) Testing on Windows would still need to be solved, Docker
probably it is not the right solution but we could find something else
to integrate for this specific use case.
2) Docker also offers a way to open a bash shell in interactive mode,
so it could be easy to run tests on a certain platform when somebody
reports a problem. Or make sure that a new set of tests runs correctly
everywhere.
3) Another use case could be to create a Dockerfile that pulls a
specific new release of httpd, installing it and running the test
suite on multiple platforms.
4) The same Docker image could also run tests suites like
https://github.com/icing/mod_h2/tree/master/test/e2e (that is really
nice, I suggest to check it if you haven't done it) to run HTTP/2
tests as well.
5) We could even think about having daily docker image builds that
take a snapshot of trunk/2.4.x and run the test suites, reporting back
any problem.

Does the above make sense or it is completely out of scope for our
project? In the former case I could spend some time on figuring out
how to create what I proposed above, otherwise I'll stop and drop the
idea :)

Last but not the least, this would be a way to help us testing changes
in a more automated way, not a replacement of community based testing
and bug reports (stating the obvious for an Apache project but better
safe than sorry :).

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Stefan Eissing
+1

I like this a lot. Having a maintained docker setup at hand to run tests in controlled setups is super good.

If you get this going for the main test suite, I will be more than happy to contribute with setups for the mod_h2 and mod_md test suites. Those are part of the github repros only.

Cheers, Stefan

> Am 25.09.2019 um 11:52 schrieb Luca Toscano <[hidden email]>:
>
> Does the above make sense or it is completely out of scope for our
> project? In the former case I could spend some time on figuring out
> how to create what I proposed above, otherwise I'll stop and drop the
> idea :)
>

Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Daniel Ferradal
+1's!

Thanks Luca, looking great!

El mié., 25 sept. 2019 a las 12:44, Stefan Eissing
(<[hidden email]>) escribió:

>
> +1
>
> I like this a lot. Having a maintained docker setup at hand to run tests in controlled setups is super good.
>
> If you get this going for the main test suite, I will be more than happy to contribute with setups for the mod_h2 and mod_md test suites. Those are part of the github repros only.
>
> Cheers, Stefan
>
> > Am 25.09.2019 um 11:52 schrieb Luca Toscano <[hidden email]>:
> >
> > Does the above make sense or it is completely out of scope for our
> > project? In the former case I could spend some time on figuring out
> > how to create what I proposed above, otherwise I'll stop and drop the
> > idea :)
> >
>


--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Mads Toftum
In reply to this post by Luca Toscano
On Wed, Sep 25, 2019 at 11:52:52AM +0200, Luca Toscano wrote:
> Does the above make sense or it is completely out of scope for our
> project? In the former case I could spend some time on figuring out
> how to create what I proposed above, otherwise I'll stop and drop the
> idea :)
>
To me it seems useful enough to put in the repo with the rest of the
test code or at least documenting somewhere on http://httpd.apache.org/dev/

If rolled into httpd test framework, it might make more sense to have a
make docker (or similar) that spins up an image with the source copied
in.

vh

Mads Toftum
--
http://flickr.com/photos/q42/
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
In reply to this post by Luca Toscano
Hi Daniel,

Il giorno mer 25 set 2019 alle ore 13:24 Daniel Ruggeri
<[hidden email]> ha scritto:
>
> I'm a big fan of this idea - it's how I do testing :-)
>
> Unfortunately, I ran into a few issues trying to build and test with Debian maintained dependencies and only ended up with one working test suite: build httpd and all dependencies from sources. But the idea as a whole is sound - we can test against any Linux-based OS distribution which is available in the Docker hub. It could also serve as a nice hint to those that would like to deploy httpd from sources (or even package maintainers) by offering a working recipe that tests clean.
>
> I also raised an inquiry to Infra (but admittedly never followed up) to understand what we'd need to do to have Docker deployed to our CI VM. I understand they use Puppet to manage said VMs so I was hoping to figure out if this would be permissible and then which scripts need to be updated.

Good point about the Debian maintained dependencies, I didn't think
about it. As first step we could simply assume that they don't exist
and build all the deps from source :)
Do you have any special ./configure arg list that you think it would
be worth to use? Or maybe a collection of them, I have seen problems
over time rising from using static vs dso modules, enabling only a
subset of modules, etc.. For the first iteration maybe something that
involves --enable-modules=reallyall is enough?

Thanks!

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
In reply to this post by Mads Toftum
Hi,

Il giorno mer 25 set 2019 alle ore 13:18 Mads Toftum <[hidden email]>
ha scritto:

>
> On Wed, Sep 25, 2019 at 11:52:52AM +0200, Luca Toscano wrote:
> > Does the above make sense or it is completely out of scope for our
> > project? In the former case I could spend some time on figuring out
> > how to create what I proposed above, otherwise I'll stop and drop the
> > idea :)
> >
> To me it seems useful enough to put in the repo with the rest of the
> test code or at least documenting somewhere on http://httpd.apache.org/dev/
> If rolled into httpd test framework, it might make more sense to have a
> make docker (or similar) that spins up an image with the source copied
> in.

Definitely, I started a separate gh project only to show the
prototype, once we have something that works and that we like it might
be better to move it elsewhere.

"make docker" could also be cool to add!

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
In reply to this post by Luca Toscano
Follow up after some work in
https://github.com/elukey/httpd_integration_testing:

* Use only one Docker image for ubuntu/debian
* Created two use cases: code snapshot testing (latest trunk and
2.4.x) vs release candidate testing
* Still some issues in installing perl deps on older Debian distro
(like Stretch), but the perl suite seems to run fine in both use cases
for Debian Buster.
* Some issues while running the HTTP/2 test suite, will follow up with Stefan.
* I'll try to add a Docker image for CentOS.

Please open pull requests if you have ideas/comments/suggestions/etc.. :)

Thanks!

Luca

Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano
<[hidden email]> ha scritto:

>
> Hi everybody,
>
> I spent some time reading the previous discussions around the concept
> of "CI" and from what I gather, it seems that we didn't reach an
> agreement about how to proceed and who is working on it (but I might
> be wrong, in case apologies!). From what I can see, there are two
> possible working fronts:
>
> 1) Simplify the usage of the current perl testing suite, adding
> docs/tests/etc.. Some people expressed the desire for a more friendly
> framework, especially when adding new tests.
> 2) Run the testing framework automatically on different environments
> to spot anomalies/bugs/etc.. in a timely manner.
>
> I am a bit ignorant about how to run a proper CI but I created a
> little prototype of a Dockerfile able to bootstrap a testing
> environment on Debian 10 (Buster) and run the perl test suite:
>
> https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile
>
> The above is only an example, it is missing a lot of things and some
> follow up work is needed. But with the following commands, on my
> laptop I was able to create a docker image and run the test suite:
>
> docker build .
> docker run $id-of-the-image make check
>
> Some thoughts:
>
> 1) The above Dockerfile is really handy since I can easily switch
> between Debian versions (Jessie/Stretch/Buster to name the last three)
> and run the test suite with different package versions (openssl,
> nghttp2, etc..). It should be also easy to create Dockerfiles for
> other OS/environments, and run make check in a similar way.
> 1-bis) Testing on Windows would still need to be solved, Docker
> probably it is not the right solution but we could find something else
> to integrate for this specific use case.
> 2) Docker also offers a way to open a bash shell in interactive mode,
> so it could be easy to run tests on a certain platform when somebody
> reports a problem. Or make sure that a new set of tests runs correctly
> everywhere.
> 3) Another use case could be to create a Dockerfile that pulls a
> specific new release of httpd, installing it and running the test
> suite on multiple platforms.
> 4) The same Docker image could also run tests suites like
> https://github.com/icing/mod_h2/tree/master/test/e2e (that is really
> nice, I suggest to check it if you haven't done it) to run HTTP/2
> tests as well.
> 5) We could even think about having daily docker image builds that
> take a snapshot of trunk/2.4.x and run the test suites, reporting back
> any problem.
>
> Does the above make sense or it is completely out of scope for our
> project? In the former case I could spend some time on figuring out
> how to create what I proposed above, otherwise I'll stop and drop the
> idea :)
>
> Last but not the least, this would be a way to help us testing changes
> in a more automated way, not a replacement of community based testing
> and bug reports (stating the obvious for an Apache project but better
> safe than sorry :).
>
> Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
Opened an issue to Infra to enable Travis support in github:
https://issues.apache.org/jira/browse/INFRA-19325

Luca

Il giorno sab 28 set 2019 alle ore 09:39 Luca Toscano
<[hidden email]> ha scritto:

>
> Follow up after some work in
> https://github.com/elukey/httpd_integration_testing:
>
> * Use only one Docker image for ubuntu/debian
> * Created two use cases: code snapshot testing (latest trunk and
> 2.4.x) vs release candidate testing
> * Still some issues in installing perl deps on older Debian distro
> (like Stretch), but the perl suite seems to run fine in both use cases
> for Debian Buster.
> * Some issues while running the HTTP/2 test suite, will follow up with Stefan.
> * I'll try to add a Docker image for CentOS.
>
> Please open pull requests if you have ideas/comments/suggestions/etc.. :)
>
> Thanks!
>
> Luca
>
> Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano
> <[hidden email]> ha scritto:
> >
> > Hi everybody,
> >
> > I spent some time reading the previous discussions around the concept
> > of "CI" and from what I gather, it seems that we didn't reach an
> > agreement about how to proceed and who is working on it (but I might
> > be wrong, in case apologies!). From what I can see, there are two
> > possible working fronts:
> >
> > 1) Simplify the usage of the current perl testing suite, adding
> > docs/tests/etc.. Some people expressed the desire for a more friendly
> > framework, especially when adding new tests.
> > 2) Run the testing framework automatically on different environments
> > to spot anomalies/bugs/etc.. in a timely manner.
> >
> > I am a bit ignorant about how to run a proper CI but I created a
> > little prototype of a Dockerfile able to bootstrap a testing
> > environment on Debian 10 (Buster) and run the perl test suite:
> >
> > https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile
> >
> > The above is only an example, it is missing a lot of things and some
> > follow up work is needed. But with the following commands, on my
> > laptop I was able to create a docker image and run the test suite:
> >
> > docker build .
> > docker run $id-of-the-image make check
> >
> > Some thoughts:
> >
> > 1) The above Dockerfile is really handy since I can easily switch
> > between Debian versions (Jessie/Stretch/Buster to name the last three)
> > and run the test suite with different package versions (openssl,
> > nghttp2, etc..). It should be also easy to create Dockerfiles for
> > other OS/environments, and run make check in a similar way.
> > 1-bis) Testing on Windows would still need to be solved, Docker
> > probably it is not the right solution but we could find something else
> > to integrate for this specific use case.
> > 2) Docker also offers a way to open a bash shell in interactive mode,
> > so it could be easy to run tests on a certain platform when somebody
> > reports a problem. Or make sure that a new set of tests runs correctly
> > everywhere.
> > 3) Another use case could be to create a Dockerfile that pulls a
> > specific new release of httpd, installing it and running the test
> > suite on multiple platforms.
> > 4) The same Docker image could also run tests suites like
> > https://github.com/icing/mod_h2/tree/master/test/e2e (that is really
> > nice, I suggest to check it if you haven't done it) to run HTTP/2
> > tests as well.
> > 5) We could even think about having daily docker image builds that
> > take a snapshot of trunk/2.4.x and run the test suites, reporting back
> > any problem.
> >
> > Does the above make sense or it is completely out of scope for our
> > project? In the former case I could spend some time on figuring out
> > how to create what I proposed above, otherwise I'll stop and drop the
> > idea :)
> >
> > Last but not the least, this would be a way to help us testing changes
> > in a more automated way, not a replacement of community based testing
> > and bug reports (stating the obvious for an Apache project but better
> > safe than sorry :).
> >
> > Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
Some updates:

- We have support for httpd in travis - https://travis-ci.org/apache/httpd
- In order to configure automatic builds, a travis.yaml file is needed
in the base path of the repository, in every branch that needs to be
tested IIUC.
- My idea is currently to add in trunk a travis.yaml initial
configuration file, that builds a Docker image like [1] and runs the
perl test suite.
- Building the Docker image in [1] takes quite a while now (between 9
and 13 mins on my laptop, depending also on network bandwidth etc..)
so it will need more time before it could be as fast as we need, but
we have to start from somewhere :)
- In the travis.yaml file we'd need to put config options about what
Docker image to build and with what parameters. Ideally I'd like to
store the Docker images in the httpd-framework repository, and
reference them in the Travis config of the httpd branches, but not
sure if it will be possible. Worst case scenario we'll need to add the
Docker image in each httpd branch that we want to test (possibly in a
dedicated dir, like "docker" or "testing".

If you are strongly against the above or have more suggestions and
knowledge about Travis please let me know!

Luca


[1] https://github.com/elukey/httpd_integration_testing/blob/master/docker/code-testing/debian/Dockerfile

Il giorno mar 22 ott 2019 alle ore 22:04 Luca Toscano
<[hidden email]> ha scritto:

>
> Opened an issue to Infra to enable Travis support in github:
> https://issues.apache.org/jira/browse/INFRA-19325
>
> Luca
>
> Il giorno sab 28 set 2019 alle ore 09:39 Luca Toscano
> <[hidden email]> ha scritto:
> >
> > Follow up after some work in
> > https://github.com/elukey/httpd_integration_testing:
> >
> > * Use only one Docker image for ubuntu/debian
> > * Created two use cases: code snapshot testing (latest trunk and
> > 2.4.x) vs release candidate testing
> > * Still some issues in installing perl deps on older Debian distro
> > (like Stretch), but the perl suite seems to run fine in both use cases
> > for Debian Buster.
> > * Some issues while running the HTTP/2 test suite, will follow up with Stefan.
> > * I'll try to add a Docker image for CentOS.
> >
> > Please open pull requests if you have ideas/comments/suggestions/etc.. :)
> >
> > Thanks!
> >
> > Luca
> >
> > Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano
> > <[hidden email]> ha scritto:
> > >
> > > Hi everybody,
> > >
> > > I spent some time reading the previous discussions around the concept
> > > of "CI" and from what I gather, it seems that we didn't reach an
> > > agreement about how to proceed and who is working on it (but I might
> > > be wrong, in case apologies!). From what I can see, there are two
> > > possible working fronts:
> > >
> > > 1) Simplify the usage of the current perl testing suite, adding
> > > docs/tests/etc.. Some people expressed the desire for a more friendly
> > > framework, especially when adding new tests.
> > > 2) Run the testing framework automatically on different environments
> > > to spot anomalies/bugs/etc.. in a timely manner.
> > >
> > > I am a bit ignorant about how to run a proper CI but I created a
> > > little prototype of a Dockerfile able to bootstrap a testing
> > > environment on Debian 10 (Buster) and run the perl test suite:
> > >
> > > https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile
> > >
> > > The above is only an example, it is missing a lot of things and some
> > > follow up work is needed. But with the following commands, on my
> > > laptop I was able to create a docker image and run the test suite:
> > >
> > > docker build .
> > > docker run $id-of-the-image make check
> > >
> > > Some thoughts:
> > >
> > > 1) The above Dockerfile is really handy since I can easily switch
> > > between Debian versions (Jessie/Stretch/Buster to name the last three)
> > > and run the test suite with different package versions (openssl,
> > > nghttp2, etc..). It should be also easy to create Dockerfiles for
> > > other OS/environments, and run make check in a similar way.
> > > 1-bis) Testing on Windows would still need to be solved, Docker
> > > probably it is not the right solution but we could find something else
> > > to integrate for this specific use case.
> > > 2) Docker also offers a way to open a bash shell in interactive mode,
> > > so it could be easy to run tests on a certain platform when somebody
> > > reports a problem. Or make sure that a new set of tests runs correctly
> > > everywhere.
> > > 3) Another use case could be to create a Dockerfile that pulls a
> > > specific new release of httpd, installing it and running the test
> > > suite on multiple platforms.
> > > 4) The same Docker image could also run tests suites like
> > > https://github.com/icing/mod_h2/tree/master/test/e2e (that is really
> > > nice, I suggest to check it if you haven't done it) to run HTTP/2
> > > tests as well.
> > > 5) We could even think about having daily docker image builds that
> > > take a snapshot of trunk/2.4.x and run the test suites, reporting back
> > > any problem.
> > >
> > > Does the above make sense or it is completely out of scope for our
> > > project? In the former case I could spend some time on figuring out
> > > how to create what I proposed above, otherwise I'll stop and drop the
> > > idea :)
> > >
> > > Last but not the least, this would be a way to help us testing changes
> > > in a more automated way, not a replacement of community based testing
> > > and bug reports (stating the obvious for an Apache project but better
> > > safe than sorry :).
> > >
> > > Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Joe Orton
On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote:
> Some updates:
>
> - We have support for httpd in travis - https://travis-ci.org/apache/httpd

Nice, thanks Luca & infra!

> - In order to configure automatic builds, a travis.yaml file is needed
> in the base path of the repository, in every branch that needs to be
> tested IIUC.
> - My idea is currently to add in trunk a travis.yaml initial
> configuration file, that builds a Docker image like [1] and runs the
> perl test suite.
>
> - Building the Docker image in [1] takes quite a while now (between 9
> and 13 mins on my laptop, depending also on network bandwidth etc..)
> so it will need more time before it could be as fast as we need, but
> we have to start from somewhere :)
> - In the travis.yaml file we'd need to put config options about what
> Docker image to build and with what parameters. Ideally I'd like to
> store the Docker images in the httpd-framework repository, and

Do you mean Dockerfiles rather than images here?

> reference them in the Travis config of the httpd branches, but not
> sure if it will be possible. Worst case scenario we'll need to add the
> Docker image in each httpd branch that we want to test (possibly in a
> dedicated dir, like "docker" or "testing".

I expected two goals for testing:

1. configure, build, and test using a variety of different configure
options in the standard Ubuntu environment which Travis provides (enable
different MPMs, different module sets, different
--enable-foo/--with-foo, different gcc versions etc etc)

2. configure build and test in a variety of OS environments - this would
make sense using container images.

I don't have much experience testing inside containers from Travis, but
if we can do both (1) and (2) inside containers that might make sense?  
Or can we do both separately?  If you have a .travis.yml which works
then I'd say go commit it and we can work from there.

Regards, Joe

> If you are strongly against the above or have more suggestions and
> knowledge about Travis please let me know!


Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
Il giorno mar 29 ott 2019 alle ore 12:23 Joe Orton <[hidden email]>
ha scritto:

>
> On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote:
> > Some updates:
> >
> > - We have support for httpd in travis - https://travis-ci.org/apache/httpd
>
> Nice, thanks Luca & infra!
>
> > - In order to configure automatic builds, a travis.yaml file is needed
> > in the base path of the repository, in every branch that needs to be
> > tested IIUC.
> > - My idea is currently to add in trunk a travis.yaml initial
> > configuration file, that builds a Docker image like [1] and runs the
> > perl test suite.
> >
> > - Building the Docker image in [1] takes quite a while now (between 9
> > and 13 mins on my laptop, depending also on network bandwidth etc..)
> > so it will need more time before it could be as fast as we need, but
> > we have to start from somewhere :)
> > - In the travis.yaml file we'd need to put config options about what
> > Docker image to build and with what parameters. Ideally I'd like to
> > store the Docker images in the httpd-framework repository, and
>
> Do you mean Dockerfiles rather than images here?

Correct yes, my bad!

>
> > reference them in the Travis config of the httpd branches, but not
> > sure if it will be possible. Worst case scenario we'll need to add the
> > Docker image in each httpd branch that we want to test (possibly in a
> > dedicated dir, like "docker" or "testing".
>
> I expected two goals for testing:
>
> 1. configure, build, and test using a variety of different configure
> options in the standard Ubuntu environment which Travis provides (enable
> different MPMs, different module sets, different
> --enable-foo/--with-foo, different gcc versions etc etc)

This is also what I was trying to understand in these days, namely if
travis could build and test directly without Docker.

> 2. configure build and test in a variety of OS environments - this would
> make sense using container images.
>
> I don't have much experience testing inside containers from Travis, but
> if we can do both (1) and (2) inside containers that might make sense?
> Or can we do both separately?  If you have a .travis.yml which works
> then I'd say go commit it and we can work from there.

I have currently a Docker file that works for Debian/Ubuntu (and
another one for CentOs), and IIUC the same could also be done for
other OSes (so having different Docker files for OSX and Win, since
Travis seems to support them). The main problem is that it currently
takes minutes to build the Docker images, and only after that tests
will be able to run. I guess that this problem will be the same if we
run directly tests in Travis, but we can give it a try. Another
downside about Travis, in my opinion, is that testing locally seems to
be more difficult that building Docker images (but I admit my
ignorance so it might be different), this is why I preferred Docker in
the first place.

> Or can we do both separately?  If you have a .travis.yml which works
> then I'd say go commit it and we can work from there.

I currently don't have a working travis yaml, but I'll come up with
something during the next days. What I am currently trying to
understand is if the Docker files could be stored elsewhere (like in
the httpd-framework repo) and downloaded in Travis on demand, to have
only one the Travis config in the httpd branches (as opposed to also
having Docker files etc..).

Let me know your suggestions/thoughts :)

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
Hi Daniel,

thanks for following up!

Il giorno mer 30 ott 2019 alle ore 13:05 Daniel Ruggeri
<[hidden email]> ha scritto:

>
>
>
> On October 29, 2019 8:12:25 AM CDT, Luca Toscano <[hidden email]> wrote:
> >Il giorno mar 29 ott 2019 alle ore 12:23 Joe Orton <[hidden email]>
> >ha scritto:
> >>
> >> On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote:
> >> > Some updates:
> >> >
> >> > - We have support for httpd in travis -
> >https://travis-ci.org/apache/httpd
> >>
> >> Nice, thanks Luca & infra!
> >>
> >> > - In order to configure automatic builds, a travis.yaml file is
> >needed
> >> > in the base path of the repository, in every branch that needs to
> >be
> >> > tested IIUC.
> >> > - My idea is currently to add in trunk a travis.yaml initial
> >> > configuration file, that builds a Docker image like [1] and runs
> >the
> >> > perl test suite.
> >> >
> >> > - Building the Docker image in [1] takes quite a while now (between
> >9
> >> > and 13 mins on my laptop, depending also on network bandwidth
> >etc..)
> >> > so it will need more time before it could be as fast as we need,
> >but
> >> > we have to start from somewhere :)
> >> > - In the travis.yaml file we'd need to put config options about
> >what
> >> > Docker image to build and with what parameters. Ideally I'd like to
> >> > store the Docker images in the httpd-framework repository, and
> >>
> >> Do you mean Dockerfiles rather than images here?
> >
> >Correct yes, my bad!
> >
> >>
> >> > reference them in the Travis config of the httpd branches, but not
> >> > sure if it will be possible. Worst case scenario we'll need to add
> >the
> >> > Docker image in each httpd branch that we want to test (possibly in
> >a
> >> > dedicated dir, like "docker" or "testing".
> >>
> >> I expected two goals for testing:
> >>
> >> 1. configure, build, and test using a variety of different configure
> >> options in the standard Ubuntu environment which Travis provides
> >(enable
> >> different MPMs, different module sets, different
> >> --enable-foo/--with-foo, different gcc versions etc etc)
> >
> >This is also what I was trying to understand in these days, namely if
> >travis could build and test directly without Docker.
> >
> >> 2. configure build and test in a variety of OS environments - this
> >would
> >> make sense using container images.
> >>
> >> I don't have much experience testing inside containers from Travis,
> >but
> >> if we can do both (1) and (2) inside containers that might make
> >sense?
> >> Or can we do both separately?  If you have a .travis.yml which works
> >> then I'd say go commit it and we can work from there.
> >
> >I have currently a Docker file that works for Debian/Ubuntu (and
> >another one for CentOs), and IIUC the same could also be done for
> >other OSes (so having different Docker files for OSX and Win, since
> >Travis seems to support them). The main problem is that it currently
> >takes minutes to build the Docker images, and only after that tests
> >will be able to run. I guess that this problem will be the same if we
> >run directly tests in Travis, but we can give it a try. Another
> >downside about Travis, in my opinion, is that testing locally seems to
> >be more difficult that building Docker images (but I admit my
> >ignorance so it might be different), this is why I preferred Docker in
> >the first place.
>
> This very closely matches my own testing setup. I *really* like this. Thank you for driving this!
>
> >
> >> Or can we do both separately?  If you have a .travis.yml which works
> >> then I'd say go commit it and we can work from there.
> >
> >I currently don't have a working travis yaml, but I'll come up with
> >something during the next days. What I am currently trying to
> >understand is if the Docker files could be stored elsewhere (like in
> >the httpd-framework repo) and downloaded in Travis on demand, to have
> >only one the Travis config in the httpd branches (as opposed to also
> >having Docker files etc..).
> >
> >Let me know your suggestions/thoughts :)
> >
> >Luca
>
> The most I know about Travis is how to spell it, but...
> What if the various Dockerfile-based testing environments were treated as a separate "CI-able" item? That is, what if we monitored for upstream changes then proactively rebuilt and pushed the testing images as "httpdtest-centos-7" and "httpdtest-ubuntu-trusty", etc? That way, the testing image is built/ready so the $work needed on check-in is just to build and test httpd.
>
> A significant downside is that these discreet test "enviroments" will all have their own lifecycles and maintenance burden. I found on my Debian-based Docker test environment that subtle upstream changes caused unexpected breakage. This was pretty disruptive to workflow because I had to get out of my "httpd tester" hat and don my "Debian administrator" hat in the middle of testing whatever I was working on to resolve the breakage. With that in mind, I view the pre-build mechanism as a way to avoid breakages during a testing cycle.

I like the idea, just asked in the task to infra if something like the
following is feasible:

- have a separate github repo for the Dockerfiles (that could even be
the one of the test suite replicated in github in theory)
- after each commit to that repo, an automatic Docker build process is
launched to create base test images in Dockerhub (how to handle
authentication etc..? Open question)
- Add travis config to each httpd branch that we need to test, that
simply pulls the latest images from Dockerhub, checks out the httpd
git commit that is under test, builds it with some ./configure and
runs the perl test suite. This step would be great to be run on
different platforms.

I took a look to Jira and other Apache projects in Github, and didn't
find anything similar to our use case (even if I am pretty sure we are
not the first ones :).

An alternative could be to have something like the following:

- Travis used directly (as Joe mentioned) on multiple platforms to
build httpd after each commit, maybe with different configure options
etc.. Apache Arrow uses the Travis matrix functionality in a similar
way https://github.com/apache/arrow/blob/master/.travis.yml
- add the "make docker platform-x" functionality (as previously
suggested) to the test suite's repo, and have people to run manually
docker tests when needed for multiple platforms.

So basically extending buildbot using Travis, and enrich our testing tooling.

Anyway, I'll wait a bit for infra and keep going with my research, and
as test I'll try to push a Travis config to trunk during the next days
(to have an idea about wait times etc..).

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Joe Orton
Here is a proof of concept -

https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml

This runs using the full test suite using a few different configurations
and also does builds with -Werror and maintainer-mode -

https://travis-ci.org/notroj/httpd/builds/607213820

...this should be very easy to extend with more configurations.

Can we merge the docker way with this kind of matrix type travis
configuration?

Regards, Joe

Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
Hi Joe,

Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <[hidden email]>
ha scritto:

>
> Here is a proof of concept -
>
> https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
>
> This runs using the full test suite using a few different configurations
> and also does builds with -Werror and maintainer-mode -
>
> https://travis-ci.org/notroj/httpd/builds/607213820
>
> ...this should be very easy to extend with more configurations.
>
> Can we merge the docker way with this kind of matrix type travis
> configuration?
>

The proof of concept looks great, I wanted to test something similar
especially to get timings. It looks like a single test takes from 2 to
5 minutes maximum, that is really impressive, I thought it would have
been more. With Docker in theory we could use the os matrix, but the
documentation doesn't really suggest any best practice. My idea was to
pull different docker images (previously built and uploaded to Docker
Hub), and depending on the os in the matrix run the correct docker
command in the "script" section (to get an idea
https://github.com/apache/arrow/blob/master/.travis.yml, even if they
don't use docker).

While I research though it might be good to follow your solution and
just have a simple Travis file for say ubuntu and windows. Even if it
takes 30 mins to complete, it would be a good feedback for anybody
committing code to trunk/2.4.x. And it could be possible that this
solution is good enough for our use cases, only testing will tell us!
So feel free to commit your Travis config to trunk, maybe following
the Apache Arrow example? If so others, more experienced with Windows,
might be able to chime in and add the missing configuration.

Thanks!

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Stefan Eissing
In reply to this post by Joe Orton
Nice work!

Some weeks ago, I played around with travis to get the mod_h2 tests running, but all OS platforms offered included a too old Apache httpd version. Which meant I would need to drag in a recent httpd as well. I got distracted before doing so and this got put back on the useful-things-to-do-list.

Maybe I should fork your fork and add these tests there...

Do you intend to PR this back to httpd one day?

Cheers, Stefan

> Am 04.11.2019 um 17:16 schrieb Joe Orton <[hidden email]>:
>
> Here is a proof of concept -
>
> https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
>
> This runs using the full test suite using a few different configurations
> and also does builds with -Werror and maintainer-mode -
>
> https://travis-ci.org/notroj/httpd/builds/607213820
>
> ...this should be very easy to extend with more configurations.
>
> Can we merge the docker way with this kind of matrix type travis
> configuration?
>
> Regards, Joe
>

Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
In reply to this post by Luca Toscano
Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano
<[hidden email]> ha scritto:

>
> Hi Joe,
>
> Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <[hidden email]>
> ha scritto:
> >
> > Here is a proof of concept -
> >
> > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
> >
> > This runs using the full test suite using a few different configurations
> > and also does builds with -Werror and maintainer-mode -
> >
> > https://travis-ci.org/notroj/httpd/builds/607213820
> >
> > ...this should be very easy to extend with more configurations.
> >
> > Can we merge the docker way with this kind of matrix type travis
> > configuration?
> >
>
> The proof of concept looks great, I wanted to test something similar
> especially to get timings. It looks like a single test takes from 2 to
> 5 minutes maximum, that is really impressive, I thought it would have
> been more. With Docker in theory we could use the os matrix, but the
> documentation doesn't really suggest any best practice. My idea was to
> pull different docker images (previously built and uploaded to Docker
> Hub), and depending on the os in the matrix run the correct docker
> command in the "script" section (to get an idea
> https://github.com/apache/arrow/blob/master/.travis.yml, even if they
> don't use docker).
>
> While I research though it might be good to follow your solution and
> just have a simple Travis file for say ubuntu and windows. Even if it
> takes 30 mins to complete, it would be a good feedback for anybody
> committing code to trunk/2.4.x. And it could be possible that this
> solution is good enough for our use cases, only testing will tell us!
> So feel free to commit your Travis config to trunk, maybe following
> the Apache Arrow example? If so others, more experienced with Windows,
> might be able to chime in and add the missing configuration.

This is my idea:

https://github.com/elukey/httpd/blob/trunk/.travis.yml

https://travis-ci.org/elukey/httpd/builds/607536348

Basically what Joe did, but following what the Apache Arrow Project
did. There is a little bit of repetition, but in theory with this
config people are free to add tests for other OSes (osx, windows) and
I will probably be able to add custom ones with Docker to see the
difference in execution timings etc.. (Apache Arrow's test do use
docker, I didn't see it a first).

Luca
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Joe Orton
On Tue, Nov 05, 2019 at 10:06:23AM +0100, Luca Toscano wrote:
> Basically what Joe did, but following what the Apache Arrow Project
> did. There is a little bit of repetition, but in theory with this
> config people are free to add tests for other OSes (osx, windows) and
> I will probably be able to add custom ones with Docker to see the
> difference in execution timings etc.. (Apache Arrow's test do use
> docker, I didn't see it a first).

I haven't tried to follow all of what the Arrow .yml is doing (too much
Java ;) but moving some of the logic into the repo itself out of
.travis.yml seems like a good idea.  e.g. moving all of script: into
something like ./test/travis_script_linux.sh or something for the Linux
based tests, does that make sense?

This is working now -

https://travis-ci.org/apache/httpd/builds

and I guess it will work for forks and PRs too, haven't tried yet.

Anybody feel free to extend, play, hack, whatever with the travis.yml, I
am no expert so may be doing something non-standard/wrong, feel free to
correct/improve ;)

The jobs are quite quick but the queue time for the "apache" repos is
long (10+ minutes) before they get started.  I guess it is doing
notifications to committers directly, but we won't find out until a job
fails?  Maybe let's keep experimenting with this a bit longer and then
start sending mail to dev@ for state changes (exactly as buildbot does)
when it seems stable?

Regards, Joe

Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Luca Toscano
In reply to this post by Luca Toscano
Il giorno mar 5 nov 2019 alle ore 10:06 Luca Toscano
<[hidden email]> ha scritto:

>
> Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano
> <[hidden email]> ha scritto:
> >
> > Hi Joe,
> >
> > Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <[hidden email]>
> > ha scritto:
> > >
> > > Here is a proof of concept -
> > >
> > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
> > >
> > > This runs using the full test suite using a few different configurations
> > > and also does builds with -Werror and maintainer-mode -
> > >
> > > https://travis-ci.org/notroj/httpd/builds/607213820
> > >
> > > ...this should be very easy to extend with more configurations.
> > >
> > > Can we merge the docker way with this kind of matrix type travis
> > > configuration?
> > >
> >
> > The proof of concept looks great, I wanted to test something similar
> > especially to get timings. It looks like a single test takes from 2 to
> > 5 minutes maximum, that is really impressive, I thought it would have
> > been more. With Docker in theory we could use the os matrix, but the
> > documentation doesn't really suggest any best practice. My idea was to
> > pull different docker images (previously built and uploaded to Docker
> > Hub), and depending on the os in the matrix run the correct docker
> > command in the "script" section (to get an idea
> > https://github.com/apache/arrow/blob/master/.travis.yml, even if they
> > don't use docker).
> >
> > While I research though it might be good to follow your solution and
> > just have a simple Travis file for say ubuntu and windows. Even if it
> > takes 30 mins to complete, it would be a good feedback for anybody
> > committing code to trunk/2.4.x. And it could be possible that this
> > solution is good enough for our use cases, only testing will tell us!
> > So feel free to commit your Travis config to trunk, maybe following
> > the Apache Arrow example? If so others, more experienced with Windows,
> > might be able to chime in and add the missing configuration.
>
> This is my idea:
>
> https://github.com/elukey/httpd/blob/trunk/.travis.yml
>
> https://travis-ci.org/elukey/httpd/builds/607536348
>
> Basically what Joe did, but following what the Apache Arrow Project
> did. There is a little bit of repetition, but in theory with this
> config people are free to add tests for other OSes (osx, windows) and
> I will probably be able to add custom ones with Docker to see the
> difference in execution timings etc.. (Apache Arrow's test do use
> docker, I didn't see it a first).
>

I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems,
so the 'addons' will be used by travis only if the os can use them (I
was afraid of having windows and apt combined). The only thing that I
would add to travis.yml are the ifs to separate
before_install/script/etc.. for every os, I think that we should add
windows as soon as possible.. Do we have a list of build commands to
test in Travis?

Luca
Reply | Threaded
Open this post in threaded view
|

AW: Integration tests running on Docker

Pluem, Ruediger, Vodafone Group
In reply to this post by Joe Orton



C2 General

> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <[hidden email]>
> Gesendet: Mittwoch, 6. November 2019 10:53
> An: Apache HTTP Server Development List <[hidden email]>
> Betreff: Re: Integration tests running on Docker
>
> On Tue, Nov 05, 2019 at 10:06:23AM +0100, Luca Toscano wrote:
> > Basically what Joe did, but following what the Apache Arrow Project
> > did. There is a little bit of repetition, but in theory with this
> > config people are free to add tests for other OSes (osx, windows) and
> > I will probably be able to add custom ones with Docker to see the
> > difference in execution timings etc.. (Apache Arrow's test do use
> > docker, I didn't see it a first).
>
> I haven't tried to follow all of what the Arrow .yml is doing (too much
> Java ;) but moving some of the logic into the repo itself out of
> .travis.yml seems like a good idea.  e.g. moving all of script: into
> something like ./test/travis_script_linux.sh or something for the Linux
> based tests, does that make sense?
>
> This is working now -
>
> https://travis-ci.org/apache/httpd/builds
>
> and I guess it will work for forks and PRs too, haven't tried yet.
>
> Anybody feel free to extend, play, hack, whatever with the travis.yml, I
> am no expert so may be doing something non-standard/wrong, feel free to
> correct/improve ;)
>
> The jobs are quite quick but the queue time for the "apache" repos is
> long (10+ minutes) before they get started.  I guess it is doing
> notifications to committers directly, but we won't find out until a job
> fails?  Maybe let's keep experimenting with this a bit longer and then
> start sending mail to dev@ for state changes (exactly as buildbot does)
> when it seems stable?

+1. Thanks for taking care.

Regards

Rüdiger
Reply | Threaded
Open this post in threaded view
|

Re: Integration tests running on Docker

Joe Orton
In reply to this post by Luca Toscano
On Wed, Nov 06, 2019 at 10:54:56AM +0100, Luca Toscano wrote:
> I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems,
> so the 'addons' will be used by travis only if the os can use them (I
> was afraid of having windows and apt combined). The only thing that I
> would add to travis.yml are the ifs to separate
> before_install/script/etc.. for every os, I think that we should add
> windows as soon as possible.. Do we have a list of build commands to
> test in Travis?

I have no idea how to build on anything but Linux :) Someone could do
osx too, travis supports that...

So it works for PRs and successfully failed for a bogus commit -

https://travis-ci.org/apache/httpd/pull_requests

This one is sensible?

https://github.com/apache/httpd/pull/71/commits/4a46327813e87f3d662e65361e6d9240ce855db5

I don't know if anything other than Linux can actually be tested like
this tbh.  (Also trying to context switch between svn & git doing this
stuff is going to melt my brain!)

12