Skip to content

Commit

Permalink
Switch from using Ginkgo to using only the Go test framework (#778)
Browse files Browse the repository at this point in the history
As explored in the prototypes in #253, we have decided that although Ginkgo is the best of the Go test frameworks, we do not want to be tied to BDD and feel these tests will be cleaner and easier to contribute to if we use no framework.

I have also refactored the supporting logic of the tests into a `test` library which can be reused for other e2e tests, documented in test/adding_tests.md

* Improved docs on running tests: now all docs on running are in test/README.md, there is only one place to look to see how to run all of the tests. Instructions on how to run conformance tests are hopefully more clear.
* Updated links that pointed at CONTRIBUTING.md since the content moved

Fixes #253
  • Loading branch information
bobcatfish authored May 1, 2018
1 parent c5af8a1 commit c2c0120
Show file tree
Hide file tree
Showing 239 changed files with 965 additions and 177,899 deletions.
13 changes: 5 additions & 8 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
# Development

This doc explains how to setup a development environment so you can get started
[contributing](./CONTRIBUTING.md) to `Elafros`. Also take a look at [the
development workflow](./CONTRIBUTING.md#workflow) and [the test docs](./test/README.md).
[contributing](./community/CONTRIBUTING.md) to `Elafros`. Also take a look at:

* [The pull request workflow](./community/CONTRIBUTING.md#pull-requests)
* [How to add and run tests](./test/README.md)
* [Iterating](#iterating)

## Getting started

Expand Down Expand Up @@ -168,12 +171,6 @@ bazel run :controller.apply
Or you can [clean it up completely](./README.md#clean-up) and [completely
redeploy `Elafros`](./README.md#start-elafros).

## Tests

Tests are run automatically for every PR. For more details, see [the development workflow](./CONTRIBUTING.md#prow).

For more details about the tests themselves and how to run them, see [the test docs](./test/README.md).

## Clean up

You can delete all of the service components with:
Expand Down
61 changes: 1 addition & 60 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions test/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
name = "go_default_library",
srcs = [
"cleanup.go",
"clients.go",
"crd.go",
"crd_polling.go",
"e2e_flags.go",
"request.go",
"states.go",
],
importpath = "github.com/elafros/elafros/test",
visibility = ["//visibility:public"],
deps = [
"//pkg/apis/ela/v1alpha1:go_default_library",
"//pkg/client/clientset/versioned:go_default_library",
"//pkg/client/clientset/versioned/typed/ela/v1alpha1:go_default_library",
"//vendor/k8s.io/api/core/v1:go_default_library",
"//vendor/k8s.io/api/extensions/v1beta1:go_default_library",
"//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",
"//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library",
"//vendor/k8s.io/client-go/kubernetes:go_default_library",
"//vendor/k8s.io/client-go/kubernetes/typed/extensions/v1beta1:go_default_library",
"//vendor/k8s.io/client-go/rest:go_default_library",
"//vendor/k8s.io/client-go/tools/clientcmd:go_default_library",
],
)
181 changes: 168 additions & 13 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,125 @@

This directory contains tests and testing docs for `Elafros`:

* [Unit tests](#running-unit-tests) currently reside in the codebase alongside the code they test.
* [Unit tests](#running-unit-tests) currently reside in the codebase alongside the code they test
* [Conformance tests](#running-conformance-tests) in [`/test/conformance`](./conformance)
* [End-to-end tests](#running-end-to-end-tests)
* [Conformance tests](./conformance/README.md) are in [`./test/conformance`](./conformance)

If you want to add more tests, see [adding_tests.md](./adding_tests.md).

## Running unit tests

The tests can be run using bazel directly:
Use bazel:

```shell
bazel test //pkg/... --test_output=errors
```

Or can be run using `go test`:
Or `go test`:

```shell
go test -v ./pkg/...
```
## Running conformance tests

To run [the conformance tests](./conformance), you need to have a running environment that meets
[the conformance test environment requirements](#conformance-test-environment-requirements).

To run the conformance tests against the current cluster in `~/.kube/config`
using `go test` using the environment specified in [your environment
variables](/DEVELOPMENT.md#environment-setup):

Since these tests are fairly slow (~1 minute), running them with logging
enabled is recommended:

```bash
go test -v ./test/conformance
```

You can [use test flags](#flags) to control the environment
your tests run against, i.e. override [your environment variables](/DEVELOPMENT.md#environment-setup):

```bash
go test -v ./test/conformance --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
```

If you are running against an environment with no loadbalancer for the ingress, at the moment
your only option is to use a domain which will resolve to the IP of the running node (see
[#609](https://github.com/elafros/elafros/issues/609)):

```bash
go test -v ./test/conformance --resolvabledomain
```

## Conformance test environment requirements

These tests require:

1. [A running `Elafros` cluster.](/DEVELOPMENT.md#getting-started)
2. The namespace `pizzaplanet` to exist in the cluster: `kubectl create namespace pizzaplanet`
3. A docker repo contianing [the conformance test images](#conformance-test-images)

### Conformance test images

The configuration for the images used for the existing conformance tests lives in
[`test_images_node`](./conformance/test_images_node).

[`upload-test-images.sh`](./upload-test-images.sh) can be used to build and push the
docker images. It requires:

* [`DOCKER_REPO_OVERRIDE`](/DEVELOPMENT.md#environment-setup) to be set
* You to be [authenticated with your
`DOCKER_REPO_OVERRIDE`](/docs/setting-up-a-docker-registry.md)
* [`docker`](https://docs.docker.com/install/) to be installed

To run the script:

```bash
./test/conformance/upload-test-images.sh
```

### Running conformance tests with Bazel

To run the conformance tests with `bazel` you must:

* Provide a `kubeconfig` file. This file must be a `data` dependency of the test in
[`BUILD.bazel`](./conformance/BUILD.bazel). By default [`BUILD.bazel`](./conformance/BUILD.bazel)
is configured to use [`test/conformance/kubeconfig`](/test/conformance/kubeconfig).
* Provide a docker repo from which the built images will be pulled. This is done
via the `--dockerrepo` argument.

_The `bazel` execution environment will not contain your environment variables, so you must
explicitly specify them with [command line args](#flags)._

To run the tests with `bazel` (assuming you have populated [`./kubeconfig`](./conformance/kubeconfig)
and your [`DOCKER_REPO_OVERRIDE`](/DEVELOPMENT.md#environment-setup) is configured
to the location where [you have pushed the conformance test images](#conformance-test-images)):

```bash
bazel test //test/... --test_arg=--dockerrepo=$DOCKER_REPO_OVERRIDE --test_arg=--kubeconfig=./kubeconfig
```

## Running end-to-end tests

You can either run all the end-to-end tests in an isolated, hermetic cluster, or use your already existing cluster for that.
The script [`e2e-tests.sh`](/test/e2e-tests.sh) can be used to run all the end to end tests
(including [the conformance tests](#running-conformance-tests)) either:

* [Using an existing cluster](#running-against-an-existing-cluster)
* [In an isolated, hermetic GCP cluster](#running-against-an-isolated-hermetic-gcp-cluster)

### Running against an existing cluster

Assuming you have [`K8S_USER_OVERRIDE`, `K8S_CLUSTER_OVERRIDE` and
`DOCKER_REPO_OVERRIDE` set](/DEVELOPMENT.md#environment-setup), run:

```bash
./tests/e2e-tests.sh --run-tests
```

### Running against an isolated, hermetic cluster
### Running against an isolated, hermetic GCP cluster

In order to run the end-to-end tests, make sure you:
This will start a cluster for you in GCP using [kubetest](https://github.com/kubernetes/test-infra/tree/master/kubetest).
Make sure you:

1. Have `kubetest` installed:
```
Expand All @@ -36,14 +130,75 @@ In order to run the end-to-end tests, make sure you:
```
2. Have the `PROJECT_ID` environment variable set to a GCP project you own.

The end-to-end tests can be run by simply executing the `e2e-tests.sh` script.
Run:

### Running against an already existing cluster
```bash
./tests/e2e-tests.sh
```

In order to run the end-to-end tests, make sure you have the `K8S_USER_OVERRIDE`, `K8S_CLUSTER_OVERRIDE` and `DOCKER_REPO_OVERRIDE` environment variables correctly set.
## Flags

The end-to-end tests can be run by executing `e2e-tests.sh --run-tests` in the command line.
These flags are useful for running against an existing cluster, making use of your existing
[environment setup])(/DEVELOPMENT.md#environment-setup).

## Running conformance tests
Tests importing [`github.com/elafros/elafros/test`](adding_tests.md#test-library) recognize these flags:

* [`--kubeconfig`](#specifying-kubeconfig)
* [`--cluster`](#specifying-cluster)
* [`--dockerrepo`](#overriding-docker-repo)
* [`--resolvabledomain`](#using-a-resolvable-domain)

#### Specifying kubeconfig

By default the tests will use the [kubeconfig
file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/)
at `~./kube/config`.
You can specify a different config file with the argument `--kubeconfig`.

To run the conformance tests with a non-default kubeconfig file:

```bash
go test ./test/conformance --kubeconfig /my/path/kubeconfig
```

#### Specifying cluster

The `--cluster` argument lets you use a different cluster than [your specified
kubeconfig's](#specifying-kubeconfig) active context. This will default to the value
of your [`K8S_CLUSTER_OVERRIDE` environment variable](/DEVELOPMENT.md#environment-setup)
if not specified.

```bash
go test ./test/conformance --cluster your-cluster-name
```

The current cluster names can be obtained by running:

```bash
kubectl config get-clusters
```

#### Overridding docker repo

The `--dockerrepo` argument lets you specify the docker repo from which images used
by your tests should be pulled. This will default to the value
of your [`DOCKER_REPO_OVERRIDE` environment variable](/DEVELOPMENT.md#environment-setup)
if not specified.

```bash
go test ./test/conformance --dockerrepo gcr.myhappyproject
```

#### Using a resolvable domain

If you setup your cluster using [the getting started
docs](../../DEVELOPMENT.md#getting-started), Routes created in the test will
use the domain `demo-domain.com`, unless the route has label `app=prod` in which
case they will use the domain `prod-domain.com`. Since these domains will not be
resolvable to deployments in your test cluster, in order to make a request
against the endpoint, the test use the IP assigned to the istio `*-ela-ingress`
and spoof the `Host` in the header.

See [conformance test docs](./conformance/README.md).
If you have configured your cluster to use a resolvable domain, you can use the
`--resolvabledomain` flag to indicate that the test should make requests directly against
`Route.Status.Domain` and does not need to spoof the `Host`.
Loading

0 comments on commit c2c0120

Please sign in to comment.