Testing

To run tests in the test suite you have two options. You may go the easy way and just run the test suite in docker. But for some tasks you could also need to install the test suite natively, which requires a little more setup since PHP and some dependencies need to be installed.

Both ways to run tests with the test suites are described here.

Let’s see what is available. Invoke the following command from within the root of the oCIS repository.

make -C tests/acceptance/docker help

Basically we have two sources for feature tests and test suites:

• oCIS feature test and test suites
• tests and test suites transferred from ownCloud core, they have prefix coreApi

At the moment, both can be applied to oCIS since the api of oCIS is designed to be compatible with ownCloud.

As a storage backend, we offer oCIS native storage, also called ocis. This stores files directly on disk. Along with that we also provide S3 storage driver.

You can invoke two types of test suite runs:

• run a full test suite, which consists of multiple feature tests
• run a single feature or single scenario in a feature

The names of the full test suite make targets have the same naming as in the CI pipeline. See the available local oCIS specific test suites here. They can be run with ocis storage and S3 storage.

For example, command:

make -C tests/acceptance/docker localApiTests-apiGraph-ocis

runs the same tests as the localApiTests-apiGraph-ocis CI pipeline, which runs the oCIS test suite “apiGraph” against the oCIS server with ocis storage.

And command:

make -C tests/acceptance/docker localApiTests-apiGraph-s3ng

runs the oCIS test suite apiGraph against the oCIS server with s3 storage.

While running the tests, oCIS server is started with ociswrapper (i.e. WITH_WRAPPER=true) by default. In order to run the tests without ociswrapper, provide WITH_WRAPPER=false when running the tests. For example:

WITH_WRAPPER=false \
BEHAT_FEATURE='tests/acceptance/features/apiGraphUserGroup/createUser.feature:26' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

But some test suites that are tagged with @env-config require the oCIS server to be run with ociswrapper. So, running those tests require WITH_WRAPPER=true (default setting).

To run the tests that require an email server (tests tagged with @email), you need to provide START_EMAIL=true while running the tests.

START_EMAIL=true \
BEHAT_FEATURE='tests/acceptance/features/apiNotification/emailNotification.feature' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

To run the tests that require tika service (tests tagged with @tikaServiceNeeded), you need to provide START_TIKA=true while running the tests.

START_TIKA=true \
BEHAT_FEATURE='tests/acceptance/features/apiSearchContent/contentSearch.feature' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

To run the tests that require an antivirus service (tests tagged with @antivirus), you need to provide the following environment variables while running the tests.

START_ANTIVIRUS=true \
OCIS_ASYNC_UPLOADS=true \
OCIS_ADD_RUN_SERVICES=antivirus \
POSTPROCESSING_STEPS=virusscan \
BEHAT_FEATURE='tests/acceptance/features/apiAntivirus/antivirus.feature' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

Command make -C tests/acceptance/docker Core-API-Tests-ocis-storage-3 runs the same tests as the Core-API-Tests-ocis-storage-3 CI pipeline, which runs the third (out of ten) test suite groups transferred from ownCloud core against the oCIS server with ocis storage.

And make -C tests/acceptance/docker Core-API-Tests-s3ng-storage-3 runs the third (out of ten) test suite groups transferred from ownCloud core against the oCIS server with s3 storage.

The tests for a single feature (a feature file) can also be run against the different storage backends. To do that, multiple make targets with the schema test-<test-source>-feature-<storage-backend> are available. To select a single feature you have to add an additional BEHAT_FEATURE=<path-to-feature-file> parameter when invoking the make command.

For example;

BEHAT_FEATURE='tests/acceptance/features/apiGraphUserGroup/createUser.feature' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

BEHAT_FEATURE must be pointing to a valid feature file

And to run a single scenario in a feature, you can do:

A specific scenario from a feature can be run by adding :<line-number> at the end of the feature file path. For example, to run the scenario at line 26 of the feature file apiGraphUserGroup/createUser.feature, simply add the line number like this: apiGraphUserGroup/createUser.feature:26. Note that the line numbers mentioned in the examples might not always point to a scenario, so always check the line numbers before running the test.

BEHAT_FEATURE='tests/acceptance/features/apiGraphUserGroup/createUser.feature:26' \
make -C tests/acceptance/docker test-ocis-feature-ocis-storage

Similarly, with S3 storage;

# run a whole feature
BEHAT_FEATURE='tests/acceptance/features/apiGraphUserGroup/createUser.feature' \
make -C tests/acceptance/docker test-ocis-feature-s3ng-storage

# run a single scenario
BEHAT_FEATURE='tests/acceptance/features/apiGraphUserGroup/createUser.feature:26' \
make -C tests/acceptance/docker test-ocis-feature-s3ng-storage

In the same way, tests transferred from ownCloud core can be run as:

# run a whole feature
BEHAT_FEATURE='tests/acceptance/features/coreApiAuth/webDavAuth.feature' \
make -C tests/acceptance/docker test-core-feature-ocis-storage

# run a single scenario
BEHAT_FEATURE='tests/acceptance/features/coreApiAuth/webDavAuth.feature:15' \
make -C tests/acceptance/docker test-core-feature-ocis-storage

The test suites transferred from ownCloud core have coreApi prefixed

By default, the tests will be run against the docker image built from your current working state of the oCIS repository. For some purposes it might also be handy to use an oCIS image from Docker Hub. Therefore, you can provide the optional flag OCIS_IMAGE_TAG=... which must contain an available docker tag of the owncloud/ocis registry on Docker Hub (e.g. ’latest’).

OCIS_IMAGE_TAG=latest \
make -C tests/acceptance/docker localApiTests-apiGraph-ocis

While a test is running or when it is finished, you can attach to the logs generated by the tests.

make -C tests/acceptance/docker show-test-logs

The log output is opened in less. You can navigate up and down with your cursors. By pressing “F” you can follow the latest line of the output.

During testing we start a redis and oCIS docker container. These will not be stopped automatically. You can stop them with:

make -C tests/acceptance/docker clean

We have two sets of tests:

• test-acceptance-api set was created for oCIS. Mainly for testing spaces features.

• test-acceptance-from-core-api set was transferred from ownCloud core repository. The suite name of all tests transferred from the ownCloud core repository starts with coreApi

Create an up-to-date oCIS binary by building oCIS

To start oCIS:

IDM_ADMIN_PASSWORD=admin \
ocis/bin/ocis init --insecure true

OCIS_INSECURE=true PROXY_ENABLE_BASIC_AUTH=true \
ocis/bin/ocis server

PROXY_ENABLE_BASIC_AUTH will allow the acceptance tests to make requests against the provisioning api (and other endpoints) using basic auth.

make test-acceptance-api \
TEST_SERVER_URL=https://localhost:9200 \

make test-acceptance-from-core-api \
TEST_SERVER_URL=https://localhost:9200 \

Useful environment variables:

TEST_SERVER_URL: oCIS server url. Please, adjust the server url according to your setup.

BEHAT_FEATURE: to run a single feature

A specific scenario from a feature can be run by adding :<line-number> at the end of the feature file path. For example, to run the scenario at line 26 of the feature file apiGraphUserGroup/createUser.feature, simply add the line number like this: apiGraphUserGroup/createUser.feature:26. Note that the line numbers mentioned in the examples might not always point to a scenario, so always check the line numbers before running the test.

Example:

BEHAT_FEATURE=tests/acceptance/features/apiGraphUserGroup/createUser.feature

Or

BEHAT_FEATURE=tests/acceptance/features/apiGraphUserGroup/createUser.feature:13

BEHAT_SUITE: to run a single suite

Example:

BEHAT_SUITE=apiGraph

STORAGE_DRIVER: to run tests with a different user storage driver. Available options are ocis (default), owncloudsql and s3ng

Example:

STORAGE_DRIVER=owncloudsql

STOP_ON_FAILURE: to stop running tests after the first failure

Example:

STOP_ON_FAILURE=true

As a lot of scenarios from test-acceptance-from-core-api are written for oC10, we can use those tests for Behaviour driven development in oCIS. Every scenario that does not work in oCIS with “ocis” storage, is listed in tests/acceptance/expected-failures-API-on-OCIS-storage.md with a link to the related issue.

Those scenarios are run in the ordinary acceptance test pipeline in CI. The scenarios that fail are checked against the expected failures. If there are any differences then the CI pipeline fails.

The tests are not currently run in CI with the OWNCLOUD or EOS storage drivers, so there are no expected-failures files for those.

If you want to work on a specific issue

1. locally run each of the tests marked with that issue in the expected failures file.

   E.g.:

   make test-acceptance-from-core-api \
   TEST_SERVER_URL=https://localhost:9200 \
   STORAGE_DRIVER=OCIS \
   BEHAT_FEATURE='tests/acceptance/features/coreApiVersions/fileVersions.feature:141'
   

2. the tests will fail, try to understand how and why they are failing

3. fix the code

4. go back to 1. and repeat till the tests are passing.

5. remove those tests from the expected failures file

6. make a PR that has the fixed code, and the relevant lines removed from the expected failures file.

Test suites tagged with @env-config are used to test the environment variables that are used to configure oCIS. These tests are special tests that require the oCIS server to be run using ociswrapper.

# working dir: ocis repo root dir

# init oCIS
IDM_ADMIN_PASSWORD=admin \
ocis/bin/ocis init --insecure true

# build the wrapper
cd tests/ociswrapper
make build

# run oCIS
PROXY_ENABLE_BASIC_AUTH=true \
./bin/ociswrapper serve --bin=../../ocis/bin/ocis

OCIS_WRAPPER_URL=http://localhost:5200 \
TEST_SERVER_URL="https://localhost:9200" \
BEHAT_FEATURE=tests/acceptance/features/apiAsyncUpload/delayPostprocessing.feature \
make test-acceptance-api

While writing tests for a new oCIS ENV configuration, please make sure to follow these guidelines:

1. Tag the test suite (or test scenarios) with @env-config
2. Use OcisConfigHelper.php for helper functions - provides functions to reconfigure the running oCIS instance.
3. Recommended: add the new step implementations in OcisConfigContext.php

Test suites that are tagged with @email require an email service. We use inbucket as the email service in our tests.

Run the following command to setup inbucket

docker run -d -p9000:9000 -p2500:2500 --name inbucket inbucket/inbucket

Documentation for environment variables is available here

# init oCIS
IDM_ADMIN_PASSWORD=admin \
ocis/bin/ocis init --insecure true

# run oCIS
PROXY_ENABLE_BASIC_AUTH=true \
NOTIFICATIONS_SMTP_HOST=localhost \
NOTIFICATIONS_SMTP_PORT=2500 \
NOTIFICATIONS_SMTP_INSECURE=true \
ocis/bin/ocis server

Run the acceptance test with the following command:

TEST_SERVER_URL="https://localhost:9200" \
EMAIL_HOST="localhost" \
EMAIL_PORT=9000 \
BEHAT_FEATURE="tests/acceptance/features/apiNotification/emailNotification.feature" \
make test-acceptance-api

Test suites that are tagged with @tikaServiceNeeded require tika service.

Run the following docker command to setup tika service

docker run -d -p 127.0.0.1:9998:9998 apache/tika

Documentation related to the content based search and tika extractor can be found here

# init oCIS
IDM_ADMIN_PASSWORD=admin \
ocis/bin/ocis init --insecure true

# run oCIS
PROXY_ENABLE_BASIC_AUTH=true \
OCIS_INSECURE=true \
SEARCH_EXTRACTOR_TYPE=tika \
SEARCH_EXTRACTOR_TIKA_TIKA_URL=http://localhost:9998 \
SEARCH_EXTRACTOR_CS3SOURCE_INSECURE=true \
ocis/bin/ocis server

Run the acceptance test with the following command:

TEST_SERVER_URL="https://localhost:9200" \
BEHAT_FEATURE="tests/acceptance/features/apiSearchContent/contentSearch.feature" \
make test-acceptance-api

Instruction on setup is available here

Edit the .env file and uncomment this line:

COMPOSE_FILE=docker-compose.yml:testing/docker-compose-additions.yml

Start the docker stack with the following command:

docker-compose up -d

All the test helpers are located in the core repo.

git clone https://github.com/owncloud/core.git

Run the acceptance tests with the following command from the root of the oCIS repository:

make test-paralleldeployment-api \
TEST_SERVER_URL="https://cloud.owncloud.test" \
TEST_OC10_URL="http://localhost:8080" \
TEST_PARALLEL_DEPLOYMENT=true \
TEST_WITH_LDAP=true \
PATH_TO_CORE="<path_to_core>" \
SKELETON_DIR="<path_to_core>/apps/testing/data/apiSkeleton"

Replace <path_to_core> with the actual path to the root directory of core repo that you have cloned earlier.

In order to run a single test, use the BEHAT_FEATURE environment variable.

make test-paralleldeployment-api \
... \
BEHAT_FEATURE="tests/parallelDeployAcceptance/features/apiShareManagement/acceptShares.feature"

Test suites that are tagged with @antivirus require antivirus service. The available antivirus and the configuration related to them can be found here. This documentation is only going to use clamAv as antivirus.

Run the following command to set up calmAV and clamAV daemon

sudo apt install clamav clamav-daemon -y

Make sure that the clamAV daemon is up and running

sudo service clamav-daemon status

The commands are ubuntu specific and may differ according to your system. You can find information related to installation of clamAV in their official documentation here.

Install ClamAV using here Start ClamAV daemon

/your/location/to/brew/Cellar/clamav/1.1.0/sbin/clamd

Run clamAV through docker

docker run -d -p 3310:3310 owncloudci/clamavd

docker run -d -p 3310:3310 -v /your/local/filesystem/path/to/clamav/:/var/lib/clamav mkodockx/docker-clamav:alpine

As antivirus service is not enabled by default we need to enable the service while running oCIS server. We also need to enable async upload and as virus scan is performed in post-processing step, we need to set it as well. Documentation for environment variables related to antivirus is available here

# run oCIS
PROXY_ENABLE_BASIC_AUTH=true \
ANTIVIRUS_SCANNER_TYPE="clamav" \
ANTIVIRUS_CLAMAV_SOCKET="tcp://host.docker.internal:3310" \
POSTPROCESSING_STEPS="virusscan" \
OCIS_ASYNC_UPLOADS=true \
OCIS_ADD_RUN_SERVICES="antivirus"
ocis/bin/ocis server

The value for ANTIVIRUS_CLAMAV_SOCKET is an example which needs adaption according your OS.

For antivirus running localy on Linux OS, use ANTIVIRUS_CLAMAV_SOCKET= "/var/run/clamav/clamd.ctl". For antivirus running localy on Mac OS, use ANTIVIRUS_CLAMAV_SOCKET= "/tmp/clamd.socket". For antivirus running with docker, use ANTIVIRUS_CLAMAV_SOCKET= "tcp://host.docker.internal:3310"

Run the acceptance test with the following command:

TEST_SERVER_URL="https://localhost:9200" \
BEHAT_FEATURE="tests/acceptance/features/apiAntivirus/antivirus.feature" \
make test-acceptance-api

There are some tests that check the text preview of files containing Unicode characters. The oCIS server by default cannot generate the thumbnail of such files correctly but it provides an environment variable to allow the use of custom fonts that support Unicode characters. So to run such tests successfully, we have to run the oCIS server with this environment variable.

...
THUMBNAILS_TXT_FONTMAP_FILE="/path/to/fontsMap.json"
ocis/bin/ocis server

The sample fontsMap.json file is located in tests/config/drone/fontsMap.json.

{
  "defaultFont": "/path/to/ocis/tests/config/drone/NotoSans.ttf"
}