ARTICLE AD BOX
I am currently working in a project using Java 17, Cucumber 7 and JUnit5. I use this project to run integration tests (API tests using REST Assured and UI tests using Selenium). I use the Maven Failsafe plugin to run the tests (via mvn clean verify command). I have almost 30 feature files in the project with hundreds of scenarios and scenario outlines and thousands of cucumber examples for these scenarios. The tests are running correctly but for some reason, whenever at least one test case fails, Failsafe (I assume it is Failsafe because I've kind of ruled out other possibilities with the help of some AIs) prints some nonsensical stuff in my console output. One example:
[ERROR] Failures: [ERROR] com.company.myproject.runners.CucumberRunner.Example #1.1 [INFO] Run 1: PASS [INFO] Run 2: PASS [INFO] Run 3: PASS [INFO] Run 4: PASS (many lines here) [INFO] Run 158: PASS [INFO] Run 159: PASS [ERROR] Run 160: CucumberRunner.Example #1.1 expected: <555> but was: <200> [INFO] Run 161: PASS [INFO] Run 162: PASS (many lines here) [INFO] Run 298: PASS [INFO] Run 299: PASS [INFO] Run 300: PASS [INFO] [ERROR] com.lenovo.lspptestautomation.runners.CucumberRunner.Example #1.2 [INFO] Run 1: PASS [INFO] Run 2: PASS [INFO] Run 3: PASS [INFO] Run 4: PASS (many lines here) [INFO] Run 158: PASS [INFO] Run 159: PASS [ERROR] Run 160: CucumberRunner.Example #1.2 expected: <555> but was: <200> [INFO] Run 161: PASS [INFO] Run 162: PASS [INFO] Run 163: PASS (many lines here) [INFO] Run 299: PASS [INFO] Run 300: PASSIn this example I run only one cucumber scenario outline (that is a simple API test) with two examples and I made them fail on porpuse so that I could see this happening.
I am pretty sure that no reruns are being executed but still I am always presented with 300 runs every time I run the tests. The test is quite fast, 12 seconds to run or less, if all those reruns were being executed, it would take a very long time to finish.
This is my Cucumber runner:
package com.company.myproject.runners; import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME; import static io.cucumber.junit.platform.engine.Constants.PLUGIN_PROPERTY_NAME; import org.junit.platform.suite.api.ConfigurationParameter; import org.junit.platform.suite.api.IncludeEngines; import org.junit.platform.suite.api.SelectPackages; import org.junit.platform.suite.api.Suite; import org.junit.platform.suite.api.SuiteDisplayName; @Suite @SuiteDisplayName("Test Automation") @IncludeEngines("cucumber") @SelectPackages("features") @ConfigurationParameter( key = GLUE_PROPERTY_NAME, value = "com.company.myproject.stepdefinitions, " + "com.company.myproject.environment, " + "com.company.myproject.utils" ) @ConfigurationParameter( key = PLUGIN_PROPERTY_NAME, value = "net.serenitybdd.cucumber.core.plugin.SerenityReporterParallel, pretty, summary, " + "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm, " + "com.company.myproject.plugins.StepPrinterPlugin" ) public class CucumberRunner { }This is my pom.xml:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.company</groupId> <artifactId>test-automation</artifactId> <version>1.0</version> <name>Test Automation</name> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <maven.compiler.source>17</maven.compiler.source> <maven.compiler.target>17</maven.compiler.target> <junit.version>6.0.1</junit.version> <springboot.version>6.2.11</springboot.version> <selenium.version>4.40.0</selenium.version> <rest-assured.version>5.5.2</rest-assured.version> <log4j.version>2.25.3</log4j.version> <json.version>20240303</json.version> <networknt.version>1.5.6</networknt.version> <cucumber.version>7.33.0</cucumber.version> <serenity.version>5.0.4</serenity.version> <allure.version>2.32.0</allure.version> <maven-surefire-junit-platform.version>3.5.4</maven-surefire-junit-platform.version> <maven-resources-plugin.version>3.1.0</maven-resources-plugin.version> <maven-compiler-plugin.version>3.13.0</maven-compiler-plugin.version> <maven-surefire-plugin.version>3.5.2</maven-surefire-plugin.version> <maven-jar-plugin.version>3.4.2</maven-jar-plugin.version> <maven-failsafe-plugin.version>3.5.4</maven-failsafe-plugin.version> <serenity-maven-plugin.version>4.2.8</serenity-maven-plugin.version> <allure-maven-plugin.version>2.15.2</allure-maven-plugin.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.junit</groupId> <artifactId>junit-bom</artifactId> <version>${junit.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-dependencies-bom</artifactId> <version>${selenium.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-bom</artifactId> <version>${log4j.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-bom</artifactId> <version>${cucumber.version}</version> <scope>import</scope> <type>pom</type> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-bom</artifactId> <version>${allure.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.junit.platform</groupId> <artifactId>junit-platform-suite</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.platform</groupId> <artifactId>junit-platform-launcher</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter-api</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter-engine</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-core</artifactId> <version>${springboot.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>net.bytebuddy</groupId> <artifactId>byte-buddy</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-manager</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.rest-assured</groupId> <artifactId>rest-assured</artifactId> <version>${rest-assured.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.json</groupId> <artifactId>json</artifactId> <version>${json.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>com.networknt</groupId> <artifactId>json-schema-validator</artifactId> <version>${networknt.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-junit-platform-engine</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-java</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-core</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-gherkin</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>net.bytebuddy</groupId> <artifactId>byte-buddy</artifactId> <version>1.15.4</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-core</artifactId> <version>${serenity.version}</version> <scope>test</scope> <exclusions> <exclusion> <groupId>net.bytebuddy</groupId> <artifactId>byte-buddy</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-junit5</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-reports</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-single-page-report</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-navigator-report</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-rest-assured</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-cucumber</artifactId> <version>${serenity.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-junit-platform</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-cucumber7-jvm</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-junit-platform</artifactId> <version>${maven-surefire-junit-platform.version}</version> <scope>test</scope> </dependency> </dependencies> <build> <testResources> <testResource> <directory>src/test/resources</directory> <filtering>true</filtering> <excludes> <exclude>json_schemas/*</exclude> <exclude>payload_jsons/*</exclude> <exclude>upload_files/*</exclude> </excludes> </testResource> </testResources> <filters> <filter>src/test/resources/filtered_properties/env_${env}.properties</filter> </filters> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-resources-plugin</artifactId> <version>${maven-resources-plugin.version}</version> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven-compiler-plugin.version}</version> <configuration> <source>${maven.compiler.source}</source> <target>${maven.compiler.target}</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${maven-surefire-plugin.version}</version> <configuration> <skip>true</skip> </configuration> </plugin> <plugin> <artifactId>maven-jar-plugin</artifactId> <version>${maven-jar-plugin.version}</version> <configuration> <skipIfEmpty>true</skipIfEmpty> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>${maven-failsafe-plugin.version}</version> <configuration> <includes> <include>**/CucumberRunner.java</include> </includes> <printSummary>false</printSummary> </configuration> <executions> <execution> <id>failsafe-integration-tests</id> <phase>integration-test</phase> <goals> <goal>integration-test</goal> </goals> </execution> </executions> <dependencies> <dependency> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-junit-platform</artifactId> <version>${maven-surefire-junit-platform.version}</version> </dependency> </dependencies> </plugin> <plugin> <groupId>net.serenity-bdd.maven.plugins</groupId> <artifactId>serenity-maven-plugin</artifactId> <version>${serenity-maven-plugin.version}</version> <dependencies> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-single-page-report</artifactId> <version>${serenity.version}</version> </dependency> <dependency> <groupId>net.serenity-bdd</groupId> <artifactId>serenity-navigator-report</artifactId> <version>${serenity.version}</version> </dependency> </dependencies> <executions> <execution> <id>check-gherkin</id> <phase>process-test-resources</phase> <goals> <goal>check-gherkin</goal> </goals> </execution> <execution> <id>serenity-reports</id> <phase>verify</phase> <goals> <goal>aggregate</goal> </goals> </execution> </executions> <configuration> <reports>single-page-html, navigator</reports> </configuration> </plugin> <plugin> <groupId>io.qameta.allure</groupId> <artifactId>allure-maven</artifactId> <version>${allure-maven-plugin.version}</version> <executions> <execution> <id>allure-reports</id> <phase>verify</phase> <goals> <goal>report</goal> </goals> <configuration> <reportDirectory> ${project.basedir}/reports/allure/ </reportDirectory> </configuration> </execution> </executions> </plugin> </plugins> </build> <repositories> <repository> <id>maven_central</id> <name>Maven Central</name> <url>https://repo.maven.apache.org/maven2/</url> </repository> </repositories> <profiles> <profile> <id>LOCAL</id> <properties> <env>LOCAL</env> </properties> </profile> <profile> <id>DEV</id> <properties> <env>dev</env> </properties> </profile> <profile> <id>QA</id> <properties> <env>qa</env> </properties> </profile> <profile> <id>SIT</id> <properties> <env>sit</env> </properties> <activation> <activeByDefault>true</activeByDefault> </activation> </profile> </profiles> </project>I have tried dozens of different solutions to circumvent this issue asking help from Gemini and ChatGPT, but until now nothing worked to make these "Runs" disappear from my output whenever some test case fails, these logs make no sense at tall.
Is there some way to configure the Failsafe plugin to avoid this?
