Migrate the testing environment to Vitest: rename equalMarkup to toEqualMarkup and remove the automated test runner

.changelog/20260706081519_ci_4595_to_equal_markup_vitest_matcher.md

@@ -0,0 +1,13 @@
+---
+type: Major breaking change
+
+scope:
+  - ckeditor5-dev-tests
+
+closes:
+  - ckeditor/ckeditor5-internal#4595
+---
+
+Replaced the `equalMarkup` Chai assertion with the `toEqualMarkup()` Vitest matcher to match the Vitest naming style. Update the call sites (`expect( a ).to.equalMarkup( b )` → `expect( a ).toEqualMarkup( b )`) and register the matcher in a Vitest setup file via `expect.extend( { toEqualMarkup } )`, importing it from `@ckeditor/ckeditor5-dev-tests`.
+
+The main entry point of the package now exposes only the custom Vitest matchers, so it can be imported in a browser context (for example, in a Vitest setup file). It no longer exports the `runAutomatedTests()`, `runManualTests()`, and `parseArguments()` functions, which were only used internally by the test runner binaries. Additionally, the unused `attribute` Chai assertion was removed.

.changelog/20260706082819_remove_karma_test_runner.md

@@ -0,0 +1,17 @@
+---
+type: Major breaking change
+
+scope:
+  - ckeditor5-dev-tests
+  - ckeditor5-dev-utils
+
+see:
+  - ckeditor/ckeditor5-internal#4595
+---
+
+Removed the automated test runner: the `ckeditor5-dev-tests-run-automated` binary and the entire Karma testing environment. Automated tests are now executed directly with Vitest, using the packages' own `test` scripts. The manual test server (`ckeditor5-dev-tests-run-manual`) is unaffected.
+
+Along with the Karma environment, the following were removed:
+
+* The IntelliJ Karma runner integration and desktop notifications support (`node-notifier`).
+* The `loaders.getCoverageLoader()` function from `@ckeditor/ckeditor5-dev-utils`.

eslint.config.js

@@ -69,9 +69,7 @@ export default defineConfig( [
 
 		languageOptions: {
 			globals: {
-				...globals.mocha,
 				...globals.browser,
-				'__karma__': true,
 				CKEditorInspector: true,
 				io: true
 			}

packages/ckeditor5-dev-tests/README.md

@@ -4,7 +4,9 @@ CKEditor 5 testing environment
 [![npm version](https://badge.fury.io/js/%40ckeditor%2Fckeditor5-dev-tests.svg)](https://www.npmjs.com/package/@ckeditor/ckeditor5-dev-tests)
 [![CircleCI](https://circleci.com/gh/ckeditor/ckeditor5-dev.svg?style=shield)](https://app.circleci.com/pipelines/github/ckeditor/ckeditor5-dev?branch=master)
 
-Testing environment for [CKEditor 5](https://ckeditor.com). It's based on [Karma](https://karma-runner.github.io/) and [webpack](https://webpack.github.io/) and it's normally used in the [CKEditor 5 development environment](https://github.com/ckeditor/ckeditor5). Read more about [CKEditor 5's testing environment](https://docs.ckeditor.com/ckeditor5/latest/framework/guides/contributing/testing-environment.html).
+Testing environment for [CKEditor 5](https://ckeditor.com). It provides the manual test server (manual tests are compiled and served with [webpack](https://webpack.js.org/)) and the custom [Vitest](https://vitest.dev/) matchers used by the automated tests. It's normally used in the [CKEditor 5 development environment](https://github.com/ckeditor/ckeditor5). Read more about [CKEditor 5's testing environment](https://docs.ckeditor.com/ckeditor5/latest/framework/guides/contributing/testing-environment.html).
+
+Automated tests are executed directly with Vitest using the packages' own `test` scripts — this package no longer provides an automated test runner.
 
 More information about development tools packages can be found at the following URL: <https://github.com/ckeditor/ckeditor5-dev>.
 
@@ -16,91 +18,35 @@ First, you need to install the package:
 npm i --save-dev @ckeditor/ckeditor5-dev-tests
 ```
 
-An example npm task to test CKEditor 5 packages (used e.g. in https://github.com/ckeditor/ckeditor5) can look like this:
+An example npm task to serve manual tests of CKEditor 5 packages (used e.g. in https://github.com/ckeditor/ckeditor5) can look like this:
 
 ```json
 {
   "scripts": {
-      "test": "node ./node_modules/.bin/ckeditor5-dev-tests-run-automated",
       "manual": "node ./node_modules/.bin/ckeditor5-dev-tests-run-manual"
   }
 }
 ```
 
-If you encounter problems with big test folders pass `--max_old_space_size=4096` option to node runner:
-
-```json
-{
-  "scripts": {
-      "test": "node --max_old_space_size=4096 ./node_modules/.bin/ckeditor5-dev-tests-run-automated"
-  }
-}
-```
-
-You can also use the bin script for testing a package:
-
-```bash
-# For running all the tests (for the current package and all dependencies).
-./node_modules/.bin/ckeditor5-dev-tests --files=*
-
-# For running tests in the current package.
-./node_modules/.bin/ckeditor5-dev-tests
-```
-
 #### CLI options
 
-* `browsers` - Browsers which will be used to run the tests. Also available as an alias: `-b`.
-* `coverage` - Whether to generate code coverage. Also available as an alias: `-c`.
-* `debug` - Allows specifying custom debug flags. For example, the `--debug engine` option uncomments the `// @if CK_DEBUG_ENGINE //` lines in the code. By default `--debug` is set to true even if you did not specify it. This enables the base set of debug logs (`// @if CK_DEBUG //`) which should always be enabled in the testing environment. You can completely turn off the debug mode by setting the `--debug false` option or `--no-debug`.
-* `files` - Package names, directories or files to tests. Also available as an alias: `-f`. Read more about this option in the [Rules for using the `--files` option](https://ckeditor.com/docs/ckeditor5/latest/framework/guides/contributing/testing-environment.html#rules-for-using-the-files-option) section.
-* `language` – Specifies a language that will be used while building tests. By default it is `en`.
-* `production` - Run strictest set of checks. E.g. it fails test run when there are [console calls](https://github.com/ckeditor/ckeditor5/issues/1996) or [DOM leaks](https://github.com/ckeditor/ckeditor5/issues/6002).
-* `repositories` (`-r`) - Specifies names of repositories containing packages that should be tested. Those repositories should be cloned into the `external/` directory in the root directory of the project. It's a shortcut of the `--files` option as these repository packages' names will be read by the tool automatically.
-* `reporter` - Mocha reporter – either `mocha` (default) or `dots` (less verbose one).
-* `server` - Whether to run the server without opening any browser.
-* `source-map` - Whether to generate the source maps. Also available as an alias: `-s`.
-* `verbose` - Whether to informs about Webpack's work. Also available as an alias: `-v`.
-* `watch` - Whether to watch the files and executing tests whenever any file changes. Also available as an alias: `-w`.
-
-#### Examples
-
-Test the `ckeditor5-enter` and `ckeditor5-paragraph` packages and generate code coverage report:
+* `files` - Package names, directories or files to serve. Also available as an alias: `-f`. Read more about this option in the [Rules for using the `--files` option](https://ckeditor.com/docs/ckeditor5/latest/framework/guides/contributing/testing-environment.html#rules-for-using-the-files-option) section.
+* `repositories` (`-r`) - Specifies names of repositories containing packages that should be served. Those repositories should be cloned into the `external/` directory in the root directory of the project. It's a shortcut of the `--files` option as these repository packages' names will be read by the tool automatically.
+* `debug` (`-d`) - Allows specifying custom debug flags. For example, the `--debug engine` option uncomments the `// @if CK_DEBUG_ENGINE //` lines in the code. By default `--debug` is set to true even if you did not specify it. This enables the base set of debug logs (`// @if CK_DEBUG //`) which should always be enabled in the testing environment. You can completely turn off the debug mode by setting the `--debug false` option or `--no-debug`.
 
-```bash
-$ npm t -- -c --files=enter,paragraph
-```
-
-Run `tests/view/**/*.js` tests from `ckeditor5-engine` and rerun them once any file change (the watch mode):
-
-```bash
-$ npm t -- -w --files=engine/view/
-```
-
-Test specified files in `ckeditor5-basic-styles` on two browsers (Chrome and Firefox):
-
-```bash
-$ npm t -- --browsers=Chrome,Firefox --files=basic-styles/bold,basic-styles/italic
-```
+Run with `--help` to see all available options.
 
-Test all installed packages:
+## Custom Vitest matchers
 
-```bash
-$ npm t -- --files=*
-```
+The `toEqualMarkup()` matcher tests whether two given strings containing markup language are equal, formatting the markup before showing a diff. Register it in a Vitest setup file:
 
-Test all installed packages except one (or more):
+```js
+import { expect } from 'vitest';
+import { toEqualMarkup } from '@ckeditor/ckeditor5-dev-tests';
 
-```bash
-$ npm t -- --files='!(engine)'
-$ npm t -- --files='!(engine|ui)'
+expect.extend( { toEqualMarkup } );
 ```
 
-## IDE integrations
-
-The CKEditor 5 can be integrated with IDEs via integrations.
-
-Currently only the IntelliJ based IDEs are supported (WebStorm, PHPStorm, etc). Detailed information are provided in [`bin/intellijkarmarunner/README.md`](./bin/intellijkarmarunner/README.md).
-
 ## Changelog
 
 See the [`CHANGELOG.md`](https://github.com/ckeditor/ckeditor5-dev/blob/master/packages/ckeditor5-dev-tests/CHANGELOG.md) file.

packages/ckeditor5-dev-tests/bin/testmanual.js

@@ -8,9 +8,10 @@
 import fs from 'node:fs';
 import { styleText } from 'node:util';
 import { loadEnvFile } from 'node:process';
-import * as tests from '../lib/index.js';
+import runManualTests from '../lib/tasks/runmanualtests.js';
+import parseArguments from '../lib/utils/parsearguments.js';
 
-const options = tests.parseArguments( process.argv.slice( 2 ), {
+const options = parseArguments( process.argv.slice( 2 ), {
 	allowDefaultIdentityFile: true,
 	commandName: 'ckeditor5-dev-tests-run-manual'
 } );
@@ -19,7 +20,7 @@ if ( fs.existsSync( '.env' ) ) {
 	loadEnvFile( '.env' );
 }
 
-tests.runManualTests( options )
+runManualTests( options )
 	.catch( error => {
 		const message = error instanceof Error ? error.stack : error.toString();
 		console.log( styleText( 'red', message ) );

packages/ckeditor5-dev-tests/lib/utils/automated-tests/leaksdetectorbefore.js → packages/ckeditor5-dev-tests/lib/index.d.ts

@@ -3,6 +3,4 @@
  * For licensing, see LICENSE.md.
  */
 
-beforeEach( function() {
-	this._lastDomElements = new WeakSet( document.body.children );
-} );
+export { toEqualMarkup, MarkupMatcherResult } from './vitest/matchers.js';

packages/ckeditor5-dev-tests/lib/index.js

@@ -3,6 +3,4 @@
  * For licensing, see LICENSE.md.
  */
 
-export { default as runAutomatedTests } from './tasks/runautomatedtests.js';
-export { default as runManualTests } from './tasks/runmanualtests.js';
-export { default as parseArguments } from './utils/parsearguments.js';
+export { toEqualMarkup } from './vitest/matchers.js';