1==========
2CRUD Tests
3==========
4
5.. contents::
6
7----
8
9Introduction
10============
11
12The YAML and JSON files in this directory tree are platform-independent tests
13that drivers can use to prove their conformance to the CRUD spec.
14
15Running these integration tests will require a running MongoDB server or
16cluster with server versions 2.6.0 or later. Some tests have specific server
17version requirements as noted by the ``runOn`` section, if provided.
18
19Subdirectories for Test Formats
20-------------------------------
21
22This document describes a legacy format for CRUD tests: legacy-v1, which dates back
23to the first version of the CRUD specification. New CRUD tests should be written
24in the `unified test format <../../../../unified-test-format/unified-test-format.rst>`_
25and placed under ``unified/``. Until such time that all original tests have been ported
26to the unified test format, tests in each format will be grouped in their own subdirectory:
27
28- ``v1/``: Legacy-v1 format tests
29- ``unified/``: Tests using the `unified test format <../../../../unified-test-format/unified-test-format.rst>`_
30
31Since some drivers may not have a unified test runner capable of executing tests
32in all two formats, segregating tests in this manner will make it easier for
33drivers to sync and feed test files to different test runners.
34
35Legacy-v1 Test Format for Single Operations
36-------------------------------------------
37
38*Note: this section pertains to test files in the "v1" directory.*
39
40The test format above supports both multiple operations and APM expectations,
41and is consistent with the formats used by other specifications. Previously, the
42CRUD spec tests used a simplified format that only allowed for executing a
43single operation. Notable differences from the legacy-v2 format are as follows:
44
45- Instead of a ``tests[i].operations`` array, a single operation was defined as
46 a document in ``tests[i].operation``. That document consisted of only the
47 ``name``, ``arguments``, and an optional ``object`` field.
48
49- Instead of ``error`` and ``result`` fields within each element in the
50 ``tests[i].operations`` array, the single operation's error and result were
51 defined under the ``tests[i].outcome.error`` and ``tests[i].outcome.result``
52 fields.
53
54- Instead of a top-level ``runOn`` field, server requirements are denoted by
55 separate top-level ``minServerVersion`` and ``maxServerVersion`` fields. The
56 minimum server version is an inclusive lower bound for running the test. The
57 maximum server version is an exclusive upper bound for running the test. If a
58 field is not present, it should be assumed that there is no corresponding bound
59 on the required server version.
60
61The legacy-v1 format should not conflict with the newer, multi-operation format
62used by other specs (e.g. Transactions). It is possible to create a unified test
63runner capable of executing both legacy formats (as some drivers do).
64
65Error Assertions for Bulk Write Operations
66==========================================
67
68When asserting errors (e.g. ``errorContains``, ``errorCodeName``) for bulk write
69operations, the test harness should inspect the ``writeConcernError`` and/or
70``writeErrors`` properties of the bulk write exception. This may not be needed for
71``errorContains`` if a driver concatenates all write and write concern error
72messages into the bulk write exception's top-level message.
73
74Test Runner Implementation
75==========================
76
77This section provides guidance for implementing a test runner for legacy-v1
78tests. See the `unified test format spec <../../../../unified-test-format/unified-test-format.rst>`_ for how to run tests under
79``unified/``.
80
81Before running the tests:
82
83- Create a global MongoClient (``globalMongoClient``) and connect to the server.
84 This client will be used for executing meta operations, such as checking
85 server versions and preparing data fixtures.
86
87For each test file:
88
89- Using ``globalMongoClient``, check that the current server version satisfies
90 one of the configurations provided in the top-level ``runOn`` field in the test
91 file (if applicable). If the
92 requirements are not satisfied, the test file should be skipped.
93
94- Determine the collection and database under test, utilizing the top-level
95 ``collection_name`` and/or ``database_name`` fields if present.
96
97- For each element in the ``tests`` array:
98
99 - Using ``globalMongoClient``, ensure that the collection and/or database
100 under test is in a "clean" state, as needed. This may be accomplished by
101 dropping the database; however, drivers may also decide to drop individual
102 collections as needed (this may be more performant).
103
104 - If the top-level ``data`` field is present in the test file, insert the
105 corresponding data into the collection under test using
106 ``globalMongoClient``.
107
108 - If the the ``failPoint`` field is present, use ``globalMongoClient`` to
109 configure the fail point on the primary server. See
110 `Server Fail Point <../../transactions/tests#server-fail-point>`_ in the
111 Transactions spec test documentation for more information.
112
113 - Create a local MongoClient (``localMongoClient``) and connect to the server.
114 This client will be used for executing the test case.
115
116 - If ``clientOptions`` is present, those options should be used to create
117 the client. Drivers MAY merge these options atop existing defaults (e.g.
118 reduced ``serverSelectionTimeoutMS`` value for faster test failures) at
119 their own discretion.
120
121 - Activate command monitoring for ``localMongoClient`` and begin capturing
122 events. Note that some events may need to be filtered out if the driver
123 uses global listeners or reports internal commands (e.g. ``hello``, legacy
124 hello, authentication).
125
126 - For each element in the ``operations`` array:
127
128 - Using ``localMongoClient``, select the appropriate ``object`` to execute
129 the operation. Default to the collection under test if this field is not
130 present.
131
132 - If ``collectionOptions`` is present, those options should be used to
133 construct the collection object.
134
135 - Given the ``name`` and ``arguments``, execute the operation on the object
136 under test. Capture the result of the operation, if any, and observe
137 whether an error occurred. If an error is encountered that includes a
138 result (e.g. BulkWriteException), extract the result object.
139
140 - If ``error`` is present and true, assert that the operation encountered an
141 error. Otherwise, assert that no error was encountered.
142
143 - if ``result`` is present, assert that it matches the operation's result.
144
145 - Deactivate command monitoring for ``localMongoClient``.
146
147 - If the ``expectations`` array is present, assert that the sequence of
148 emitted CommandStartedEvents from executing the operation(s) matches the
149 sequence of ``command_started_event`` objects in the ``expectations`` array.
150
151 - If the ``outcome`` field is present, assert the contents of the specified
152 collection using ``globalMongoClient``.
153 Note the server does not guarantee that documents returned by a find
154 command will be in inserted order. This find MUST sort by ``{_id:1}``.
155
156Evaluating Matches
157------------------
158
159The expected values for results (e.g. ``result`` for an operation
160operation, ``command_started_event.command``, elements in ``outcome.data``) are
161written in `Extended JSON <../../extended-json.rst>`_. Drivers may adopt any of
162the following approaches to comparisons, as long as they are consistent:
163
164- Convert ``actual`` to Extended JSON and compare to ``expected``
165- Convert ``expected`` and ``actual`` to BSON, and compare them
166- Convert ``expected`` and ``actual`` to native representations, and compare
167 them
168
169Extra Fields in Actual Documents
170~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
171
172When comparing ``actual`` and ``expected`` *documents*, drivers should permit
173``actual`` documents to contain additional fields not present in ``expected``.
174For example, the following documents match:
175
176- ``expected`` is ``{ "x": 1 }``
177- ``actual`` is ``{ "_id": { "$oid" : "000000000000000000000001" }, "x": 1 }``
178
179In this sense, ``expected`` may be a subset of ``actual``. It may also be
180helpful to think of ``expected`` as a form of query criteria. The intention
181behind this rule is that it is not always feasible for the test to express all
182fields in the expected document(s) (e.g. session and cluster time information
183in a ``command_started_event.command`` document).
184
185This rule for allowing extra fields in ``actual`` only applies for values that
186correspond to a document. For instance, an actual result of ``[1, 2, 3, 4]`` for
187a ``distinct`` operation would not match an expected result of ``[1, 2, 3]``.
188Likewise with the ``find`` operation, this rule would only apply when matching
189documents *within* the expected result array and actual cursor.
190
191Note that in the case of result objects for some CRUD operations, ``expected``
192may condition additional, optional fields (see:
193`Optional Fields in Expected Result Objects`_).
194
195Fields that must NOT be present in Actual Documents
196~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197
198Some command-started events in ``expectations`` include ``null`` values for
199optional fields such as ``allowDiskUse``.
200Tests MUST assert that the actual command **omits** any field that has a
201``null`` value in the expected command.
202
203Optional Fields in Expected Result Objects
204~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
205
206Some ``expected`` results may include fields that are optional in the CRUD
207specification, such as ``insertedId`` (for InsertOneResult), ``insertedIds``
208(for InsertManyResult), and ``upsertedCount`` (for UpdateResult). Drivers that
209do not implement these fields should ignore them when comparing ``actual`` with
210``expected``.
211
212Prose Tests
213===========
214
215The following tests have not yet been automated, but MUST still be tested.
216
2171. WriteConcernError.details exposes writeConcernError.errInfo
218--------------------------------------------------------------
219
220Test that ``writeConcernError.errInfo`` in a command response is propagated as
221``WriteConcernError.details`` (or equivalent) in the driver.
222
223Using a 4.0+ server, set the following failpoint:
224
225.. code:: javascript
226
227 {
228 "configureFailPoint": "failCommand",
229 "data": {
230 "failCommands": ["insert"],
231 "writeConcernError": {
232 "code": 100,
233 "codeName": "UnsatisfiableWriteConcern",
234 "errmsg": "Not enough data-bearing nodes",
235 "errInfo": {
236 "writeConcern": {
237 "w": 2,
238 "wtimeout": 0,
239 "provenance": "clientSupplied"
240 }
241 }
242 }
243 },
244 "mode": { "times": 1 }
245 }
246
247Then, perform an insert operation and assert that a WriteConcernError occurs and
248that its ``details`` property is both accessible and matches the ``errInfo``
249object from the failpoint.
250
2512. WriteError.details exposes writeErrors[].errInfo
252---------------------------------------------------
253
254Test that ``writeErrors[].errInfo`` in a command response is propagated as
255``WriteError.details`` (or equivalent) in the driver.
256
257Using a 5.0+ server, create a collection with
258`document validation <https://www.mongodb.com/docs/manual/core/schema-validation/>`_
259like so:
260
261.. code:: javascript
262
263 {
264 "create": "test",
265 "validator": {
266 "x": { $type: "string" }
267 }
268 }
269
270Enable `command monitoring <../../command-monitoring/command-monitoring.rst>`_
271to observe CommandSucceededEvents. Then, insert an invalid document (e.g.
272``{x: 1}``) and assert that a WriteError occurs, that its code is ``121``
273(i.e. DocumentValidationFailure), and that its ``details`` property is
274accessible. Additionally, assert that a CommandSucceededEvent was observed and
275that the ``writeErrors[0].errInfo`` field in the response document matches the
276WriteError's ``details`` property.
View as plain text