1======================================
2Client Side Operations Timeouts Tests
3======================================
4
5.. contents::
6
7----
8
9Introduction
10============
11
12This document describes the tests that drivers MUST run to validate the behavior of the timeoutMS option. These tests
13are broken up into automated YAML/JSON tests and additional prose tests.
14
15Spec Tests
16==========
17
18This directory contains a set of YAML and JSON spec tests. Drivers MUST run these as described in the "Unified Test
19Runner" specification. Because the tests introduced in this specification are timing-based, there is a risk that some
20of them may intermittently fail without any bugs being present in the driver. As a mitigatio, drivers MAY execute
21these tests in two new Evergreen tasks that use single-node replica sets: one with only authentication enabled and
22another with both authentication and TLS enabled. Drivers that choose to do so SHOULD use the ``single-node-auth.json``
23and ``single-node-auth-ssl.json`` files in the ``drivers-evergreen-tools`` repository to create these clusters.
24
25Prose Tests
26===========
27
28There are some tests that cannot be expressed in the unified YAML/JSON format. For each of these tests, drivers MUST
29create a MongoClient without the ``timeoutMS`` option set (referred to as ``internalClient``). Any fail points set
30during a test MUST be unset using ``internalClient`` after the test has been executed. All MongoClient instances
31created for tests MUST be configured with read/write concern ``majority``, read preference ``primary``, and command
32monitoring enabled to listen for ``command_started`` events.
33
34Multi-batch writes
35~~~~~~~~~~~~~~~~~~
36
37This test MUST only run against server versions 4.4 and higher.
38
39#. Using ``internalClient``, drop the ``db.coll`` collection.
40#. Using ``internalClient``, set the following fail point:
41
42 .. code:: javascript
43
44 {
45 configureFailPoint: "failCommand",
46 mode: {
47 times: 2
48 },
49 data: {
50 failCommands: ["insert"],
51 blockConnection: true,
52 blockTimeMS: 15
53 }
54 }
55
56#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
57#. Using ``client``, insert 100,001 empty documents in a single ``insertMany`` call.
58
59 - Expect this to fail with a timeout error.
60
61#. Verify that two ``insert`` commands were executed against ``db.coll`` as part of the ``insertMany`` call.
62
63maxTimeMS is not set for commands sent to mongocryptd
64~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
65
66This test MUST only be run against enterprise server versions 4.2 and higher.
67
68#. Launch a mongoryptd process on 23000.
69#. Create a MongoClient (referred to as ``client``) using the URI ``mongodb://localhost:23000/?timeoutMS=1000``.
70#. Using ``client``, execute the ``{ ping: 1 }`` command against the ``admin`` database.
71#. Verify via command monitoring that the ``ping`` command sent did not contain a ``maxTimeMS`` field.
72
73ClientEncryption
74~~~~~~~~~~~~~~~~
75
76Each test under this category MUST only be run against server versions 4.4 and higher. In these tests,
77``LOCAL_MASTERKEY`` refers to the following base64:
78
79.. code:: javascript
80
81 Mng0NCt4ZHVUYUJCa1kxNkVyNUR1QURhZ2h2UzR2d2RrZzh0cFBwM3R6NmdWMDFBMUN3YkQ5aXRRMkhGRGdQV09wOGVNYUMxT2k3NjZKelhaQmRCZGJkTXVyZG9uSjFk
82
83For each test, perform the following setup:
84
85#. Using ``internalClient``, drop and create the ``keyvault.datakeys`` collection.
86#. Create a MongoClient (referred to as ``keyVaultClient``) with ``timeoutMS=10``.
87#. Create a ``ClientEncryption`` object that wraps ``keyVaultClient`` (referred to as ``clientEncryption``). Configure this object with ``keyVaultNamespace`` set to ``keyvault.datakeys`` and the following KMS providers map:
88
89 .. code:: javascript
90
91 {
92 "local": { "key": <base64 decoding of LOCAL_MASTERKEY> }
93 }
94
95createDataKey
96`````````````
97
98#. Using ``internalClient``, set the following fail point:
99
100 .. code:: javascript
101
102 {
103 configureFailPoint: "failCommand",
104 mode: {
105 times: 1
106 },
107 data: {
108 failCommands: ["insert"],
109 blockConnection: true,
110 blockTimeMS: 15
111 }
112 }
113
114#. Call ``clientEncryption.createDataKey()`` with the ``local`` KMS provider.
115
116 - Expect this to fail with a timeout error.
117
118#. Verify that an ``insert`` command was executed against to ``keyvault.datakeys`` as part of the ``createDataKey`` call.
119
120encrypt
121```````
122
123#. Call ``client_encryption.createDataKey()`` with the ``local`` KMS provider.
124
125 - Expect a BSON binary with subtype 4 to be returned, referred to as ``datakeyId``.
126
127#. Using ``internalClient``, set the following fail point:
128
129 .. code:: javascript
130
131 {
132 configureFailPoint: "failCommand",
133 mode: {
134 times: 1
135 },
136 data: {
137 failCommands: ["find"],
138 blockConnection: true,
139 blockTimeMS: 15
140 }
141 }
142
143#. Call ``clientEncryption.encrypt()`` with the value ``hello``, the algorithm ``AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic``, and the keyId ``datakeyId``.
144
145 - Expect this to fail with a timeout error.
146
147#. Verify that a ``find`` command was executed against the ``keyvault.datakeys`` collection as part of the ``encrypt`` call.
148
149decrypt
150```````
151
152#. Call ``clientEncryption.createDataKey()`` with the ``local`` KMS provider.
153
154 - Expect this to return a BSON binary with subtype 4, referred to as ``dataKeyId``.
155
156#. Call ``clientEncryption.encrypt()`` with the value ``hello``, the algorithm ``AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic``, and the keyId ``dataKeyId``.
157
158 - Expect this to return a BSON binary with subtype 6, referred to as ``encrypted``.
159
160#. Close and re-create the ``keyVaultClient`` and ``clientEncryption`` objects.
161
162#. Using ``internalClient``, set the following fail point:
163
164 .. code:: javascript
165
166 {
167 configureFailPoint: "failCommand",
168 mode: {
169 times: 1
170 },
171 data: {
172 failCommands: ["find"],
173 blockConnection: true,
174 blockTimeMS: 15
175 }
176 }
177
178#. Call ``clientEncryption.decrypt()`` with the value ``encrypted``.
179
180 - Expect this to fail with a timeout error.
181
182#. Verify that a ``find`` command was executed against the ``keyvault.datakeys`` collection as part of the ``decrypt`` call.
183
184Background Connection Pooling
185~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
186
187The tests in this section MUST only be run if the server version is 4.4 or higher and the URI has authentication
188fields (i.e. a username and password). Each test in this section requires drivers to create a MongoClient and then wait
189for some CMAP events to be published. Drivers MUST wait for up to 10 seconds and fail the test if the specified events
190are not published within that time.
191
192timeoutMS used for handshake commands
193`````````````````````````````````````
194
195#. Using ``internalClient``, set the following fail point:
196
197 .. code:: javascript
198
199 {
200 configureFailPoint: "failCommand",
201 mode: {
202 times: 1
203 },
204 data: {
205 failCommands: ["saslContinue"],
206 blockConnection: true,
207 blockTimeMS: 15,
208 appName: "timeoutBackgroundPoolTest"
209 }
210 }
211
212#. Create a MongoClient (referred to as ``client``) configured with the following:
213
214 - ``minPoolSize`` of 1
215 - ``timeoutMS`` of 10
216 - ``appName`` of ``timeoutBackgroundPoolTest``
217 - CMAP monitor configured to listen for ``ConnectionCreatedEvent`` and ``ConnectionClosedEvent`` events.
218
219#. Wait for a ``ConnectionCreatedEvent`` and a ``ConnectionClosedEvent`` to be published.
220
221timeoutMS is refreshed for each handshake command
222`````````````````````````````````````````````````
223
224#. Using ``internalClient``, set the following fail point:
225
226 .. code:: javascript
227
228 {
229 configureFailPoint: "failCommand",
230 mode: "alwaysOn",
231 data: {
232 failCommands: ["isMaster", "saslContinue"],
233 blockConnection: true,
234 blockTimeMS: 15,
235 appName: "refreshTimeoutBackgroundPoolTest"
236 }
237 }
238
239#. Create a MongoClient (referred to as ``client``) configured with the following:
240
241 - ``minPoolSize`` of 1
242 - ``timeoutMS`` of 20
243 - ``appName`` of ``refreshTimeoutBackgroundPoolTest``
244 - CMAP monitor configured to listen for ``ConnectionCreatedEvent`` and ``ConnectionReady`` events.
245
246#. Wait for a ``ConnectionCreatedEvent`` and a ``ConnectionReady`` to be published.
247
248Blocking Iteration Methods
249~~~~~~~~~~~~~~~~~~~~~~~~~~
250
251Tests in this section MUST only be run against server versions 4.4 and higher and only apply to drivers that have a
252blocking method for cursor iteration that executes ``getMore`` commands in a loop until a document is available or an
253error occurs.
254
255Tailable cursors
256````````````````
257
258#. Using ``internalClient``, drop the ``db.coll`` collection.
259#. Using ``internalClient``, insert the document ``{ x: 1 }`` into ``db.coll``.
260#. Using ``internalClient``, set the following fail point:
261
262 .. code:: javascript
263
264 {
265 configureFailPoint: "failCommand",
266 mode: "alwaysOn",
267 data: {
268 failCommands: ["getMore"],
269 blockConnection: true,
270 blockTimeMS: 15
271 }
272 }
273
274#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
275#. Using ``client``, create a tailable cursor on ``db.coll`` with ``cursorType=tailable``.
276
277 - Expect this to succeed and return a cursor with a non-zero ID.
278
279#. Call either a blocking or non-blocking iteration method on the cursor.
280
281 - Expect this to succeed and return the document ``{ x: 1 }`` without sending a ``getMore`` command.
282
283#. Call the blocking iteration method on the resulting cursor.
284
285 - Expect this to fail with a timeout error.
286
287#. Verify that a ``find`` command and two ``getMore`` commands were executed against the ``db.coll`` collection during the test.
288
289Change Streams
290``````````````
291
292#. Using ``internalClient``, drop the ``db.coll`` collection.
293#. Using ``internalClient``, set the following fail point:
294
295 .. code:: javascript
296
297 {
298 configureFailPoint: "failCommand",
299 mode: "alwaysOn",
300 data: {
301 failCommands: ["getMore"],
302 blockConnection: true,
303 blockTimeMS: 15
304 }
305 }
306
307#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
308#. Using ``client``, use the ``watch`` helper to create a change stream against ``db.coll``.
309
310 - Expect this to succeed and return a change stream with a non-zero ID.
311
312#. Call the blocking iteration method on the resulting change stream.
313
314 - Expect this to fail with a timeout error.
315
316#. Verify that an ``aggregate`` command and two ``getMore`` commands were executed against the ``db.coll`` collection during the test.
317
318GridFS - Upload
319~~~~~~~~~~~~~~~
320
321Tests in this section MUST only be run against server versions 4.4 and higher.
322
323uploads via openUploadStream can be timed out
324`````````````````````````````````````````````
325
326#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
327#. Using ``internalClient``, set the following fail point:
328
329 .. code:: javascript
330
331 {
332 configureFailPoint: "failCommand",
333 mode: { times: 1 },
334 data: {
335 failCommands: ["insert"],
336 blockConnection: true,
337 blockTimeMS: 15
338 }
339 }
340
341#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
342#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database.
343#. Call ``bucket.open_upload_stream()`` with the filename ``filename`` to create an upload stream (referred to as ``uploadStream``).
344
345 - Expect this to succeed and return a non-null stream.
346
347#. Using ``uploadStream``, upload a single ``0x12`` byte.
348#. Call ``uploadStream.close()`` to flush the stream and insert chunks.
349
350 - Expect this to fail with a timeout error.
351
352Aborting an upload stream can be timed out
353``````````````````````````````````````````
354
355This test only applies to drivers that provide an API to abort a GridFS upload stream.
356
357#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
358#. Using ``internalClient``, set the following fail point:
359
360 .. code:: javascript
361
362 {
363 configureFailPoint: "failCommand",
364 mode: { times: 1 },
365 data: {
366 failCommands: ["delete"],
367 blockConnection: true,
368 blockTimeMS: 15
369 }
370 }
371
372#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
373#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database with ``chunkSizeBytes=2``.
374#. Call ``bucket.open_upload_stream()`` with the filename ``filename`` to create an upload stream (referred to as ``uploadStream``).
375
376 - Expect this to succeed and return a non-null stream.
377
378#. Using ``uploadStream``, upload the bytes ``[0x01, 0x02, 0x03, 0x04]``.
379#. Call ``uploadStream.abort()``.
380
381 - Expect this to fail with a timeout error.
382
383GridFS - Download
384~~~~~~~~~~~~~~~~~
385
386This test MUST only be run against server versions 4.4 and higher.
387
388#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
389#. Using ``internalClient``, insert the following document into the ``db.fs.files`` collection:
390
391 .. code:: javascript
392
393 {
394 "_id": {
395 "$oid": "000000000000000000000005"
396 },
397 "length": 10,
398 "chunkSize": 4,
399 "uploadDate": {
400 "$date": "1970-01-01T00:00:00.000Z"
401 },
402 "md5": "57d83cd477bfb1ccd975ab33d827a92b",
403 "filename": "length-10",
404 "contentType": "application/octet-stream",
405 "aliases": [],
406 "metadata": {}
407 }
408
409#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
410#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database.
411#. Call ``bucket.open_download_stream`` with the id ``{ "$oid": "000000000000000000000005" }`` to create a download stream (referred to as ``downloadStream``).
412
413 - Expect this to succeed and return a non-null stream.
414
415#. Using ``internalClient``, set the following fail point:
416
417 .. code:: javascript
418
419 {
420 configureFailPoint: "failCommand",
421 mode: { times: 1 },
422 data: {
423 failCommands: ["find"],
424 blockConnection: true,
425 blockTimeMS: 15
426 }
427 }
428
429#. Read from the ``downloadStream``.
430
431 - Expect this to fail with a timeout error.
432
433#. Verify that two ``find`` commands were executed during the read: one against ``db.fs.files`` and another against ``db.fs.chunks``.
434
435Server Selection
436~~~~~~~~~~~~~~~~
437
438serverSelectionTimeoutMS honored if timeoutMS is not set
439````````````````````````````````````````````````````````
440
441#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?serverSelectionTimeoutMS=10``.
442
443#. Using ``client``, execute the command ``{ ping: 1 }`` against the ``admin`` database.
444
445 - Expect this to fail with a server selection timeout error after no more than 15ms.
446
447timeoutMS honored for server selection if it's lower than serverSelectionTimeoutMS
448``````````````````````````````````````````````````````````````````````````````````
449
450#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=10&serverSelectionTimeoutMS=20``.
451
452#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.
453
454 - Expect this to fail with a server selection timeout error after no more than 15ms.
455
456serverSelectionTimeoutMS honored for server selection if it's lower than timeoutMS
457``````````````````````````````````````````````````````````````````````````````````
458
459#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=20&serverSelectionTimeoutMS=10``.
460
461#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.
462
463 - Expect this to fail with a server selection timeout error after no more than 15ms.
464
465serverSelectionTimeoutMS honored for server selection if timeoutMS=0
466````````````````````````````````````````````````````````````````````
467
468#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=0&serverSelectionTimeoutMS=10``.
469
470#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.
471
472 - Expect this to fail with a server selection timeout error after no more than 15ms.
473
474timeoutMS honored for connection handshake commands if it's lower than serverSelectionTimeoutMS
475```````````````````````````````````````````````````````````````````````````````````````````````
476
477This test MUST only be run if the server version is 4.4 or higher and the URI has authentication fields (i.e. a
478username and password).
479
480#. Using ``internalClient``, set the following fail point:
481
482 .. code:: javascript
483
484 {
485 configureFailPoint: failCommand,
486 mode: { times: 1 },
487 data: {
488 failCommands: ["saslContinue"],
489 blockConnection: true,
490 blockTimeMS: 15
491 }
492 }
493
494#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10`` and ``serverSelectionTimeoutMS=20``.
495#. Using ``client``, insert the document ``{ x: 1 }`` into collection ``db.coll``.
496
497 - Expect this to fail with a timeout error after no more than 15ms.
498
499serverSelectionTimeoutMS honored for connection handshake commands if it's lower than timeoutMS
500```````````````````````````````````````````````````````````````````````````````````````````````
501
502This test MUST only be run if the server version is 4.4 or higher and the URI has authentication fields (i.e. a
503username and password).
504
505#. Using ``internalClient``, set the following fail point:
506
507 .. code:: javascript
508
509 {
510 configureFailPoint: failCommand,
511 mode: { times: 1 },
512 data: {
513 failCommands: ["saslContinue"],
514 blockConnection: true,
515 blockTimeMS: 15
516 }
517 }
518
519#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20`` and ``serverSelectionTimeoutMS=10``.
520#. Using ``client``, insert the document ``{ x: 1 }`` into collection ``db.coll``.
521
522 - Expect this to fail with a timeout error after no more than 15ms.
523
524endSession
525~~~~~~~~~~
526
527This test MUST only be run against replica sets and sharded clusters with server version 4.4 or higher. It MUST be
528run three times: once with the timeout specified via the MongoClient ``timeoutMS`` option, once with the timeout
529specified via the ClientSession ``defaultTimeoutMS`` option, and once more with the timeout specified via the
530``timeoutMS`` option for the ``endSession`` operation. In all cases, the timeout MUST be set to 10 milliseconds.
531
532#. Using ``internalClient``, drop the ``db.coll`` collection.
533#. Using ``internalClient``, set the following fail point:
534
535 .. code:: javascript
536
537 {
538 configureFailPoint: failCommand,
539 mode: { times: 1 },
540 data: {
541 failCommands: ["abortTransaction"],
542 blockConnection: true,
543 blockTimeMS: 15
544 }
545 }
546
547#. Create a new MongoClient (referred to as ``client``) and an explicit ClientSession derived from that MongoClient (referred to as ``session``).
548#. Execute the following code:
549
550 .. code:: typescript
551
552 coll = client.database("db").collection("coll")
553 session.start_transaction()
554 coll.insert_one({x: 1}, session=session)
555
556#. Using ``session``, execute ``session.end_session``
557
558 - Expect this to fail with a timeout error after no more than 15ms.
559
560Convenient Transactions
561~~~~~~~~~~~~~~~~~~~~~~~
562
563Tests in this section MUST only run against replica sets and sharded clusters with server versions 4.4 or higher.
564
565timeoutMS is refreshed for abortTransaction if the callback fails
566`````````````````````````````````````````````````````````````````
567
568#. Using ``internalClient``, drop the ``db.coll`` collection.
569#. Using ``internalClient``, set the following fail point:
570
571 .. code:: javascript
572
573 {
574 configureFailPoint: failCommand,
575 mode: { times: 2 },
576 data: {
577 failCommands: ["insert", "abortTransaction"],
578 blockConnection: true,
579 blockTimeMS: 15
580 }
581 }
582
583#. Create a new MongoClient (referred to as ``client``) configured with ``timeoutMS=10`` and an explicit ClientSession derived from that MongoClient (referred to as ``session``).
584#. Using ``session``, execute a ``withTransaction`` operation with the following callback:
585
586 .. code:: typescript
587
588 def callback() {
589 coll = client.database("db").collection("coll")
590 coll.insert_one({ _id: 1 }, session=session)
591 }
592
593#. Expect the previous ``withTransaction`` call to fail with a timeout error.
594#. Verify that the following events were published during the ``withTransaction`` call:
595
596 #. ``command_started`` and ``command_failed`` events for an ``insert`` command.
597 #. ``command_started`` and ``command_failed`` events for an ``abortTransaction`` command.
598
599Unit Tests
600==========
601
602The tests enumerated in this section could not be expressed in either spec or prose format. Drivers SHOULD implement
603these if it is possible to do so using the driver's existing test infrastructure.
604
605- Operations should ignore ``waitQueueTimeoutMS`` if ``timeoutMS`` is also set.
606- If ``timeoutMS`` is set for an operation, the remaining ``timeoutMS`` value should apply to connection checkout after a server has been selected.
607- If ``timeoutMS`` is not set for an operation, ``waitQueueTimeoutMS`` should apply to connection checkout after a server has been selected.
608- If a new connection is required to execute an operation, ``min(remaining computedServerSelectionTimeout, connectTimeoutMS)`` should apply to socket establishment.
609- For drivers that have control over OCSP behavior, ``min(remaining computedServerSelectionTimeout, 5 seconds)`` should apply to HTTP requests against OCSP responders.
610- If ``timeoutMS`` is unset, operations fail after two non-consecutive socket timeouts.
611- The remaining ``timeoutMS`` value should apply to HTTP requests against KMS servers for CSFLE.
612- The remaining ``timeoutMS`` value should apply to commands sent to mongocryptd as part of automatic encryption.
613- When doing ``minPoolSize`` maintenance, ``connectTimeoutMS`` is used as the timeout for socket establishment.
View as plain text