luna - wolfssl/README-async.md

  1# wolfSSL / wolfCrypt Asynchronous Support
  2
  3The asynchronous code was previously maintained at https://github.com/wolfSSL/wolfAsyncCrypt but was integrated into wolfSSL proper starting with release v5.8.4.
  4
  5This feature is enabled using:
  6`./configure --enable-asynccrypt` or `#define WOLFSSL_ASYNC_CRYPT`.
  7
  8If async crypto is enabled but no hardware backend is enabled or if `WOLFSSL_ASYNC_CRYPT_SW` is defined, a software backend using wolfCrypt is used instead. This software backend can simulate periodic hardware delays using the macro `WOLF_ASYNC_SW_SKIP_MOD`, which is on by default if `DEBUG_WOLFSSL` is defined.
  9
 10## Design
 11
 12Each crypto algorithm has its own `WC_ASYNC_DEV` structure, which contains a `WOLF_EVENT`, local crypto context and local hardware context.
 13
 14For SSL/TLS the `WOLF_EVENT` context is the `WOLFSSL*` and the type is `WOLF_EVENT_TYPE_ASYNC_WOLFSSL`. For wolfCrypt operations the `WOLF_EVENT` context is the `WC_ASYNC_DEV*` and the type is `WOLF_EVENT_TYPE_ASYNC_WOLFCRYPT`.
 15
 16A generic event system has been created using a `WOLF_EVENT` structure when `HAVE_WOLF_EVENT` is defined. The event structure resides in the `WC_ASYNC_DEV`.
 17
 18The asynchronous crypto system is modeled after epoll. The implementation uses `wolfSSL_AsyncPoll` or `wolfSSL_CTX_AsyncPoll` to check if any async operations are complete.
 19
 20## Hardware Backends
 21
 22Supported hardware backends:
 23
 24* Intel QuickAssist with QAT 1.6 or QAT 1.7 driver. See README.md in `wolfcrypt/src/port/intel/README.md`.
 25* Cavium Nitrox III and V. See README.md in `wolfcrypt/src/port/cavium/README.md`.
 26
 27## wolfCrypt Backend
 28
 29The wolfCrypt backend uses the same API as the hardware backends do. Once an asynchronous operation is initiated with the software backend, subsequent calls to `wolfSSL_AsyncPoll` will call into wolfCrypt to complete the operation. If non-blocking is enabled, for example, for ECC (via `WC_ECC_NONBLOCK`), each `wolfSSL_AsyncPoll` will do a chunk of work for the operation and return, to minimize blocking time.
 30
 31## API's
 32
 33### ```wolfSSL_AsyncPoll```
 34```
 35int wolfSSL_AsyncPoll(WOLFSSL* ssl, WOLF_EVENT_FLAG flags);
 36```
 37
 38Polls the provided WOLFSSL object's reference to the WOLFSSL_CTX's event queue to see if any operations outstanding for the WOLFSSL object are done. Return the completed event count on success.
 39
 40### ```wolfSSL_CTX_AsyncPoll```
 41```
 42int wolfSSL_CTX_AsyncPoll(WOLFSSL_CTX* ctx, WOLF_EVENT** events, int maxEvents, WOLF_EVENT_FLAG flags, int* eventCount)
 43```
 44
 45Polls the provided WOLFSSL_CTX context event queue to see if any pending events are done. If the `events` argument is provided then a pointer to the `WOLF_EVENT` will be returned up to `maxEvents`. If `eventCount` is provided then the number of events populated will be returned. The `flags` allows for `WOLF_POLL_FLAG_CHECK_HW` to indicate if the crypto backend (i.e. hardware or wolfCrypt, if the software implementation is being used) should be polled again or just return more events.
 46
 47### ```wolfAsync_DevOpen```
 48```
 49int wolfAsync_DevOpen(int *devId);
 50```
 51
 52Open the async device and returns an `int` device id for it.
 53
 54### ```wolfAsync_DevOpenThread```
 55```
 56int wolfAsync_DevOpenThread(int *devId, void* threadId);
 57```
 58Opens the async device for a specific thread. A crypto instance is assigned and thread affinity set.
 59
 60### ```wolfAsync_DevClose```
 61```
 62void wolfAsync_DevClose(int *devId)
 63```
 64
 65Closes the async device.
 66
 67### ```wolfAsync_DevCopy```
 68```
 69int wolfAsync_DevCopy(WC_ASYNC_DEV* src, WC_ASYNC_DEV* dst);
 70```
 71
 72Copy async device memory safe (not pointers to old device).
 73
 74### ```wolfAsync_DevCtxInit```
 75```
 76int wolfAsync_DevCtxInit(WC_ASYNC_DEV* asyncDev, word32 marker, void* heap, int devId);
 77```
 78
 79Initialize the device context and open the device hardware using the provided `WC_ASYNC_DEV ` pointer, marker and device id (from wolfAsync_DevOpen).
 80
 81### ```wolfAsync_DevCtxFree```
 82```
 83void wolfAsync_DevCtxFree(WC_ASYNC_DEV* asyncDev);
 84```
 85
 86Closes and free's the device context.
 87
 88
 89### ```wolfAsync_EventInit```
 90```
 91int wolfAsync_EventInit(WOLF_EVENT* event, enum WOLF_EVENT_TYPE type, void* context, word32 flags);
 92```
 93
 94Initialize an event structure with provided type and context. Sets the pending flag and the status code to `WC_PENDING_E`. Current flag options are `WC_ASYNC_FLAG_NONE` and `WC_ASYNC_FLAG_CALL_AGAIN` (indicates crypto needs called again after WC_PENDING_E).
 95
 96### ```wolfAsync_EventWait ```
 97```
 98int wolfAsync_EventWait(WOLF_EVENT* event);
 99```
100
101Waits for the provided event to complete.
102
103### ```wolfAsync_EventPoll```
104```
105int wolfAsync_EventPoll(WOLF_EVENT* event, WOLF_EVENT_FLAG event_flags);
106```
107
108Polls the provided event to determine if its done.
109
110### ```wolfAsync_EventPop ```
111
112```
113int wolfAsync_EventPop(WOLF_EVENT* event, enum WOLF_EVENT_TYPE event_type);
114```
115
116This will check the event to see if the event type matches and the event is complete. If it is then the async return code is returned. If not then `WC_NOT_PENDING_E` is returned.
117
118
119### ```wolfAsync_EventQueuePush```
120```
121int wolfAsync_EventQueuePush(WOLF_EVENT_QUEUE* queue, WOLF_EVENT* event);
122```
123
124Pushes an event to the provided event queue and assigns the provided event.
125
126### ```wolfAsync_EventQueuePoll```
127```
128int wolfAsync_EventQueuePoll(WOLF_EVENT_QUEUE* queue, void* context_filter,
129    WOLF_EVENT** events, int maxEvents, WOLF_EVENT_FLAG event_flags, int* eventCount);
130```
131
132Polls all events in the provided event queue. Optionally filters by context. Will return pointers to the done events.
133
134### ```wc_AsyncHandle```
135```
136int wc_AsyncHandle(WC_ASYNC_DEV* asyncDev, WOLF_EVENT_QUEUE* queue, word32 flags);
137```
138
139This will push the event inside asyncDev into the provided queue.
140
141### ```wc_AsyncWait```
142```
143int wc_AsyncWait(int ret, WC_ASYNC_DEV* asyncDev, word32 flags);
144```
145
146This will wait until the provided asyncDev is done (or error).
147
148### ```wolfAsync_HardwareStart```
149```
150int wolfAsync_HardwareStart(void);
151```
152
153If using multiple threads this allows a way to start the hardware before using `wolfAsync_DevOpen` to ensure the memory system is setup. Ensure that `wolfAsync_HardwareStop` is called on exit. Internally there is a start/stop counter, so this can be called multiple times, but stop must also be called the same number of times to shutdown the hardware.
154
155### ```wolfAsync_HardwareStop```
156```
157void wolfAsync_HardwareStop(void);
158```
159
160Stops hardware if internal `--start_count == 0`.
161
162## Examples
163
164### TLS Server Example
165
166```c
167int devId = INVALID_DEVID;
168
169ret = wolfAsync_DevOpen(&devId);
170if (ret != 0) {
171    err_sys("Async device open failed");
172}
173wolfSSL_CTX_SetDevId(ctx, devId);
174
175do {
176    err = 0; /* reset error */
177    ret = wolfSSL_accept(ssl, msg, msgSz, &msgSz);
178    if (ret <= 0) {
179        err = wolfSSL_get_error(ssl, 0);
180        if (err == WC_PENDING_E) {
181            ret = wolfSSL_AsyncPoll(ssl, WOLF_POLL_FLAG_CHECK_HW);
182            if (ret < 0) break;
183        }
184    }
185} while (err == WC_PENDING_E);
186if (ret != WOLFSSL_SUCCESS) {
187    err_sys("SSL_connect failed");
188}
189
190wolfAsync_DevClose(&devId);
191```
192
193### wolfCrypt RSA Example
194
195```c
196static int devId = INVALID_DEVID;
197RsaKey key;
198
199ret = wolfAsync_DevOpen(&devId);
200if (ret != 0)
201    err_sys("Async device open failed");
202
203wc_InitRsaKey_ex(&key, HEAP_HINT, devId);
204if (ret == 0) {
205    ret = wc_RsaPrivateKeyDecode(tmp, &idx, &key, (word32)bytes);
206    do {
207        ret = wc_AsyncWait(ret, &key.asyncDev, WC_ASYNC_FLAG_CALL_AGAIN);
208        if (ret >= 0)
209            ret = wc_RsaPublicEncrypt(in, inLen, out, outSz, &key, &rng);
210    } while (ret == WC_PENDING_E);
211    wc_FreeRsaKey(&key);
212}
213
214wolfAsync_DevClose(&devId);
215```
216
217## Build Options
218
2191. Async multi-threading can be disabled by defining `WC_NO_ASYNC_THREADING`. This only disables internal async threading functions. You are free to use other threading APIs or paradigms in your application.
2202. Software benchmarks can be disabled by defining `NO_SW_BENCH`.
2213. The `WC_ASYNC_THRESH_NONE` define can be used to disable the cipher thresholds, which are tunable values to determine at what size hardware should be used vs. software.
2224. Use `WOLFSSL_DEBUG_MEMORY` and `WOLFSSL_TRACK_MEMORY` to help debug memory issues. QAT also supports `WOLFSSL_DEBUG_MEMORY_PRINT`.
223
224
225## References
226
227### TLS Client/Server Async Example
228
229We have a full TLS client/server async examples here:
230
231* [https://github.com/wolfSSL/wolfssl-examples/blob/master/tls/server-tls-epoll-perf.c](https://github.com/wolfSSL/wolfssl-examples/blob/master/tls/server-tls-epoll-perf.c)
232
233* [https://github.com/wolfSSL/wolfssl-examples/blob/master/tls/client-tls-perf.c](https://github.com/wolfSSL/wolfssl-examples/blob/master/tls/client-tls-perf.c)
234
235#### TLS Threaded epoll Example Building
236
237```sh
238git clone git@github.com:wolfSSL/wolfssl-examples.git
239cd wolfssl-examples
240cd tls
241# For QuickAssist: Uncomment QAT lines at top of Makefile
242make
243```
244
245#### TLS Threaded epoll Example Usage
246
247```sh
248$ ./client-tls-perf -?
249perf 4.5.0 (NOTE: All files relative to wolfSSL home dir)
250-?          Help, print this usage
251-p <num>    Port to listen on, not 0, default 11111
252-v <num>    SSL version [0-3], SSLv3(0) - TLS1.2(3)), default 3
253-l <str>    Cipher suite list (: delimited)
254-c <file>   Certificate file,           default ../certs/client-cert.pem
255-k <file>   Key file,                   default ../certs/client-key.pem
256-A <file>   Certificate Authority file, default ../certs/ca-cert.pem
257-r          Resume session
258-n <num>    Benchmark <num> connections
259-N <num>    <num> concurrent connections
260-R <num>    <num> bytes read from client
261-W <num>    <num> bytes written to client
262-B <num>    Benchmark <num> written bytes
263```
264
265#### TLS Threaded epoll Example Output
266
267```sh
268$ sudo ./server-tls-epoll-threaded -n 10000
269$ sudo ./client-tls-perf -n 10000
270
271wolfSSL Server Benchmark 16384 bytes
272    Num Conns         :     10000
273    Total             : 18575.800 ms
274    Total Avg         :     1.858 ms
275    t/s               :   538.335
276    Accept            : 35848.428 ms
277    Accept Avg        :     3.585 ms
278    Total Read bytes  : 163840000 bytes
279    Total Write bytes : 163840000 bytes
280    Read              :   402.212 ms (  388.476 MBps)
281    Write             :   591.469 ms (  264.173 MBps)
282```
283
284## Change Log
285
286### wolfSSL Async Release v5.8.0 (May 01, 2025)
287* Includes all wolfSSL v5.8.0 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-580-apr-24-2025
288* Update for libwolfssl_sources.h refactor. (https://github.com/wolfSSL/wolfAsyncCrypt/pull/77)
289
290### wolfSSL Async Release v5.7.4 (Oct 29, 2024)
291* Includes all wolfSSL v5.7.4 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-574-oct-24-2024
292 - Plus fixes for asynchronous release - SHA3/HMAC devId (https://github.com/wolfSSL/wolfssl/pull/8119)
293* Fix for Intel QuickAssist RSA Key generation exponent result. (https://github.com/wolfSSL/wolfAsyncCrypt/pull/75)
294
295### wolfSSL Async Release v5.7.0 (Mar 21, 2023)
296* Includes all wolfSSL v5.7.0 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-570-mar-20-2024
297
298### wolfSSL Async Release v5.6.6 (Dec 20, 2023)
299* Includes all wolfSSL v5.6.6 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-566-dec-19-2023
300 - Plus wolfSSL PR 7085 fix for invalid `dh_ffdhe_test` for even P when using Intel QuickAssist. https://github.com/wolfSSL/wolfssl/pull/7085
301* Fix for missing `IntelQaFreeFlatBuffer` with DH enabled and no keygen. (broken in PR #71)
302* Add return code checking for wc_AsyncThreadCreate_ex in exit_fail section for pthread_attr_destroy. (PR #72)
303
304### wolfSSL Async Release v5.6.4 (Oct 30, 2023)
305* Fixes for support async with crypto or pk callbacks.
306* Rename `WC_NOT_PENDING_E` -> `WC_NO_PENDING_E`
307
308### wolfSSL Async Release v5.6.3 (June 16, 2023)
309* Includes all wolfSSL v5.6.3 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-563-jun-16-2023
310* Add sanity check of index devId before accessing array
311* Use the blocking call from the async test
312
313### wolfSSL Async Release v5.6.0 (Mar 29, 2023)
314* Includes all wolfSSL v5.6.0 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-560-mar-24-2023
315* wolfAsyncCrypt github repository became public.
316
317### wolfSSL Async Release v5.5.4 (Dec 22, 2022)
318
319* Includes all wolfSSL v5.5.4 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-554-dec-21-2022
320* Use the `wc_ecc_shared_secret_ex` version for async test. Requires https://github.com/wolfSSL/wolfssl/pull/5868
321
322### wolfSSL Async Release v5.5.3 (Nov 8, 2022)
323
324* Includes all wolfSSL v5.5.1-v5.5.3 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-553-nov-2-2022
325* Fix for Intel QAT handling of sign R when cofactor is not 1. https://github.com/wolfSSL/wolfssl/pull/5737 and https://github.com/wolfSSL/wolfAsyncCrypt/pull/54
326* Fix check scalar bits for ECC cofactor. https://github.com/wolfSSL/wolfssl/pull/5737
327* Fixes for async sniffer: https://github.com/wolfSSL/wolfssl/pull/5734
328  - Handling of packets with multiple TLS messages.
329  - Multiple back to back sessions.
330  - Ensure all pending queued packets are finished before ending pcap processing.
331* Fix for various tests that do not properly handle `WC_PENDING_E`. https://github.com/wolfSSL/wolfssl/pull/5773
332* Revert "Fix for sniffer to decode out of order packets". https://github.com/wolfSSL/wolfssl/pull/5771
333
334### wolfSSL Async Release v5.5.0 (Sep 2, 2022)
335
336* Includes all wolfSSL v5.5.0 fixes. See ChangeLog.md here: https://github.com/wolfSSL/wolfssl/blob/master/ChangeLog.md#wolfssl-release-550-aug-30-2022
337* Fix for handling return codes from `pthread_attr_destroy`.
338* Fix for async session tickets. https://github.com/wolfSSL/wolfssl/pull/5534
339* Fix for async with OCSP non-blocking in ProcessPeerCerts. https://github.com/wolfSSL/wolfssl/pull/5539
340
341### wolfSSL Async Release v5.4.0 (July 11, 2022)
342* Fix for DH trim of leading zeros to use memmove.
343* Fix to print errors to stderr.
344* Fix to consistently return the status of failed pthreads funcs.
345* Move async device pointer (https://github.com/wolfSSL/wolfssl/pull/5149)
346
347### wolfSSL Async Release v5.3.0 (May 5, 2022)
348
349* Added Intel QuickAssist ECC Key Generation acceleration. Specifically point multiplication similar to our `wc_ecc_mulmod_ex2`.
350* Fix for building Intel QAT with SP math all
351* Fix for `error: unused function 'IntelQaFreeFlatBuffer'`.
352* Fix for handling the Koblitz curve param "a", which is all zeros.
353* Fixes for scan-build warnings.
354* Includes wolfSSL PR https://github.com/wolfSSL/wolfssl/pull/5101
355
356### wolfSSL Async Release v5.2.0 (Feb 21, 2022)
357
358* Adds `WC_NO_ASYNC_SLEEP` option to hide wc_AsyncSleep for platforms that do not need it.
359* Fix for async test anonymous union on some platforms (`#pragma anon_unions` and `HAVE_ANONYMOUS_INLINE_AGGREGATES`)
360* Fixes for invalidPrintfArgType_sint (cppcheck) and readability-redundant-preprocessor (clang-tidy).
361
362### wolfSSL Async Release v5.1.0 (Jan 3rd, 2022)
363
364
365### wolfSSL Async Release v5.0.0 (11/01/2021)
366
367* Fix for issue with QAT AES GCM input buffer already NUMA and not aligned.
368
369### wolfSSL Async Release v4.8.0 (07/14/2021)
370
371* Fix for new QAT 1.7 hash types warning.
372* Updated Intel QAT 1.7 build instructions.
373* Includes possible HAVE_WOLF_BIGINT leaks in PR https://github.com/wolfSSL/wolfssl/pull/4208
374
375### wolfSSL Async Release v4.7.0 (02/20/2021)
376
377* Fix for ARC4 macro typo
378
379### wolfSSL Async Release v4.6.0 (12/21/2020)
380
381* Documentation updates.
382* Fixes for Cavium Nitrox and Intel Quick Assist (wolfSSL/wolfssl#3577) with TLS v1.3
383
384### wolfSSL Async Release v4.4.0 (04/24/2020)
385
386* Fix for uninitialized `supSha3` warning.
387* Fix for use of incorrect devId for wolfSSL_SHA3_256_Init.
388* Fix for QAT with Shake256.
389* Fix for QAT example `./build.sh`.
390
391### wolfSSL Async Release v4.3.0 (12/20/2019)
392
393* Fix for async date override callback issue.
394* Updates to Octeon README.
395
396### wolfSSL Async Release v4.2.0 (10/22/2019)
397
398* Fix for QuickAssist DH Agree issue with leading zero bytes.
399* Fix for QuickAssist AES CBC issue with previous IV on back-to-back operations.
400* Updates to QuickAssist README.md for latest QAT v1.7 driver.
401* Instructions for Octeon III (CN7300) use.
402
403### wolfSSL Async Release v4.0.0 (03/25/2019)
404
405* Fix for building with QuickAssist v1.7 driver (4.4.0-00023) (was missing usdm_drv during configure with check).
406* Fix for building async with file system disabled.
407* Fix for SHA-3 runtime detection for not supported in hardware.
408
409### wolfSSL Async Release v3.15.8 (03/01/2019) - Intermediate release
410
411* Performance improvements for QuickAssist.
412* Added new build option `QAT_POLL_RESP_QUOTA` to indicate maximum number of callbacks to service per poll. The default is 0 (all), was previously 8.
413* Added useful QAT_DEBUG logging for ECC and DH operations.
414* Cleanup whitespace in quickassist.c.
415* Enhanced the Cavium macros for `CAVIUM_MAX_PENDING` and `CAVIUM_MAX_POLL` over-ridable.
416* Added build-time override for benchmark thread count `WC_ASYNC_BENCH_THREAD_COUNT`.
417* Fixes for wolfCrypt test with asynchronous support enabled and `--enable-nginx`.
418* Fix to use QAT for ECC sign and verify when SP is enabled and key was initialized with devId.
419* Fixes issues with wolfCrypt test and QAT not properly calling "again" for the ECC sign, verify and shared secret.
420* Correct the output for multi-threaded benchmark using `-base10` option.
421* Fixes to QAT HMAC enables in benchmark tool.
422* Adds new `NO_HW_BENCH` to support using multi-threaded software only benchmarks.
423
424### wolfSSL Async Release v3.15.7 (12/27/2018)
425
426* Fixes for various analysis warnings (https://github.com/wolfSSL/wolfssl/pull/2003).
427* Added QAT v1.7 driver support.
428* Added QAT SHA-3 support.
429* Added QAT RSA Key Generation support.
430* Added support for new usdm memory driver.
431* Added support for detecting QAT version and features.
432* Added `QAT_ENABLE_RNG` option to disable QAT TRNG/DRBG.
433* Added alternate hashing method to cache all updates (avoids using partial updates).
434
435### wolfSSL Async Release v3.15.5 (11/09/2018)
436
437* Fixes for various analysis warnings (https://github.com/wolfSSL/wolfssl/pull/1918).
438* Fix for QAT possible double free case where `ctx->symCtx` is not trapped.
439* Improved QAT debug messages when using `QAT_DEBUG`.
440* Fix for QAT RNG to allow zero length. This resolves PSS case where `wc_RNG_GenerateBlock` is called for saltLen == 0.
441
442
443### wolfSSL Async Release v3.15.3 (06/20/2018)
444
445* Fixes for fsantize tests with Cavium Nitrox V.
446* Removed typedef for `CspHandle`, since its already defined.
447* Fixes for a couple of fsanitize warnings.
448* Fix for possible leak with large request to `IntelQaDrbg`.
449
450### wolfSSL Async Release v3.14.4 (04/13/2018)
451
452* Added Nitrox V ECC.
453* Added Nitrox V SHA-224 and SHA-3
454* Added Nitrox V AES GCM
455* Added Nitrox III SHA2 384/512 support for HMAC.
456* Added error code handling for signature check failure.
457* Added error translate for `ERR_PKCS_DECRYPT_INCORRECT`
458* Added useful `WOLFSSL_NITROX_DEBUG` and show count for pending checks.
459* Cleanup of Nitrox symmetric processing to use single while loops.
460* Cleanup to only include some headers in cavium_nitrox.c port.
461* Fixes for building against Nitrox III and V SDK.
462* Updates to README.md with required CFLAGS/LDFLAGS when building without ./configure.
463* Fix for Intel QuickAssist HMAC to use software for unsupported hash algorithms.
464
465
466### wolfSSL Async Release v3.12.2 (10/22/2017)
467
468* Fix for HMAC QAT when block size aligned. The QAT HMAC final without any buffers will fail incorrectly (bug in QAT 1.6).
469* Nitrox fix for rename of `ContextType` to `context_type_t`. Updates to Nitrox README.md.
470* Workaround for `USE_QAE_THREAD_LS` issue with realloc from a different thread.
471* Fix for hashing to allow zero length. This resolves issue with new empty hash tests.
472* Fix bug with blocking async where operation was being free'd before completion. Set freeFunc prior to performing operation and check ret code in poll.
473* Fix leak with cipher symmetric context close.
474* Fix QAT_DEBUG partialState offset.
475* Fixes for symmetric context caching.
476* Refactored async event initialization so its done prior to making possible async calls.
477* Fix to resolve issue with QAT callbacks and multi-threading.
478* The cleanup is now handled in polling function and the event is only marked done from the polling thread that matches the originating thread.
479* Fix possible mem leak with multiple threads `g_qatEcdhY` and `g_qatEcdhCofactor1`.
480* Fix the block polling to use `ret` instead of `status`.
481* Change order of `IntelQaDevClear` and setting `event->ret`.
482* Fixes to better handle threading with async.
483* Refactor of async event state.
484* Refactor to initialize event prior to operation (in case it finishes before adding to queue).
485* Fixes issues with AES GCM decrypt that can corrupt up to authTag bytes at end of output buffer provided.
486* Optimize the Hmac struct to replace keyRaw with ipad.
487* Enhancement to allow reuse of the symmetric context for ciphers.
488* Fixes for QuickAssist (QAT) multi-threading. Fix to not set return code until after callback cleanup.
489* Disable thread binding to specific CPU by default (enabled now with `WC_ASYNC_THREAD_BIND`).
490* Added optional define `QAT_USE_POLLING_CHECK ` to have only one thread polling at a time (not required and doesn't improve performance).
491* Reduced default QAT_MAX_PENDING for benchmark to 15 (120/num_threads).
492* Fix for IntelQaDrbg to handle buffer over 0xFFFF in length.
493* Added working DRBG and TRNG implementations for QAT.
494* Fix to set callback status after ret and output have been set. Cleanup of the symmetric context.
495* Updates to support refactored dynamic types.
496* Fix for QAT symmetric to allow NULL authTag.
497* Fix GCC 7 build warning with braces.
498* Cleanup formatting.
499
500### wolfSSL Async Release v3.11.0 (05/05/2017)
501
502* Fixes for Cavium Nitrox III/V.
503    - Fix with possible crash when using a request Id that is already complete, due to partial submissions not marking event done.
504    - Improvements to max buffer lengths.
505    - Fixes to handle various return code patterns with CNN55XX-SDK.
506    - All Nitrox V tests and benchmarks pass. Bench: RSA 2048-bit public 336,674 ops/sec and private (CRT) 66,524 ops/sec.
507
508* Intel QuickAssist support and various async fixes/improvements:
509    - Added support for Intel QuickAssist v1.6 driver with QuickAssist 8950 hardware
510    - Added QAE memory option to use static memory list instead of dynamic list using `USE_QAE_STATIC_MEM`.
511    - Added tracking of deallocs and made the values signed long.
512    - Improved code for wolf header check and expanded to 16-byte alignment for performance improvement with TLS.
513    - Added ability to override limit dev access parameters and all configurable QAT fields.
514    - Added async simulator tests for DH, DES3 CBC and AES CBC/GCM.
515    - Rename AsyncCryptDev to WC_ASYNC_DEV.
516    - Refactor to move WOLF_EVENT into WC_ASYNC_DEV.
517    - Refactor the async struct/enum names to use WC_ naming.
518    - Refactor of the async event->context to use WOLF_EVENT_TYPE_ASYNC_WOLFSSL or WOLF_EVENT_TYPE_ASYNC_WOLFCRYPT to indicate the type of context pointer.
519    - Added flag to WOLF_EVENT which is used to determine if the async complete should call into operation again or goto next `WC_ASYNC_FLAG_CALL_AGAIN`.
520    - Cleanup of the "wolfAsync_DevCtxInit" calls to make sure asyncDev is always cleared if invalid device id is used.
521    - Eliminated WOLFSSL_ASYNC_CRYPT_STATE.
522    - Removed async event type WOLF_EVENT_TYPE_ASYNC_ANY.
523    - Enable the random extra delay option by default for simulator as it helps catch bugs.
524    - Cleanup for async free to also check marker.
525    - Refactor of the async wait and handle to reduce duplicate code.
526    - Added async simulator test for RSA make key.
527    - Added WC_ASYNC_THRESH_NONE to allow bypass of threshold for testing
528    - Added static numbers for the async sim test types, for easier debugging of the “testDev->type” value.
529    - Populate heap hint into asyncDev struct.
530    - Enhancement to cache the asyncDev to improve poll performance.
531    - Added async threading helpers and new wolfAsync_DevOpenThread.
532    - Added WC_NO_ASYNC_THREADING to prevent async threading.
533    - Added new API “wc_AsyncGetNumberOfCpus” for getting number of CPU’s.
534    - Added new “wc_AsyncThreadYield” API.
535    - Added WOLF_ASYNC_MAX_THREADS.
536    - Added new API for wolfAsync_DevCopy.
537    - Fix to make sure an async init failure sets the deviceId to INVALID_DEVID.
538    - Fix for building with async threading support on Mac.
539    - Fix for using simulator so it supports multiple threads.
540
541* Moved Intel QuickAssist and Cavium Nitrox III/V code into async repo.
542* Added new WC_ASYNC_NO_* options to allow disabling of individual async algorithms.
543    - New defines are: WC_ASYNC_NO_CRYPT, WC_ASYNC_NO_PKI and WC_ASYNC_NO_HASH.
544    - Additionally each algorithm has a WC_ASYNC_NO_[ALGO] define.
545
546
547### wolfSSL Async Release v3.9.8 (07/25/2016)
548
549* Asynchronous wolfCrypt and Cavium Nitrox V support.
550
551### wolfSSL Async Release v3.9.0 (03/04/2016)
552
553* Initial version with async simulator and README.md.
554
555
556## Support
557
558For questions email wolfSSL support at support@wolfssl.com