Merge mbedtls 2.16.6 into baremetal

Conflicts:
mbedtls.doxyfile - PROJECT_NAME - mbed TLS v2.16.6 chosen.
doc_mainpage.h - mbed TLS v2.16.6 version chosen.
hmac_drbg.h - line 260, extended description chosen.
            - line 313, extended description chosen.
            - line 338, extended description chosen.
version.h - 2.16.6 chosen.
CMakeLists.txt - 2.16.6 chosen.
test_suite_version.data - 2.16.6 chosen.
Makefile - 141 - manual correction - baremetal version of C_SOURCE_FILES
                 with variables for directories plus 2.16.6 CTAGS addition.
pkparse.c - lines 846 onwards - the asn1_get_nonzero_mpi implementation chosen.
ssl_tls.c - line 5269 - edited manually, left the ret=0, because baremetal has
            a different behaviour since commit 87b5626, but added a debug
            message that's new in 2.16.6.    
all.sh:
- component_build_deprecated - chosen the refactored version from 2.16.6,
                               but with extra flags from baremetal.
- rest of the _no_xxx tests - merged make options to have PTHREAD=1 and
                              other changes from 2.16.6 (like -O1 instead of -O0).
- component_build_arm_none_eabi_gcc_no_64bit_multiplication - added 
                              TINYCRYPT_BUILD=0 to the 2.16.6 version of make.

x509/req_app.c - left baremetal log but with mbedtls_exit( 0 ) call.
x509/crl_app.c - left baremetal log but with mbedtls_exit( 0 ) call.
x509/cert_app.c - left baremetal log but with mbedtls_exit( 0 ) call.
ssl/ssl_mail_client.c - left baremetal log but with mbedtls_exit( 0 ) call.
ssl/ssl_pthread_server.c - left baremetal log but with mbedtls_exit( 0 ) call.
ssl/ssl_fork_server.c - left baremetal log but with mbedtls_exit( 0 ) call.
ssl_client1.c - line 54 - left baremetal log but with mbedtls_exit( 0 ) call.
ssl_client2.c - line 54 - left baremetal log but with mbedtls_exit( 0 ) call.
              - line 132 - new options of both branches added.
              - skip close notify handled as in 2.16.6, but with `ssl` instead of `&ssl`.
              - Merged the 2.16.6 usage split with additional baremetal usages.
              - Merged options from baremetal and 2.16.6.
ssl_server.c - left baremetal log but with mbedtls_exit( 0 ) call.
ssl_server2.c - Merged the 2.16.6 usage split with additional baremetal usages.
config.pl - fixed missing defines from the documentation, removed duplicates,
            and reorganised so that the documentation and excluded list
            are ordered in the same way.
test_suite_x509parse.data - only added the two new pathlen tests.
x509_crt.c - change the return code by removing
             MBEDTLS_ERR_X509_INVALID_EXTENSIONS, since it's added by
             x509_crt_frame_parse_ext not by an "or", but by "+=".
Changelog - Assigned all entries to appropriate sections.
ssl-opt.sh - line 8263 - merged options.
           - removed lines 1165 - 1176 - there was a duplicate test, probably
             an artifact of previous merges.
check-files.py - sticked to old formatting.

Signed-off-by: Andrzej Kurek <andrzej.kurek@arm.com>

scripts/abi_check.py

@@ -29,7 +29,7 @@ from types import SimpleNamespace
 import xml.etree.ElementTree as ET
 
 
-class AbiChecker(object):
+class AbiChecker:
     """API and ABI checker."""
 
     def __init__(self, old_version, new_version, configuration):

scripts/assemble_changelog.py

@@ -0,0 +1,505 @@
+#!/usr/bin/env python3
+
+"""Assemble Mbed TLS change log entries into the change log file.
+
+Add changelog entries to the first level-2 section.
+Create a new level-2 section for unreleased changes if needed.
+Remove the input files unless --keep-entries is specified.
+
+In each level-3 section, entries are sorted in chronological order
+(oldest first). From oldest to newest:
+* Merged entry files are sorted according to their merge date (date of
+  the merge commit that brought the commit that created the file into
+  the target branch).
+* Committed but unmerged entry files are sorted according to the date
+  of the commit that adds them.
+* Uncommitted entry files are sorted according to their modification time.
+
+You must run this program from within a git working directory.
+"""
+
+# Copyright (C) 2019, Arm Limited, All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file is part of Mbed TLS (https://tls.mbed.org)
+
+import argparse
+from collections import OrderedDict, namedtuple
+import datetime
+import functools
+import glob
+import os
+import re
+import subprocess
+import sys
+
+class InputFormatError(Exception):
+    def __init__(self, filename, line_number, message, *args, **kwargs):
+        message = '{}:{}: {}'.format(filename, line_number,
+                                     message.format(*args, **kwargs))
+        super().__init__(message)
+
+class CategoryParseError(Exception):
+    def __init__(self, line_offset, error_message):
+        self.line_offset = line_offset
+        self.error_message = error_message
+        super().__init__('{}: {}'.format(line_offset, error_message))
+
+class LostContent(Exception):
+    def __init__(self, filename, line):
+        message = ('Lost content from {}: "{}"'.format(filename, line))
+        super().__init__(message)
+
+# The category names we use in the changelog.
+# If you edit this, update ChangeLog.d/README.md.
+STANDARD_CATEGORIES = (
+    b'API changes',
+    b'Default behavior changes',
+    b'Requirement changes',
+    b'New deprecations',
+    b'Removals',
+    b'Features',
+    b'Security',
+    b'Bugfix',
+    b'Changes',
+)
+
+CategoryContent = namedtuple('CategoryContent', [
+    'name', 'title_line', # Title text and line number of the title
+    'body', 'body_line', # Body text and starting line number of the body
+])
+
+class ChangelogFormat:
+    """Virtual class documenting how to write a changelog format class."""
+
+    @classmethod
+    def extract_top_version(cls, changelog_file_content):
+        """Split out the top version section.
+
+        If the top version is already released, create a new top
+        version section for an unreleased version.
+
+        Return ``(header, top_version_title, top_version_body, trailer)``
+        where the "top version" is the existing top version section if it's
+        for unreleased changes, and a newly created section otherwise.
+        To assemble the changelog after modifying top_version_body,
+        concatenate the four pieces.
+        """
+        raise NotImplementedError
+
+    @classmethod
+    def version_title_text(cls, version_title):
+        """Return the text of a formatted version section title."""
+        raise NotImplementedError
+
+    @classmethod
+    def split_categories(cls, version_body):
+        """Split a changelog version section body into categories.
+
+        Return a list of `CategoryContent` the name is category title
+        without any formatting.
+        """
+        raise NotImplementedError
+
+    @classmethod
+    def format_category(cls, title, body):
+        """Construct the text of a category section from its title and body."""
+        raise NotImplementedError
+
+class TextChangelogFormat(ChangelogFormat):
+    """The traditional Mbed TLS changelog format."""
+
+    _unreleased_version_text = b'= mbed TLS x.x.x branch released xxxx-xx-xx'
+    @classmethod
+    def is_released_version(cls, title):
+        # Look for an incomplete release date
+        return not re.search(br'[0-9x]{4}-[0-9x]{2}-[0-9x]?x', title)
+
+    _top_version_re = re.compile(br'(?:\A|\n)(=[^\n]*\n+)(.*?\n)(?:=|$)',
+                                 re.DOTALL)
+    @classmethod
+    def extract_top_version(cls, changelog_file_content):
+        """A version section starts with a line starting with '='."""
+        m = re.search(cls._top_version_re, changelog_file_content)
+        top_version_start = m.start(1)
+        top_version_end = m.end(2)
+        top_version_title = m.group(1)
+        top_version_body = m.group(2)
+        if cls.is_released_version(top_version_title):
+            top_version_end = top_version_start
+            top_version_title = cls._unreleased_version_text + b'\n\n'
+            top_version_body = b''
+        return (changelog_file_content[:top_version_start],
+                top_version_title, top_version_body,
+                changelog_file_content[top_version_end:])
+
+    @classmethod
+    def version_title_text(cls, version_title):
+        return re.sub(br'\n.*', version_title, re.DOTALL)
+
+    _category_title_re = re.compile(br'(^\w.*)\n+', re.MULTILINE)
+    @classmethod
+    def split_categories(cls, version_body):
+        """A category title is a line with the title in column 0."""
+        if not version_body:
+            return []
+        title_matches = list(re.finditer(cls._category_title_re, version_body))
+        if not title_matches or title_matches[0].start() != 0:
+            # There is junk before the first category.
+            raise CategoryParseError(0, 'Junk found where category expected')
+        title_starts = [m.start(1) for m in title_matches]
+        body_starts = [m.end(0) for m in title_matches]
+        body_ends = title_starts[1:] + [len(version_body)]
+        bodies = [version_body[body_start:body_end].rstrip(b'\n') + b'\n'
+                  for (body_start, body_end) in zip(body_starts, body_ends)]
+        title_lines = [version_body[:pos].count(b'\n') for pos in title_starts]
+        body_lines = [version_body[:pos].count(b'\n') for pos in body_starts]
+        return [CategoryContent(title_match.group(1), title_line,
+                                body, body_line)
+                for title_match, title_line, body, body_line
+                in zip(title_matches, title_lines, bodies, body_lines)]
+
+    @classmethod
+    def format_category(cls, title, body):
+        # `split_categories` ensures that each body ends with a newline.
+        # Make sure that there is additionally a blank line between categories.
+        if not body.endswith(b'\n\n'):
+            body += b'\n'
+        return title + b'\n' + body
+
+class ChangeLog:
+    """An Mbed TLS changelog.
+
+    A changelog file consists of some header text followed by one or
+    more version sections. The version sections are in reverse
+    chronological order. Each version section consists of a title and a body.
+
+    The body of a version section consists of zero or more category
+    subsections. Each category subsection consists of a title and a body.
+
+    A changelog entry file has the same format as the body of a version section.
+
+    A `ChangelogFormat` object defines the concrete syntax of the changelog.
+    Entry files must have the same format as the changelog file.
+    """
+
+    # Only accept dotted version numbers (e.g. "3.1", not "3").
+    # Refuse ".x" in a version number where x is a letter: this indicates
+    # a version that is not yet released. Something like "3.1a" is accepted.
+    _version_number_re = re.compile(br'[0-9]+\.[0-9A-Za-z.]+')
+    _incomplete_version_number_re = re.compile(br'.*\.[A-Za-z]')
+
+    def add_categories_from_text(self, filename, line_offset,
+                                 text, allow_unknown_category):
+        """Parse a version section or entry file."""
+        try:
+            categories = self.format.split_categories(text)
+        except CategoryParseError as e:
+            raise InputFormatError(filename, line_offset + e.line_offset,
+                                   e.error_message)
+        for category in categories:
+            if not allow_unknown_category and \
+               category.name not in self.categories:
+                raise InputFormatError(filename,
+                                       line_offset + category.title_line,
+                                       'Unknown category: "{}"',
+                                       category.name.decode('utf8'))
+            self.categories[category.name] += category.body
+
+    def __init__(self, input_stream, changelog_format):
+        """Create a changelog object.
+
+        Populate the changelog object from the content of the file
+        input_stream.
+        """
+        self.format = changelog_format
+        whole_file = input_stream.read()
+        (self.header,
+         self.top_version_title, top_version_body,
+         self.trailer) = self.format.extract_top_version(whole_file)
+        # Split the top version section into categories.
+        self.categories = OrderedDict()
+        for category in STANDARD_CATEGORIES:
+            self.categories[category] = b''
+        offset = (self.header + self.top_version_title).count(b'\n') + 1
+        self.add_categories_from_text(input_stream.name, offset,
+                                      top_version_body, True)
+
+    def add_file(self, input_stream):
+        """Add changelog entries from a file.
+        """
+        self.add_categories_from_text(input_stream.name, 1,
+                                      input_stream.read(), False)
+
+    def write(self, filename):
+        """Write the changelog to the specified file.
+        """
+        with open(filename, 'wb') as out:
+            out.write(self.header)
+            out.write(self.top_version_title)
+            for title, body in self.categories.items():
+                if not body:
+                    continue
+                out.write(self.format.format_category(title, body))
+            out.write(self.trailer)
+
+
+@functools.total_ordering
+class EntryFileSortKey:
+    """This classes defines an ordering on changelog entry files: older < newer.
+
+    * Merged entry files are sorted according to their merge date (date of
+      the merge commit that brought the commit that created the file into
+      the target branch).
+    * Committed but unmerged entry files are sorted according to the date
+      of the commit that adds them.
+    * Uncommitted entry files are sorted according to their modification time.
+
+    This class assumes that the file is in a git working directory with
+    the target branch checked out.
+    """
+
+    # Categories of files. A lower number is considered older.
+    MERGED = 0
+    COMMITTED = 1
+    LOCAL = 2
+
+    @staticmethod
+    def creation_hash(filename):
+        """Return the git commit id at which the given file was created.
+
+        Return None if the file was never checked into git.
+        """
+        hashes = subprocess.check_output(['git', 'log', '--format=%H',
+                                          '--follow',
+                                          '--', filename])
+        m = re.search(b'(.+)$', hashes)
+        if not m:
+            # The git output is empty. This means that the file was
+            # never checked in.
+            return None
+        # The last commit in the log is the oldest one, which is when the
+        # file was created.
+        return m.group(0)
+
+    @staticmethod
+    def list_merges(some_hash, target, *options):
+        """List merge commits from some_hash to target.
+
+        Pass options to git to select which commits are included.
+        """
+        text = subprocess.check_output(['git', 'rev-list',
+                                        '--merges', *options,
+                                        b'..'.join([some_hash, target])])
+        return text.rstrip(b'\n').split(b'\n')
+
+    @classmethod
+    def merge_hash(cls, some_hash):
+        """Return the git commit id at which the given commit was merged.
+
+        Return None if the given commit was never merged.
+        """
+        target = b'HEAD'
+        # List the merges from some_hash to the target in two ways.
+        # The ancestry list is the ones that are both descendants of
+        # some_hash and ancestors of the target.
+        ancestry = frozenset(cls.list_merges(some_hash, target,
+                                             '--ancestry-path'))
+        # The first_parents list only contains merges that are directly
+        # on the target branch. We want it in reverse order (oldest first).
+        first_parents = cls.list_merges(some_hash, target,
+                                        '--first-parent', '--reverse')
+        # Look for the oldest merge commit that's both on the direct path
+        # and directly on the target branch. That's the place where some_hash
+        # was merged on the target branch. See
+        # https://stackoverflow.com/questions/8475448/find-merge-commit-which-include-a-specific-commit
+        for commit in first_parents:
+            if commit in ancestry:
+                return commit
+        return None
+
+    @staticmethod
+    def commit_timestamp(commit_id):
+        """Return the timestamp of the given commit."""
+        text = subprocess.check_output(['git', 'show', '-s',
+                                        '--format=%ct',
+                                        commit_id])
+        return datetime.datetime.utcfromtimestamp(int(text))
+
+    @staticmethod
+    def file_timestamp(filename):
+        """Return the modification timestamp of the given file."""
+        mtime = os.stat(filename).st_mtime
+        return datetime.datetime.fromtimestamp(mtime)
+
+    def __init__(self, filename):
+        """Determine position of the file in the changelog entry order.
+
+        This constructor returns an object that can be used with comparison
+        operators, with `sort` and `sorted`, etc. Older entries are sorted
+        before newer entries.
+        """
+        self.filename = filename
+        creation_hash = self.creation_hash(filename)
+        if not creation_hash:
+            self.category = self.LOCAL
+            self.datetime = self.file_timestamp(filename)
+            return
+        merge_hash = self.merge_hash(creation_hash)
+        if not merge_hash:
+            self.category = self.COMMITTED
+            self.datetime = self.commit_timestamp(creation_hash)
+            return
+        self.category = self.MERGED
+        self.datetime = self.commit_timestamp(merge_hash)
+
+    def sort_key(self):
+        """"Return a concrete sort key for this entry file sort key object.
+
+        ``ts1 < ts2`` is implemented as ``ts1.sort_key() < ts2.sort_key()``.
+        """
+        return (self.category, self.datetime, self.filename)
+
+    def __eq__(self, other):
+        return self.sort_key() == other.sort_key()
+
+    def __lt__(self, other):
+        return self.sort_key() < other.sort_key()
+
+
+def check_output(generated_output_file, main_input_file, merged_files):
+    """Make sanity checks on the generated output.
+
+    The intent of these sanity checks is to have reasonable confidence
+    that no content has been lost.
+
+    The sanity check is that every line that is present in an input file
+    is also present in an output file. This is not perfect but good enough
+    for now.
+    """
+    generated_output = set(open(generated_output_file, 'rb'))
+    for line in open(main_input_file, 'rb'):
+        if line not in generated_output:
+            raise LostContent('original file', line)
+    for merged_file in merged_files:
+        for line in open(merged_file, 'rb'):
+            if line not in generated_output:
+                raise LostContent(merged_file, line)
+
+def finish_output(changelog, output_file, input_file, merged_files):
+    """Write the changelog to the output file.
+
+    The input file and the list of merged files are used only for sanity
+    checks on the output.
+    """
+    if os.path.exists(output_file) and not os.path.isfile(output_file):
+        # The output is a non-regular file (e.g. pipe). Write to it directly.
+        output_temp = output_file
+    else:
+        # The output is a regular file. Write to a temporary file,
+        # then move it into place atomically.
+        output_temp = output_file + '.tmp'
+    changelog.write(output_temp)
+    check_output(output_temp, input_file, merged_files)
+    if output_temp != output_file:
+        os.rename(output_temp, output_file)
+
+def remove_merged_entries(files_to_remove):
+    for filename in files_to_remove:
+        os.remove(filename)
+
+def list_files_to_merge(options):
+    """List the entry files to merge, oldest first.
+
+    "Oldest" is defined by `EntryFileSortKey`.
+    """
+    files_to_merge = glob.glob(os.path.join(options.dir, '*.txt'))
+    files_to_merge.sort(key=EntryFileSortKey)
+    return files_to_merge
+
+def merge_entries(options):
+    """Merge changelog entries into the changelog file.
+
+    Read the changelog file from options.input.
+    Read entries to merge from the directory options.dir.
+    Write the new changelog to options.output.
+    Remove the merged entries if options.keep_entries is false.
+    """
+    with open(options.input, 'rb') as input_file:
+        changelog = ChangeLog(input_file, TextChangelogFormat)
+    files_to_merge = list_files_to_merge(options)
+    if not files_to_merge:
+        sys.stderr.write('There are no pending changelog entries.\n')
+        return
+    for filename in files_to_merge:
+        with open(filename, 'rb') as input_file:
+            changelog.add_file(input_file)
+    finish_output(changelog, options.output, options.input, files_to_merge)
+    if not options.keep_entries:
+        remove_merged_entries(files_to_merge)
+
+def show_file_timestamps(options):
+    """List the files to merge and their timestamp.
+
+    This is only intended for debugging purposes.
+    """
+    files = list_files_to_merge(options)
+    for filename in files:
+        ts = EntryFileSortKey(filename)
+        print(ts.category, ts.datetime, filename)
+
+def set_defaults(options):
+    """Add default values for missing options."""
+    output_file = getattr(options, 'output', None)
+    if output_file is None:
+        options.output = options.input
+    if getattr(options, 'keep_entries', None) is None:
+        options.keep_entries = (output_file is not None)
+
+def main():
+    """Command line entry point."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--dir', '-d', metavar='DIR',
+                        default='ChangeLog.d',
+                        help='Directory to read entries from'
+                             ' (default: ChangeLog.d)')
+    parser.add_argument('--input', '-i', metavar='FILE',
+                        default='ChangeLog',
+                        help='Existing changelog file to read from and augment'
+                             ' (default: ChangeLog)')
+    parser.add_argument('--keep-entries',
+                        action='store_true', dest='keep_entries', default=None,
+                        help='Keep the files containing entries'
+                             ' (default: remove them if --output/-o is not specified)')
+    parser.add_argument('--no-keep-entries',
+                        action='store_false', dest='keep_entries',
+                        help='Remove the files containing entries after they are merged'
+                             ' (default: remove them if --output/-o is not specified)')
+    parser.add_argument('--output', '-o', metavar='FILE',
+                        help='Output changelog file'
+                             ' (default: overwrite the input)')
+    parser.add_argument('--list-files-only',
+                        action='store_true',
+                        help=('Only list the files that would be processed '
+                              '(with some debugging information)'))
+    options = parser.parse_args()
+    set_defaults(options)
+    if options.list_files_only:
+        show_file_timestamps(options)
+        return
+    merge_entries(options)
+
+if __name__ == '__main__':
+    main()

scripts/config.pl

@@ -21,11 +21,13 @@
 #
 #   MBEDTLS_TEST_NULL_ENTROPY
 #   MBEDTLS_DEPRECATED_REMOVED
+#   MBEDTLS_DEPRECATED_WARNING
 #   MBEDTLS_HAVE_SSE2
 #   MBEDTLS_PLATFORM_NO_STD_FUNCTIONS
 #   MBEDTLS_ECP_DP_M221_ENABLED
 #   MBEDTLS_ECP_DP_M383_ENABLED
 #   MBEDTLS_ECP_DP_M511_ENABLED
+#   MBEDTLS_MEMORY_DEBUG
 #   MBEDTLS_MEMORY_BACKTRACE
 #   MBEDTLS_MEMORY_BUFFER_ALLOC_C
 #   MBEDTLS_NO_DEFAULT_ENTROPY_SOURCES
@@ -34,7 +36,7 @@
 #   MBEDTLS_REMOVE_3DES_CIPHERSUITES
 #   MBEDTLS_SHA256_NO_SHA224
 #   MBEDTLS_SSL_HW_RECORD_ACCEL
-#   MBEDTLS_SSL_PROTO_NO_DTLS
+#   MBEDTLS_SSL_PROTO_NO_TLS
 #   MBEDTLS_SSL_NO_SESSION_CACHE
 #   MBEDTLS_SSL_NO_SESSION_RESUMPTION
 #   MBEDTLS_RSA_NO_CRT
@@ -54,8 +56,13 @@
 #   MBEDTLS_AES_ONLY_128_BIT_KEY_LENGTH
 #   MBEDTLS_AES_ONLY_ENCRYPT
 #   MBEDTLS_AES_SCA_COUNTERMEASURES
+#   MBEDTLS_CTR_DRBG_USE_128_BIT_KEY
 #   and any symbol beginning _ALT
 #
+# The baremetal configuration excludes options that require a library or
+# operating system feature that is typically not present on bare metal
+# systems. Features that are excluded from "full" won't be in "baremetal"
+# either.
 
 use warnings;
 use strict;
@@ -98,9 +105,9 @@ EOU
 my @excluded = qw(
 MBEDTLS_TEST_NULL_ENTROPY
 MBEDTLS_DEPRECATED_REMOVED
+MBEDTLS_DEPRECATED_WARNING
 MBEDTLS_HAVE_SSE2
 MBEDTLS_PLATFORM_NO_STD_FUNCTIONS
-MBEDTLS_CTR_DRBG_USE_128_BIT_KEY
 MBEDTLS_ECP_DP_M221_ENABLED
 MBEDTLS_ECP_DP_M383_ENABLED
 MBEDTLS_ECP_DP_M511_ENABLED
@@ -109,7 +116,6 @@ MBEDTLS_MEMORY_BACKTRACE
 MBEDTLS_MEMORY_BUFFER_ALLOC_C
 MBEDTLS_NO_DEFAULT_ENTROPY_SOURCES
 MBEDTLS_NO_PLATFORM_ENTROPY
-MBEDTLS_RSA_NO_CRT
 MBEDTLS_REMOVE_ARC4_CIPHERSUITES
 MBEDTLS_REMOVE_3DES_CIPHERSUITES
 MBEDTLS_SHA256_NO_SHA224
@@ -117,6 +123,7 @@ MBEDTLS_SSL_HW_RECORD_ACCEL
 MBEDTLS_SSL_PROTO_NO_TLS
 MBEDTLS_SSL_NO_SESSION_CACHE
 MBEDTLS_SSL_NO_SESSION_RESUMPTION
+MBEDTLS_RSA_NO_CRT
 MBEDTLS_USE_TINYCRYPT
 MBEDTLS_X509_ALLOW_EXTENSIONS_NON_V3
 MBEDTLS_X509_ALLOW_UNSUPPORTED_CRITICAL_EXTENSION
@@ -129,34 +136,38 @@ MBEDTLS_ZLIB_SUPPORT
 MBEDTLS_PKCS11_C
 MBEDTLS_NO_UDBL_DIVISION
 MBEDTLS_NO_64BIT_MULTIPLICATION
-MBEDTLS_USE_TINYCRYPT
 MBEDTLS_AES_ONLY_128_BIT_KEY_LENGTH
 MBEDTLS_AES_ONLY_ENCRYPT
 MBEDTLS_AES_SCA_COUNTERMEASURES
+MBEDTLS_CTR_DRBG_USE_128_BIT_KEY
 _ALT\s*$
 );
 
 # Things that should be disabled in "baremetal"
 my @excluded_baremetal = qw(
-MBEDTLS_NET_C
-MBEDTLS_TIMING_C
-MBEDTLS_FS_IO
 MBEDTLS_ENTROPY_NV_SEED
+MBEDTLS_FS_IO
+MBEDTLS_HAVEGE_C
 MBEDTLS_HAVE_TIME
 MBEDTLS_HAVE_TIME_DATE
-MBEDTLS_DEPRECATED_WARNING
-MBEDTLS_HAVEGE_C
-MBEDTLS_THREADING_C
-MBEDTLS_THREADING_PTHREAD
 MBEDTLS_MEMORY_BACKTRACE
 MBEDTLS_MEMORY_BUFFER_ALLOC_C
-MBEDTLS_PLATFORM_TIME_ALT
+MBEDTLS_NET_C
 MBEDTLS_PLATFORM_FPRINTF_ALT
+MBEDTLS_PLATFORM_NV_SEED_ALT
+MBEDTLS_PLATFORM_TIME_ALT
+MBEDTLS_THREADING_C
+MBEDTLS_THREADING_PTHREAD
+MBEDTLS_TIMING_C
 );
 
-# Things that should be enabled in "full" even if they match @excluded
+# Things that should be enabled in "full" even if they match @excluded.
+# Platform ALTs enable global variables that allow configuring the behavior
+# but default to the default behavior, except for PLATFORM_SETUP_TEARDOWN_ALT
+# which requires the application to provide relevant functions like
+# non-platform ALTs.
 my @non_excluded = qw(
-PLATFORM_[A-Z0-9]+_ALT
+PLATFORM_(?!SETUP_TEARDOWN_)[A-Z_0-9]+_ALT
 );
 
 # Things that should be enabled in "baremetal"

scripts/data_files/vs2010-app-template.vcxproj

@@ -18,10 +18,11 @@
       <Platform>x64</Platform>
     </ProjectConfiguration>
   </ItemGroup>
-  <ItemGroup>
+  <ItemGroup>
+    <SOURCES>
   </ItemGroup>
   <ItemGroup>
-  <ItemGroup>
+    <ProjectReference Include="mbedTLS.vcxproj">
       <Project>{46cf2d25-6a36-4189-b59c-e4815388e554}</Project>
       <LinkLibraryDependencies>true</LinkLibraryDependencies>
     </ProjectReference>
@@ -70,22 +71,22 @@
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
   </ImportGroup>
   <PropertyGroup Label="UserMacros" />
-  <PropertyGroup Label="UserMacros" />
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
-    <LinkIncremental>true</LinkIncremental>
-    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
-  </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
-    <LinkIncremental>true</LinkIncremental>
-    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
-  </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
-    <LinkIncremental>false</LinkIncremental>
-    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
-  </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
-    <LinkIncremental>false</LinkIncremental>
-    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+    <LinkIncremental>true</LinkIncremental>
+    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <LinkIncremental>true</LinkIncremental>
+    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+    <LinkIncremental>false</LinkIncremental>
+    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <LinkIncremental>false</LinkIncremental>
+    <IntDir>$(Configuration)\$(TargetName)\</IntDir>
+  </PropertyGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <ClCompile>
       <PrecompiledHeader>
@@ -99,7 +100,7 @@
       <SubSystem>Console</SubSystem>
       <GenerateDebugInformation>true</GenerateDebugInformation>
       <ShowProgress>NotSet</ShowProgress>
-      <ShowProgress>NotSet</ShowProgress>
+      <AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <AdditionalLibraryDirectories>Debug</AdditionalLibraryDirectories>
     </Link>
     <ProjectReference>
@@ -119,7 +120,7 @@
       <SubSystem>Console</SubSystem>
       <GenerateDebugInformation>true</GenerateDebugInformation>
       <ShowProgress>NotSet</ShowProgress>
-      <ShowProgress>NotSet</ShowProgress>
+      <AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <AdditionalLibraryDirectories>Debug</AdditionalLibraryDirectories>
     </Link>
     <ProjectReference>
@@ -143,7 +144,7 @@
       <EnableCOMDATFolding>true</EnableCOMDATFolding>
       <OptimizeReferences>true</OptimizeReferences>
       <AdditionalLibraryDirectories>Release</AdditionalLibraryDirectories>
-      <AdditionalLibraryDirectories>Release</AdditionalLibraryDirectories>
+      <AdditionalDependencies>kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
     </Link>
   </ItemDefinitionGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">

scripts/data_files/vs2010-main-template.vcxproj

@@ -21,7 +21,7 @@
   <PropertyGroup Label="Globals">
     <ProjectGuid>{46CF2D25-6A36-4189-B59C-E4815388E554}</ProjectGuid>
     <Keyword>Win32Proj</Keyword>
-    <RootNamespace>mbedTLS</RootNamespace>
+    <RootNamespace>mbedTLS</RootNamespace>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
@@ -45,7 +45,6 @@
     <UseDebugLibraries>false</UseDebugLibraries>
     <WholeProgramOptimization>true</WholeProgramOptimization>
     <CharacterSet>Unicode</CharacterSet>
-    <PlatformToolset>Windows7.1SDK</PlatformToolset>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
   <ImportGroup Label="ExtensionSettings">
@@ -85,7 +84,7 @@
       </PrecompiledHeader>
       <WarningLevel>Level3</WarningLevel>
       <Optimization>Disabled</Optimization>
-      <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>../../include</AdditionalIncludeDirectories>
       <CompileAs>CompileAsC</CompileAs>
     </ClCompile>
@@ -100,7 +99,7 @@
       </PrecompiledHeader>
       <WarningLevel>Level3</WarningLevel>
       <Optimization>Disabled</Optimization>
-      <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>../../include</AdditionalIncludeDirectories>
       <CompileAs>CompileAsC</CompileAs>
     </ClCompile>
@@ -117,7 +116,7 @@
       <Optimization>MaxSpeed</Optimization>
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
-      <PreprocessorDefinitions>WIN32;NDEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>WIN32;NDEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>../../include</AdditionalIncludeDirectories>
     </ClCompile>
     <Link>
@@ -135,7 +134,7 @@
       <Optimization>MaxSpeed</Optimization>
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
-      <PreprocessorDefinitions>WIN64;NDEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>WIN64;NDEBUG;_WINDOWS;_USRDLL;MBEDTLS_EXPORTS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>../../include</AdditionalIncludeDirectories>
     </ClCompile>
     <Link>

scripts/data_files/vs2010-sln-template.sln

@@ -1,7 +1,7 @@
 
 Microsoft Visual Studio Solution File, Format Version 11.00
 # Visual C++ Express 2010
-Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mbedTLS", "mbedTLS.vcxproj", "{46CF2D25-6A36-4189-B59C-E4815388E554}"
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mbedTLS", "mbedTLS.vcxproj", "{46CF2D25-6A36-4189-B59C-E4815388E554}"
 EndProject
 APP_ENTRIES
 Global

scripts/data_files/vs6-main-template.dsp

@@ -1,24 +1,24 @@
-# Microsoft Developer Studio Project File - Name="mbedtls" - Package Owner=<4>
+# Microsoft Developer Studio Project File - Name="mbedtls" - Package Owner=<4>
 # Microsoft Developer Studio Generated Build File, Format Version 6.00
 # ** DO NOT EDIT **
 
 # TARGTYPE "Win32 (x86) Static Library" 0x0104
 
-CFG=mbedtls - Win32 Debug
+CFG=mbedtls - Win32 Debug
 !MESSAGE This is not a valid makefile. To build this project using NMAKE,
 !MESSAGE use the Export Makefile command and run
 !MESSAGE 
-!MESSAGE NMAKE /f "mbedtls.mak".
+!MESSAGE NMAKE /f "mbedtls.mak".
 !MESSAGE 
 !MESSAGE You can specify a configuration when running NMAKE
 !MESSAGE by defining the macro CFG on the command line. For example:
 !MESSAGE 
-!MESSAGE NMAKE /f "mbedtls.mak" CFG="mbedtls - Win32 Debug"
+!MESSAGE NMAKE /f "mbedtls.mak" CFG="mbedtls - Win32 Debug"
 !MESSAGE 
 !MESSAGE Possible choices for configuration are:
 !MESSAGE 
-!MESSAGE "mbedtls - Win32 Release" (based on "Win32 (x86) Static Library")
-!MESSAGE "mbedtls - Win32 Debug" (based on "Win32 (x86) Static Library")
+!MESSAGE "mbedtls - Win32 Release" (based on "Win32 (x86) Static Library")
+!MESSAGE "mbedtls - Win32 Debug" (based on "Win32 (x86) Static Library")
 !MESSAGE 
 
 # Begin Project
@@ -28,7 +28,7 @@ CFG=mbedtls - Win32 Debug
 CPP=cl.exe
 RSC=rc.exe
 
-!IF  "$(CFG)" == "mbedtls - Win32 Release"
+!IF  "$(CFG)" == "mbedtls - Win32 Release"
 
 # PROP BASE Use_MFC 0
 # PROP BASE Use_Debug_Libraries 0
@@ -51,7 +51,7 @@ LIB32=link.exe -lib
 # ADD BASE LIB32 /nologo
 # ADD LIB32 /nologo
 
-!ELSEIF  "$(CFG)" == "mbedtls - Win32 Debug"
+!ELSEIF  "$(CFG)" == "mbedtls - Win32 Debug"
 
 # PROP BASE Use_MFC 0
 # PROP BASE Use_Debug_Libraries 1
@@ -78,8 +78,8 @@ LIB32=link.exe -lib
 
 # Begin Target
 
-# Name "mbedtls - Win32 Release"
-# Name "mbedtls - Win32 Debug"
+# Name "mbedtls - Win32 Release"
+# Name "mbedtls - Win32 Debug"
 # Begin Group "Source Files"
 
 # PROP Default_Filter "cpp;c;cxx;rc;def;r;odl;idl;hpj;bat"

scripts/generate_visualc_files.pl

@@ -93,10 +93,10 @@ sub gen_app {
     $path =~ s!/!\\!g;
     (my $appname = $path) =~ s/.*\\//;
 
-    my $srcs = "\n    <ClCompile Include=\"..\\..\\programs\\$path.c\" \/>\r";
+    my $srcs = "<ClCompile Include=\"..\\..\\programs\\$path.c\" \/>";
     if( $appname eq "ssl_client2" or $appname eq "ssl_server2" or
         $appname eq "query_compile_time_config" ) {
-        $srcs .= "\n    <ClCompile Include=\"..\\..\\programs\\ssl\\query_config.c\" \/>\r";
+        $srcs .= "\r\n    <ClCompile Include=\"..\\..\\programs\\ssl\\query_config.c\" \/>";
     }
 
     my $content = $template;

scripts/output_env.sh

@@ -23,11 +23,15 @@ print_version()
     shift
     ARGS="$1"
     shift
-    FAIL_MSG="$1"
+    VARIANT="$1"
     shift
 
-    if ! `type "$BIN" > /dev/null 2>&1`; then
-        echo "* $FAIL_MSG"
+    if [ -n "$VARIANT" ]; then
+        VARIANT=" ($VARIANT)"
+    fi
+
+    if ! type "$BIN" > /dev/null 2>&1; then
+        echo " * ${BIN##*/}$VARIANT: Not found."
         return 0
     fi
 
@@ -41,81 +45,127 @@ print_version()
         VERSION_STR=`echo "$VERSION_STR" | $FILTER`
     done
 
-    echo "* ${BIN##*/}: $BIN: $VERSION_STR"
+    if [ -z "$VERSION_STR" ]; then
+        VERSION_STR="Version could not be determined."
+    fi
+
+    echo " * ${BIN##*/}$VARIANT: ${BIN} : ${VERSION_STR} "
 }
 
+echo "** Platform:"
+echo
+
+if [ `uname -s` = "Linux" ]; then
+    echo "Linux variant"
+    lsb_release -d -c
+else
+    echo "Unknown Unix variant"
+fi
+
+echo
+
 print_version "uname" "-a" ""
+
+echo
+echo
+echo "** Tool Versions:"
 echo
 
 if [ "${RUN_ARMCC:-1}" -ne 0 ]; then
     : "${ARMC5_CC:=armcc}"
-    print_version "$ARMC5_CC" "--vsn" "armcc not found!" "head -n 2"
+    print_version "$ARMC5_CC" "--vsn" "" "head -n 2"
     echo
 
     : "${ARMC6_CC:=armclang}"
-    print_version "$ARMC6_CC" "--vsn" "armclang not found!" "head -n 2"
+    print_version "$ARMC6_CC" "--vsn" "" "head -n 2"
     echo
 fi
 
-print_version "arm-none-eabi-gcc" "--version" "gcc-arm not found!" "head -n 1"
+print_version "arm-none-eabi-gcc" "--version" "" "head -n 1"
 echo
 
-print_version "gcc" "--version" "gcc not found!" "head -n 1"
+print_version "gcc" "--version" "" "head -n 1"
 echo
 
-print_version "clang" "--version" "clang not found" "head -n 2"
+print_version "clang" "--version" "" "head -n 2"
 echo
 
-print_version "ldd" "--version"                     \
-    "No ldd present: can't determine libc version!" \
-    "head -n 1"
+print_version "ldd" "--version" "" "head -n 1"
 echo
 
-print_version "valgrind" "--version" "valgrind not found!"
+print_version "valgrind" "--version" ""
+echo
+
+print_version "gdb" "--version" "" "head -n 1"
+echo
+
+print_version "perl" "--version" "" "head -n 2" "grep ."
+echo
+
+print_version "python" "--version" "" "head -n 1"
+echo
+
+# Find the installed version of Pylint. Installed as a distro package this can
+# be pylint3 and as a PEP egg, pylint. In test scripts We prefer pylint over
+# pylint3
+if type pylint >/dev/null 2>/dev/null; then
+    print_version "pylint" "--version" "" "sed /^.*config/d" "grep pylint"
+elif type pylint3 >/dev/null 2>/dev/null; then
+    print_version "pylint3" "--version" "" "sed /^.*config/d" "grep pylint"
+else
+    echo " * pylint or pylint3: Not found."
+fi
 echo
 
 : ${OPENSSL:=openssl}
-print_version "$OPENSSL" "version" "openssl not found!"
+print_version "$OPENSSL" "version" "default"
 echo
 
 if [ -n "${OPENSSL_LEGACY+set}" ]; then
-    print_version "$OPENSSL_LEGACY" "version" "openssl legacy version not found!"
-    echo
+    print_version "$OPENSSL_LEGACY" "version" "legacy"
+else
+    echo " * openssl (legacy): Not configured."
 fi
+echo
 
 if [ -n "${OPENSSL_NEXT+set}" ]; then
-    print_version "$OPENSSL_NEXT" "version" "openssl next version not found!"
-    echo
+    print_version "$OPENSSL_NEXT" "version" "next"
+else
+    echo " * openssl (next): Not configured."
 fi
+echo
 
 : ${GNUTLS_CLI:=gnutls-cli}
-print_version "$GNUTLS_CLI" "--version" "gnuTLS client not found!" "head -n 1"
+print_version "$GNUTLS_CLI" "--version" "default" "head -n 1"
 echo
 
 : ${GNUTLS_SERV:=gnutls-serv}
-print_version "$GNUTLS_SERV" "--version" "gnuTLS server not found!" "head -n 1"
+print_version "$GNUTLS_SERV" "--version" "default" "head -n 1"
 echo
 
 if [ -n "${GNUTLS_LEGACY_CLI+set}" ]; then
-    print_version "$GNUTLS_LEGACY_CLI" "--version" \
-        "gnuTLS client legacy version not found!"  \
-        "head -n 1"
-    echo
-fi
-
-if [ -n "${GNUTLS_LEGACY_SERV+set}" ]; then
-    print_version "$GNUTLS_LEGACY_SERV" "--version" \
-        "gnuTLS server legacy version not found!"   \
-        "head -n 1"
-    echo
-fi
-
-if `hash dpkg > /dev/null 2>&1`; then
-    echo "* asan:"
-    dpkg -s libasan2 2> /dev/null | grep -i version
-    dpkg -s libasan1 2> /dev/null | grep -i version
-    dpkg -s libasan0 2> /dev/null | grep -i version
+    print_version "$GNUTLS_LEGACY_CLI" "--version" "legacy" "head -n 1"
 else
-    echo "* No dpkg present: can't determine asan version!"
+     echo " * gnutls-cli (legacy): Not configured."
+fi
+echo
+
+if [ -n "${GNUTLS_LEGACY_SERV+set}" ]; then
+    print_version "$GNUTLS_LEGACY_SERV" "--version" "legacy" "head -n 1"
+else
+    echo " * gnutls-serv (legacy): Not configured."
+fi
+echo
+
+echo " * Installed asan versions:"
+if type dpkg-query >/dev/null 2>/dev/null; then
+    if ! dpkg-query -f '${Status} ${Package}: ${Version}\n' -W 'libasan*' |
+         awk '$3 == "installed" && $4 !~ /-/ {print $4, $5}' |
+         grep .
+    then
+        echo "   No asan versions installed."
+    fi
+else
+    echo "  Unable to determine the asan version without dpkg."
 fi
 echo

scripts/windows_msbuild.bat

@@ -0,0 +1,20 @@
+@rem Build and test Mbed TLS with Visual Studio using msbuild.
+@rem Usage: windows_msbuild [RETARGET]
+@rem   RETARGET: version of Visual Studio to emulate
+@rem             https://docs.microsoft.com/en-us/cpp/build/how-to-modify-the-target-framework-and-platform-toolset
+
+@rem These parameters are hard-coded for now.
+set "arch=x64" & @rem "x86" or "x64"
+set "cfg=Release" & @rem "Debug" or "Release"
+set "vcvarsall=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat"
+
+if not "%~1"=="" set "retarget=,PlatformToolset=%1"
+
+@rem If the %USERPROFILE%\Source directory exists, then running
+@rem vcvarsall.bat will silently change the directory to that directory.
+@rem Setting the VSCMD_START_DIR environment variable causes it to change
+@rem to that directory instead.
+set "VSCMD_START_DIR=%~dp0\..\visualc\VS2010"
+
+"%vcvarsall%" x64 && ^
+msbuild /t:Rebuild /p:Configuration=%cfg%%retarget% /m mbedTLS.sln