{"id":10907,"date":"2017-04-07T20:11:59","date_gmt":"2017-04-07T19:11:59","guid":{"rendered":"https:\/\/sysprogs.com\/w\/?p=10907"},"modified":"2017-04-07T22:32:10","modified_gmt":"2017-04-07T21:32:10","slug":"mbed-settings-extractor","status":"publish","type":"post","link":"https:\/\/sysprogs.com\/w\/mbed-settings-extractor\/","title":{"rendered":"Extracting mbed build settings for offline builds"},"content":{"rendered":"<p><a href=\"https:\/\/developer.mbed.org\/\">ARM Mbed<\/a> is a powerful embedded framework that supports many modern devices and allows creating complex firmware (like USB devices or Bluetooth LE\u00a0beacons) using easy high-level C++ API. Although mbed comes with a powerful build system that makes sense of the numerous targets and\u00a0features, it is\u00a0optimized for the online compiler, so\u00a0building the projects offline (and debugging them) could be tricky.<\/p>\n<p>So we decided to make a tool that integrates into the Python-based mbed build system, polls it for supported targets, features and libraries and stores the\u00a0recovered information in a structured XML file that can be used\u00a0to create Visual Studio projects for various mbed targets.<!--more--><\/p>\n<h2>Mbed build system<\/h2>\n<p>The mbed build system is a part of the <a href=\"https:\/\/github.com\/ARMmbed\/mbed-os\">mbed Github repository<\/a> (located in the tools folder). It is pretty sophisticated, so I will show\u00a0you how\u00a0to properly hook into it in order to get the structured\u00a0information about the\u00a0build process.<\/p>\n<p>In order to build mbed for a certain target (or to create a project doing that), you\u00a0essentially need the following information:<\/p>\n<ul>\n<li>The exact list of source files (many of them are\u00a0platform-specific, so\u00a0just building everything won&#8217;t work)<\/li>\n<li>The list of preprocessor macros that the mbed code would expect for that target (like\u00a0DEVICE_SPI=1)<\/li>\n<li>The list of directories to search for header files<\/li>\n<li>The linker script used to produce the final executable<\/li>\n<\/ul>\n<p>It is also nice to have a list of\u00a0configurable options (like UART baud rate) and to be able to change them via GUI.<\/p>\n<p>Mbed internally determines\u00a0those settings from a combination of JSON files and directory naming rules, so it could be tricky to\u00a0try replicating that logic\u00a0outside mbed. Thankfully, the mbed build system is based on Python and provides a\u00a0relatively easy API for querying that information. E.g. the following Python script will query and display basic build settings for all mbed targets:<\/p>\n<pre class=\"\">import sys\r\nfrom copy import deepcopy\r\nfrom os.path import join, dirname, basename\r\n\r\nimport tools.build_api\r\nfrom tools.export import EXPORTERS\r\nfrom tools.libraries import LIBRARIES\r\nfrom tools.paths import MBED_HEADER\r\nfrom tools.settings import ROOT\r\n\r\nfor target in EXPORTERS['gcc_arm'].TARGETS:\r\n toolchain = tools.build_api.prepare_toolchain([ROOT], \"\", target, 'GCC_ARM', silent=True)\r\n fullRes = toolchain.scan_resources(ROOT)\r\n\r\n print fullRes<\/pre>\n<p>If you run this script under debugger, you will notice that calling toolchain.scan_resources() makes mbed search every subdirectory of the mbed source tree, locate the files that apply to the current target and store them in a convenient &#8216;resources&#8217; object:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-10912\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources-1024x779.png\" alt=\"python-resources\" width=\"474\" height=\"361\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources-1024x779.png 1024w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources-300x228.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources-768x584.png 768w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/python-resources.png 1278w\" sizes=\"(max-width: 474px) 100vw, 474px\" \/><\/a><\/p>\n<p>The major problem, however, is that the resources object will\u00a0also include sources and settings from\u00a0many auxiliary libraries that are not needed for many projects and will slow down the build if included.\u00a0Luckily, there is an easy way to scan them separately from the main sources. Each library has a corresponding &#8216;mbed_lib.json&#8217; file listed in the &#8216;json_files&#8217; field of the resources, object so\u00a0doing another scan with the library directories excluded will get the absolute minimum build environment for the selected target:<\/p>\n<pre class=\"\"> mbed_library_dirs = [os.path.dirname(j) for j in fullRes.json_files if os.path.basename(j) == 'mbed_lib.json']\r\n mbed_library_dirs = [lib for lib in mbed_library_dirs if lib != os.path.join(ROOT, 'platform') and not os.path.basename(lib).startswith(\"TARGET_\")]\r\n minimal_res = toolchain.scan_resources(ROOT, exclude_paths = [os.path.join(ROOT, 'features')] + mbed_library_dirs)<\/pre>\n<p><a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-10913\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json-1024x779.png\" alt=\"json\" width=\"474\" height=\"361\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json-1024x779.png 1024w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json-300x228.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json-768x585.png 768w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/json.png 1281w\" sizes=\"(max-width: 474px) 100vw, 474px\" \/><\/a><\/p>\n<h2>Features<\/h2>\n<p>The mbed framework has a concept of features. An mbed feature is a collection of source files and\u00a0preprocessor macros that\u00a0can be added to a project to support certain functionality (like\u00a0Bluetooth LE). You can easily get a list of features for each target by querying the\u00a0<strong>resources.features<\/strong> dictionary:<\/p>\n<pre class=\"\"> for feature in fullRes.features:\r\n     featureRes = fullRes.features[feature]\r\n     print featureRes<\/pre>\n<p>The keys in the dictionary\u00a0are feature names; the values are &#8216;resources&#8217; object\u00a0equivalent to the ones\u00a0created for the targets.<\/p>\n<p><a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-10914\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat-1024x779.png\" alt=\"feat\" width=\"474\" height=\"361\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat-1024x779.png 1024w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat-300x228.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat-768x584.png 768w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/feat.png 1278w\" sizes=\"(max-width: 474px) 100vw, 474px\" \/><\/a><\/p>\n<h2>Libraries<\/h2>\n<p>In addition to features and the implicit libraries, mbed also defines 8 libraries (via the tools.libraries.LIBRARIES object) that extend your projects with functionality like USB host\/device or RTOS. Getting\u00a0structured information about them is\u00a0also as simple as calling toolchain.scan_resources():<\/p>\n<pre class=\"\">for lib in LIBRARIES:\r\n sourceDirs = lib['source_dir']\r\n if isinstance(sourceDirs, str):\r\n  sourceDirs = [sourceDirs]\r\n\r\nfor srcDir in sourceDirs:\r\n libToolchain = deepcopy(toolchain)\r\n libRes = libToolchain.scan_resources(srcDir)<\/pre>\n<h2>Configurable Properties<\/h2>\n<p>Each target, feature or library can define\u00a0configurable properties that will affect the behavior of the project. They can be queried by first loading the resources via toolchain.config.load_resources() and then calling toolchain.config.get_config_data():<\/p>\n<pre class=\"\">for feature in fullRes.features:\r\n featureToolchain = deepcopy(toolchain)\r\n featureRes = fullRes.features[feature]\r\n featureToolchain.config.load_resources(featureRes)\r\n (settings, macros) = featureToolchain.config.get_config_data()\r\n print settings<\/pre>\n<p>The settings are often annotated, so you can easily see\u00a0what each of them does:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-10915\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1-1024x779.png\" alt=\"cfg\" width=\"474\" height=\"361\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1-1024x779.png 1024w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1-300x228.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1-768x584.png 768w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/cfg-1.png 1278w\" sizes=\"(max-width: 474px) 100vw, 474px\" \/><\/a><\/p>\n<h2>Putting it all together<\/h2>\n<p>So having extracted the detailed\u00a0build settings for each mbed target, we made a simple\u00a0tool that\u00a0compiled 3 basic projects on each of them using the extracted settings:<\/p>\n<ul>\n<li>The simplest &#8220;Blinking LED&#8221;<\/li>\n<li>A\u00a0multi-threaded version of &#8220;Blinking LED&#8221; using the RTOS included with mbed<\/li>\n<li>A\u00a0virtual COM port over USB (CDC device class)<\/li>\n<\/ul>\n<p>The results were not bad at all: LEDBlink\u00a0built successfully on 142 out of 152 targets (the ones that failed mostly had minor bugs in the target drivers). The\u00a0virtual COM port succeeded on all supported targets except EFM32 that seems to\u00a0have a mismatch between the USB driver\u00a0and the underlying low-level driver. The multi-threaded blinking LED was too large to fit into\u00a0the smallest devices, but otherwise built successfully.<\/p>\n<h2>Trying it out<\/h2>\n<p>You can download the source code for\u00a0the tool that extracts mbed build settings\u00a0from\u00a0<a href=\"https:\/\/github.com\/sysprogs\/BSPTools\/blob\/master\/generators\/mbed\/data\/BuildConfigExtractor.py\">our\u00a0repository on\u00a0Gihtub<\/a>. Or if you prefer things that work out-of-the-box, give <a href=\"https:\/\/visualgdb.com\/\">VisualGDB<\/a> a try and simply create a buildable and debuggable mbed-based Visual Studio project using its wizard:\u00a0<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/wizard.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-10916\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/wizard.png\" alt=\"wizard\" width=\"822\" height=\"642\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/wizard.png 822w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/wizard-300x234.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2017\/04\/wizard-768x600.png 768w\" sizes=\"(max-width: 822px) 100vw, 822px\" \/><\/a><\/p>\n<p>We are always interested in your feedback, so feel free to drop us a line via <a href=\"mailto:support@sysprogs.com\">support@sysprogs.com<\/a> if you have questions or suggestions regarding our mbed integration.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>ARM Mbed is a powerful embedded framework that supports many modern devices and allows creating complex firmware (like USB devices or Bluetooth LE\u00a0beacons) using easy high-level C++ API. Although mbed comes with a powerful build system that makes sense of the numerous targets and\u00a0features, it is\u00a0optimized for the online compiler, so\u00a0building the projects offline (and &hellip; <a href=\"https:\/\/sysprogs.com\/w\/mbed-settings-extractor\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Extracting mbed build settings for offline builds<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/10907"}],"collection":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/comments?post=10907"}],"version-history":[{"count":6,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/10907\/revisions"}],"predecessor-version":[{"id":10932,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/10907\/revisions\/10932"}],"wp:attachment":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/media?parent=10907"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/categories?post=10907"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/tags?post=10907"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}