Saturday, 10 February 2018

One MapGuide/FDO build system to rule them all?

Before the bombshell of the death of Autodesk Infrastructure Map Server was dropped, I was putting the final finishing touches of making (pun intended) the MapGuide/FDO build experience on Linux a more pleasant experience.

Namely, I had it with autotools (some may describe it as autohell) as the way to build MapGuide/FDO on Linux. For FDO, this was the original motivation for introducing CMake as an alternative. CMake is a more pleasant way to build software on Linux over autotools because:
  1. CMake builds your sources outside of the source tree. If you're doing development on a SVN working copy this is an absolute boon as it means when it comes time to commit any changes, you don't have to deal with sifting through tons of autotools-generated junk that is left in your actual SVN wc.
  2. CMake builds are faster than their autotools counterpart.
  3. It is much easier to find and consume external libraries with CMake than it is through autotools, which makes build times faster because we can just source system-installed copies of thirdparty libraries we use, instead of waste time having to build these copies (in our internal thirdparty source tree) ourselves. If we are able to use system-installed copies of libraries when building FDO, then we can take advantage of SVN sparse checkouts and be able to skip downloading whole chunks of thirdparty library sources that we never have to build!
Sadly, while this sounds nice in theory, the CMake way to build FDO had fallen into a state of disrepair. My distaste for autotools was sufficient motivation to get the CMake build back into working condition. Several weeks of bashing at various CMakeLists.txt files later, the FDO CMake build was operational again and had some several major advantages over the autotools build (in addition to what was already mentioned):
  • We can setup the CMake to generate build configurations for Ninja instead of standard make. A ninja-powered CMake build is faster than standard make ^.
  • On Ubuntu 14.04 LTS (the current Ubuntu version we're targeting), all the thirdparty libraries we use were available for us to apt-get install in the right version ranges, and the CMake build can take advantage of all of them. Not a single internal thirdparty library copy needs to be built!
  • We can easily light up compiler features like AddressSanitizer and linking with the faster gold instead of ld. AddressSanitizer in particular easily helped us catch some issues that have flew under the radar.
  • All of the unit tests are build-able and more importantly ... executable outside the source tree, making it easier to fix up whatever was failing.
Although we now had a functional FDO CMake build. MapGuide still was built on Linux using autotools. So for the same reasons and motivations, I started the process of introducing CMake to the MapGuide build system for Linux.

Unlike FDO, MapGuide still needed some of the internal thirdparty libraries built.
  • DBXML - No ubuntu package available, though we can get it to build against a system-provided version of xerces, so we can at least skip building that part of DBXML.
  • Apache HTTPD - Ubuntu package available, but having MapGuide be able to integrate with an Ubuntu-provided httpd installation was not in the scope of this work, even though this is a nice thing to have.
  • PHP - Same reasons as Apache HTTPD
  • Antigrain Geometry - No ubuntu package available. Also the AGG sources are basically wedded to our Renderers project anyways.
  • DWF Toolkit - No ubuntu package available
  • CS-Map - No ubuntu package available
For everything else, Ubuntu provided the package in the right version ranges for CMake to take advantage of. Another few weeks of bashing various CMakeLists.txt files into shape and we had FDO and MapGuide both build-able on Linux via CMake. To solve the problem of still needing to build some internal thirdparty libs, but still be able to retain the CMake quality of everything is built outside the source tree, some wrapper shell scripts are provided that will copy applicable thirdparty library sources out of the current directory, build them in their copied directories and then invoke CMake and pass in all the required parameters so that it will know where to look for the internal libraries to link against when it comes to build MapGuide proper.

This was also backported to FDO, so that on distros where we do not have all our required thirdparty libraries available, we can selectively build internal copies and be able to find/link the rest, and have CMake take care of all of that for us.

So what's with the title of this post?

Remember when I wrote about how interesting vcpkg was?

What is best used with vcpkg to easily consume thirdparty libraries on Windows? Why CMake of course! Now building MapGuide on Windows via CMake is not on the immediate horizon. We'll still be maintaining Visual Studio project files by hand (instead of auto-generating them with CMake) for the immediate future, but can you imagine being able to build FDO and MapGuide on both Windows and Linux with CMake and not have to waste time on huge SVN checkouts and building thirdparty libraries? That future is starting to look real possible now!

For the next major release of MapGuide Open Source, it is my plan to use CMake over autotools as the way to build both MapGuide and FDO on Linux.

^ Well, the ninja-powered CMake build used to be blazing fast until Meltdown and Spectre happened. My dev environment got the OS security patches and whatever build performance gains that were made through ninja and CMake were instantly wiped out and we were back to square one in terms of build time. Still, the autotools build performed worse after the meltdown patches, so while CMake still beats the autotools build in terms of build time, we ultimately gained nothing on this front.

Thanks Intel!!!

Thursday, 11 January 2018

RIP: Autodesk Infrastructure Map Server (2006-2018)

Prepare the burial plot in the Autodesk Graveyard, the news has come out which I had long suspected, but is now official: Autodesk has ceased development of Infrastructure Map Server, the commercial counterpart of MapGuide Open Source.

However unlike Autodesk's other ill-fated products, Infrastructure Map Server has the unique lifeline of being able to live on through the MapGuide Open Source project because AIMS is built on top of MapGuide Open Source. Just because AIMS has died does not mean the same fate has to apply to MapGuide Open Source. This project has been in existence for 12 years and counting. The future of the MapGuide Open Source project can be as bright as the community allows for it.

If you are an AIMS customer wondering what your options are in light of this announcement, you should subscribe to the mapguide-users mailing list (if you haven't already) and share your thoughts, questions and concerns.

If you provide support/consulting/development for MapGuide/AIMS you should also subscribe and advertise your services.

I'll make some announcements on the mailing lists about future plans for MapGuide Open Source.

Rest in peace Autodesk Infrastructure Map Server, formerly known as MapGuide Enterprise (2006 - 2018)


Sunday, 24 December 2017

MapGuide tidbits: Windows 10 Fall Creators Update

In case anyone else might encounter this problem.

I had a copy of MapGuide Open Source 3.1 installed on my laptop running Windows 10. Recently it was upgraded to the Fall Creator's update and suddenly, the web tier (which was an IIS/.net configuration) of this installation stopped working.

The non-functional web tier would manifest as HTTP 503 errors when I try to connect with MapGuide Maestro



If you go into IIS manager and try to recycle the associated Application Pool, you may see an error like this



If we take a look at the Windows Event Viewer, we see log entries like this



It just so happens if we search online for "\\?\", we find a similar issue but relating to a completely different product (Team Foundation Server). Still, that link gives us a few useful leads.

Per that link, it appears to be some symlink issue regarding the Application Pool. So to verify this issue also applies to us, if we check C:\inetpub\temp\appPools we indeed see a symlink for our Application Pool name.



If we try to open it, windows throws an error



So with this confirmation of a broken symlink, we can fix the issue by following the prescription in that linked issue:
  1. Delete the broken symlink in question under C:\inetpub\temp\appPools
  2. Restart IIS and the Application Pool in question.
After this, I can connect to the Web Tier again with Maestro confirming the resolution of the issue.


Thursday, 23 November 2017

FDO road test: SQL Server 2017 on Linux

You can consider this post as the 2017 edition of this post.

So for some background. There's been several annoyances I've been personally experiencing with the SQL Server FDO provider that have given me sufficient motivation to fix the problem right at the source (code). However, before I can go down that road, I needed to set up a local dev installation of SQL Server as my dev environment is more geared towards MapGuide than individual FDO providers.

But just like my previous adventure with the King Oracle FDO provider, I didn't want to have to actually find/download a SQL Server installer and proceed to pollute my dev environment with a whole assortment of junk and bloat. We now live in the era of docker containers! Spinning up a SQL Server environment should be a docker pull away and when I no longer need the environment, I can cleanly blow it away without leaving lots of junk behind.

And it just so happens that with the latest release of SQL Server 2017, not only is running it inside a docker container a first-class user story, it is also the first release of SQL Server that natively runs on Linux.

So through the exercise of spinning up a SQL Server 2017 linux container we can kill multiple birds with one stone:

  • We'll know if MapGuide/FDO in its current form can work with SQL Server 2017
  • We'll also know how well it works with the Linux version of SQL Server (given its feature set is not at parity with the equivalent Windows version)
  • If MapGuide/FDO works, we'd then have a SQL Server environment ready to go which can be spun up and torn down on demand to then start fixing various problems with the FDO provider.

Spinning up the SQL Server 2017 linux docker container

This was easy because Microsoft provides an official docker image. So it was a case of just pulling down the docker image and adjusting some environment parameters to use a custom SQL Server sa login when we go to docker run the container and also define port mappings so we can connect to this container from the docker host OS.

The FDO Toolbox bootstrapping test

This was an easy way to determine if the SQL Server FDO provider works with SQL Server 2017. FDO Toolbox has the ability to:
  1. Create a SQL Server data store
  2. Bulk Copy spatial data into it
  3. Query/Preview data from it
If we can do all 3 things above in FDO Toolbox against the freshly spun up SQL Server 2017 linux container, that's a very good sign that everything works.

Creating the FDO data store

FDO Toolbox has a specialized UI for creating SQL Server data stores that is accessible by right-clicking the FDO Data Sources node and choosing Create Data Store - Create SQL Server


This gives us the UI to set up a new SQL Server data store


The first real test is to see if the FDO provider can connect to our SQL Server container, which is a case of filling in all the required connection properties and clicking the Test button, which gives us:


So far so good. Now that we know the FDO provider can connect to the container, we can fill out the data store parameters and click OK to create the data store, which gave us another good sign:


Now just to be sure that the FDO provider did actually create the database, I connected to this SQL Server instance through alternative tools (such as the new SQL Operations Studio) and we can see that the database is indeed there.


So now we can bulk copy some spatial data into it, which will be a nice solid verification that the feature and schema manipulation functionality of the FDO provider work in SQL Server 2017.

So I set up a bulk copy using a whole bunch of test SHP files. A few moments later, we got another positive sign:


Again, for verification we can look at this database in a different tool and can see that the FDO provider correctly created the database tables.


And that data was actually being copied in


Just as an aside: SQL Operations Studio doesn't do spatial data previews like its big brother SQL Server Management Studio.

A shame really. Oh well, at least we can do that in FDO Toolbox :)


Which is also confirmation that FDO is getting the geometry data out of our SQL Server 2017 linux container without any problems.

So based on all these findings, I feel comfortable in saying that FDO (and applications using it like MapGuide) works just fine with SQL Server 2017, especially its Linux version.

Now to deal with these actual annoyances in the FDO provider itself ...

An introduction to MgTileSeeder

I previously said I'd cover this tool in a future post, and that future is now.

MgTileSeeder (introduced as a standalone companion release to MapGuide Maestro 6.0m8) is a new command-line tile seeding application that is the successor to the current MgCooker tile seeder.

This tool is the offspring of an original thought experiment about how one could possibly build a multi-threaded tile seeder using 2017-era .net libraries and tools. It turns out the actual implementation didn't differ that much from my hypothetical code sample from the original post!

But besides being a ground-up rewrite, MgTileSeeder has the following unique features over MgCooker:

  • If your MapGuide Server is 2.6 or newer, we will use CREATERUNTIMEMAP to automatically infer the required meters-per-unit value that is critical in determining how many tiles we need to actually seed.
  • MgTileSeeder is a cross-platform and self-contained .net core application taking advantage of the newly netstandard-ized Maestro API.
  • More importantly, MgTileSeeder finally supports seeding of XYZ tilesets. In fact, the way this support has been designed, you can use MgTileSeeder as a generic tile cache seeder for any XYZ tileset, not just ones served by MapGuide itself.
Seeding standard tiled maps

The minimal command to start seeding a tiled map is simply:

MgTileSeeder mapguide -m --map

Here's an example MgTileSeeder invocation to seed a tile set

MgTileSeeder mapguide -m http://localhost/mapguide/mapagent/mapagent.fcgi --map Library://Samples/Sheboygan/TileSets/Sheboygan.TileSetDefinition

This will use CREATERUNTIMEMAP to auto-infer the required meters-per-unit (for tile sets, we make a temporary Map Definition that links to the tile set and run CREATERUNTIMEMAP against that) and then proceeds to display a running progress that updates every second:


There are other options available, such as:
  • Restricting tile seeding to a specific extent
  • Restricting tile seeding to specific base layer groups
  • Manually passing in the meters-per-unit value

Seeding XYZ tile sets

Seeding XYZ tile sets uses a completely different set of parameters. The minimal command to seed an XYZ tile set is:

MgTileSeeder xyz --url --minx --miny --maxx --maxy

An example of tiling a XYZ tile set (eg. Library://Samples/Sheboygan/TileSets/SheboyganXYZ.TileSetDefinition) in MapGuide would look like this:

MgTileSeeder xyz --url "http://localhost/mapguide/mapagent/mapagent.fcgi?OPERATION=GETTILEIMAGE&VERSION=1.2.0&CLIENTAGENT=OpenLayers&USERNAME=Anonymous&MAPDEFINITION=Library://Samples/Sheboygan/TileSets/SheboyganXYZ.TileSetDefinition&BASEMAPLAYERGROUPNAME=Base+Layer+Group&TILECOL={y}&TILEROW={x}&SCALEINDEX={z}" --minx -87.7978 --miny 43.6868 --maxx -87.6645 --maxy 43.8037

Unlike the standard tiling mode you are required to define the bounds (in lat/long) of the area you wish to seed. Also you can see here that the XYZ tiling mode accepts any arbitrary URL that has {x}, {y} and {z} placeholders. This means you can use MgTileSeeder for tiling any XYZ tile set (eg. Your own custom OpenStreetMap tile set), not just ones served by MapGuide. You just need to make sure your URL provides the required XYZ placeholders.



And that concludes our introduction to the MgTileSeeder tool.

Happy tiling!


Friday, 17 November 2017

Announcing: MapGuide Maestro 6.0m8

Here's another new milestone of MapGuide Maestro 6.0. This release is somewhat light in new features, with more emphasis on changes under-the-hood and the surrounding ecosystem.

Let's start with the new features first.

Feature Count for Thematic Rules

When dealing with thematic layers, sometimes one might want to know exactly how many features are covered by each thematic layer rule. There's now a Feature Count button to crunch those numbers for you.


Clicking it will crunch the feature counts of each individual style rule with a filter (default rule is omitted) and present the totals in a new dialog.


MgTileSeeder (the successor to MgCooker)

Not bundled with Maestro yet, but included as a standalone package available for download alongside this release is MgTileSeeder, a new command-line tile seeding application that is the successor to MgCooker and will eventually replace it in a future release.

I'll cover this tool in more detail in a future post.

New project site

Since MapGuide Maestro is now on GitHub, I've activated the GitHub Pages feature and spun up a new project web site for it.

On this site you will also find the user guide, developer's guide and the API reference for Maestro API and friends.

So speaking of Maestro API ...

Where's the SDK package?

The SDK story is going through a bit of churn at the moment. This milestone release is primarily focused around Maestro (the application) and not the API/SDK, so whatever things I had intended to finish regarding the Maestro API/SDK have taken a back seat so I can get Maestro (the application) out the door.

So as it stands, there is no SDK package with this release and there never will be with any future releases. This is due to major under-the-hood work to port the MapGuide Maestro API and supporting libraries over to target .net standard.

The end result of this is that the primary way to acquire the Maestro API is now via a NuGet package

And since the API reference is now online, this makes the SDK package somewhat redundant.

The various sample code and supporting tools in the SDK have been shipped off to a separate repository, that will be revealed in due course once they have all been updated to work in this new .net world we live in.

If you are an existing consumer of the Maestro API, it should be as simple as removing all your current assembly references to Maestro API and friends and installing the NuGet packages in the affected projects.

.net Framework 4.6.1 required

Due to porting the Maestro API to target .net standard 2.0, .net Framework 4.6.1 is the minimum version of the .net Framework required.

The Windows installer will automatically download and install this for you if you don't have it. It will also automatically install the Visual C++ redistributable so the local connection (mg-desktop) mode will also work out of the box.

Other changes/fixes
  • Fix a long standing annoyance where setting WMS bounds on a published layer will set the coordinate system to EPSG:???? requiring you to manually enter in the EPSG code. This should now be automatic most of the time. It will also be automatically transformed to EPSG:4326 bounds if required.
  • Now uses ICSharpCode.TextEditor for dialog to edit raw resource header XML
  • New resources validation rules around WMS-published Layer Definitions
  • Basic line styles no longer trashed on cancellation of the Edit Style dialog
  • Can now read configuration documents where FDO-related attributes have incorrect casing
  • No-op any map viewer rendering requests if any display parameter is <= 0
  • Disable local map preview if connecting to a MapGuide Server older than 2.1
  • Fusion editor no longer adds obsolete VirtualEarthScript element when adding Bing Maps layers
  • Now gracefully handles invalid resources with open editors instead of crashing out to desktop.