Friday, 18 July 2014

Other new things (or things I forgot to talk about) in Fusion for MapGuide 2.6

This post has been stuck in my draft queue for several months now. I guess I forgot about this post so some of the content in this post may have already been talked about. Still, there some content here that I'm certain hasn't been talked about, and since I currently don't have idea at the moment what has and hasn't been talked about, I've decided to just publish the whole post.

So here's some new features (or features I've probably already talked about) in Fusion for MapGuide Open Source 2.6

OpenLayers updated to 2.13.1

OpenLayers in Fusion has been updated to the latest stable release, which is 2.13.1. This gives us important fixes for tiled layers in IE10, among other fixes and improvements from the 2.12 release.

Short of a few specific customizations to the Google layer (to disable tilt at lower levels), you'll be glad to hear that the OpenLayers included with Fusion is now mostly vanilla with very few modifications. Making your own custom OpenLayers build should be a painless affair.

Built with Closure

fusionSF-compressed.js is now built with the Google Closure Compiler.

Besides giving us a 50kb reduction in file size over YUI compressor with default settings, building through Closure gives us Source Map support for modern browsers.

What this means is you can now debug and step through fusionSF-compressed.js in its uncompressed form with full comments without having to modify the template files to switch to a debug version of Fusion. Current versions of Chrome and Firefox both support source maps in their respective developer tools.

With advanced optimizations, fusionSF-compressed.js file size can be reduced by up to 500kb! Unfortunately, building with closure's advanced optimizations may actually alter application behaviour and requires a thorough audit of the code in Fusion first before we can determine if it is safe for us to flick the switch. So for now, we build with default settings.

Delayed refresh on layer/group toggle in legend

When you toggle a layer or group on/off in the legend, it triggers a map refresh. If you do this on multiple layers/groups in quick succession, it's going to fire off that many refresh requests. Not very efficient.

For this release, we've added support for a new DrawDelay extension property in the Map element of the MapGroup. Setting this property will cause any toggle in the legend to delay the refresh for that amount of time (in ms).



Having this property set will allow you to toggle multiple layers/groups in the legend in quick succession, but only trigger a single refresh request at the end as long as each individual toggle falls within the delay period. This feature is opt-in. By default, the delay is 0 which means legend widget will behave like before where each toggle triggers a refresh immediately.

More QuickPlot tweaks

The legal disclaimer is now a configurable element


Of course, you may want this element to be mandatory, so we've added a new AllowDisclaimerToggle extension property that determines whether this checkbox will be shown.

In addition, you can configure the initial checked state of these elements through the following new extension properties:
  • ShowLegend
  • ShowNorthArrow
  • ShowCoordinates
  • ShowScaleBar
  • ShowLegalDisclaimer (does nothing if there is no legal disclaimer attached or AllowDisclaimerToggle = false)

Making MBTiles databases with a little help from mapguide-rest

So allow me to explain what this screenshot represents


What you are looking at is tiles stored in a MBTiles database. The Android app in question is Locus Map, which supports MBTiles databases. MBTiles is an ideal format for delivering tiled maps to mobile devices. It's just a SQLite database that uses the same intuitive XYZ tiling scheme used by OpenStreetMap and friends.

This post will show you how to make such MBTiles databases with a little help from the mapguide-rest extension.

[NOTE: These instructions were used against the current master branch of mapguide-rest which supercedes the 0.8 release, and has several fixes for tile caching. These instructions may or may not work on the 0.8 release]

First thing you need is to download and install Portable Basemap Server.

In the directory where you installed Portable Basemap Server, edit the CustomOnlineMaps.xml and add a new map source to any XYZ tileset accessible from mapguide-rest

  <onlineMapSource>
    <name>mapguide-rest Sheboygan</name>
    <url><![CDATA[http://localhost/mapguide/rest/library/Samples/Sheboygan/MapsTiled/SheboyganNoWatermark.MapDefinition/xyz/Base Layer Group/{$z}/{$x}/{$y}/tile.png]]></url>
    <tileFormat>png</tileFormat>
    <servers></servers>
  </onlineMapSource>

Note the {$x}, {$y} and {$z} placeholder tokens in the URL. These placeholders are required and Portable Basemap Server will be replacing them with actual X, Y and Z values when fetching tiles.

Once you've edited and saved CustomOnlineMaps.xml, run the PortableBasemapServer.exe as an administrator.

What is unique about Portable Basemap Server (and something I could not find in any other free/OSS packages), is that Portable Basemap Server includes a convenient function to create an MBTiles database from any online tiled map (that presumably follows a XYZ tiling scheme). This function is accessible from the Format Convert menu.


In the Format Convert window, change the Data Source Type to the map source you registered in CustomOnlineMaps.xml. If your tiled map does not span the entire world, you may want to hone in to the specific region your tiled map is in with one of the existing map sources before switching over to your mapguide-rest map source.



Now do the following:

  • Specify the path of the MBTiles database you want to create
  • Trace a box around the area you want to fetch the tiles for
  • Tick the Download Levels (aka. Zoom levels) you want to fetch tiles for. The map preview shows the current zoom level, so generally you want to tick that level and every level above that.
  • Tick Compact? to optimize storage of redundant tiles

Once you are satisfied with the settings, click Start to initiate the tile seeding process. Portable Basemap Server will spit out lots of useful progress information while building the MBTiles database.


Once the tile seeding process is complete, you can now transfer this MBTiles database file to your mobile device and load it into your MBTiles supported app.

As a bonus, the actual process of building an MBTiles database also seeds the XYZ tile cache on the server-side. So this process can also serve as a XYZ tile cache seeder if you have no use for MBTiles.

If you're interested in what platforms support MBTiles, check out the MBTiles wiki.

Announcing: MapGuide Open Source 2.6 RC2

The number of critical issues revolving around v2.6 QUERYMAPFEATURES that have cropped up since the MapGuide 2.6 RC1 release have necessitated a second release candidate.

Here's a list of changes since RC1:
  • Fix: Bad sample code and data in light of the Sheboygan dataset update.
  • Fix: v2.6 QUERYMAPFEATURES cannot append/update viewer selections via shift-click in AJAX and Fusion viewers.
  • Fix: v2.6 QUERYMAPFEATURES cannot handle dates before Jan 1, 1970.
  • Fix: GEOMETRY parameter in QUERYMAPFEATURES operation should be optional. QUERYMAPFEATURES should be able to accept a query geometry (via GEOMETRY) or an XML selection string (via FEATUREFILTER) when updating viewer selections.
  • Remove setselection AJAX viewer helper script which is made redundant with the above fix.
  • Fusion: Add int16 identity property support for search widget.
  • Fusion: Allow blank titles in print widget.
  • Fusion: Preserve inner stack trace (if possible) when reporting back un-handled AJAX errors.

Friday, 11 July 2014

Tuesday, 1 July 2014

How we make our builds of MapGuide

For this post, I'm going to shed a little insight (for those interested) into how we produce our builds of MapGuide Open Source for Windows and Linux.

Windows

We use Jenkins to coordinate our Windows builds of MapGuide. Jenkins is a Java-based continuous integration tool, though in our case we generally emphasise the continuous aspect of CI more than the integration aspect due to the nature of C++ code. Jenkins was chosen because of its ease of use and its massive array of available plugins that enhance the functionality of Jenkins. In addition to MapGuide and FDO, Maestro is also built with Jenkins.



MapGuide and FDO are built with completely free tools. For 2.5 and older releases, we used Visual C++ 2010 Express edition and some installation gymnastics with the Windows 7.1 SDK to get access to the 64-bit Visual C++ compiler. Thanks to the migration to Visual Studio 2012 as the de-facto windows compiler for MapGuide and FDO, building MapGuide/FDO with free tools is now a totally painless experience as the Windows Desktop SKU of VS2012 Express satisfies all our requirements, including both x86 and x64 C++ compilers needed for us to produce 32-bit and 64-bit builds.

Sadly, since the majority of the code we're building is C++, that does not lend itself to fast iteration cycles where build times are generally expected to be several minutes. Our build times for MapGuide and FDO are several hours which generally hampers the quick feedback cycles that are expected of normal CI processes.

So our MapGuide and FDO build jobs are manually triggered instead of triggered off of polling a remote SCM repository. Maestro however, builds a lot faster (being .net code) so it is configured to trigger off of new commits meaning we can get fresh builds of Maestro several minutes after a svn commit.

So when it comes to put out a new build of MapGuide, we simply fire up Jenkins, launch the FDO build jobs (that will trigger the downstream MapGuide build job when completed), and check back in several hours later. If a build fails, we pore through the various log files to find out why and see how it can be rectified. This process is not perfect, but it's good enough given what we have to work with.

Linux

For 2.5 and older releases, producing Linux builds of MapGuide was a somewhat manual process of making a CentOS or Ubuntu VM in VirtualBox, setting up any required tools and libraries and running a set of build scripts to compile MapGuide and FDO for that particular distro. This is why the Linux builds always lagged several days behind the Windows builds.

For 2.6, I wanted to vastly streamline this process. Like the Windows build process, the Linux build process should be something we can trigger with a press of a button or running a script and we can check back several hours later to collect the various build artifacts for upload or additional testing.

To this end, I caught wind of this wonderful tool called vagrant, which made the process of setting up and configuring VMs (a process they call provisioning) much simpler. Some leg work was required to set up the CentOS and Ubuntu base boxes for vagrant but once that was done, producing a CentOS or Ubuntu build of MapGuide was a simple "vagrant up" away.

The use of vagrant means VMs are now truly disposable entities. Our Linux VMs are created as part of the build process and are disposed of after building MapGuide to ensure a clean slate VM is spun up every time for subsequent builds. This means MapGuide/FDO will always be built from a clean environment and means we won't encounter problems due to tainted build environments.

Like the Windows build process, this one is not perfect. Some of the MapGuide unit tests involving threads have a very high chance of locking up the shell provisioner (I don't know why this happens), so these tests have to be disabled and run elsewhere in a manually supervised VM.

Also editing Vagrantfiles requires careful review and scrutiny as a typo or syntax error in the shell script code in a Vagrantfile means several hours lost as you wonder and investigate why "vagrant up" didn't produce your set of build artifacts after leaving it for several hours.

Still, the process as a whole is a lot better than what we had before which is what ultimately counts.

In closing

If you're interested in setting up your own CI environment for MapGuide, you can find the configurations I've used:
Now when I'm ready to put out a build of MapGuide, it's just a case of:
  • Triggering the FDO build jobs in Jenkins
  • Running "vagrant up" on my CentOS and Ubuntu Vagrantfiles
I don't think it can get much simpler than that!

Friday, 27 June 2014

Announcing: MapGuide Open Source 2.6 RC1

Here's the first (and hopefully last) release candidate for MapGuide Open Source 2.6

Key changes from Beta 1 include:

Improved Ubuntu installer

Although we had to fall back to Ubuntu 12.04 as the supported version for stability reasons, we had time to at least improve the installation experience a bit with some more options.

You can now configure server IP and listening ports


And the listening ports for Apache and Tomcat



Updated Sheboygan Sample Dataset

The venerable Sheboygan sample data set has been updated:
API Documentation

RC1 fixes missing .net/Java API documentation for some constants which weren't formatted properly in the C++ code causing our API documentation transfer process to skip over them

Similarly, un-published APIs (ie. anything under EXTERNAL_API in C++ code) will now have warning labels in their respective API documentation telling you of this fact.


Ubuntu

The Ubuntu build of MapGuide now uses CentOS-compiled FDO binaries to workaround stability issues with Ubuntu-compiled FDO. We'll be doing this for the foreseeable future.

Updated Sample Code

The various sample code that's included with the installer have been updated to not use any deprecated classes and methods in the MapGuide API and to fix various PHP warnings.

Similarly, Fusion and various applications in the web tier have also been updated in the same manner.

Fixes from Beta 1
  • Fix inconsistent results when making v2.6 QUERYMAPFEATURES requests
  • Fix bounds of selected features not being transformed to the Map's coordinate system in v2.6 QUERYMAPFEATURES, breaking functions like zoom to selection in AJAX viewer and Fusion.
  • CentOS/Ubuntu install scripts now set the proper permissions for the web tier temp dir
  • Fix some PHP deprecation errors in the Feature Source previewer
  • Fix broken links in Site Administrator help documentation
  • Fusion: Fix EPSG 3857 not being recognized in production mode
  • Fusion: Fix tiled maps not rendering properly
  • Fusion: Fix legend/map inconsistency with multiple base layer groups
  • Fusion: Fix PHP deprecation warnings in Selection.php
  • Fusion: Include XXE security hotfix
  • FDO: Fix incorrect SQL Server queries for layers with filters
  • FDO: Fix WMS provider not being able to connect to HTTPS services
If nothing show-stopping happens a week and a bit from this RC1 release, that's when we'll put out 2.6 final.

Thursday, 19 June 2014

Announcing: MapGuide Maestro 6.0 Alpha 3

Here's the 3rd alpha release because I was tired of running my installed copy and keep finding out it's the previous release :)

This release has 2 main changes.

Firstly, the code base has undergone lots of breaking changes to support resources of unknown types. This was done to "future-proof" Maestro ensuring it can work with future versions of MapGuide that introduce new resource types. From the user-perspective, nothing much should change. If you're building applications with the Maestro API, expect some minor pain fixing up all the compiler errors.

In fact, let me extend that disclaimer to all iterations of the Maestro API that will be put out before 6.0 goes final. If an API looks crufty, we'll probably either tweak it, remove it or replace it and breaking applications as a result. As mentioned already, I intend for 6.0 to be my last major release, so if we have the opportunity to do things right (even if things break along the way), we should.

Secondly, in the similar vein of future proofing, the Map Group component of the Fusion editor has been re-designed. The UI now reflects the logical structure of the Map Group itself.



Adding commercial layers is now a case of picking the available layer types from the dropdown menu.



Note that the ordering of these items is somewhat important, as it reflects the draw order these items will be loaded in by Fusion. For all intents and purposes, the "MapGuide" layer should always be at the top and the external layer you want as the default backdrop should be the next topmost item after that. The other external layers can be in any order, as you will only ever be able to see one external layer at any time. Using this screenshot as an example.


OpenStreetMap with the Mapnik tileset, will be the default backdrop.

For custom external layers, you can use the Generic Layer option and edit the raw XML needed to initialize this custom layer.



Download