A quick note on Piggy patterns and comparison with XPath

XPath (1, 2, 3) is a language for finding nodes in an XML tree, and has a long history in AST search. Maletic et al. (4) is probably the first paper on XPath used on ASTs, using Antlr. It was further researched and is now part of the OSS world (5). In 2014, Parr added to Antlr releases an XPath API to search Antlr-generated ASTs (6). Src-d uses XPath and an engine for “universal” ASTs (7).

Continue reading

Getting Xamarin.Forms apps working with .NET Standard and Roslyn

Recently, I was trying to write a Xamarin.Forms app that uses Roslyn, Microsoft’s NET languages compiler framework. But, no matter what I did, it seemed as though it wasn’t possible. But, I tried a few things, and noticed that the Roslyn library could link with Android and iOS applications. That’s when I realized it might be possible.

That said, it depends on what you will be interested in of the Roslyn API. While one can parse and compile code from the .NET Standard/PCL library, Assembly.Load is essentially unavailable. Full functionality is available in Xamarin Android version 7. If you want to run compiled code, you need to use a bait/switch layer to accomplish that, in which case, you don’t need to make your PCL into a .NET Standard library, but it’s a good idea anyways.

Also, I’ve noticed VS 2015 is quite brittle in setting up an app to use .NET Standard. The following steps show that it can be done. Note: after each step, verify with a build and run.

  1. To set up a Xamarin Forms app to use the .NET Standard, open Visual Studio 2015 and create a blank cross-platform app that uses portable class libraries. You will see a portable class library and apps for each of the platforms.
  2. In the Project Explorer, Remove the Windows, Windows Phone apps, and leave the Android and iOS apps.
  3. Set the default project to the Android project. Build and run to make sure all is OK.
  4. Update the Xamarin.Forms in the three projects, to the latest version of Xamarin.Forms (2.3.3.175). Rebuild and run to make sure it works.
  5. In the Project Explorer, right-click on the References for the Xamarin Forms app portable class library, and “Manage NuGet Packages.” Remove the Xamarin.Forms library.
  6. Right-click on the portable class library, select Properties. Click on the Target .NET Standard hyperlink, then select NET Standard 1.5. Note, if you use .NET Standard 1.6, the References suddenly drops all DLL references, so don’t use that! In VS 2017, that appears to be fixed. Once VS has made the conversion, project.json will appear. After making the conversion, the default project has been reset. Set it back to the Android project.
  7. In the file project.json, change the ‘frameworks’ section with the following code, which allows the .NET Standard library to be compatible with certain PCL libraries.
 "frameworks": {
   "netstandard1.5": {
     "imports": [
       "portable-net45+wpa81+wp8+win8"
     ]
   }
 }
  1. Go back to the NuGet package management for the Xamarin Forms app PCL, add in Xamarin.Forms (the latest version).
  2. Rebuild and test. It should all work fine. If not, there is probably something wrong with your VS. Make sure it has the latest updates.

Roslyn works with .NET Standard, but adding it blindly causes a dependency/version issue. Roslyn requires Microsoft.Composition, but the version that it requires is old and incompatible with the .NET Standard (Package Microsoft.Composition 1.0.27 is not compatible with netstandard1.4 (.NETStandard,Version=v1.4). Package Microsoft.Composition 1.0.27 supports: portable-net45+win8+wp8+wpa81 (.NETPortable,Version=v0.0,Profile=Profile259)). Instead, add Microsoft.Composition first, then Microsoft.CodeAnalysis.

  1. In the Project Explorer, right-click on the References for the portable class library, and “Manage NuGet Packages.” Add in Microsoft.Composition (1.0.30), then Microsoft.CodeAnalysis (1.3.2). Make sure to add that to the Android project as well or you get “Severity Code Description Project File Line Source Suppression State
    Error Exception while loading assemblies: System.IO.FileNotFoundException: Could not load assembly ‘Microsoft.CodeAnalysis, Version=1.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’. Perhaps it doesn’t exist in the Mono for Android profile….”

For a check, add source referencing some Roslyn classes, build and run.

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;

namespace App15
{
    public class Class1
    {
        public Class1()
        { }

        public static void YoDoit()
        {
            SyntaxTree tree = CSharpSyntaxTree.ParseText(@"
            public class MyClass
            {
                public int MyMethod()
                {
                    return 1+2;
                }
            }");
        }
    }
}

This code is in Github: https://github.com/kaby76/XamForms-Roslyn-Example

Cross-Debugging Mono C# Executables Running on Linux

Debugging C# programs that run on a Linux server can be a challenge. On Ubuntu, a server may not have the Unity GUI installed, so you can’t use MonoDevelop. Even if the server had the GUI, MonoDevelop often does not work with a C# project because it doesn’t support many Visual Studio 2015 projects. Visual Studio supposedly can cross-debug a program running on Linux from a Windows box, but I haven’t been able to get one solution to work. (It does work for C++.Xamarin Studio works, but the steps are somewhat convoluted, and won’t work if you naively follow what has been outlined by others for a Windows box (e.g., in (a), you cannot simply build the app on Linux using xbuild, because the paths for files between server and debugger host differ). Here, I note how one can get Xamarin Studio working on Windows.

  1. Download a copy of Xamarin Studio for Windows, and install it. Xamarin Studio for Windows is, in fact, no longer officially available, but you can still find the download here as of July 2016. I don’t know how long this link will last; you can’t download Xamarin Studio for Windows though Xamarin.com.
  2. In your Linux server, make sure to have it set up with a firewall, SSH, and Mono.
  3. Create a variable in your environment to enable Xamarin Studio “Custom Soft Debugging.” Create “MONODEVELOP_SDB_TEST” with the value “1” (no quotes) through the System Properties box.
  4. Start Xamarin Studio, and open or create a C# test program, like “Hello World.” Set the configuration to Debug and build the program.
  5. Mono does not understand PDB files, so you have to create the equivalent MDB files. Start a command-line Bash shell in the directory ‘bin/Debug/”. Type: for i in *.exe *.dll;do echo $i; pdb2mdb $i; done. If pdb2mdb is not found, make sure Mono/bin is on the search path.
  6. Open a command-line Bash shell on your Linux box using “ssh”, e.g., “ssh root@192.168.1.10”.
  7. In the Windows Bash shell, type: tar -cvf – Debug | ssh root@192.168.1.10 ‘cat – > Debug.tar’.
  8. In the Linux Bash shell, type: tar -xvf Debug.tar; cd Debug; mono –debug –debugger-agent=transport=dt_socket,address=0.0.0.0:12345,server=y HelloWorld.exe, replacing HelloWorld.exe with the name of the executable you want to debug.
  9. In Xamarin Studio, open Run -> Run with -> Custom Command Mono Soft Debugger2016-07-14 (1)
  10. In the pop up, enter the IP address of the Linux box, then click on “Connect”. 2016-07-14 (9)
  11. You should now be able to debug your program. Note: you don’t need to compile the program on Linux. Note: if you can’t set breakpoints, it’s because you haven’t copied the .mdb files from Windows to Linux. Path names in the .mdb files are full paths, and Linux path names do not work.

 

Further Information

  1. https://eladnava.com/debug-remote-mono-apps-via-xamarin-studio/
  2. http://www.mono-project.com/docs/advanced/runtime/docs/soft-debugger/
  3. http://www.jeffongames.com/2012/03/debugging-embedded-mono/
  4. http://tirania.org/blog/archive/2008/Sep-04.html
  5. https://blogs.msdn.microsoft.com/vcblog/2015/11/18/announcing-the-vs-gdb-debugger-extension/

Getting a C# Server Working with Dokku and Vagrant

Developing a self-hosted web server in C# to run on Linux is like solo climbing a mountain: not a lot of people will be accompanying you; and the route will be unmarked, unmaintained, and often dangerous. In the digital world, there are two groups of programmers: people who use Node.js, Java, Ruby, etc. hosted on Linux; and people who use C# hosted on Windows. Building a C# server to run on Linux isn’t usually done because there is a perception that Mono is buggy. Most of those arguments are from experiences over 5 years ago, and Mono has come a long way. Of course, there is always room for improvement.

In particular, if you are a developer that wants to create services without focusing on all the crap associated with deployment, then Dokku is the way to go. I definitely prefer a command-line deployment rather than working with IDEs and browsers. Dokku, modeled after Heroku, is a PAAS model whereby the deployment of the server is simply a “git push.”  Unfortunately, the problem with Dokku and C# programming is that the easy “git push” deployment is offset by the difficulty in setting up the environment.

With that in mind, my goal was:

Prerequisites

Dokku requires you to work with SSH (both client and server), so you need to understand SSH very well. It is likely you have to understand how to configure your DHCP router, as I could not get Vagrant to send the DHCP router the name of the virtual machine. Also, if you aren’t familiar with Heroku, you might like to give that a spin with the Java example in order to understand some of the concepts.

Method

  1. Create a vanilla Ubuntu virtual machine, 20 GB disk space, 2048 memory, user “ubuntu”.
    1. Download the latest Ubuntu release from http://www.ubuntu.com.
    2. Start Virtualbox, and create a new virtual machine with a unique virtual_box_name running Ubuntu.
    3. Note: use the 64-bit “desktop” version of the OS. Make sure you have enough disk space for the large ISO file. You will also need at least 10GB of free disk space for the virtual machine itself.
    4. Install Ubuntu. Add user, e.g., “ubuntu”. Note: The Dokku installation scripts create user “dokku”. In order to login, you will want another account other than dokku.
    5. Restart the virtual machine.
    6. Install the VirtualBox Guest Additions.
    7. Restart the virtual machine.
    8. Start a terminal command-line interface in the virtual machine.
    9. sudo apt-get install openssh-server
    10. sudo visudo
      1. Add user ALL=(ALL) NOPASSWD: ALL to the file. This updates user with superuser rights so you don’t have to give a password for every sudo command.
    11. Shutdown the virtual machine.
  2. Create a Vagrant package of the vanilla Ubuntu virtual machine.
    vagrant box remove my-box
    vagrant init my-box
    vagrant package --base virtual_box_name
    vagrant box add my-box package.box
  3. Start the Vagrant virtual machine.
    1. Get a copy of my ‘kukku’ vagrant setup files at https://ken_domino@bitbucket.org/ken_domino/kukku.git
    2. cd kukku
    3. In the Vagrantfile, note the lines for setting up the user id and password for signing into the virtual machine, etc. Make sure they are in agreement with the names you gave in setting up the virtual machine.
      # -*- mode: ruby -*-
      # vi: set ft=ruby :
      # encoding: UTF-8
      
      BOX_NAME = ENV["BOX_NAME"] || "my-box"
      DOKKU_DOMAIN = ENV["DOKKU_DOMAIN"] || "kukku.home"
      # DOKKU_IP = ENV["DOKKU_IP"] || "10.0.0.2"
      PUBLIC_KEY_PATH = "#{Dir.home}/.ssh/id_rsa.pub"
      
      Vagrant.configure(2) do |config|
      
        # Set the name of the vagrant image file.
        config.vm.box = BOX_NAME
      
        # Display the Virtualbox GUI.
        config.vm.provider "virtualbox" do |v|
          v.gui = true
        end
      
        # Declare the name of the VM folder.
        config.vm.provider "virtualbox" do |v|
          v.name = "kukku-fun"
        end
      
        # Set the network to use bridge adapter, with a static IP, and known MAC.
        # My router does not handle the name of the guest correctly, so this bypasses
        # the problem by fixing it all. Note, in the router, I create an entry for
        # "whack" because the name isn't being broadcasted from the guest to the router.
        config.vm.network "public_network", ip: "192.168.1.33", :mac => "080027EA2F24"
        
        # Tell Vagrant to not muck around with the .ssh/authorized_hosts file.
        # It should not exist, or if it does, it should be empty. This is make sure
        # Dokku ssh works.
        config.ssh.insert_key = false
        config.ssh.private_key_path = File.expand_path("../.ssh/vagrant", __FILE__)
      
        # Define user id and password for login.
        config.ssh.username = "ubuntu"
        config.ssh.password = "ubuntu"
        
        # Provision the box with Dokku.
        config.vm.provision "shell", inline: >>-SHELL
      
          wget https://raw.githubusercontent.com/dokku/dokku/v0.4.14/bootstrap.sh
          sudo DOKKU_TAG=v0.4.14 bash bootstrap.sh
      
        SHELL
      end
      
      
    4. vagrant up
  4. Set up Dokku, and run the C# server.
    1. In the virtual machine…open a browser, and enter the address “localhost/”. It should be the Dokku set up screen.
    2. In the setup screen, enter the contents of your ~/.ssh/id_rsa.pub file, and set the name of the box to ‘whack’ or what ever machine you set for the machine. Click the button to set up Dokku.
    3. In the host machine…
      1. Verify that you see the help screen for Dokku via ‘ssh dokku@whack’. Otherwise, the ~dokku/.ssh/authorized_keys file on the virtual machine has the wrong entry. Edit that file, or type cat ~/.ssh/id_rsa.pub | ssh dokku@whack "sudo sshcommand acl-add dokku foobar"
      2. ssh dokku@whack apps:create fun
      3. ssh dokku@whack config:set fun BUILDPACK_URL=https://github.com/AdamBurgess/heroku-buildpack-mono
        1. Note: Adam Burgess’ C# buildpack is the only buildpack that seems to work. There are at least 4 others which I tried and don’t work. This buildpack uses xbuild for the build process.
      4. Get a copy of my “Hello World!” https://ken_domino@bitbucket.org/ken_domino/hn.git
        1. Note, you can substitute your program, but it should be an http server, and it must access the environmental variable PORT for the port to open. Dokku monitors that port to verify that the program is operating.
      5. cd hn
      6. git remote add foo dokku@whack:fun
      7. git push foo master
      8. Note: you should see output like the following.
        Ken@Win8-GAZ77X /cygdrive/c/Users/Ken/Documents/hn
        $ git push foo master
        Counting objects: 32, done.
        Delta compression using up to 4 threads.
        Compressing objects: 100% (25/25), done.
        Writing objects: 100% (32/32), 916.64 KiB | 0 bytes/s, done.
        Total 32 (delta 3), reused 0 (delta 0)
        -----> Cleaning up...
        -----> Building fun from herokuish...
        -----> Adding BUILD_ENV to build environment...
        -----> Fetching custom buildpack
        -----> .NET app detected
        -----> Installing mono version 636c0c95cc78-minimal
        -----> Downloading https://github.com/AdamBurgess/heroku-buildpack-mono/releases/download/636c0c95cc78/mono-minimal.tar.xz
        -----> Installing SSL certificates
        -----> Installing NuGet packages
               Installing 'Nancy.Hosting.Self 1.4.1'.
               Installing 'Nancy 1.4.3'.
               Successfully installed 'Nancy.Hosting.Self 1.4.1'.
               Successfully installed 'Nancy 1.4.3'.
        -----> Compiling .NET application
               XBuild Engine Version 14.0
               Mono, Version 4.3.3.0
               Copyright (C) 2005-2013 Various Mono authors
        
               Build started 03/04/2016 19:07:53.
               __________________________________________________
               Project "/tmp/build/hn.sln" (default target(s)):
               Target ValidateSolutionConfiguration:
               Building solution configuration "Release|Any CPU".
               Target Build:
               Project "/tmp/build/hn.csproj" (default target(s)):
               Target PrepareForBuild:
               Configuration: Release Platform: AnyCPU
               Created directory "obj/Release/"
               Target CopyFilesMarkedCopyLocal:
               Copying file from '/tmp/build/packages/Nancy.1.4.3/lib/net40/Nancy.dll' to '/tmp/build/Nancy.dll'
               Copying file from '/tmp/build/packages/Nancy.Hosting.Self.1.4.1/lib/net40/Nancy.Hosting.Self.dll' to '/tmp/build/Nancy.Hosting.Self.dll'
               Target GenerateSatelliteAssemblies:
               No input files were specified for target GenerateSatelliteAssemblies, skipping.
               Target CoreCompile:
               Tool /tmp/build/mono/lib/mono/4.5/mcs.exe execution started with arguments: /noconfig /debug:pdbonly /optimize+ /out:obj/Release/hn.exe Program.cs Properties/AssemblyInfo.cs obj/Release/.NETFramework,Version=v4.5.AssemblyAttribute.cs /target:exe /define:TRACE /nostdlib /platform:AnyCPU /reference:packages/Nancy.1.4.3/lib/net40/Nancy.dll /reference:packages/Nancy.Hosting.Self.1.4.1/lib/net40/Nancy.Hosting.Self.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Xml.Linq.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Data.DataSetExtensions.dll /reference:/tmp/build/mono/lib/mono/4.5-api/Microsoft.CSharp.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Data.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Net.Http.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Xml.dll /reference:/tmp/build/mono/lib/mono/4.5-api/System.Core.dll /reference:/tmp/build/mono/lib/mono/4.5-api/mscorlib.dll /warn:4
               Program.cs(41,17): warning CS0162: Unreachable code detected
               Target _CopyAppConfigFile:
               Copying file from '/tmp/build/App.config' to '/tmp/build/hn.exe.config'
               Target DeployOutputFiles:
               Copying file from '/tmp/build/obj/Release/hn.exe.mdb' to '/tmp/build/hn.exe.mdb'
               Copying file from '/tmp/build/obj/Release/hn.exe' to '/tmp/build/hn.exe
               Done building project "/tmp/build/hn.csproj".
               Done building project "/tmp/build/hn.sln".
        
               Build succeeded.
        
               Warnings:
        
               /tmp/build/hn.sln (default targets) ->
               (Build target) ->
               /tmp/build/hn.csproj (default targets) ->
               /tmp/build/mono/lib/mono/xbuild/14.0/bin/Microsoft.CSharp.targets (CoreCompile target) ->
        
               Program.cs(41,17): warning CS0162: Unreachable code detected
        
               1 Warning(s)
               0 Error(s)
        
               Time Elapsed 00:00:00.4912070
        -----> Discovering process types
               Procfile declares types -> web
        -----> Releasing fun (dokku/fun:latest)...
        -----> Deploying fun (dokku/fun:latest)...
        -----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...
        -----> New DOKKU_SCALE file generated
        =====> web=1
        -----> Running pre-flight checks
               For more efficient zero downtime deployments, create a file CHECKS.
               See http://dokku.viewdocs.io/dokku/checks-examples.md for examples
               CHECKS file not found in container: Running simple container check...
        -----> Waiting for 10 seconds ...
        -----> Default container check successful!
        =====> fun container output:
               http://localhost:5000/
        =====> end fun container output
        -----> Running post-deploy
        =====> renaming container (d6dc5cffa4c9) nauseous_yonath to fun.web.1
        -----> Setting config vars
               DOKKU_NGINX_PORT: 80
        -----> Creating http nginx.conf
        -----> Running nginx-pre-reload
               Reloading nginx
        -----> Setting config vars
               DOKKU_APP_RESTORE: 1
        =====> Application deployed:
               http://dokku:32769 (container)
               http://dokku:80 (nginx)
        
        To dokku@dokku:fun
         * [new branch]      master -> master
        
        Ken@Win8-GAZ77X /cygdrive/c/Users/Ken/Documents/hn
        
    4. In a browser, type ‘whack/’. You should see “Hello world!”.

    Further Information

     

    An Important Side Note

    Part of the motivation for me was to re-examine what is available in website hosting. Currently, I use GoDaddy.com . After years of enduring GoDaddy ads that employ sophomoric humor, themes of misogyny, homophobia, and animal cruelty take on another light when juxtaposed by the actions of the son of Republican GoDaddy founder. It’s particularly important for me because I started a non-profit called “Hikers and Climbers Against Domestic Violence,” which tries to bring attention to the violence some women endure. While GoDaddy does offer good services at a low price, perceptions matter, and I’m shopping around to see what are my best options.

A Short, Practical Review of Foreign Function Calls in Java, C#, C++

Android is a popular platform for smartphones, with ten of thousands of applications developed for it every year. For coding an Android app, there are a number of programming languages, but Java is the most popular. However, Xamarin provides tools for writing applications in C# if you prefer. If you use Xamarin, but sometimes want to use an open-source library written in Java, you still can. The state of currently used, practical Foreign Function Interfaces (FFI) on most platforms is similar. Of course, the difficulty is in the details.

Generally, there are two ways for code in one language to call another: use a native function to call the foreign function; or, use reflection to call the foreign function. A bindings interface offers native functions to the whole foreign API.

Java Native Interface (JNI) provides a native interface to call non-Java code from Java. It also provides an API for use in C++ to invoke Java code, structured like Java’s reflection API.

There are a number of tutorials on how to use JNI, but they are somewhat out of date. This article brings it forward–if only to a small degree, with “how to” explanations.

Continue reading