Calling Roslyn from Net Framework and Net Core

It never ceases to amaze me how people can write a huge API and never bother to document how to use it. But, it’s been that way for as long as I can remember, going back 35 years. In my latest adventures, I’ve been trying to compile, link, and run C# code dynamically using Roslyn for Piggy, my transformational system. If you’ve ever used Roslyn in C#, you’ve probably discovered that it can be such a pain in the arse to use because Microsoft gives doc for the APIdoes give some tutorials, but I can’t find a simple example for compiling, linking, and running C#. I don’t need to know all the details yet, just a starting point framework. Unfortunately, the solution is quite sensitive to whether you use Net Core or Net Framework.

Continue reading

Posted in Tip

MSBuild rules for Antlr4 grammars using Antlr4BuildTasks

In order to better support Piggy, which uses Antlr4, I’ve added a NuGet package called Antlr4BuildTasks. This package is a pared-down derivative of the excellent work of Sam Hartwell Antlr4cs code generator package, and includes just the rules and code needed to do builds in MSBuild, Dotnet, or Visual Studio 2017 IDE–just no Antlr4 tool itself. This package decouples the build rules from the Antlr4 tool and runtime, so you can build Antlr programs using the latest Java-based Antlr tool and runtime release. To use this package, make a reference to this package as if you would to any NuGet package; make sure to also reference the Antlr4.Runtime.Standard package, install Java, the Java-based Antler tool, and set JAVA_HOME and Antlr4ToolPath. The tool works with Net Core, Net Framework, or Net Standard code, and on Windows or Linux.

Continue reading

Posted in Tip

This little piggy is not at the market yet

With a bit of hacking for the last month or two, and I can finally see that I am making progress on Piggy, a new kind of p/invoke generator. Some might say “Why in the world are you wasting time writing a p/invoke generator? Aren’t there tools already that do this?” Well, yeah, there are other generators, but they all…how should I say…suck! I need a p/invoke generator for Campy, a compiler and runtime for C# for GPUs, which I am still working on, but had to place on the back burner to work on this. Campy uses LLVM and CUDA. Because these libraries are large and constantly changing, I have to have an automated way of handling new releases.

Continue reading

Posted in Tip

Porting Re2/j to C#

For these last few weeks, I’ve been trying to grapple with the problem of p/invoke–the nasty but must-use feature in C#. While one could write these declarations out by hand, some libraries are too large, and change too often, so people use p/invoke generators, like SWIG. However, the is no generator that is easy to use or generates 100% correct C# declarations. So, as every software developer does, so I go to re-invent the wheel.

Continue reading

Posted in Tip

Useful websites for algorithm designers and programmers

I’ve been lax the last six months on my blog, working instead on Campy (a C#/GPU programming language extension). Now that that is slightly under control, time to get back to the blog. And, by the way, the whole reason for Campy is to implement popular algorithms to run on a GPU, I thought I’d take some time to review what information is available on the internet on algorithms. The following is a list I’ve been working on for a few months. It is by no means a complete list. But, I hope it covers some of the more popular sites. The entries are not in any particular order. Note, this list does not include parallel algorithms, which will be a post unto itself, nor the seemingly required AI algorithms you must have nowadays.

Continue reading

Posted in Tip

Windows recover tools for an idiot

I’m going to tell a story that probably many of you can relate to.

Last week, I experienced a slow motion catastrophic failure in Windows. It surprise me because ever since Windows 7, I’ve had pretty good luck in rolling forward with the updates and upgrades of the OS. But, that changed when I moved to the Windows 10 Creator Update.

As Microsoft posted in its website, “Why wait? Download Creator’s Update now.” So I did.

Unfortunately, it started with problems with an old NVIDIA Ti 470 graphics card. More importantly, I couldn’t run a Samsung Android virtual machine with VirtualBox due to a problem in a VirtualBox network adapter driver.

I tried the usual: reinstalling VBox, some registry edits of some VBox settings, and sanity checks on the OS and disk drives. But, no matter what I tried, I couldn’t get VBox working.

Biting the bullet, I decided on increasingly drastic measures: a roll back to a previous restore point; a Windows 10 Reset (keeping user files). While the VBox problem was fixed, Windows broke in the process: I couldn’t use Windows 10 Start and type in “cmd” to run a shell.

In desperation, I decided to try a full Windows 10 Reset (removing all user files), but stopped immediately when Windows prompted whether I wanted to delete all data from all drivers in the machine! Really?? Being old school, I decided just to do a format/install fresh copy of the OS instead. I know, so “old school”.

In the end, I was able to get the PC back up and restore my files. But, I learned a few things about the tools MS offers to fix a Windows.

System File Integrity

SFC — System File Checker

Verifies the validity of Windows system files.

C:\Windows\system32>sfc /help

Microsoft (R) Windows (R) Resource Checker Version 6.0
Copyright (C) Microsoft Corporation. All rights reserved.

Scans the integrity of all protected system files and replaces incorrect versions with
correct Microsoft versions.

SFC [/SCANNOW] [/VERIFYONLY] [/SCANFILE=<file>] [/VERIFYFILE=<file>]
    [/OFFWINDIR=<offline windows directory> /OFFBOOTDIR=<offline boot directory>]

/SCANNOW        Scans integrity of all protected system files and repairs files with
                problems when possible.
/VERIFYONLY     Scans integrity of all protected system files. No repair operation is
                performed.
/SCANFILE       Scans integrity of the referenced file, repairs file if problems are
                identified. Specify full path <file>
/VERIFYFILE     Verifies the integrity of the file with full path <file>.  No repair
                operation is performed.
/OFFBOOTDIR     For offline repair specify the location of the offline boot directory
/OFFWINDIR      For offline repair specify the location of the offline windows directory

e.g.

        sfc /SCANNOW
        sfc /VERIFYFILE=c:\windows\system32\kernel32.dll
        sfc /SCANFILE=d:\windows\system32\kernel32.dll /OFFBOOTDIR=d:\ /OFFWINDIR=d:\windows
        sfc /VERIFYONLY

C:\Windows\system32>

Basic File System Integrity

Chkdsk — Checks a disk and displays a status report

This tool checks the validity of a file system. It used to check for bad sectors in a drive, but that is no longer performed by the tool.

C:\Windows\system32>chkdsk /?
Checks a disk and displays a status report.


CHKDSK [volume[[path]filename]]] [/F] [/V] [/R] [/X] [/I] [/C] [/L[:size]] [/B] [/scan] [/spotfix]


  volume              Specifies the drive letter (followed by a colon),
                      mount point, or volume name.
  filename            FAT/FAT32 only: Specifies the files to check for
                      fragmentation.
  /F                  Fixes errors on the disk.
  /V                  On FAT/FAT32: Displays the full path and name of every
                      file on the disk.
                      On NTFS: Displays cleanup messages if any.
  /R                  Locates bad sectors and recovers readable information
                      (implies /F, when /scan not specified).
  /L:size             NTFS only:  Changes the log file size to the specified
                      number of kilobytes.  If size is not specified, displays
                      current size.
  /X                  Forces the volume to dismount first if necessary.
                      All opened handles to the volume would then be invalid
                      (implies /F).
  /I                  NTFS only: Performs a less vigorous check of index
                      entries.
  /C                  NTFS only: Skips checking of cycles within the folder
                      structure.
  /B                  NTFS only: Re-evaluates bad clusters on the volume
                      (implies /R)
  /scan               NTFS only: Runs an online scan on the volume
  /forceofflinefix    NTFS only: (Must be used with "/scan")
                      Bypass all online repair; all defects found
                      are queued for offline repair (i.e. "chkdsk /spotfix").
  /perf               NTFS only: (Must be used with "/scan")
                      Uses more system resources to complete a scan as fast as
                      possible. This may have a negative performance impact on
                      other tasks running on the system.
  /spotfix            NTFS only: Runs spot fixing on the volume
  /sdcleanup          NTFS only: Garbage collect unneeded security descriptor
                      data (implies /F).
  /offlinescanandfix  Runs an offline scan and fix on the volume.
  /freeorphanedchains FAT/FAT32/exFAT only: Frees any orphaned cluster chains
                      instead of recovering their contents.
  /markclean          FAT/FAT32/exFAT only: Marks the volume clean if no
                      corruption was detected, even if /F was not specified.

The /I or /C switch reduces the amount of time required to run Chkdsk by
skipping certain checks of the volume.

C:\Windows\system32>

Bad Disk Drive Checks

There are a number of free tools which check for bad sectors on a hard disk drive (HDD). You should not use these tools on solid state drives (SSD) as these checks shorten the life of the SSD.

Many of the tools are manufacturer-specific programs. For example, Seatools is specific for Seagate drives. Sandisk SSD Dashboard is specific for Sandisk. Macrorit Disk Scanner seems pretty good tool with a colorful GUI to boot, and is not manufacturer specific.

Windows trouble shooter

Windows provides some trouble shooting programs to help detect and fix problems. You can operate them from the GUI, or the command line.

msdt.exe /id name-of-check

Available checks:
AeroDiagnostic
NetworkDiagnosticsDA
DeviceDiagnostic
HomeGroupDiagnostic
NetworkDiagnosticsInbound
NetworkDiagnosticsWeb
IEDiagnostic
IESecurityDiagnostic
NetworkDiagnosticsNetworkAdapter
PerformanceDiagnostic
AudioPlaybackDiagnostic
PowerDiagnostic
PrinterDiagnostic
PCWDiagnostic
AudioRecordingDiagnostic
SearchDiagnostic
NetworkDiagnosticsFileShare
MaintenanceDiagnostic
WindowsMediaPlayerDVDDiagnostic
WindowsMediaPlayerLibraryDiagnostic
WindowsMediaPlayerConfigurationDiagnostic
WindowsUpdateDiagnostic

 

DISM — Deployment Image Servicing and Management tool

DISM enumerates, installs, uninstalls, configures, and updates features
and packages in Windows images.

C:\Windows\system32>dism /help

Deployment Image Servicing and Management tool
Version: 10.0.15063.0


DISM.exe [dism_options] {Imaging_command} [<Imaging_arguments>]
DISM.exe {/Image:<path_to_offline_image> | /Online} [dism_options]
         {servicing_command} [<servicing_arguments>]

DESCRIPTION:

  DISM enumerates, installs, uninstalls, configures, and updates features
  and packages in Windows images. The commands that are available depend
  on the image being serviced and whether the image is offline or running.

GENERIC IMAGING COMMANDS:

  /Split-Image            - Splits an existing .wim or .ffu file into multiple
                            read-only split WIM/FFU files.
  /Apply-Image            - Applies an image.
  /Get-MountedImageInfo   - Displays information about mounted WIM and VHD
                            images.
  /Get-ImageInfo          - Displays information about images in a WIM or VHD
                            file.
  /Commit-Image           - Saves changes to a mounted WIM or VHD image.
  /Unmount-Image          - Unmounts a mounted WIM or VHD image.
  /Mount-Image            - Mounts an image from a WIM or VHD file.
  /Remount-Image          - Recovers an orphaned image mount directory.
  /Cleanup-Mountpoints    - Deletes resources associated with corrupted
                            mounted images.
WIM COMMANDS:

  /Apply-CustomDataImage  - Dehydrates files contained in the custom data image.
  /Capture-CustomImage    - Captures customizations into a delta WIM file on a
                            WIMBoot system. Captured directories include all
                            subfolders and data.
  /Get-WIMBootEntry       - Displays WIMBoot configuration entries for the
                            specified disk volume.
  /Update-WIMBootEntry    - Updates WIMBoot configuration entry for the
                            specified disk volume.
  /List-Image             - Displays a list of the files and folders in a
                            specified image.
  /Delete-Image           - Deletes the specified volume image from a WIM file
                            that has multiple volume images.
  /Export-Image           - Exports a copy of the specified image to another
                            file.
  /Append-Image           - Adds another image to a WIM file.
  /Capture-Image          - Captures an image of a drive into a new WIM file.
                            Captured directories include all subfolders and
                            data.
  /Get-MountedWimInfo     - Displays information about mounted WIM images.
  /Get-WimInfo            - Displays information about images in a WIM file.
  /Commit-Wim             - Saves changes to a mounted WIM image.
  /Unmount-Wim            - Unmounts a mounted WIM image.
  /Mount-Wim              - Mounts an image from a WIM file.
  /Remount-Wim            - Recovers an orphaned WIM mount directory.
  /Cleanup-Wim            - Deletes resources associated with mounted WIM
                            images that are corrupted.

IMAGE SPECIFICATIONS:

  /Online                 - Targets the running operating system.
  /Image                  - Specifies the path to the root directory of an
                            offline Windows image.

DISM OPTIONS:

  /English                - Displays command line output in English.
  /Format                 - Specifies the report output format.
  /WinDir                 - Specifies the path to the Windows directory.
  /SysDriveDir            - Specifies the path to the system-loader file named
                            BootMgr.
  /LogPath                - Specifies the logfile path.
  /LogLevel               - Specifies the output level shown in the log (1-4).
  /NoRestart              - Suppresses automatic reboots and reboot prompts.
  /Quiet                  - Suppresses all output except for error messages.
  /ScratchDir             - Specifies the path to a scratch directory.

For more information about these DISM options and their arguments, specify an
option immediately before /?.

  Examples:
    DISM.exe /Mount-Wim /?
    DISM.exe /ScratchDir /?
    DISM.exe /Image:C:\test\offline /?
    DISM.exe /Online /?



C:\Windows\system32>

Additional examples:

dism /online /cleanup-image /checkhealth
dism /online /cleanup-image /scanhealth

Windows Driver Verifier

Verifier.exe is a tool to check drivers installed on your system. It is a Desktop GUI application. Be very careful with this program: you can easily trash your system using the tool–from the voice of experience! Make sure to create a backup before proceeding. A good intro is here.

Recreate a Profile

Sometimes recreating a profile may solve your problems. See https://community.spiceworks.com/how_to/121165-re-create-user-profile-windows

How to Start Windows in Safe Mode with Command Prompt

Once in a while, you may need to go into “Safe Mode” when booting Windows. If the PC can boot, try this at the login screen: hold down the Shift key and click on the power button and then click on Restart. If the PC cannot boot, you must boot from a recovery disk. Make sure you do that before trouble hits! Plug in a flash drive, open Control Panel’s Recovery tool, then click Create a recovery drive.

 

Posted in Tip

Windows backups for an idiot

Today, I’m writing this note about backup procedures for Windows. Up to now, I’ve been backing up my Windows machines (operating currently five) in an ad hoc manner. Cringe you may, but I’d only made a backup once a month or so. Well, for the many decades I’ve been working with computers, I’ve never lost that much, through disk failures, unexpected file deletions, and upgrades. If a machine got trashed, I could mount the driver in another machine, boot a good OS, copy the files I’m interested in, and start afresh with a reformatted/reinstalled OS. But, my luck changed unexpectedly last week.

The recent Windows 10 Creator’s Update (which is actually an upgrade) bit me big time. And, it took me a week to recover from the mess MS dealt me.

It seems that many updates from Microsoft these days are poorly tested. Updates to a box are shoved down your throat whether you like it or not. The “new” Microsoft is repeating the lessons it should have learned with Windows Vista.

Thus, backups are more important now than ever!

Backups in Windows 10

Microsoft says that Windows 10 provides a “backup” through “file history.” But this is NOT a backup. A backup should include system files and partitions, not just your personal files. Windows 10 also provides the “Windows 7 backup”, but I really do not understand why it’s called that, and not “Windows 10 backup.” Is it going to be supported in the future? Maybe not!

Online Backups

An online backup is a good alternative. There are plenty of commercial products available, some of which were reviewed just a few days ago (http://www.pcmag.com/article2/0,2817,2288745,00.asp).  However, online backups have several problems: the services require a yearly subscription; there are limits on the size or duration of the backups saved; transferring data to and from the online service places demands on the internet link; online backup providers may not be reliable in the long run.

Backup to a spare PC on a LAN

Because I have a spare box, several large disks installed in it, with the machine attached to a gigabit LAN, I went with Acronis True Image 2017 to the spare box. The cost of Acronis was $60, which isn’t too bad. On each machine, a script is run to wake up the spare server, run Acronis True Image, then shutdown the server at the end of the backup.

The script is executed by the Windows Task Scheduler, which has the ability to wake up the computer from sleep mode, or start the process when the computer is first turned on. This script is written in Powershell. I would have preferred to write it in Bash, but WSL Bash does not work with the Windows Task Scheduler.

function grep {
    $input | out-string -stream | select-string $args
}

$var = tasklist | grep TrueImageHostNotify.exe

if ($var)
{
    'There is a backup currently running.'
    'Cannot start another backup because it might interfere with existing backup.'
    Exit 1
}
else
{
    'ok'
}

'No backups currently underway.'
' Turning on server...'

.\wol\wol\bin\Debug\netcoreapp1.1\win10-x64\publish\wol.exe -mac 44-8a-5b-ca-6b-be -mount \\llano\e\ -hostname llano

if ($?)
{
    'Backup server is now up.'
}
else
{
    'Wol of the backup server failed for some reason.'
    Exit 1
}


# Invoke backup. Note, the script name is generated by Acronis True Image when you create a backup plan.

&"c:\Program Files (x86)\Acronis\TrueImageHome\TrueImageLauncher.exe" "/script:C8E6E46F-2075-4916-9AE5-04977FDBA37F"
if ($?)
{
}
else
{
    'Acronis True Image launcher failed for some reason.'
    Exit 1
}

# Wait until the backup to begin.
while ($true)
{
    $var = tasklist.exe | grep TrueImageHomeNotify.exe
    if ($var)
    {
        break
    }
    'Waiting for backup to begin'
    Start-Sleep -s 5
}


# Wait for backup to finish.
while ($true)
{
    $var = tasklist.exe | grep TrueImageHomeNotify.exe
    if ($var)
    {
    }
    else
    {
        break
    }
    'Waiting for backup to end'
    Start-Sleep -s 5
}


# Turn off the remote backup server.

'Turning off server...'
$uid = Get-Content .\uid.txt -Raw
$pw = Get-Content .\pw.txt -Raw
.\psshutdown.exe -d '\\llano' -u $uid -p $pw

Exit 0

A simple WOL program in Net Core

To wake up the backup spare box, I wrote a “WOL” program in C# Net Core. It sends out a UDP “magic packet” to wake up the backup server when needed, which must be configured in both the BIOS and the network controller driver to accept WOL packets.

namespace WOL
{
    using System.IO;
    using System;
    using System.Collections.Generic;
    using System.Net;
    using System.Net.Sockets;
    using System.Threading;
    using System.Linq;
    using System.Runtime.InteropServices;

    internal class Program
    {
        private static int _count = 1;
        private static int _interval = 1;
        private static byte[] _mac;
        private static bool _sleep = false;
        private static string _mount;
        private static string _hostname;

        public static void Main(string[] args)
        {
            try
            {
                ParseArgs(args);
                if (_mac != null)
                {
                    List<byte> result = new List<byte>();
                    result.AddRange(Enumerable.Repeat((byte) 0xff, 6));
                    result.AddRange(Enumerable.Repeat(_mac, 16).SelectMany(arr => arr));
                    byte[] buffer = result.ToArray();
                    IPEndPoint ep = new IPEndPoint(IPAddress.Broadcast, 9);
                    UdpClient client = new UdpClient();
                    client.EnableBroadcast = true;
                    while (_count > 0)
                    {
                        Console.WriteLine("Sending packet...");
                        client.SendAsync(buffer, buffer.Length, _hostname, 9);
                        if (_count > 1)
                        {
                            Thread.Sleep(_interval * 1000);
                        }
                        _count--;
                    }
                }

                if (_mount != null) Mount(_mount);

                if (_sleep) Sleeper.Suspend();
            }
            catch (MacException e)
            {
                Console.WriteLine(e.Message);
                Environment.Exit(1);
            }
            catch (UsageException)
            {
                Usage();
                Environment.Exit(1);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Environment.Exit(1);
            }
        }

        private static void Mount(string path)
        {
            var x = Directory.GetDirectories(path);
            System.Console.WriteLine(path + " size = " + x.Length);
        }

        private static void Usage()
        {
            Console.WriteLine($@"wol - Wake-On-Lan command-line utility
-c <int>              Number of packets to send to slave machine.
-i <int>              Number of seconds between packets sent.
-mac <string>         MAC address of the slave machine to wake up.
-mount <string>       UNC of the path of a slave file system to check if accessible.
-hostname <string>    Name of the slave computer on the LAN.
-s                    Indicates to place the host machine in sleep.
");
        }

        private static void ParseArgs(string[] args)
        {
            // Just in case, set up something.
            if (args.Length == 0)
            {
                throw new UsageException("No args. Check usage.");
            }
            int argc = 0;
            while (argc < args.Length)
            {
                if (args[argc] == "-mac")
                {
                    argc++;
                    string[] macParts = args[argc].Split(':', '-');
                    if (macParts.Length != 6)
                    {
                        throw new MacException(args[0]);
                    }

                    byte[] mac = new byte[6];
                    for (int i = 0; i < 6; i++)
                    {
                        mac[i] = Byte.Parse(macParts[i], System.Globalization.NumberStyles.HexNumber);
                    }

                    _mac = mac;
                    argc++;
                }
                else if (args[argc] == "-c")
                {
                    argc++;
                    _count = Int32.Parse(args[argc]);
                    argc++;
                }
                else if (args[argc] == "-i")
                {
                    argc++;
                    _interval = Int32.Parse(args[argc]);
                    argc++;
                }
                else if (args[argc] == "-s")
                {
                    argc++;
                    _sleep = true;
                }
                else if (args[argc] == "-mount")
                {
                    argc++;
                    _mount = args[argc];
                    argc++;
                }
                else if (args[argc] == "-hostname")
                {
                    argc++;
                    _hostname = args[argc];
                    argc++;
                }
                else
                    throw new UsageException("Unknown arg. Check command parameters.");
            }
        }
    }

    internal class MacException : ArgumentException
    {
        public MacException(string address)
            : base(address + " is not a valid MAC Address!")
        {
        }
    }
    internal class UsageException : ArgumentOutOfRangeException
    {
        public UsageException(string message)
            : base(message)
        {
        }
    }

    public class Sleeper
    {
        [DllImport("Powrprof.dll", SetLastError = true)]
        static extern bool SetSuspendState(bool hibernate, bool forceCritical, bool disableWakeEvent);

        public static void Suspend()
        {
            SetSuspendState(false, false, false);
        }
    }
}

 

Enjoy.

Posted in Tip