Microsoft Bob

Just a short, simple blog for Bob to share some tips and tricks.

Be sure to check out my non-technical blog at www.bobsbasement.net.

Month List

How to Merge a Folder of MP4 Files with FFmpeg

I ran into an interesting situation the other day: I had a bunch of H.264 MP4 video files which I had created with Handbrake that I needed to combine, and I didn't want to use my normal video editor (Sony Vegas) to perform the merge. I'm a big fan of FFmpeg, so I figured that there was some way to automate the merge without having to use an editor.

I did some searching around the Internet, and I couldn't find anyone who was doing exactly what I was doing, so I wrote my own batch file that combines some tricks that I have used to automate FFmpeg in the past with some ideas that I found through some video hacking forums. Here is the resulting batch file, which will combine all of the MP4 files in a directory into a single MP4 file named "foobar.mp4", which can be renamed to something else:

@echo off

REM Clean up any pre-existing temporary files
if exist foobar.mp4 del foobar.mp4
if exist foobar.tmp del foobar.tmp
if exist *.ts del *.ts

REM Remux MP4 files into individual transport streams
for /f "usebackq delims=|" %%a in (`dir /on /b *.mp4`) do (
ffmpeg.exe -i "%%a" -c copy -bsf h264_mp4toannexb -f mpegts "%%a.ts"
)

REM Create a file list of individual transport streams
for /f "usebackq delims=|" %%a in (`dir /b *.ts`) do (
echo file %%a>>foobar.tmp
)

REM Remux transport streams from the file list into merged MP4 file
ffmpeg.exe -f concat -i foobar.tmp -c copy -bsf aac_adtstoasc foobar.mp4

REM Clean up temporary files
if exist foobar.tmp del foobar.tmp
if exist *.ts del *.ts

The merging process in this batch file is performed in two steps:

  • First, all of the individual MP4 files are remuxed into individual transport streams
  • Second, all of the individual transport streams are remuxed into a merged MP4 file

Here are the URLs for the official documentation on each of the FFmpeg switches and parameters that I used:

By the way, I realize that there may be better ways to do this with FFmpeg, so I am open to suggestions. ;-]

Posted: Jul 06 2014, 00:04 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Some Useful and Obscure FTP Configuration Settings

I get a lot of question about various configuration settings for the IIS FTP service, and most of the settings that I discuss with people can be configured through the FTP features in the IIS Manager. That being said, there are some useful configuration settings for the FTP service which I periodical send to people that have no user interface for setting them. With that in mind, I thought I would write a quick blog to point out a few of these obscure settings that I personally use the most-often or I send to other people.

Using Domain Name Syntax as an FTP Virtual Hostname

I use this setting on all of my FTP servers because it seems a little more natural to me. Here's the scenario: the IIS FTP service supports two kinds of hostnames:

  • "Real" FTP hostnames - these are real FTP hostnames that are specified by using the FTP HOST command (as defined in RFC 7151)
  • "Virtual" ftp hostnames - these are FTP hostnames that are specified as part of the FTP USER command

Real FTP hostnames are pretty straight-forward: an FTP client specifies the hostname with a HOST command when a user is connecting to the server. Once the IIS FTP service receives that command, the FTP service routes the FTP session to the correct FTP site.

That being said, the FTP HOST command is still rather new, so only a handful of FTP clients currently support it. Because of that, you can use FTP "virtual" hostnames with the IIS FTP service. By default that syntax uses the "vertical line" or "pipe" character to differentiate between the hostname and user name. For example:

  • "ftp.contoso.com|username"
  • "ftp.fabrikam.com|username"

When you are specifying your FTP credentials in your FTP client, you would enter your username like the preceding examples. While this syntax is valid for both the IIS FTP service and the underlying FTP protocol, it seems a little odd to most users (including me). With that in mind, we added a configuration setting for the FTP service that will allow you to use the more-familiar domain\username syntax like the following examples:

  • "ftp.contoso.com\username"
  • "ftp.fabrikam.com\username"

To enable this feature, use the following steps:

  1. Open a command prompt as an administrator.
  2. Type the following commands:
    cd /d "%SystemRoot%\System32\Inetsrv"
    appcmd.exe set config -section:system.ftpServer/serverRuntime /hostNameSupport.useDomainNameAsHostName:"True" /commit:apphost
    net.exe stop FTPSVC
    net.exe start FTPSVC
  3. Close the command prompt.

More information about this feature is available in the IIS configuration reference at the following URL:

FTP Credential Caching

The FTP service caches user credentials for successfully-authenticated user sessions in order to help improve login performance, and I wrote the following detailed blog about this a couple of years ago:

Credential Caching in FTP 7.0 and FTP 7.5

I don't want to re-post an old blog, but I have sent several people to that blog over the years, so I thought that it was worth mentioning here since it seems to be referenced quite often. The problem that people seem to run into the most is that their old password is still valid for FTP after they have changed it, and this is caused by the FTP service caching their user credentials.

This is especially annoying for me personally when I am working on a development computer where I am creating an authentication provider. Unless I disable credential caching on my development computer, I can never seem to get any work done. To resolve this issue, I disable credential caching for the FTP service by using the following steps:

  1. Open a command prompt as an administrator.
  2. Type the following commands:
    cd /d "%SystemRoot%\System32\Inetsrv"
    appcmd.exe set config -section:system.ftpServer/caching /credentialsCache.enabled:"False" /commit:apphost
    net.exe stop FTPSVC
    net.exe start FTPSVC
  3. Close the command prompt.

The blog which I mentioned earlier goes into more detail about setting a custom timeout interval for credential caching instead of disabling the feature entirely, and all of the settings for FTP credential caching are in the IIS configuration reference at the following URLs:

FTP Client Certificate Authentication

FTP Client Certificate Authentication is an often-overlooked feature of the IIS FTP service, and I think that this is due to two reasons:

  1. There is no user interface to configure the required settings
  2. Configuring the required settings is very difficult

My second reason cannot be understated; I usually have to set up FTP Client Certificate Authentication once or twice a year in order to test various scenarios, and each time I do so I am reminded of just how difficult it can be to get everything right, and equally how easy it is to get something wrong.

Fortunately I took the time a couple of years ago to write a blog which documents everything that it takes to configure the FTP service, and I have used my notes in that blog several times. In complement to my blog on the subject, Vivek Kumbhar wrote an excellent blog series with additional steps to configure your Active Directory for certificate authentication. With that in mind, here are all of the requisite blog posts that you would need to set up this feature:

As I have mentioned before, configuring this feature is not for the faint-of-heart, but it can be very beneficial from a security standpoint.

For more information about the settings that are required for FTP Client Certificate Authentication, see the following articles in the IIS configuration reference:

That wraps it up for today's post. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

Posted: Jun 30 2014, 16:37 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Configuration | FTP | IIS
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Case Study: Migrating Microsoft’s .NET Community Websites to Microsoft Azure

Have you ever wondered how much work is involved when migrating a traditionally-hosted production website to Microsoft Azure? If so, the following case study might be of interest to you:

Microsoft Azure Migration: Microsoft’s .NET Community Websites
Migrating Microsoft’s ASP.NET and IIS.NET Community Websites to Microsoft Azure

Here's a little background information on this migration case study: last fall Microsoft worked with two of it's hosting partners, Neudesic and Orcsweb, to migrate the www.asp.net and www.iis.net websites from a traditional web hosting scenario (e.g. websites hosted on physical servers) to virtual machines that are hosted in the cloud on Microsoft Azure. Here's what the web farm looked like before the migration:

After the migration, Microsoft had reduced both the hosting costs and the number of servers required by almost 50%.  Here's what the web farm looked like when the migration had been completed:

There are a lot of people who helped make this migration a success - and there are far too many to name here - but I would like to say a special "thanks" to everyone at Neudesic and Orcsweb for making this migration process as painless as possible.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: May 30 2014, 10:34 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: ASP.NET | Azure | IIS
Tags:
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Simple Utility to Calculate File Hashes

I have to download various files from time-to-time, and it's nice when websites provide checksum hashes so I can validate that the file I just downloaded matches the version on the server. (ON a related note, I wrote a blog several years ago which showed how to create a provider for the IIS FTP service which automatically creates checksum files when files are uploaded to a server; see my Automatically Creating Checksum Files for FTP Uploads blog post for the details.)

In order to calculate hashes for files that I have downloaded, several years ago I wrote a simple command-line application for Windows which uses several of the built-in algorithms in .NET's System.Security.Cryptography. And while I realize that there are probably other tools that provide this same functionality, I have used this little utility for years, and I've had several people ask me for copies. With that in mind, I thought that it might make a nice blog topic if I shared the code with everyone. (Note: It's a really simple sample; the .NET framework does all the real work for this application.)

Without further fanfare, here's the source code. In order to use this code sample, you need to create a new C# project in Visual Studio and choose the Console Application template. When the new project opens, replace the template's code with the following:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Security.Cryptography;

class Hash
{
    static void Main(string[] args)
    {
        // Verify the correct number of command-line arguments.
        if (args.Length != 2)
        {
            // Show the help message if an incorrect number of arguments was specified.
            ShowHelp();
            return;
        }
        else
        {
            byte[] hashValue = null;
            // Verify that the specified file exists.
            if (!File.Exists(args[1]))
            {
                // Show the help message if a non-existent filename was specified.
                ShowHelp();
                return;
            }
            else
            {
                try
                {
                    // Create a fileStream for the file.
                    FileStream fileStream = File.OpenRead(args[1]);
                    // Be sure it's positioned to the beginning of the stream.
                    fileStream.Position = 0;
                    // Use the specified hash algorithm.
                    switch (args[0].ToUpper())
                    {
                        case "MD5":
                            // Compute the MD5 hash of the fileStream.
                            hashValue = MD5.Create().ComputeHash(fileStream);
                            break;
                        case "SHA1":
                            // Compute the SHA1 hash of the fileStream.
                            hashValue = SHA1.Create().ComputeHash(fileStream);
                            break;
                        case "SHA256":
                            // Compute the SHA256 hash of the fileStream.
                            hashValue = SHA256.Create().ComputeHash(fileStream);
                            break;
                        case "SHA384":
                            // Compute the SHA384 hash of the fileStream.
                            hashValue = SHA384.Create().ComputeHash(fileStream);
                            break;
                        case "SHA512":
                            // Compute the SHA512 hash of the fileStream.
                            hashValue = SHA512.Create().ComputeHash(fileStream);
                            break;
                        case "BASE64":
                            // Compute the BASE64 hash of the fileStream.
                            byte[] binaryData = new Byte[fileStream.Length];
                            long bytesRead = fileStream.Read(binaryData, 0, (int)fileStream.Length);
                            if (bytesRead != fileStream.Length)
                            {
                                throw new Exception(String.Format("Number of bytes read ({0}) does not match file size ({1}).", bytesRead, fileStream.Length));
                            }
                            string base64String = System.Convert.ToBase64String(binaryData, 0, binaryData.Length);
                            Console.WriteLine("File: {0}\r\nBASE64 Hash: {1}", fileStream.Name, base64String);
                            hashValue = null;
                            break;
                        default:
                            // Display the help message if an unrecognized hash algorithm was specified.
                            ShowHelp();
                            return;
                    }
                    if (hashValue != null)
                    {
                        // Write the hash value to the Console.
                        PrintHashData(args[0].ToUpper(), fileStream.Name, hashValue);
                    }
                    // Close the file.
                    fileStream.Close();
                }
                catch (Exception ex)
                {
                    Console.WriteLine("Error: {0}", ex.Message);
                }
            }
        }
    }

    // Display the help message.
    private static void ShowHelp()
    {/>        Console.WriteLine("HASH.exe <hash algorithm> <file name>\n\n" +
            "\tWhere <hash algorithm> is one of the following:\n" +
            "\t\tBASE64\n\t\tMD5\n\t\tSHA1\n\t\tSHA256\n\t\tSHA384\n\t\tSHA512\n");
    }

    // Print the hash data in a readable format.
    private static void PrintHashData(string algorithm, string fileName, byte[] array)
    {
        Console.Write("File: {0}\r\n{1} Hash: ", fileName,algorithm);
        for (int i = 0; i < array.Length; i++)
        {
            Console.Write(String.Format("{0:X2}", array[i]));
        }
        Console.WriteLine();
    }/>}

When you compile and run the application, you will see following help message when you specify no command-line parameters:


HASH.exe <hash algorithm> <file name> Where <hash algorithm> is one of the following: BASE64 MD5 SHA1 SHA256 SHA384 SHA512

When you specify one of the supported hashing algorithms and a filename, the application will display something like the following example:


C:\>hash.exe SHA1 foobar.zip File: C:\foobar.zip SHA1 Hash: 57686F6120447564652C20426F6220526F636B73

That's all there is to it. As I mentioned earlier, it's a pretty simple sample. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: May 15 2014, 19:22 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

What I Do For A Living

It seems that I have always had a difficult time explaining to people what I do at Microsoft. It's not that I'm unsure about what I do - the details of my job have always been crystal-clear to me and I love what I am doing. It's just that I can't find a way to explain things in a way that doesn't result in blank stares from anyone who isn't a geek. (This problem isn't limited to me, though; my non-technical wife simply responds "I have no idea what he does" when someone asks her what I do for a living.)

Here's a perfect example: when I was a Program Manager on the Internet Information Services (IIS) team, people would often ask me what I did for Microsoft, and I would reply with something like, "I help design and implement the web publishing protocols for Microsoft's web server."

Other Person: [Blank Stare]

I would attempt to remedy the situation by adding, "You know, I design Microsoft's implementation of Internet technologies like the File Transfer Protocol, WebDAV, and the FrontPage Server Extensions."

Other Person: [Blank Stare]

In a sometimes-futile effort to salvage the conversation from complete disaster, I would interject, "You like to use the Internet, right? Well, your computer is on one side of the Internet, and my team helps build the other side of the Internet. That's kind of what I do."

That comment would usually be met with a slight spark of recognition, which was sometimes followed by a half-muttered, "That's nice."

At one time or other during my tenure as a Program Manager on the IIS team I was responsible for a smattering of disparate technologies; things like FTP, WebDAV, FPSE, FastCGI, PHP, URL Rewrite, IIS Express, Log Parser, etc. Most of those technologies garnered little to no interest for the average person, and many of my coworkers found them pretty boring as well. Just the same, I personally found every one of those technologies completely fascinating. (Why else would I spend eight years trying to get just one new command added to FTP?)

A couple of years ago I left the IIS program management team and I joined the writing team which is responsible for documenting Microsoft's ASP.NET framework; and if you have to ask what that means, then you are probably not interested in the answer.

Still, people would ask me what I do for Microsoft, and I would try to explain my job with statements like, "I document the Application Programming Interfaces (or APIs) for Microsoft's ASP.NET."

Other Person: [Blank Stare]

I would try to nudge the conversation along by saying things like, "I help people write web code."

Other Person: [Blank Stare]

Skipping ahead in the conversation, I would usually make a last-ditch attempt by stating, "Let's say you wanted to create a website; if so, you might read something that I wrote in order to help you get started."

Sometimes this remark would illicit a hint of acknowledgment, but usually I just got another blank stare.

This leads me to a few days ago. My wife and I were at dinner, and a waiter asked me what I did for a living. In the back of my mind I started to say something like, "Well, these days I'm documenting a set of APIs that Java programmers will use with Microsoft Azure technologies [blah blah blah]..."

But what actually came out of my mouth was, "I could explain it to you, but I'm pretty sure you wouldn't want me to. Trust me."

I like that answer. I think I'll stick with it in the future. :-)

Posted: Apr 28 2014, 23:47 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Random Thoughts
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Cleaning Up Your Windows System When QuickTime Has Screwed Up Your Media Settings

So here's the deal: I don't use anything from Apple. I have no iPod, no iPhone, no Mac, etc. I buy all of my MP3s through Xbox Music and Amazon. :-] Because of this, I have had no real need to install iTunes or QuickTime in years.

But unfortunately it seemed that I had to install either iTunes or QuickTime at one time or other, mainly because some of my digital cameras recorded video in QuickTime *.MOV format. But over the years I learned to detest both iTunes and QuickTime because of the undesirable ways in which they modified my system; both iTunes and QuickTime would remap all of media settings to open in their @#$% player, which I didn't really want in the first place.

Now that Windows supports the *.MOV format natively, and I can easily convert *.MOV files into something infinitely more useful and universal like *.MP4 format, I really never see the need for installing either iTunes or QuickTime.

However, just the other day I installed a new video editor (which shall remain nameless) and it quietly installed QuickTime on my system. I presume that this was to make it easier to import files in *.MOV format into the video editor, but I was pretty upset when I discovered that QuickTime had been installed. What's more, I was angry when I discovered that QuickTime had once again messed up all of my media settings.

In all of this misery is one saving grace: QuickTime has the decency to preserve your original settings. I am assuming that the backups are for when you uninstall QuickTime and attempt to reclaim your system from being hijacked by Apple, but just the same - that little nicety allowed me to fix my system with a little bit of scripting.

So without further introduction - first the script, and then the explanation:

Const HKEY_CLASSES_ROOT = &H80000000
Const strQuickTimeBAK = "QuickTime.bak"

Set objRegistry = GetObject("winmgmts:" & _
  "{impersonationLevel=impersonate}" & _
  "!\\.\root\default:StdRegProv")
 
objRegistry.EnumKey HKEY_CLASSES_ROOT, "", arrSubKeys

For Each objSubkey in arrSubKeys
  If Len(objSubkey)>2 Then
    If Left(objSubkey,1)="." Then
      objRegistry.EnumValues HKEY_CLASSES_ROOT, _
        objSubkey, arrEntryNames, arrValueTypes
      If IsArray(arrEntryNames) Then
        For i = 0 To UBound(arrEntryNames)
          If StrComp(arrEntryNames(i), strQuickTimeBAK, vbTextCompare)=0 Then
            intReturnValue = objRegistry.GetStringValue( _
              HKEY_CLASSES_ROOT, objSubkey, strQuickTimeBAK, strEntryValue)
            If intReturnValue = 0 Then
              intReturnValue = objRegistry.SetStringValue( _
                HKEY_CLASSES_ROOT, objSubkey, "", strEntryValue)
            End If
          End If
        Next
      End If
    End If
  End If
Next

Here's what this script does: first the script enumerates all of the keys under HKEY_CLASSES_ROOT and looks for file extension mappings, then it looks for mappings which have been modified and backed up by QuickTime. When it locates file extensions which have been modified, it copies the value which was backed up into the default location where it belongs.

All-in-all, it's a pretty straight-forward script, but it sucks that I had to write it.

Posted: Apr 11 2014, 00:31 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Scripting | Windows
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

FTP ETW Tracing and IIS 8 - Part 2

Shortly after I published my FTP ETW Tracing and IIS 8 blog post, I was using the batch file from that blog to troubleshoot an issue that I was having with a custom FTP provider. One of the columns which I display in my results is Clock-Time, which is obviously a sequential timestamp that is used to indicate the time and order in which the events occurred.

(Click the following image to view it full-size.)

At first glance the Clock-Time values might appear to be a range of useless numbers, but I use Clock-Time values quite often when I import the data from my ETW traces into something like Excel and I need to sort the data by the various columns.

That being said, apart from keeping the trace events in order, Clock-Time isn't a very user-friendly value. However, LogParser has some great built-in functions for crunching date/time values, so I decided to update the script to take advantage of some LogParser coolness and reformat the Clock-Time value into a human-readable Date/Time value.

My first order of business was to figure out how to decode the Clock-Time value; since Clock-Time increases for each event, it is obviously an offset from some constant, and after a bit of searching I found that the Clock-Time value is the offset in 100-nanosecond intervals since midnight on January 1, 1601. (Windows uses that value in a lot of places, not just ETW.) Once I had that information, it was pretty easy to come up with a LogParser formula to convert the Clock-Time value into the local time for my system, which is much easier to read.

With that in mind, here is the modified batch file:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], TO_LOCALTIME(ADD(TO_TIMESTAMP('1601-01-01 00:00:00', 'yyyy-MM-dd hh:mm:ss'), TO_TIMESTAMP(DIV([Clock-Time],10000000)))) AS [Date/Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

When you run this new batch file, it will display an additional "Date/Time" column with a more-informative value in local time for the sever where you captured the trace.

(Click the following image to view it full-size.)

The new Date/Time column is considerably more practical, so I'll probably keep it in the batch file that I use when I am troubleshooting. You will also notice that I kept the original Clock-Time column; I chose to do so because I will undoubtedly continue to use that column for sorting when I import the data into something else, but you can safely remove that column if you would prefer to use only the new Date/Time value.

That wraps it up for today's post. :-)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Apr 09 2014, 03:17 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

FTP ETW Tracing and IIS 8

In the past I have written a couple of blogs about using the FTP service's Event Tracing for Windows (ETW) features to troubleshoot issues; see FTP and ETW Tracing and Troubleshooting Custom FTP Providers with ETW for details. Those blog posts contain batch files which use the built-in Windows LogMan utility to capture an ETW trace, and they use downloadable LogParser utility to parse the results into human-readable form. I use the batch files from those blogs quite often, and I tend to use them a lot when I am developing custom FTP providers which add new functionality to my FTP servers.

Unfortunately, sometime around the release of Windows 8 and Windows Server 2012 I discovered that the ETW format had changed, and the current version of LogParser (version 2.2) cannot read the new ETW files. When you try to use the batch files from my blog with IIS 8, you see the following errors:

Verifying that LogParser.exe is in the path...
Done.

Starting the ETW session for full FTP tracing...
The command completed successfully.

Now reproduce your problem.

After you have reproduced your issue, hit any key to close the FTP tracing session. Your trace events will be displayed automatically.

Closing the ETW session for full FTP tracing...
The command completed successfully.

Parsing the results - this may take a long time depending on the size of the trace...
Task aborted.
Cannot open <from-entity>: Trace file "C:\temp\ftp.etl" has been created on a OS version (6.3) that is not compatible with the current OS version


Statistics:
-----------
Elements processed: 0
Elements output: 0
Execution time: 0.06 seconds

I meant to research a workaround at the time, but one thing led to another and I simply forgot about doing so. But I needed to use ETW the other day when I was developing something, so that seemed like a good time to quit slacking and come up with an answer. :-)

With that in mind, I came up with a very easy workaround, which I will present here. Once again, this batch file has a requirement on LogParser being installed on your system, but for the sake of brevity I have removed the lines from this version of the batch file which check for LogParser. (You can copy those lines from my previous blog posts if you want that functionality restored.)

Here's the way that this workaround is implemented: instead of creating an ETW log and then parsing it directly with LogParser, this new batch file invokes the built-in Windows TraceRpt command to parse the ETW file and save the results as a CSV file, which is then read by LogParser to view the results in a datagrid like the batch files in my previous blogs:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

Here's another great thing about this new batch file - it will also work down-level on Windows 7 and Windows Server 2008; so if you have been using my previous batch files with IIS 7 - you can simply replace your old batch file with this new version. You will see a few differences between the results from my old batch files and this new version, namely that I included a couple of extra columns that I like to use for troubleshooting.

(Click the following image to view it full-size.)

There is one last thing which I would like to mention in closing: I realize that it would be much easier on everyone if Microsoft simply released a new version of LogParser which works with the new ETW format, but unfortunately there are no plans at the moment to release a new version of LogParser. And trust me - I'm just as depressed about that fact as anyone else. :-(

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Apr 08 2014, 08:35 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

Personalizing Removable Drive Icons for Windows Explorer

Like most people these days, I tend to swap a lot of removable storage devices between my ever-growing assortment of computing devices. The trouble is, I also have an ever-growing collection of removable storage devices, so it gets difficult keeping track of which device is which when I view them in Windows Explorer. The default images are pretty generic, and even though I try to use meaningful names, most of the drives look the same:

By using a simple and under-used Windows feature, I have been personalizing my drives so that they have meaningful icons in Windows Explorer that will be displayed when I plug them into any of my computing devices:

Here's how this works - you just need to store two files in the root folder of each removable drive, both of which will be discussed in more detail later:

  • autorun.inf - which defines the icon to use
  • icon.ico - which is the icon/image to use for the drive

Creating the Autorun.inf File

The autorun.inf file defines the icon that will be used in Windows Explorer, and its syntax very simple:

[autorun]
icon=icon.ico

Paste the above code into Windows Notepad and save it as autorun.inf in the root folder of your removable drive.

Creating the Icon.ico File

This part is a little trickier because you have to find an image and convert it to an icon. I find all of my images by using http://images.bing.com/ to search for a particular removable drive - see http://tinyurl.com/mztbald for an example. What I am looking for is a specific image for the removable drive that I am using, and if I can't find a specific image then I will look for a generic image that works. The following image illustrates that idea:

Once I have an image, I need to convert it to an Icon file. To do my conversions, I use AveIconifier 2.1 by Andreas Verhoeven, which you can download through the Internet Archive at the following URL:

http://web.archive.org/web/20060613232414/http://mpj.tomaatnet.nl/Aveicon.zip

When you open the application, it will prompt you to drag and drop a PNG file into it.

If you were only able to find a JPG or GIF file, don't worry - you can open the image in Windows Paint and click File -> Save As -> PNG Picture to save it as a PNG image:

Once you drag a PNG image into AveIconifier, you can drag out the ICO file that you will need to rename to Icon.ico and save that the root folder of your removable drive.

Click the following image to see what a completed icon that was created with AveIconifier to show looks like.

Hiding the Autorun.inf and Icon.ico Files

One last thing that I do is optional, which is to hide and protect the autorun.inf and icon.ico files. To do so, open a command prompt and change directory to the root of your removable drive, then enter the following commans:

  • attrib +r +h +s autorun.inf
  • attrib +r +h +s icon.ico

These two commands will make the files as read-only, hidden, system files, which should normally prevent you from seeing them when you open your drive in Windows Explorer, and it should prevent them from being accidentally deleted.

Viewing the Changes

After you have saved both the autorun.inf and icon.ico files to the root of your removable drive, you will need to eject the drive and re-attach it to your system in order to see the effects. But as you can see in my earlier illustration, personalization of the drives makes them much easier to identify.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Mar 21 2014, 17:52 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: Windows
Tags:
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us

RFC 7151 - File Transfer Protocol HOST Command for Virtual Hosts

I received an email yesterday from the RFC Editor that a new Request for Comments (RFC) document has just been published, RFC 7151, which adds support for a new "HOST" command to FTP. This new command allows hosting multiple FTP sites on a single IP address, much like what Host Headers provide for HTTP.

Here's the URL to the new RFC on the RFC Editor website:

File Transfer Protocol HOST Command for Virtual Hosts
http://www.ietf.org/rfc/rfc7151.txt

Or you can see the HTML-based version at the following URL:

http://tools.ietf.org/html/rfc7151

One minor point which I would like to clarify is that this adds a new command for FTP to specify which virtual host to connect to. I periodically hear people referring to this as "FTP Host Headers", but that is technically incorrect since FTP does not have request headers like HTTP. Here's a simple example of what the communications flow looks like when the HOST command is used:

CLIENT> HOST ftp.example.com
SERVER> 220 Host accepted
CLIENT> USER foo
SERVER> 331 Password required
CLIENT> PASS bar
SERVER> 230 User logged in

I need to make sure that I thank my co-author for this RFC, Paul Hethmon, who has authored other FTP-related RFCs over the years. For example, Paul wrote RFC 3659, and he co-wrote RFC 2389 with Robert Elz. As a result, the Internet community has Paul and Robert to thank for several great FTP command extensions in the past. (e.g. FEAT, OPTS, MDTM, SIZE, REST, MLST, MLSD, etc.) Paul and I co-wrote RFC 7151 over the past several years, and it was great working with him.

Support for the HOST command has been built-in to Microsoft's FTP service since IIS 7.0, but now that the RFC has been officially published I hope that this feature will be adopted by other FTP servers and clients. That being said, IIS is not the only implementation of the FTP HOST command; at the time of this blog post, these are the server and client implementations that I am aware of which already provide support for this new command. (Note: there may be more than I have listed here; these are just the implementations that I currently know about.)

In addition to the clients listed above, if you have been reading my series on FTP clients over the past few years, I have posted details on how to use the FTP HOST command with some other FTP clients which do not provide built-in support. For example, the Core FTP Client allows you to specific pre-login commands as part of an FTP site's connection properties, so you can manually type in the HOST command and save it along the site's settings.

A Little Bit of History

When I joined the feature team which was creating the FTP service for Windows Server 2008, one of the things that bothered us was that there was no way at the protocol level to host multiple FTP sites on the same IP address. There were several ways that FTP server implementations were approximating that sort of functionality, for example the User Isolation features that we ship with FTP for IIS, but each FTP server seemed to be implementing its own workaround and there was no standardization.

Because of this limitation, our team received a lot of requests to add "FTP Host Headers," although as I explained earlier FTP has no concept of request headers. To help address some of the questions which I was often seeing, I explained the lack of hostname support for FTP in detail in the comments section of my FTP User Isolation with Multiple User Accounts blog that I posted back in 2006, which was shortly before we began work on implementing the HOST command. I will paraphrase some of my comments here:

While I realize that the ability host multiple FTP sites on the same IP address and port like HTTP is a desired configuration, the simple answer is that FTP does not currently support this at the protocol level. To put things in perspective, RFC 959 is the governing document for FTP, and that was published in October of 1985. FTP was simply not designed for the Internet as we use and understand it today, even though it is a generally reliable protocol that many people will continue to use for some time. HTTP/1.1 was designed much later and resolved this problem, but only for HTTP requests.

There are three ways that you can create unique bindings for a web or HTTP site: IP address, port, or host header. FTP can create unique bindings by IP address or port, but the FTP protocol does not currently provide support for hostnames.

Here's why: HTTP packets consist of a set of request headers and possibly a block of data. Here's an example of a simple GET request:

GET /default.aspx HTTP/1.0 [CrLf]
Accept: */* [CrLf]
[CrLf]

When HTTP 1.1 was published in RFC 2068 and RFC 2616, it defined a header for specifying a "host" name in a separate name/value pair:

GET /default.aspx HTTP/1.1 [CrLf]
Host: example.com [CrLf]
Accept: */* [CrLf]
[CrLf]

The "Host" header allows multiple HTTP virtual servers ("hosts") on the same IP address and port that are differentiated by host name. While this works great for the HTTP protocol, FTP currently has no comparable functionality. As such, the FTP protocol would have to be updated to allow multiple hosts on the same IP address and port, then FTP servers and clients would need to be updated to accommodate the changes to FTP.

While my explanation may have clarified root cause of the FTP limitation for anyone who was asking about it, I personally thought the situation was unacceptable. This inspired me to research the addition of a new command for FTP which would allow FTP clients to specify hostnames. As I was researching how to propose a new RFC document to the IETF, I discovered that Paul Hethmon had been researching the same problem a few years earlier. I contacted Paul and offered to combine our work, and he agreed. After several years of work and a great deal of supportive assistance from dozens of great people whom I met through the IETF, RFC 7151 has finally been published.

There are a lot of people besides Paul whom I should thank, and we mention them in the acknowledgments section of our RFC, which you can read at the following URL:

http://tools.ietf.org/html/rfc7151#appendix-B

One final note - two of my coworkers, Jaroslav Dunajsky and Wade Hilmo, are mentioned in the acknowledgments section of the RFC. Jaroslav is the developer who implemented the FTP HOST command for IIS, and Wade is a senior developer on the IIS team who graciously allowed me to bounce ideas off him while I was doing my research over the past few years. (I probably I owe him a lunch or two.)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/
Posted: Mar 13 2014, 21:32 by Bob | Comments (0)
  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Filed under: IIS | FTP | SSL | HOST
Tags:
Social Bookmarks: E-mail | Kick it! | DZone it! | del.icio.us