Prerequisites

An installed version of the OpenSim libraries: these are either built from source then installed or downloaded and included with some distribution of OpenSim (that is consistent with the source you're using) If building from source, make sure you choose to BUILD_JAVA_WRAPPING
Download NetBeans IDE with JDK Bundle, e.g., at http://www.oracle.com/technetwork/java/javase/downloads/jdk-netbeans-jsp-142931.html.
OpenSim is built on top of the Netbeans platform which is included with the NetBeans IDE. Ensure you install a distribution of NetBeans (version 7.0.1 or later) that is bundled with a compatible Java JDK (so you do not have to separately install a possibly incompatible NetBeans and Java JDK). Ensure you download the 32-bit JDK version if you're using 32 bit libraries (or 64-bit if using 64-bit tools/installers).
If you anticipate that you may need to make changes to the OpenSim API, download and install SWIG (version 2.0.9 is recommended) from http://www.swig.org
Visualization libraries: For now these are VTK libraries and they are included in the bin directory of OpenSim distributions and a Windows version is available in the OpenSim repository for both 32 and 64 bit. The version of VTK in use is 5.0.6.
Access to the OpenSim repository. See Contributing to the OpenSim Source Code

How to Build the OpenSim GUI

The GUI is built on top of the C++ API, after wrapping it using SWIG. Unless you need to recreate the Java wrapping (e.g to change signature of methods or to expose new classes) there's no need to have swig installed. The output from SWIG (java classes in package org.opensim.modeling and the C++ files OpenSimJNI_wrap.cxx) is already checked into the OpenSim repository. If you plan to change the API then do the following:

Check the box for BUILD_JAVA_WRAPPING in CMake while building OpenSim.
Build the project: "Java Bindings - Generate" (in VisualStudio or build system)
Build the library "Java Bindings - osimJavaJNI" and install it.

Running Netbeans and Launching the GUI

Start Netbeans, and the click on File->Open Project
Browse (starting with the folder containing the OpenSim source code) to one of its OpenSim sub-folder's named Gui/ and then open the "project" opensim.
Click on the green arrow button (run project button) to automatically compile all the source and launch the OpenSim GUI (which may look a bit different and contain extra menu items/windows depending on your version of Netbeans -- but the functionality is there).
If you have an OpenSim installation consistent with the GUI you built and the bin directory of this installation is on your library path (env. var PATH on windows, LD_LIBRARY_PATH on linux, DYLD_LIBRARY_PATH on Mac) then the application will launch smoothly, if not you'll need to fix these environment variables and relaunch NetBeans.

Troubleshooting

We use swig 2.0.9 and have not verified compliance with newer versions.
If the GUI crashes while running through Netbeans you may have to manually delete the residual lock file contained in the OpenSim source code subfolder Gui/opensim/build/testuserdir/lock. Failing to do so may result in access violation errors during subsequent runs.

Browser not supported