Something that still shocks me this day is just how many developers out there write code without the aid of a debugger or profiler. I understand that not everyone has the time/energy to test every single aspect of their written applications, but in my mind, that’s just setting yourself up for failure. How can you possibly be sure that your code is as performant as possible without stepping through it to see what the stack/heap is every step of the way? Relying on coverage tests or unit tests alone is also not a definitive way to ensure that your code is performing as you originally intended, especially when it comes to topics such as recursion. And what about trying to diagnose problems with 3rd-party libraries, whose implementation you may not be as familiar with?
But regardless of my view on aggressive debugging during development, I’ve come to the realization that most people don’t debug their code simply because they don’t know how to do it. Oh sure, they might have set it up at one point, but then their configuration changed, or they had to re-install PyCharm, or “Autodesk changed something” and now their setup simply doesn’t work anymore, etc.
Today, I’m going to try and provide clarity on just what exactly you need to do to be able to debug your compiled C/C++ plugins in Maya, for all 3 major platforms. The first step, of course, is that we’ll need some code to actually step through. I’m using CMake for my project builds in Maya, so the first step is to setup the project builds across the major platforms.
I’ve uploaded the sample project code to Github, which is as barebones as it gets.
I use a slightly modified version of Chad Vernon’s FindMaya.cmake module, which does the brunt of the work for setting up Maya plugin builds. For this tutorial, we’re going to be compiling an example MPxGeometryFilter from the Maya devkit, namely, the identityGeomFilter.cpp file provided. You can find it yourself in devkit/plugins/identityGeomFilter.
The plug-in itself is fairly simple; it does absolutely nothing at all apart from setting the positions of the input geometry to that of the original mesh’s positions, which makes it perfect for testing.
I’m going to make the following assumptions here:
- That you know a little about C/C++ and how to use it to make Maya plug-ins (at least, enough to have already compiled your plug-in before and run it in action)
- That you know how to use your IDE that you’re comfortable in enough to be able to configure it up the wazoo to suit the instructions given
- That you won’t start talking about SCons instead in the comments, because seriously, I don’t want to hear it. (Only half-joking here)
For Windows, things aren’t as complicated as they seem…at least, at a first glance. You’ll need the following dependencies installed for Maya 2016 Extension 2:
- Visual Studio 2012 Update 4 + Win 8 SDK
- CMake (>2.8)
Personally, I run with VS 2015 as my debugging IDE and have older versions of Visual Studio installed for compiling for older versions of Maya as necessary. You could run a newer version of the compiler to compile your plugins, but there’s no telling when or what might break in specific situations (though I can’t say I’ve personally ever experienced anything of the sort yet, why tempt fate?)
To compile my plug-ins, I use CMake. To generate the build files, first create
a directory called build in the root directory and then navigate to it. Once
inside, run the following commands:
cmake -G "Visual Studio 11 2012 Win64" -DMAYA_VERSION=2016 -DCMAKE_BUILD_TYPE=Debug ../
This will generate the Visual Studio solution for the project, which can then be opened in a new Visual Studio session. You can then either build the plugin within Visual Studio itself or run the command:
cmake --build . --config Debug --target install
Once that’s done, you’re ready to debug! Open up the Visual Studio solution first. You now have two options for debugging…
The quick way (attaching to the Maya process) is as follows:
- Launch Maya.
- Run the following command:
tasklist /fi "IMAGHENAME eq maya.exe"
-
You should see something like this as your output
-
Remember the Process ID listed there. You’re going to need it.
-
Back in Visual Studio, click on Debug > Attach to Process. You should see a window pop up. Make sure your settings are the same as shown, and then click Attach. (You can use the PID that you retrieved previously to search for the process more easily)
-
Go ahead and set a breakpoint at line 93 of the source file.
-
Now load your plugin in Maya, either through bog-standard loadPlugin or whatever fancy deployment system you have. Just make sure it’s the same .mll that you actually built…
-
You should now be able to hit your breakpoint! Just create the deformer on a cube or something and watch the magic happen!
The other method, which requires a bit more setup, but is way more flexible, involves setting up Maya as your debug executable. This allows you to attach the debugger to the process as it launches, essentially putting it under the control of the debugger. This way, you can do things such as modifying the environment that it launches in, pass custom flags to the Maya command being run, etc. All very useful.
-
First, set your exampleDebugProject as the Startup Project. This is so you don’t get an error when trying to launch the debug configuration in Visual Studio.
-
Next, you want to tell Visual Studio the location of the executable to launch under debugger control. Right-click your project, choose Properties, and then go to Debugging. Change the value of Command to be the path to your Maya executable, as shown:
-
Hit OK and now hit F5 (Or click Start Debugging). Maya should now launch under the control of the debugger. Now go ahead, set your breakpoints and hit them!
-
XCode 6.1.1 with SDK 10.9 (Mavericks)
-
clang with libc++
Personally, I run with XCode 7.1 as my debugging IDE (because the App Store hangs every time I try to update to 8.0, thanks Apple!)
To generate the build files using CMake on OSX, first create a directory called
build in the root directory and then navigate to it. Once inside, run the
following commands:
cmake -G "Xcode" -DMAYA_VERSION=2016 -DCMAKE_BUILD_TYPE=Debug ../
This will generate the XCode project, which can then be opened in a new XCode session. You can then either build the plugin within XCode itself or run the command:
cmake --build . --config Debug --target install
Once that’s done, you’re ready to debug! Open up the XCode project first. You now have two options for debugging…
The quick/slightly unreliable way (attaching to the Maya process) is as follows:
- Launch Maya.
- Run the following command:
ps aux | grep -i maya
- You should see something like this as your output
sonictk 1704 0.0 4.0 6231112 667236 ?? SX 7:41PM 0:37.23 /Applications/Autodesk/maya2016/Maya.app/Contents/MacOS/Maya
- Remember the Process ID listed there. You’re going to need it.
- Back in XCode, click on **Debug > Attach to Process > Attach to Process by PID or Name. **Enter the PID of the Maya process that you found previously. Alternatively, you can also use the **Debug > Attach To Process > Maya **route directly, though I’ve found that sometimes to not be as reliable for whatever reason.
- Go ahead and set a breakpoint at line 93 of the source file in XCode.
- Now load your plugin in Maya, either through bog-standard loadPlugin or whatever fancy deployment system you have. Just make sure it’s the same .bundle that you actually built…
- You should now be able to hit your breakpoint! Just create the deformer on a cube or something and watch the magic happen!
…Wait, you weren’t able to hit your breakpoint? Welcome to the random shit that happens on OSX.
The other method, which requires a bit more setup, but is way more flexible and reliable on OSX, involves setting up Maya as your debug executable. This allows you to attach the debugger to the process as it launches, essentially putting it under the control of the debugger. This way, you can do things such as modifying the environment that it launches in, pass custom flags to the Maya command being run, etc. As a matter of fact, this is the most hassle-free way on OSX to launch Maya under a custom environment.
First, let’s set up our XCode project so that it launches Maya as well under the control of LLDB (XCode’s debugger) when we fire off a build. Go to Product > Scheme > Edit Scheme:
The next fun part is actually setting the debug “executable”. Because setting it to just Maya.app as you normally would for a OSX application results in Maya just printing out worthless help information, we’re going to have to open a new Finder window, and navigate to the Maya.app itself, right-click it, choose Show Package Contents, then navigate to Contents/bin/maya, and then drag-and-drop that to the Executable window (Make sure you choose other for the Executable first to bring up the file chooser dialog).
Now you should be able to hit your breakpoint.
(For Linux, my development environment has been tested in Fedora 23, CentOS7 and EL6.5 environments. I don’t roll with Debian-based distributions mostly because RHEL is the officially-supported platform for a lot of DCCs. However, that being said, there’s no reason the following workflow shouldn’t work on any popular Debian-based distro that you might be using)
For Linux, it’s probably the easiest of the 3 to setup. All you really need to do is run Maya under the control of the debugger you’re trying to use (which is probably gdb or ddd). You can do so with the simple command:
maya -d gdb (or ddd)
But obviously, this is very sad and puerile. So we’re going to do things my way. That is, with Emacs! (Or whatever the hell you want, really. Just make sure it has a gdb front-end or something. CLion will work as well.)
For this, you’re going to need the following dependencies:
- GCC 4.8.2
To compile my plug-ins, I use CMake. To generate the build files, first create
a directory called build in the root directory and then navigate to it. Once
inside, run the following commands:
cmake -G "Unix Makefiles" -DMAYA_VERSION=2016 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG="-Wall -ggdb" ../
This will generate the Makefile for the project, which can then be either built with Make itself or you can run the command:
cmake --build . --config Debug --target install
Once that’s done, you’re ready to debug! Open up your favorite IDE of choice which has a gdb front-end.
- Launch Maya.
- Run the following command:
ps aux | grep -i maya.bin
- You should see something like this as your output
sonictk 13696 0.0 0.0 125016 2876 tty2 S+ 23:27 0:00 /bin/csh -f /usr/autodesk/maya/bin/maya
sonictk 13709 7.6 3.6 44450148 898036 tty2 Sl+ 23:27 0:21 /usr/autodesk/maya/bin/maya.bin
You’re looking for PID 13709, in this case. (Yes, I run ZShell. Get over it.)
-
Run the following command in your IDE to attach to the Maya process:
-
gdb attach
(For Emacs, add the -i=mi flag as well in order for gud-gdb to activate)
-
Once there, set a breakpoint at the usual place:
And that’s it! You should be able to hit your breakpoint now once you reach it in Maya: So there you have it. A little bit of effort, but now you have the ability to debug infinitely more complex problems, especially tricky ones that happen sporadically as well in deform()/compute() functions!
No more excuses!
