Trying to use AjaxControlToolkit with SharePoint 2013, I managed to make this work using the following steps:
1. Download latest (.NET 4.5) version from http://ajaxcontroltoolkit.codeplex.com/releases/view/112805
2. Add Reference in your VS solution (both AjaxControlToolkit and AjaxMin dlls)
3. In Package designer, create safe control entries for both:
4. SP Web.config needs to have the AjaxControlToolkit assembly in compilation/assemblies node. I use SPWebConfigModification in feature receiver (web application scope) to do this.
5. Replace the default ScriptManager in your masterpage with the ToolkitScriptManager
6. Important: Move this section from the head tag to the body tag in your master page after the ToolkitScriptManager to avoid databind issue
That's all I had to do. I used the Ajax Combobox and bind my datasource in code behind and it is working fine.